Question

What's the recommended way to exclude test/development users

  • 2 October 2023
  • 4 replies
  • 68 views
C
Badge +3

My understanding is that while there is a concept of sandbox versus production purchases, there is no concept of a development versus production users. And since we can’t filter by custom user attributes, stats that involve factoring in the total number of users, like subscription rate, will be off in charts and experiments.

Off the top of my head, some solutions might be:

  • Mock RevenueCat outside of prod.
  • Delete development/test users when I’m done with them.
  • Accept that some metrics will be off. Perhaps use a separate analytics service for those.

Is my understanding correct? And is there an official recommendation for how to handle this?

4 replies

Ryan Glanz
Rank
Userlevel 3
Badge +8

Hi,

My understanding is that while there is a concept of sandbox versus production purchases, there is no concept of a development versus production users

Yes, that’s correct.

 

stats that involve factoring in the total number of users, like subscription rate, will be off in charts and experiments.

No, we won’t factor “revenue” from Sandbox purchases into your Charts (see here). And Experiments are run in production (we enroll users upon app installation), so those users won’t make sandbox purchases in the first place.

 

So our stats will reflect the real revenue you make, and won’t be thrown off by your testing.

C
Badge +3

Hey @Ryan Glanz ,

I’m thinking about the “New Customers” and “Start Rate” columns in the “Trial Conversion” chart (or the “Customers” and “Initial conversion rate” rows for experiments).

 

Development users would show up as Customers, inflating the value. And as a consequence, the Start Rate, if set to show as a relative percentage, would be deflated.

Ryan Glanz
Rank
Userlevel 3
Badge +8

Ah, I see. We cohort those charts by first seen date, which is essentially the install date for the app. So if you’re just testing in sandbox (not testflight), you shouldn’t see those numbers increase.

 

Another solution that a lot of developers do is to have a separate RevenueCat project for their production and development/staging apps. That way you can filter your charts by project to only see real users in the production project, and have a completely separate environment for your test users.

C
Badge +3

Ah, I see. We cohort those charts by first seen date, which is essentially the install date for the app. So if you’re just testing in sandbox (not testflight), you shouldn’t see those numbers increase.

 

Good to know. If that chart is excluding sandbox users, maybe what’s happening is that I also have a backend service in development that’s hitting the v1 REST API. And any users created through that route be assumed as production, since there isn’t a way to specify sandbox/production through the API.

Reply


Terms & ConditionsCookie settings