Experiments - need to stop adding new users but keep experiment running.

  • 22 February 2024
  • 3 replies
  • 23 views

Badge +6

I’m finding RevenueCat Experiments limited because stopping the experiment is all or nothing. If I stop the experiment, RC changes all those users back to the default offering right away and stops collecting conversion data from the users. (Except it can keep gathering some subscription renewals in some cases???).

This is limiting for apps that a strong long tail conversion (ie convert a lot subsequent to first use) and even for apps with subscription trials. Consider an experiment I just did… after 4 days it's very clear that the new paywall I’m testing performs much better. So I wan’t to stop adding users to the experiment and give all new and old users who are not in the experiment the new paywall so I can see more real $ coming in. But I don’t want to _stop_ the experiment. I want to keep the experiment running but stop adding new users to the experiment. I want to leave the experiment running for another 10 days or so to catch conversions that happen on day 2,3,4,5,6,7,8… and I want full revenue accounting for 7 day trials that end up converting. 

If I understand correctly the only way to do what I want is to leave the experiment running for 14 days. Which means lost revenue because too many users got left with the control paywall for too long. And even if I leave it for 14 days and then close it, the new users who entered the experiment on day 13 only have 1 day to convert, etc. 

I want a button ‘Close experiment to new users’ that keeps the experiment running without adding more people. And then eg 10 days later I fully stop the experiment. So an experiment gets stopped in 2 phases. 

Maybe there is a work around that does what I want?

Thanks - Mark.


This post has been closed for comments

3 replies

Badge +6

So it looks like RC _does_ have this feature. (It got added recently?) On a running experiment I can edit the experiment, change the ‘New Customers to Enroll’ slider down to 0%, save and keep the experiment running. 

Userlevel 4
Badge +6

Hey Mark!

You’re correct, we just added a few new aspects to Experiments - the ability to edit new customers after the Experiments has begun, the ability to run multiple experiments simultaneously, and a more frequent refresh time for results in the dashboard. 

Glad you were able to find the feature! Let us know if you have any other questions about it!

 

Badge +6

Great! Would be good if I could set the % down to 0 to support my use case. And I encourage explaining the 2 step experiment turn off in documentation for people to consider. 

Mark