Question

When is the right time to conclude an experiment?

  • 31 December 2022
  • 2 replies
  • 49 views

Badge +2

I’ve read the article Experiments: Debugging and best Practices for A/B Testing that was probably written when Experiments were still in beta. The documentation mentions an “LTV Chart” with a gauge to determine when you can conclude an experiment.. However, I can’t find that my RevenueCat dashboard. 

My case: 
I have two subscription groups that both contain a monthly and yearly subscription. The trial length for the subscriptions between the groups are identical. The only difference is price.

I’ve been running the experiment for around two weeks now, and see positive results. However, it’s still unclear for me when I can finish the experiment. 


2 replies

Userlevel 4
Badge +8

The documentation mentions an “LTV Chart” with a gauge to determine when you can conclude an experiment.

 Sorry for the confusion, we need to update that document.

 

However, it’s still unclear for me when I can finish the experiment. 

 It is up to you when you want to conclude it, it depends per app but you should do so whenever you feel like you have the results you want. One thing to note is that when you stop an experiment, the results will continue to be updated for 28 days to capture any additional subscription events. Please see here for our tips on running experiments: https://www.revenuecat.com/docs/experiments-overview-v1#tips-for-using-experiments

Badge +6

“One thing to note is that when you stop an experiment, the results will continue to be updated for 28 days to capture any additional subscription events.”

Can you clarify? I’m unclear what data is added to the experiment after it is stopped. If I stop an experiment and then all the users in the experiment get their offering changed to the default offering, then what data can you continue to collect that won’t mess up the test (having changed the offering)???

 

Reply