We currently have an AB test running, but the results seem confusing. The tests shows the following numbers:
Variant A has an annual price of $49.99 and Variant B $59.99. From this, it seems more intuitive to me that Variant A is outperforming because the price of B is only 20% higher than A’s, but there are more than double the amount of active subscriptions in Variant A.
Yet, the results section suggests:
“Based on our most recent run of the model, at 1 years, there is a 67% chance that Variant B has a higher LTV.”
Can you explain to me where this result comes from? Maybe there’s just a lack of additional data in the dashboard that would show the real reason, but with the data available, it seems very off for us.
Best answer by ryanView original
67% is still pretty close, I’d recommend running the experiment longer to try and increase the confidence in one variant or the other. Only 18 out of 114 trials have completed in Variant B so there may be a lot of “potential” revenue the model is still considering.
There’s some more info on the underlying model in this blog post here that may be helpful: https://www.revenuecat.com/blog/price-testing-for-mobile-apps
Ok that does make sense. It would be helpful, if the data points available would have a bit more context information. For example, it isn’t clear whether (in Variant A):
It’s a bit unclear what the rows actually mean, whether it’s a funnel or if they are inclusive / excklusive numbers etc. Maybe there could be a bit more tooltip information in the dashboard.
And thanks for the article!