Skip to main content
Solved

A/B Experiments - why are we seeing an uneven distribution?


Forum|alt.badge.img+5

We have enabled a/b experiments and are testing what experiment performs best. We also have analytics around what SKUs users are seeing. We’ve noticed that our default variant is seen 70% of the time and variant B 30%. It’s hard to tell if this is a RevenueCat issue or whether our users are that skewed in terms of who is looking at the paywall.

 

Any ideas on why we would see such an uneven distribution?

Best answer by ryan

There should be some metrics in the Experiments dashboard on the “Number of users in variant” - does that appear to be a 50/50 split? 

Are your analytics taking into account when you started running the experiment? All of the downloads from before the experiment started would still be seeing the default Offering which may be skewing your analytics in that direction. 

Once you start an experiment, RevenueCat starts splitting the default Offering for new installs, it won’t change the Offering for existing installs.

View original
Did this post help you find an answer to your question?

2 replies

ryan
RevenueCat Staff
Forum|alt.badge.img+9
  • RevenueCat Staff
  • 391 replies
  • Answer
  • August 23, 2021

There should be some metrics in the Experiments dashboard on the “Number of users in variant” - does that appear to be a 50/50 split? 

Are your analytics taking into account when you started running the experiment? All of the downloads from before the experiment started would still be seeing the default Offering which may be skewing your analytics in that direction. 

Once you start an experiment, RevenueCat starts splitting the default Offering for new installs, it won’t change the Offering for existing installs.


Forum|alt.badge.img+5
  • Author
  • New Member
  • 2 replies
  • August 23, 2021

Thanks, that looks like it may be the issue. Most of our users are older and therefore on the default offering and therefore represents a larger portion of total users.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings