Experiments should have the concept of "placement"

  • 14 September 2023
  • 2 replies
  • 49 views
C
Badge +1

Hello,

I feel like it is hard to have clean data of our experiments with revenueCat. The main reasons being:

  1. Almost every new experiment implies an app update which the user may not have done
  2. We have a paywall where the user can change his subscription that we do not want to include in the A/B test

As a result we always have “other” products in our experiment results and globally I am not at ease with the data returned.
Is there something we are doing wrong ?


It would be great to have the concept of “placement” so that we can do an A/B Test only in certain part of the app and also compare the revenues from differents places in the app.

Best regards,

2 replies

kaitlin
Rank
Userlevel 4
Badge +6

Hi @cf4ed4, 

To answer your question, it’d help to know a little more about what you were expecting to see in your experiment results vs what you actually saw. Are any of the other products that in your results also included in the Offerings that you’re testing? It’s probably more helpful to consider the test between two Offerings rather than two products, and by keeping the two Offerings largely the same besides the product that you want to test, this will help to reduce other variables. 

You are correct that at this time, we can only include results from choices made using the Offerings and not from other places in the app. I’d be happy to pass along that feedback.

C
Badge +1

Thanks for your response,

In our app we have multiple placements that display different offerings, for exemple:

Onboarding: Paywall A (offering A)

Settings: Paywall B (offering B)

Call to action: Paywall C (offering C)

 

We decide to compare the paywall A against a new paywall D (and a new offering D) in the onboarding.
We do an app update with Paywall D and in the onboarding we either display Paywall A or Paywall D based on revenue cat’s “current” offering.

 

I would like to have the experiment results for the users that did the app update and comparing only offering A and D. Instead I also get the offering B and C in my experiment result and even offering A from user that did not do the update and therefore cannot see offering D.

 

Best regards,

 

 

Reply


Terms & ConditionsCookie settings