Missing webhook deliveries for events during the Nov 23 outage

  • 26 November 2022
  • 1 reply
  • 39 views

Badge +3

Greetings, RevenueCat team & community

 

Yesterday I found out that some customers don’t have premium access anymore to my client’s app - the premium access is offered through a monthly recurring subscription.

 

Looking at their profiles in RevenueCat, a colleague noticed that the webhook delivery for the Renewal event was missing in the dashboard - please see the attached image.

 

Thus, our backend had no idea that the subscription should be extended. Now we are in a situation where some customers paid, but don’t have access anymore. I exported a custom CSV list from RevenueCat for all renewals done on Nov 23 and checked them one by one, out of about 120 renewal events, 12 had missing webhook deliveries. I provided that list to my teammate and he manually fixed them in our database.

 

However, there are probably more customers impacted than what I managed to find. Here comes the part where I need your help:

 

  1. Is there a way for us to get a complete list of impacted users so we can manually fix them in our db? (for all events, not just renewals)
  2. Is our RevenueCat project the only one impacted by missing webhooks during the outage? I read the status updates RevenueCat wrote for the planned db upgrade / incident report and I understood that there shouldn’t be missing requests. - “Additionally, we are replaying requests that failed to ensure all traffic is captured.” - however, this doesn’t seem to be the case here.
  3. Was the RevenueCat team aware of missing deliveries? If so, were there any attempts to contact owners of the impacted projects (like us) so we can at least know asap and manually fix impacted users, instead of hearing the news from our client 2 days later? I’m not trying to blame here, I understand that this upgrade was forced by AWS and everyone did their best to minimize damage. However, I think more transparency would be appreciated next time something like this happens.

 

I’m available to respond with more info, or if someone from RevenueCat wants to contact me privately, to offer me a list of impacted users, that would be greatly appreciated.

Thank you for your time!

 


1 reply

Userlevel 3
Badge +6

Hey There! 

 

Sounds like we might need some more information for this, can you please open a support ticket?

 

That way we could dive deeper into this situation, to create a support ticket please use the following link: https://app.revenuecat.com/settings/support

Reply