Resolved -
We've received notification from our billing vendor (Stripe) that they had "an outage affecting some customers between 19:09 UTC on May 26th to 18:35 UTC on May 27th where some of our merchants received an elevated 429 errors on their Stripe accounts, tied to rate limited API requests. "
These errors were an unintended result of a recent change on Stripe's end to update the rate limit configuration for certain types of requests.
Our observations are aligned with Stripe's outage description, and we believe this incident is fully resolved.
May 28, 11:31 EDT
Monitoring -
We've deployed a patch and are monitoring logs to confirm resolution.
If you were affected by this issue, please re-try any operations which previously failed. If you still experience failures while managing users, please open a support case.
May 27, 16:15 EDT
Update -
We are preparing to deploy a patch to affected customers within the next 30 minutes
May 27, 15:49 EDT
Identified -
We are working with our billing vendor to restore the integration that supports our user management operations.
We are also developing an application patch to gracefully handle billing update failures in a way that does not block user management operations.
May 27, 13:54 EDT
Investigating -
We've received a number of reports that adding / editing / deleting users is failing.
We've identified an issue with our billing platform causing synchronous rejections of updates to billing details.
We are working towards a resolution
May 27, 12:51 EDT