Resolved -
The number of active instances and HTTP traffic has returned to typical levels, warm pools are healthy and serving new instance requests, and errors are down.
While systems are now operational and this incident has been resolved, we will continue to monitor things closely.
Jul 18, 23:37 UTC
Update -
After resolving the original Prisma ORM binaries download issue we observed many users attempting to create query engines at the same time. Accelerate keeps a pool of warm instances to serve these requests quickly and when that warm pool is depleted we fallback on creating cold instances.
We identified a bug in the cold instance path which prevented us from bringing Accelerate online for all users fast enough. While some users were able to get instances from the warm path as new instances were added to the pool, it wasn't fast enough to serve the backlog of users who were requesting new instances. Once we found and patched the bug Accelerate was able to serve all instance requests within minutes.
Because this only affected the creation of new instances, users with instances already running would have not experienced any issues.
Thank you for your patience. We're already working on collecting data and insights for our postmortem which always aim to make our products and services better.
Jul 18, 23:16 UTC
Monitoring -
The issue has officially been resolved. After identifying and resolving the root cause we experienced issues while restoring Accelerate to service.
We're continuing to monitor closely and will provide another update with more details on what delayed bringing Accelerate back to service.
Jul 18, 22:52 UTC
Investigating -
We are currently investigating this issue.
Jul 18, 22:06 UTC
Update -
As we continue to monitor we have an update on the root cause and resolution of this incident:
- When a new version of the ORM is published it uploads a binary to R2 and a failover to copy to S3
- ORM users, including Accelerate and Prisma Postgres, require these binaries when starting up query engines
- The failover to S3 triggered but ORM, Accelerate, and Prisma Postgres were not able to download binaries
- We resolved the incident by overriding the failover and pointing to R2
We're investigating both the cause of the failover the issue related to downloading binaries from S3.
Jul 18, 20:43 UTC
Monitoring -
Description: We have identified and mitigated an issue with Accelerate, also affecting Prisma Postgres, where Prisma ORM binaries are unable to be downloaded. The root cause has been found and fixed and we are monitoring Accelerate and Prisma Postgres.
Affected users: All Accelerate and Prisma Postgres users, and a subset of Prisma ORM users
Root cause: Identified and fixed
Jul 18, 18:00 UTC