Elevated errors in the us region (resolved)
Between 09:05 and 09:20 UTC on 19 November, a deployment of Restate Cloud’s authentication service caused authentication checks to fail.
us: Requests that needed a new credential check were rejected with HTTP 503. Between 09:10 and 09:18 UTC, up to 80% of incoming requests failed. Services connected through a Restate Cloud tunnel lost their connection and could not reconnect. Between 09:15 and 09:17 UTC no tunnel-connected service was reachable. All tunnels had reconnected by 09:21 UTC.
eu: Up to 19% of incoming requests failed with HTTP 503 between 09:06 and 09:19 UTC. Tunnel connections were not affected.
Cause: two interdependent changes went through our deployment pipeline as one unit, and one went live before the other. We now release dependent changes separately, and we have added this check to our review process.
Resolved
Elevated errors in the us region (resolved)
Between 09:05 and 09:20 UTC on 19 November, a deployment of Restate Cloud’s authentication service caused authentication checks to fail.
us: Requests that needed a new credential check were rejected with HTTP 503. Between 09:10 and 09:18 UTC, up to 80% of incoming requests failed. Services connected through a Restate Cloud tunnel lost their connection and could not reconnect. Between 09:15 and 09:17 UTC no tunnel-connected service was reachable. All tunnels had reconnected by 09:21 UTC.
eu: Up to 19% of incoming requests failed with HTTP 503 between 09:06 and 09:19 UTC. Tunnel connections were not affected.
Cause: two interdependent changes went through our deployment pipeline as one unit, and one went live before the other. We now release dependent changes separately, and we have added this check to our review process.