Outages and root causes.

Root cause, as stated by GitHub

An automated metadata process changed a runtime configuration value on running VMs, breaking service discovery and leaving traffic routers without backends.

Root cause, as stated by CircleCI

An internal data-deletion job overlapped with a burst of customer project deletions and overloaded a data service, and missing timeouts let slow responses exhaust API workers.

Root cause, as stated by Honeycomb

A deployment changed how dataset schemas were cached in a backward-incompatible way, so cache reads failed and all lookups stampeded the database.

Root cause, as stated by Upstash

A recent guest kernel update on the host platform interacted badly with a hypervisor bug affecting log writes, stalling the Redis process.

Root cause, as stated by Clerk

A failed live migration of the virtual machine hosting Clerk's managed database caused a spike in disk latency and lock contention.

Root cause, as stated by Cloudflare

A new automated cleanup task sent an API query with an empty parameter that the Addressing API interpreted as matching all BYOIP prefixes, so it began deleting them.

Root cause, as stated by Clerk

An automatic statistics refresh in Postgres caused the planner to switch to an inefficient query plan that consumed nearly all database resources.

Root cause, as stated by Supabase

A deployment of a new internal monitoring service unintentionally turned on a region-wide AWS setting that blocks public internet access to VPCs.

Root cause, as stated by GitHub

GitHub says a configuration change to a user-settings cache triggered a flood of cache rewrites; these first overwhelmed a shared background-work component and later caused replication delays, each time exhausting connections in the Git HTTPS proxy.

Root cause, as stated by GitHub

A loss of telemetry led to security policies being wrongly applied to backend storage accounts at the compute provider, blocking the VM metadata needed to create and manage runner VMs.

Root cause, as stated by Asana

A code path overloaded a shared metadata database that the app's services depend on.

Root cause, as stated by Cloudflare

Turning off an internal WAF testing tool through the global configuration system, as part of a request-body buffer increase, triggered a latent bug in the older proxy's rules module.

Root cause, as stated by Cloudflare

A database permissions change made a query emit duplicate rows, doubling the size of a Bot Management feature file beyond a hard limit in the proxy software, which then crashed.

Root cause, as stated by Zendesk

Zendesk says the bot-management system at its content delivery network provider broke down, which disrupted logins.

Root cause, as stated by Asana

A routine configuration change made during a refactor had an unintended side effect that broke app loading.

Root cause, as stated by Microsoft Azure

Configuration changes processed by two different control-plane versions produced incompatible metadata that triggered a latent crash bug on edge servers, and validation safeguards did not catch it.

Root cause, as stated by Amazon Web Services

A latent race condition in DynamoDB's automated DNS management let a stale plan overwrite a newer one and then be cleaned up, leaving the regional endpoint with an empty DNS record.

Root cause, as stated by Vercel

The AWS us-east-1 outage triggered a cascading failure in Vercel's regional cache service and, via a feature-flag provider outage, resource exhaustion in its primary control-plane cluster.

Root cause, as stated by Docker

Docker attributes the outage to the AWS US-East-1 regional failure that its services depend on.

Root cause, as stated by PlanetScale

PlanetScale says a DNS misconfiguration at one of its service providers and the Amazon DynamoDB outage in AWS us-east-1 took out upstream dependencies of its control plane; later it could not launch new EC2 instances there and partial network partitions cut some database servers off across availability zones.

Root cause, as stated by Buildkite

Buildkite says EC2 launch failures during the AWS us-east-1 outage stopped its autoscaling from adding capacity, so it could not scale up for US business-hours traffic and used up all available compute.

Root cause, as stated by GitLab

A scheduled CI pipeline from one user hammered an expensive API endpoint, saturating database replicas and then the primary database and exhausting connection pools.

Root cause, as stated by Confluence

A new CDN configuration used a resource name that clashed with the existing customer-serving resource and overwrote it; Atlassian names the root cause as its pre-deployment validation and tests failing to catch the bug.

Root cause, as stated by Webflow

Sustained malicious traffic combined with performance problems after scaling a critical database cluster onto different hardware, plus configuration issues and a known bug in that cluster.

Root cause, as stated by Clerk

Clerk's compute containers on its cloud provider entered a restart loop, which Clerk attributes to a flaw in the provider's zonal redundancy that prevented failover to healthy capacity.

Root cause, as stated by Cloudflare

Part of the storage infrastructure behind Workers KV, hosted with a third-party cloud provider that had its own outage, failed and KV lacked sufficient redundancy to absorb it.

Root cause, as stated by Google Cloud

A policy change containing blank fields hit new Service Control code that had no error handling or feature-flag protection, causing a null-pointer crash loop in every region.

Root cause, as stated by Heroku

An automated operating-system update that should have been disabled ran in production and restarted host networking, and required routes were not reapplied because a legacy script only set them at boot.

Root cause, as stated by Convex

Background search-index compaction needlessly invalidated huge numbers of subscriptions, client reconnect logic amplified the overload, and a manual push moved the deployment onto under-provisioned hardware.

Root cause, as stated by Neon

Neon says that on 16 May a query plan change overloaded its control-plane database, so idle databases stopped suspending; pod counts rose past planned capacity and used up the IP addresses in two of the region's three subnets. On 19 May, reverting network settings changed during the first incident made IP allocation fail again, a phase Neon said it was still investigating.

Root cause, as stated by Zoom

A miscommunication between Zoom's domain registrar and the domain registry led the registry to mistakenly block the zoom.us domain.

Root cause, as stated by Cloudflare

During a credential rotation, the new credentials were deployed to the wrong environment, so the production R2 gateway lost access once the old credentials were removed.

Root cause, as stated by Slack

A maintenance action on a database system, combined with a latent defect in the caching system, overloaded the database with traffic.

Root cause, as stated by GitHub

GitHub attributes it to a configuration change tied to traffic routing and testing, which made an internal load balancer discard requests passing among the services Git operations depend on.

Root cause, as stated by OpenAI

A power failure at a cloud provider's data center took out critical services such as databases, and regional failover required manual action by the provider.

Root cause, as stated by OpenAI

A newly deployed telemetry service made every node issue expensive Kubernetes API calls at once, overwhelming the control plane in most of OpenAI's large clusters and breaking the DNS-based service discovery that depends on it.

Root cause, as stated by Canva

Canva says a network problem at its CDN provider stalled one file of a new editor release, so hundreds of thousands of waiting clients loaded it at once and hit the API gateway with about three times peak load. The gateway, already slowed by a regression in a telemetry library, ran out of memory faster than autoscaling could replace it.

Root cause, as stated by Fly.io

A specific configuration update propagated across the fleet triggered a lock-handling bug that deadlocked the proxy that routes all requests.

Root cause, as stated by GitHub

An erroneous database configuration change stopped databases answering health checks, so the routing layer marked them unhealthy and reads failed.

Root cause, as stated by Amazon Web Services

The cell management system mishandled a cell with a very large number of low-throughput shards, wrongly marking healthy hosts unhealthy and triggering shard redistribution that overloaded a connection-provisioning component.

Root cause, as stated by Microsoft Azure

A DDoS attack was the trigger, but the outage itself came from protection routes that could not be restored at one site plus a latent network misconfiguration that funnelled outside traffic into the European DDoS protection system.

Root cause, as stated by Microsoft Azure

Microsoft says an allow list of VM host addresses was built while some of its source files were unreachable, and the incomplete list was rolled out across the region's storage, which then rejected disk requests from valid hosts.

Root cause, as stated by Cloudflare

Cloudflare cites two independent events: automated traffic re-routing briefly congested its backbone, and then, with larger effect, a newly deployed DDoS mitigation rule exposed a latent bug in the rate-limiting system that sent request-handling processes into an infinite loop.

Root cause, as stated by Render

A faulty code change, which a bug in feature-flag handling had kept from being exercised in non-production environments, caused a restart of all workloads.

Root cause, as stated by Linear

A database migration used a cascading truncate statement that also emptied existing production tables linked by foreign keys.