One pod, four load balancers, seven hours forty-seven

On Monday August 17 at 13:28 UTC, GitHub began returning errors to its users, and plenty of them. At peak, roughly one web or API request in five failed, and about half of the archive and raw-content downloads went nowhere.2 The unicorn page flashed for developers on six continents, and StatusGator, a monitoring service, logged its first user report online at 12:47 UTC, more than forty minutes before GitHub's own status page said anything at all.3

The incident officially ran from 13:28 to 21:15 UTC, which comes to seven hours and forty-seven minutes of degradation heading toward eight of degradation touching Issues, pull requests, APIs, Actions, webhooks, Git operations, Pages, SAML and OIDC sign-in, SCIM provisioning, Team Sync, and Copilot, each of them at a different depth.1 2

Why study this one? The chain of failure turned out unusually legible for a simple reason once GitHub published its root cause analysis on the same day it resolved the incident, followed three days later by a postmortem signed by CTO Vlad Fedorov.1 2 A sidecar pod hit one of its limits. What was supposed to notice it watched another gauge entirely. Four load balancers fell over as a result. After that came the platform's own callers, its own editor included, which spent hours stretching the recovery by trying just a little too hard to help.

Monday, 13:28 UTC

The public chronology earns a reconstruction, since it shows how quickly one team's bad afternoon became everybody's problem.2

GitHub posted its first "investigating" notice at 13:40 UTC, twelve minutes after the start of the incident, with API requests already degrading; Actions followed within four minutes, and webhooks came next. A number appeared at 13:45: error rates around 20% across numerous experiences. Pull Requests and Issues joined the degraded list before 14:00, and at 14:24 the severity class changed, since SAML and OIDC authentication were hit along with SCIM provisioning and Team Sync. Organizations whose single sign-on ran through GitHub could no longer log in reliably. A website having a bad afternoon had become something else entirely.

Even enterprise customers who never touch github.com got pulled in, as Actions workflows running on GHEC with Data Residency depend on public workflow step definitions hosted on GitHub.com, and those stopped resolving too.2 Copilot joined the list at 14:31 and Git operations at 15:21. Between 15:42 and 16:16 the engineers applied mitigations while they were yet to find root cause. The turn arrived at 16:36, with the problematic component identified, corrective actions taken, and what GitHub called strong signs of recovery. Most services steadied as the Central US datacenter recovered, in line with the staged restoration teams described later: rerouted traffic, isolated infrastructure, services brought back step by step.1 2

Very little happened during the five hours that followed, at least officially. The resolution notice didn't land until 21:15, and that gap holds the most instructive part of the story.

The cascade

The mechanism, in order:2

Traffic hit a new peak, and in the Central US datacenter an Istio sidecar pod reached one of its concurrency ceilings. Sidecars are the small proxies attached to each service in a mesh (they carry its network traffic), and brushing their ceiling is supposed to be survivable: the mesh scales the app out, load redistributes across instances, and life continues for everyone else.

Except the autoscaling policy was watching the capacity of the host service while it ignored the limits of the sidecar itself, meaning that the part of the system responsible for adding capacity measured everything except the thing that had run out of room. Requests kept arriving while the proxy stayed pinned at its ceiling, and failure spread outward until four HAProxy nodes had exhausted their flow limits.

Those nodes sat on the gateway authentication path, and that placement turned a capacity hiccup into the outage users actually felt, with sign-ins slowing down before failing outright on services that had nothing directly to do with the original pod. Every authenticated surface inherits a problem of this kind.

One more ingredient turned a capacity event into a prolonged incident: retry behavior that GitHub itself describes as optimistic began overloading internal load balancers with repeat attempts,2 so a strained system received extra work at the worst possible moment.

Independent media recognized the absurdity here. The Register noted in its summary of the RCA that a misconfiguration and a retry storm had degraded infrastructure "on which many organizations depend."4

Any team that has watched its own dashboards melt will recognize the pattern, because storms of retries are likely to appear wherever callers lack a shared cap. Engineers spent decades learning to bound them in their own systems; August 17 shows what happens when the boundary is missing at the scale of the world's default code host.

The wrong gauge

Stop for a moment on the autoscaling bug, which is the smallest piece of the chain and also the easiest one to reproduce somewhere else.

An autoscaler reacts only to signals it has been given, and a healthy signal existed in this case, sitting right there in the concurrency of the sidecar, but the policy pointed at the host service's numbers instead.2 Nothing about the machinery of scaling itself misbehaved, since it watched faithfully a gauge that would never move while the actual bottleneck filled up.

Nobody involved looks careless afterwards, which is exactly why bugs of this kind survive review. Someone configured scaling for the host, someone else deployed the sidecars, and nobody tested the wiring between those two decisions against the scenario that mattered, the one where the sidecar runs out of room before the service does.

GitHub's list of follow-ups concedes all of this. It promises autoscaling policies corrected to account for sidecar concurrency, together with an audit of Istio request, concurrency, and scaling limits across every affected service.2 Read between the lines of that commitment and a simple plan appears: go look everywhere else the gauge and the bottleneck disagree.

Pause everything

Now comes the part that sounds backwards. Pausing HAProxy simultaneously on the four affected nodes produced what the RCA describes as an immediate recovery across the board.2

It makes sense once somebody asks what four exhausted nodes do all afternoon. They accept traffic they cannot answer, and they hand every disappointed caller a fresh reason to retry. Killing them at the same instant cut the loop, demand stopped flowing into a path that couldn't process it anyway, and everything around them got room to breathe. Some of the failing traffic had already been rerouted to Northern Virginia and was served fine there while engineers picked apart Central US.2

By 16:36 the main mitigation held. Had the story ended there, we would call this a routine incident of three and a half hours.

Ten times louder

It didn't end there. While traffic shifted to Northern Virginia, delayed replies from one internal endpoint triggered what GitHub calls a latent retry bug in Visual Studio Code.2 Slow responses during the outage made the editor generate retries far beyond anything the situation justified, and traffic to the Copilot Token Service grew roughly tenfold. Engineers had to mitigate that behavior before they could safely restore traffic.1

The figures repay a second look. That endpoint normally sees 7,000 to 9,000 requests per second. During the storm it reached 70,000 to 100,000 RPS,2 and each failed token operation could spawn extra requests, each of them capable of entering a retry loop of its own. Recovery was fighting on two fronts at that point, against the original saturation and against the platform's own editor demanding tokens at ten times its usual volume.

The countermeasures look like triage because that is what they were. Engineers temporarily reduced gateway retries, in effect shipping a pull request mid-outage asking the infrastructure to stop being so helpful,2 and load balancers started answering inbound Copilot Token Service requests with HTTP 403. Traffic ramped back up gradually, site by site, letting callers succeed instead of falling back into their old loops. The choice of 403 matters here. Unlike a timeout or a server error of the 5xx kind, it carries no polite invitation to try again.

Recovery also had uninvited guests. Scraping attacks on codeload endpoints added load during the window, and GitHub lists them among the factors that impeded progress.2 Actions cleared around 18:03, Issues stabilized by 20:22, the token service finished recovering at 21:02, and the books closed at 21:15.2

That leaves the shape of the whole thing: containment of the original failure took place in well under half of the total time, and everything after 16:36 went to wrestling amplified load, plain retries first and one specific application carrying one specific bug second.5

February again

None of this will surprise anyone who read GitHub's spring posts, because August 17 belongs to an incident family the company has been documenting since February.6 7

February 9 brought an overload of the core cluster supporting authentication and user management. Two freshly released applications had unintentionally driven a more-than-tenfold rise in read traffic, and a deploy two days earlier cut a cache refresh TTL from 12 hours to 2, multiplying writes in the process. Weekend load masked the combination until Monday arrived with regular peak, waves of app updates, and another model release all at once. Same Monday pattern, same tenfold amplification, this time coming from callers and aimed straight at a cluster.6

The month before, hosted runners had gone down in every region after a telemetry gap let security policies lock internal storage accounts. March brought a Redis failover that left a cluster with no writable primary thanks to latent configuration. On April 23 a regression in merge queue silently reverted changes in 658 repositories, touching 2,092 pull requests, and on April 27 an Elasticsearch cluster probably under botnet attack stopped returning search results, in a subsystem that GitHub admitted it hadn't yet isolated as a single point of failure.6 7

Between April and August 17 sits August 6, when a routine rollout inside the Actions backend exposed a weakness of capacity that already existed, pods crashed into each other, and at peak 71% of workflow runs hit infrastructure failures while 75% of the rest ran more than five minutes late. Runners got stuck afterwards retrying invalid jobs, which dragged recovery past nine hours, and some ARC self-hosted runners stayed offline afterwards and had to be rescued by hand with kubectl, while push or pull-request events swallowed during the window couldn't be replayed on their own, which left teams to re-trigger their workflows by hand.8

Read together, these incidents share one signature: a shared dependency under sudden growth, safeguards that look elsewhere, and callers whose self-repair instincts add fuel to the fire. In March, GitHub named its structural causes plainly: rapid growth of load, architectural coupling that lets localized issues cascade, and an inability to shed load from misbehaving callers.6 Six months and several postmortems later, all three showed up on August 17 wearing different clothes.

Criticism cuts both ways here, and honesty requires saying so. Publishing detailed RCAs within days is genuinely good practice, the specificity of the August 17 analysis sits well above the norm of the industry, and the Hacker News thread collecting 636 points mostly argued mechanisms rather than cover-ups.10 But recurrence remains the finding, because knowing your failure modes in February didn't remove them in August, and only a rebuild does that.

Three million cores

What has GitHub actually been building? The postmortem's answer is concrete.1 Since the spring commitments: more than 3 million CPU cores, 120 petabytes of high-speed storage, significant network capacity. There's a physical detail hiding inside those figures, because hardware went in only up to what available power allowed in existing datacenters. Even a platform of GitHub's size meets the wall where electricity meets floor space, which partly explains why migration to Azure matters beyond cost sheets.

Azure now serves roughly 58% of platform load and half of all Git operations, up from 12% in May. For perspective, in March the company celebrated 12.5% of traffic on Azure and targeted 50% by July. The availability report of July recorded monolith reads peaking at 52.75% on Azure Central US on July 28, Git traffic in Azure at 47%, and it flagged that the core of Actions ran entirely in GitHub's own datacenters at that point, named then as a contributing factor in the capacity problems Actions would meet six days later.6 9

Demand explains the urgency. Monthly commits went from 1.4 billion in April to 2.9 billion, doubling in the space of four months, and the charts of the postmortem show roughly 130 million merged pull requests per month alongside about 24 million new repositories, accelerating through 2025 and 2026.1 Fedorov puts it bluntly: neither August incident came out of a deploy or a settings change, and both were failures of capacity at heart. "We failed to scale critical components before demand exceeded their capacity," he wrote, adding that growth explains the pressure even if it never excuses the outages.1

Where does that kind of demand come from? The April update traced it to agentic development workflows accelerating sharply since the second half of December 2025, lifting repository creation, PR activity, API usage, automation, and monorepo workloads together.7 Capacity plans drawn for 10X growth in October 2025 were redrawn for 30X by February. When agents do the writing, each commit drags CI runs, API calls, webhooks, and token authentications behind it, and load stops tracking the typing speed of humans.

Reading without ceilings

The future architecture gets a single quietly radical sentence in the postmortem: read capacity scaling linearly with the number of readers, enabling unlimited read operations.1

Unlimited reads would sever today's coupling between popularity and pain, because fetch throughput plateaus whenever a repository spikes, whether from a viral project or from a mass clone during an incident. GitHub's chart shows a fetch path flattening near 1,000 operations per second before dropping, next to a redesigned path stepping up toward 1,800 OPS/S. Rollout begins with the largest monorepos, precisely the workloads that agentic development inflates first.1

Work on isolation runs alongside, with shared dependencies coming out of critical systems so that one saturated component can't reach across products the way August 17's auth path did. The July report compressed the strategy into a line worth keeping: GitHub is becoming less dependent on shared infrastructure and individual datacenter locations.9

Budgets for retries

August produced two immediate changes, both aimed at the amplification layer rather than any component in particular.1

First came consistent limits on retries, budgets for retries, and variable timeouts across service-to-service interactions, meant to prevent retry storms and cascading load. Second came work on reviewing lower-priority CPU and memory alerts for components that could fail during sudden spikes, shutting down the blind spot where the sidecar policy had lived.

A retry budget deserves a definition anyone can reuse, because every team building on distributed systems needs one eventually. Retries convert one failed request into several attempts, trading extra load now for resilience against transient faults, and the trade works while failures stay rare and independent. During overload, failures turn common and correlated, retries stop being insurance, and they start behaving like attack traffic from inside the building. A budget caps the trade: some share of traffic may be retries, and past that line the system fails fast instead of feeding itself. Add jittered backoff and timeouts tuned per operation, and no single slow dependency can recruit every caller against the rest.

The RCA mirrors the rule onto the other side, the caller side: reviewing limits on retries and backoff across gateways and applications, fixing the editor's retry behavior specifically, improving load-balancer capacity monitoring, building regional failover safeguards.2 Notice the word "callers". GitHub now treats retry behavior in software it ships, editors and CLIs and apps included, as part of its own reliability surface, something no hardware purchase could fix. Your outage includes what your users' tools do when you hiccup.

What builders keep

Take the brand names out, and the chain works as a checklist for any stack at any scale.2 5

Watch the gauge of whatever saturates first. In your stack that might be a worker pool, a connection count, queue depth, a rate limiter; anything that hits its ceiling before the dashboards do. Autoscaling pointed anywhere else is decoration. Budget retries on both sides of the wire as well, with caps and fail-fast behavior on the server side and with caller-side software written in the assumption that it will run during somebody else's bad day, because the lesson of the Visual Studio Code bug generalizes uncomfortably well. Code you ship becomes someone else's incident load.

Test recovery, not only failure. Recovery at GitHub worked only after operators manually broke the loops, pausing proxies together, blocking requests for tokens, ramping sites gradually, none of it automatic. If your plan assumes self-healing, inject latency into a staging game day and watch whether the retries behave.

Keep writing postmortems people can actually learn from, too. Documentation quality like this is why the incident can be studied, and recurrence across February, April, and August is why the study deserves company. Infrastructure craft, like any other craft, reveals its quality when the material pushes back.