GitHub incident: autoscaling misconfiguration and retry storm

Summary

On August 17, 2026 GitHub experienced a major outage that degraded a number of key services (Pull Requests, APIs, Actions, Issues, and Copilot). These services are crucial for developer productivity and users couldn't complete code reviews, create/read issues, and have their CI/CD pipelines run properly via GitHub hosted runners. 20% of web/API traffic experienced errors and content operations (archiving, code downloads) hit 50% error rates.

The root cause was network saturation on load balancers that GitHub ran in Central US (Azure). While GitHub has hit new peaks of traffic since AI has become heavily adopted, they have autoscaling that should have scaled their load balancers. However, Istio sidecar pods that they run as part of their service mesh in Kubernetes clusters hit concurrency limits and were not autoscaled properly. The existing autoscale policy only monitored the host metrics rather than the sidecar resource consumption. This has the effect of making it appear that the hosts running the load balancers had sufficient resources even though the sidecars running on those hosts were saturated.

The issue was exacerbated by thundering herd problems, which caused a surge in retry traffic that had to be mitigated with code/config deployments from GitHub.

What's the big deal?

Vlad Fedorov, GitHub's CTO, effectively described the impact in his blog post with this statement: "if you were trying to ship software that day, we let you down". As a developer, only local development was possible during the 7+ hrs of the incident so no one was able to deploy code.

One might ask: what's the big deal with not being able to use these services for several hours? These services would prevent teams who use GitHub (which is most teams on earth) from shipping new code. While that might seem like it only slows down releases for a day, it can do much more. There are many instances where a team needs to quickly push out a change to production to fix a critical bug or security vulnerability. If the ability to ship code is impeded for several hours, this can prevent critical changes from getting out the door. In many cases, development teams might have had to bypass their CI/CD (PRs, GitHub Actions) to get code pushed to production, forcing risks to be taken.

Timeline of the Outage

Elapsed times are measured from the start of impact at 13:28 UTC.

Between 16:36 and recovery, the mitigations stacked up rather than landing all at once: a PR to reduce retry logic at the API gateway, changes to the HTTP responses served for Copilot token requests, pausing HAProxy internal load balancer nodes, and shifting traffic out of Central US to Northern Virginia. None of those steps carry a public timestamp, which is itself worth noting.

The 21:15 end time is derived from the stated 7h 47m duration rather than published directly.

Assessment

Time-to-detect (TTD)🟡 5 / 10

The incident page indicated the issue started at 13:28 UTC but the first update was at 13:40 UTC stating that GitHub was investigating reports of impacted performance on several services. This appears on the face of things to indicate at least 12min passed between the impact starting and GitHub raising the status page. It also suggests that there were customer reports of issues before the company declared an incident.

A good rule-of-thumb for TTD is <= 5min. This means that alerts fire within 5min of impact starting and reach an on-call engineer (i.e. paging them) who can respond to the issue. Based on the timestamps, it appears this wasn't detected by GitHub before customers experienced it. Furthermore, one of the follow-up action items was improving load-balancer capacity monitoring, and the fact that it took them over 3hrs to find the issue suggests that monitoring wasn't great.

I will give GitHub the benefit of the doubt here since they leverage SLO-based alerting and since error rates on requests were exceeding 20% during this period it's very likely these alerts fired within 5min of the impact starting.

Detection sourceN/A

This is still unknown so I'll withhold judgement here, but my hunch is a combination of user reports and SLO-based alerts fired on error rates that told GitHub there was a problem somewhere within their infra.

Status page updates🟢 9 / 10

Hrishikesh Barua wrote a great blog on how to share outage updates. GitHub delivered updates very rapidly (every few minutes) at the beginning of the incident to focus on the impact to users and which services were affected. This gave customers an idea of what they should expect during the incident.

The updates became less frequent (i.e. 15-20min) though as the incident dragged on until about 3hrs in when the Istio sidecar capacity issues were identified. After that they indicated corrective actions being taken and whether services were returning to normal function.

Following operations returning to normal, they posted a root cause within 30min which is a fast turnaround. This was the highlight of the timeline.

My only suggestion here is that they deliver updates with predictable timelines. For example, they should indicate that the next update will happen in XX minutes, even if they are not sure whether they will make progress within that timeframe. It gives customers a reasonable expectation on when they should be checking back in for an update.

Time-to-mitigate (TTM)🔴 3 / 10

The first mitigation was put in place at 16:36, over 3hrs into the incident but coupled with the same update where the problematic Istio sidecar was identified. This could have been GitHub manually scaling that particular component since the autoscaling policy was misconfigured. That did result in temporary mitigation of a number of the impacted services, but retry logic put further pressure on the services and additional mitigations were needed that took a number of other attempts.

Because mitigations were only temporary and there was a cascading effect of the issues, I rated the mitigation efforts as mostly insufficient.

Time-to-resolve (TTR)🔴 2 / 10

Nearly 8hrs of significantly degraded performance (20%+ error rates) is very bad. GitHub's SLA describes an uptime of >= 99.9% over a quarter time window. For simplicity, their enterprise cloud offering uses a time-based formula:

Uptime % = (Maximum Available Minutes − Downtime) ÷ Maximum Available Minutes × 100

That gives GitHub 129.6min of downtime allowed over 129,600 available minutes in a 3-month period. They had 467 min of downtime that would make their best case for the current quarter a 99.6397% uptime, violating at least one level of the SLA.

No matter how you cut it, almost 8hrs of severely degraded availability of their core services is a very long time for such a critical piece of developer productivity infra.

Blast radius reduction🟡 4 / 10

Many of GitHub's most important services were impacted by the network saturation in the Central US Azure region. They indicated that traffic had to be manually shifted out to Northern Virginia to relieve the saturated region — which tells you the blast radius was scoped by geography rather than by any deliberate isolation boundary.

The saving grace is that the failure did not take down every GitHub surface at once: git operations over SSH and the core repository data plane stayed largely intact while the control-plane services fronted by those load balancers degraded. That is a real boundary, and it kept the incident from being total. But within the affected region, essentially every service a developer touches to ship code failed together, which is close to the worst possible correlation.

Automated remediation🔴 3 / 10

The incident was mostly resolved through manual actions (PR to address retry logic, changes to Copilot HTTP response codes, pausing HAProxy internal load balancer nodes, shifting traffic to Northern Virginia DC). While there were autoscaling policies in place for the load balancers, those did not account for Istio sidecars.

GitHub needs to make sure autoscaling policies are in place on all necessary components that are part of critical paths that are experiencing high demand. Furthermore, they should also consider making retry policies more dynamic. For example, dynamically reducing retries depending on the load on their APIs and also putting in jitter/randomness for retries to avoid load surges.

Furthermore, it appeared that traffic had to be manually shifted from Central US to Northern Virginia until the issue was resolved. Since GitHub was detecting high error rates and retry traffic, they should consider balancing traffic across multiple regions for critical infra like this. Having a multi-regional approach rather than a centralized control-plane allows for smoother redistribution of traffic.

Rollback vs. RollforwardN/A

There was no deployment that caused this issue.

Prevention mechanisms🟢 8 / 10

The follow-up actions here are solid (auditing Istio request/concurrency/scale limits, VS Code retry behaviour, reviewing retry limits + backoff behaviour across gateways and clients). However, GitHub has been hitting scalability issues and capacity limitations for the past couple of years as AI usage has exploded. I would like to better understand how they are tackling these issues across their stack more holistically and estimating necessary capacity over the next 12+ months.

CI/CD test coverageN/A

This wasn't caused by a deployment so CI/CD test coverage is less relevant here. However, they did have to ship a PR to decrease retry logic at the API gateway level and update HTTP responses to block Copilot token requests. I'd like to know more about how quickly those changes were shipped and what tests were run on them given the urgency of the incident.

Time-to-root-cause (TTRC)🟡 5 / 10

Impact began at 13:28 UTC and the root cause was identified at 16:36 UTC, a little over three hours later. I'm curious what level of observability they had instrumented on their load balancers because one of the follow-ups was "Improving load-balancer capacity monitoring".

Root cause transparency🟢 9 / 10

Great transparency in the incident page and follow-up blog from their CTO. However, it does lack some details I would like to see (internal vs. external detection, the ~3hr timeline in order to identify the problematic sidecar bottleneck).