Microsoft traces 365 outage to July gateway defect
Microsoft’s preliminary account links the 365 and Exchange outage to a July gateway defect triggered by load and reduced capacity.
Microsoft has linked the November 12 Microsoft 365 and Exchange Online outage to a defect introduced in a July network gateway update, according to a preliminary report described by The Register. At the outage’s worst, more than 80% of gateway servers were unresponsive, the report said.
The failure disrupted email and other productivity services across regions, with Exchange Online particularly exposed because it could not be moved away from the faulty gateway at scale. Microsoft said on November 14 that it had restored service to the remaining affected customers and regions.
The Register said Microsoft’s November 15 preliminary report identified two coinciding pressures: Teams had failed over to the primary gateway supporting Exchange Online, increasing load, while nearly 14% of gateway server capacity was out of service. Microsoft said the remaining capacity had enough headroom under normal conditions, but the combination triggered the latent defect.
The defect caused lock contention inside a Windows process, exhausting CPU and memory. As servers lost connectivity, traffic shifted to other servers, which encountered the same problem. The issue spread across the fleet within two hours, according to the report.
Microsoft told The Register that its July update had passed load testing and initially deployed without impact. Its tests, however, tended to use weaker hardware with fewer cores; the contention worsened on newer machines with more cores.
The reported timeline puts the Teams failover at 01:07 UTC on November 12 and the first connectivity alert at 16:03. Engineers added capacity and rerouted traffic, but restarted servers could become unresponsive again under load. Microsoft’s status updates also said it extended the email timeout to 48 hours to protect delayed messages.
A straightforward rollback was not possible because the defective update was months old, The Register reported. Instead, engineers developed an update to return the affected code path to its previous state and began applying it at 07:20 UTC on November 13; recovery followed in stages.
The full preliminary report was not independently available, and the reporting did not give a count of affected customers. The Register said a final incident report is due November 20.
Comments ()