
Microsoft says the 365 outage that hit FlySafair, SAA and Absa lasted almost 24 hours, and traces it to a network change first rolled out in July.
Microsoft has closed the Microsoft 365 outage that stopped email at FlySafair, SAA, Absa, the City of Cape Town and many other South African organisations last week. Customer impact ran from about 6.20pm on Thursday 12 November to 6.10pm on Friday 13 November, South African time: almost 24 hours.
Its explanation is that the fault was not new. In an update on Friday evening Microsoft said "the incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change". Its preliminary post-incident report, which TechCentral reported on Monday, traces that to a change first rolled out in July.
When this newsroom's earlier report was filed on Saturday morning, Microsoft had not yet declared the incident over. It did so in its final updates early on Saturday, saying it had addressed the underlying issue and restored service to the remaining affected customers and regions.
The outage was tracked as EX1550543 for Exchange Online, the service behind Outlook mail and calendars, and MO1550821 for the knock-on effects on the rest of Microsoft 365. Microsoft has said throughout that it was a technical fault and, in its words, "not related to cyber or AI / agents".
According to TechCentral's account of the preliminary report, the July change was made to servers in a networking gateway that sits in front of many Microsoft 365 services. It went through Microsoft's staged rollout and, the report says, "for more than three months, the change was deployed worldwide without impact". The defect only showed under certain loads, when tasks in an internal Windows process got stuck waiting on each other until servers ran out of processor and memory and stopped responding. Microsoft says it was worse on newer machines with more processor cores, while its tests "tended to focus on weaker hardware with fewer cores".
The trigger came early on Thursday, when Microsoft's systems automatically moved Teams traffic from a second gateway onto the one that carries Exchange Online. Nearly 14% of that gateway's capacity was already offline because of an unrelated problem the day before. When US traffic peaked, the bug fired.
Microsoft's own safety systems then made it worse. Traffic management moved users off failing servers onto healthy ones, which pushed those into the same fault. "This caused a cascading failure from server to server as the same amount of load was spread across fewer servers," the report says. The fault spread across the gateway in under two hours, and at the worst point more than 80% of its servers were down. Barriers meant to keep failures inside one region did not hold because of what Microsoft calls "a separate issue". The report does not explain that issue.
That matches the updates Microsoft posted during the night, which described servers failing health checks, a process that sets up secure connections becoming overwhelmed, and servers that came back falling over again as soon as traffic reached them. Because the faulty code was months old, Microsoft's usual rollback tools could not quickly undo it.
At 9.20am on Friday, about 15 hours after the first alert, Microsoft started deploying an update that reversed the faulty change, taking groups of servers out of service, updating and restarting them, and starting with regions in or entering business hours. TechCentral reports that Europe was back to normal by 1.35pm and webmail worldwide by 4.17pm. Microsoft also extended its email timeout window to 48 hours during the outage, so that queued messages would still be delivered rather than bounced.
Teams, OneDrive, SharePoint and the Office apps could be moved away from the failing gateway and recovered sooner. Exchange Online could not be moved "for scale reasons", which is why email was out for so much longer than anything else.
The preliminary report lists closer monitoring of how traffic is redirected, limits on how load is redistributed so that one failure cannot cascade the same way, and more capacity. Microsoft says a final report with the full list of repairs will follow by 20 November.
For local IT teams the practical question is now money. TechCentral notes that under Microsoft's service level agreement, Exchange Online customers can claim service credits when monthly availability falls below 99.9%, with larger credits below 99% and 95%. The credits are not automatic: an administrator has to lodge a claim with Microsoft support, with times, affected users and errors, by 31 December. They are capped at the month's fees and do not cover lost sales or other losses.
Microsoft has not yet explained the "separate issue" that let the failure cross regional boundaries, and its final report is not due until 20 November. We have not seen the preliminary report ourselves, and secondary accounts of Microsoft's status updates give slightly different times for its final post.
None of the South African organisations hit last week, including FlySafair, SAA, Absa, SARS and the City of Cape Town, has put a figure on what the outage cost them. In the reports we could find, none has said publicly whether it will claim service credits.
Two limits on this account should stay visible. Microsoft's own status pages could not be read directly while the updates were being checked, so the timeline above comes from copies of those updates and from news reports of them. The preliminary incident report was not available as Microsoft's own document either. Where this article describes the July gateway change, the Teams traffic shift, the cascade past 80% of one gateway's servers, and the 9.20am start of the fix, it is following a published account of that preliminary report. Sources also disagree on the clock time of Microsoft's final post, which is why this article says only that the final updates came early on Saturday.
The claim window is the part a South African administrator can act on without waiting for the final report. The credits are not paid automatically. A claim has to be lodged with Microsoft support, with the times, the affected users and the errors, and the deadline in the published account of the agreement is 31 December. The credits are capped at the month's fees. They do not pay for lost sales. No South African organisation named in the outage has said what it cost, or whether it will claim.
Source: SA Tech News




