Storm-0558’s key escaped through a crash dump, a path every agency development team also has
Microsoft’s September 2023 post-mortem described a signing key that left production in diagnostic data, tokens accepted beyond their scope, and a key that lived for years. Agencies that build their own applications carry all three risks in their own code and contracts.
Looking Back revisits one week at a time, from January 2020 to today, and asks what it meant for the banks, health systems and public agencies we serve. The series was written in 2026, with the benefit of hindsight, and each article says which week it looks back on.
The week
On September 6, 2023, Microsoft published the results of its investigation into how the China-based actor it calls Storm-0558 obtained a consumer signing key used to forge authentication tokens. In July, Microsoft had disclosed that the actor used the key to read email at about two dozen organizations, including U.S. government agencies, through Exchange Online and Outlook on the web. The September account said the key had most likely ended up in a crash dump from a signing system. The dump was then moved from an isolated production environment into a debugging environment on Microsoft’s corporate network, and the actor later compromised an engineer’s account with access to that environment. Microsoft also described a validation flaw that let its mail system accept a token signed with a consumer key for enterprise accounts.
Storm-0558 is usually discussed as a story about cloud provider accountability, and the federal review that followed in 2024 concentrated there. The September post-mortem also described failures that have nothing to do with being a hyperscale provider. Crash dumps, debugging environments and token validation exist in every organization that writes software, including federal and state agencies with their own development teams and the contractors who build systems for them. The post-mortem points to three ways a signing key turns into an attacker’s asset, and they follow the life of a key in order: how key material escapes, how far a stolen key reaches, and how long it stays useful.
How key material escapes
The most striking detail was mundane. A system crashed, a memory dump was captured for diagnosis, and the dump contained key material that should have been excluded. Automated checks meant to catch secrets in such dumps missed it. The dump then moved, as diagnostic data routinely does, to an environment where engineers could examine it, which was less protected than the production system it came from.
Agency development teams move diagnostic data this way every week: core dumps, heap snapshots, verbose application logs, database copies made for troubleshooting, and browser session recordings attached to vendor support tickets. Each can carry secrets, session tokens or personal data. Three controls address the path. First, keep signing keys in hardware security modules or managed key services, so that the private key never sits in application memory and cannot appear in a dump. Second, scrub diagnostic artifacts automatically before they leave production, and test the scrubber against seeded fake secrets as part of each release. Third, treat any environment that receives production diagnostic data as production, with the same access reviews, phishing-resistant authentication and monitoring.
The third control is the one most often skipped, because debugging environments are where engineers expect freedom. An agency can keep that freedom by giving engineers synthetic or scrubbed data by default and making access to raw production artifacts a logged, time-limited request approved by the system owner.
How far a stolen key reaches
A consumer key should never have signed a token that an enterprise mail service accepted. Microsoft attributed the failure partly to developers assuming that a shared library performed checks it did not perform. One key therefore opened doors well beyond its intended scope.
Agencies run many applications that accept tokens from identity providers: a commercial cloud identity service for staff, Login.gov or a state platform for the public, and federation with partner agencies. Each application has to check more than the signature. It must confirm the issuer, the audience, the tenant and the scopes, and reject a token that is genuine but meant for a different application or population. Libraries differ in what they verify by default, and their documentation is often vague on exactly this point.
The practical fix is to make validation testable. A security test suite for each application can include forged tokens, tokens from the wrong tenant, tokens for another audience and expired tokens, and every one should fail. Development contracts can require that suite and its results as a deliverable at each major release. Code review checklists can name the validation function and the parameters it must receive, so that a reviewer can confirm it in minutes instead of trusting a wrapper.
Ownership matters here too. When an application team assumes the identity team validates tokens, and the identity team assumes each application does, nobody does. The system security plan for each application should state which component performs each check, and the authorizing official should see that statement before signing an authorization to operate.
How long it stays useful
The key Storm-0558 used had been created years before the intrusion. A long-lived key gives whoever obtains it a long window, and a key nobody tracks may never be revoked. Rotation limits the window. An inventory makes revocation possible at all.
Many agencies already track TLS certificates because expired certificates cause visible outages. Token signing keys, API keys used for service-to-service calls and the certificates trusted for federation rarely get the same treatment. A useful inventory records each key’s purpose, the applications that trust it, where it is stored, its owner and its expiry, and it covers keys held by contractors and software-as-a-service providers on the agency’s behalf. Rotation should be scheduled and automated where possible.
The harder test is emergency rotation. If a key is suspected compromised on a Friday night, can the agency replace it and have every dependent application accept the new key without an outage? That procedure needs a written runbook and a rehearsal, ideally in a non-production environment that mirrors the real dependencies. A program director can measure progress with two numbers: the share of signing keys in the inventory with a named owner, and the time taken in the most recent rotation exercise.
With hindsight
The Cyber Safety Review Board’s report in April 2024 concluded that the intrusion was preventable and pointed to a cascade of security failures at Microsoft. Before it was published, Microsoft updated its account to say it had not found a crash dump containing the key, which left the exact path uncertain while leaving the general weaknesses clear. Microsoft’s Secure Future Initiative, announced in November 2023 and expanded in 2024, committed to moving signing keys into hardware-backed protection, automating rotation and standardizing token validation, close to the agenda above. CISA finalized its secure software development attestation form in March 2024 for software sold to federal agencies, which asks producers to attest to practices that include protecting secrets. Those rules bind vendors. The applications agencies build and commission themselves still depend on the agency’s own key inventory, validation testing and rotation drills.