A deleted file, a national ground stop, and the runbook nobody rehearsed

The week

Late on January 10, 2023, the Federal Aviation Administration’s Notice to Air Missions system failed. NOTAMs carry safety information that pilots must review before flight. Early on January 11 the FAA ordered a nationwide pause on domestic departures, the first such ground stop since September 2001. Departures resumed later that morning, but thousands of flights were delayed and more than a thousand canceled. In its preliminary review the FAA said contract personnel had unintentionally deleted files while working to correct synchronization between the live primary database and a backup database, and that it had found no evidence of a cyberattack or malicious intent. The outage came weeks after Southwest Airlines’ holiday scheduling collapse, and both events drew attention to aging technology in critical transportation systems.

The obvious conclusion from the NOTAM failure was that the FAA needed a new system, and it did. A new system alone would not have prevented that night, though. The trigger was a maintenance task on a primary database and its backup, performed in a way that damaged both. That kind of failure can happen on a platform of any age. Three separate sources of risk converged: the procedure, the people performing it, and an architecture that let one mistake reach both copies.

The procedure

Re-synchronizing a primary database with its backup is routine work. It is dangerous because it touches both copies of the data on which the system’s resilience depends. A sound procedure for that work has several parts. It needs a written runbook that someone other than its author has reviewed. It needs a fresh, verified copy of the data taken beforehand and stored where the task cannot reach it. It needs a rehearsal in a non-production environment that mirrors production closely enough to matter. And it needs a rollback plan with a fixed point at which the team stops and restores.

Legacy systems often lack some of these. Their procedures grew up informally over decades. Steps live in the memory of experienced staff, in personal notes, or in documents last revised before the most recent major change. Nobody decided to run critical maintenance this way. It is what remains when a system outlives several generations of its operators.

For a program director, the first useful question about a critical legacy system is therefore not its replacement date. It is whether every maintenance procedure that touches production data or its backups is written down, current, rehearsed and actually followed. A short audit can answer that within weeks: list the procedures, record when each was last reviewed and last executed, and compare the document with what operators really do.

The people

The FAA said contractor personnel made the error. That needs careful reading. It does not show that contractors are riskier than federal employees. It does point to a common pattern in government IT. Detailed system knowledge sits with contract staff whose teams can change at each recompetition. The federal program office holds accountability but not always the operational detail.

That arrangement needs specific safeguards. Contracts for critical systems can make current runbooks a government-owned deliverable, updated after every significant change. Changes to production data can require a second qualified person to review and approve each step in real time, with the session recorded. Transitions between contractors can include supervised handover of operational procedures, with outgoing staff observing incoming staff perform them. The government should also keep enough technical staff of its own to understand and challenge what the contractor is doing on its most important systems.

Timing matters as well. Maintenance on critical systems is often scheduled overnight to limit disruption. That reduces the operational impact of a planned outage but also means fewer experienced people are available when something goes wrong. Agencies can set rules for which tasks require senior staff on call, and for when a task must stop and wait for daylight.

None of this is free. Two-person control slows maintenance. Government-owned runbooks require someone to review them. These costs are small next to a national ground stop, but they have to be funded in operations budgets that are usually squeezed to pay for modernization.

The architecture

The third risk was that one maintenance error could affect both the primary and the backup. A backup continuously synchronized with production protects against hardware failure. It does not protect against corrupted data or a bad change, because the corruption is copied too. Protection against operator error needs a different mechanism: point-in-time copies held separately, with a tested way to restore them quickly.

Here legacy design genuinely matters. Older systems were often built when synchronized replication was the main resilience goal. Restoring from a point-in-time copy may be slow, poorly documented or never practiced. A modernization plan should make fast restoration of known-good data an explicit requirement, with a target time tied to the mission. For a safety system that grounds aircraft when it fails, that target is measured in minutes or a few hours. The design also needs a degraded mode: a way to keep the most essential information flowing, even manually, while the main system is restored. Deciding in advance what that minimum service is, and who can declare it, avoids improvising a national decision in the middle of the night.

Until a replacement exists, the old system has to be run safely, and the replacement can be damaged by a careless maintenance step as easily as the original. The most useful test applies to both. Ask the operations team to demonstrate a restore from an isolated copy and time it. The result shows where exposure sits more clearly than the system’s age does.

With hindsight

Congress responded with the NOTAM Improvement Act of 2023, which gave the FAA until September 30, 2024 to modernize the system and add a backup. According to a group of senators writing in April 2025, the backup was in place by mid-2024 but the modernized primary system was not. In May 2025 the FAA selected CGI Federal to lead a cloud-based replacement, and the first phase was reported complete in 2026. Wider air traffic control technology problems drew sustained attention through 2025, and Congress approved substantial new funding for modernization. The replacement program shows the limits of replacement as the main answer: it took years, and the old system had to be run in the meantime. The operational basics, rehearsed runbooks, two-person control for production data changes and isolated point-in-time backups, protect both old and new systems. Agencies should ask whether they have actually restored known-good data from a separate copy within the time their mission allows.