Volt Typhoon left little to scan for, so agencies had to know their own normal

The week

On May 24, 2023, Microsoft reported that a state-sponsored actor based in China, which it called Volt Typhoon, had been active since mid-2021 against critical infrastructure organizations in Guam and elsewhere in the United States. Targets spanned communications, manufacturing, utilities, transportation, construction, maritime, government, information technology and education. The same day, the NSA, CISA and the FBI, with cybersecurity agencies in Australia, Canada, New Zealand and the United Kingdom, published a joint advisory. Both described “living off the land”: using legitimate tools built into Windows and network devices instead of custom malware. Microsoft said the actor gained initial access through internet-facing Fortinet devices and routed traffic through compromised small office and home office routers. It assessed with moderate confidence that the campaign was developing capabilities that could disrupt critical communications between the United States and Asia during a future crisis.

The advisory was unusual for what it told defenders not to expect. There was little malware to scan for and few indicators that would stay useful for long. The intruders used the command-line tools, administrative accounts and remote access methods that IT staff use every day. They entered through edge devices that many organizations monitor poorly, and their traffic arrived from ordinary US home routers. For state and local agencies, water and power utilities, ports and transit authorities, this created three separate problems: knowing what normal administrative activity looks like, owning the network edge, and planning to keep services running on a network that cannot be trusted.

Knowing normal administrative activity

The joint advisory listed the kinds of commands defenders should hunt for. They included using the built-in ntdsutil utility to copy the Active Directory database, using netsh to set up port forwarding, and running Windows Management Instrumentation queries to survey systems. Every one of these is something an administrator might legitimately do. No signature can separate the intruder from the employee. Only context can: who ran the command, on which system, at what time, and whether anyone planned it.

That context depends on logging many organizations had not switched on. Process creation events with full command lines, PowerShell logging and domain controller security logs are the minimum. Retention matters as much as collection. An intruder present for months will not be found in thirty days of logs.

The most practical baseline is change management. Copying the directory database should be rare and always tied to a ticket. A rule that alerts on directory copies, new port forwarding or new remote access outside an approved change window catches the misuse that matters most. It also produces few enough alerts for a small team to review. Over time the same approach extends to other rare administrative actions. Each one becomes a matched pair of a ticket and a log entry, and an unmatched entry is a lead.

Federal agencies had a logging framework in OMB’s 2021 memorandum M-21-31. Most state and local bodies had no equivalent, and many lacked staff to read logs at all. For them the realistic route is a shared service: a state security operations program, a managed detection provider, or the services the Multi-State Information Sharing and Analysis Center offers to state and local governments. The agency still has to decide which logs to send and keep the change records the analysts need.

Owning the network edge

Two different edges were involved. The first was the victim’s own perimeter: firewalls and VPN appliances facing the internet. The second was other people’s equipment. Volt Typhoon relayed its traffic through compromised home and small office routers, many past the end of vendor support. Its connections therefore arrived from residential US addresses, which defeats blocking by country and most reputation lists.

The first edge is the agency’s to fix. Public organizations have more of it than they think: pump stations, substations, libraries, fire stations, small field offices and remote access installed by integrators for control systems. The steps are known. Keep an inventory of every internet-facing device with its owner, firmware level and support status. Set a shorter patch deadline for edge devices than for internal systems, because they are attacked first. Take management interfaces off the internet. Require multi-factor authentication on remote access, and send device logs to central monitoring. Replace devices the vendor no longer supports, which usually means a capital request rather than a ticket.

The second edge changes detection logic. A login that comes from a domestic residential address is not reassuring by itself. A VPN session for a service account, or an administrator logging in from an address never seen before, deserves review wherever it originates.

Planning to run on an untrusted network

The third problem was intent. Most public-sector continuity plans assume theft of data or ransomware, which makes systems unavailable. Microsoft’s assessment pointed to an intruder positioned to disrupt at a moment of its choosing. Such an intruder may have watched the network for years and may still be inside it during the response. That changes the plan in three ways.

First, essential services need a defined manual mode. A water utility should know whether treatment and distribution can run without network control, for how long, and with which staff. A 911 center needs a fallback if computer-aided dispatch is lost. Second, operational technology should not depend on the business network for basic operation, so that losing one does not stop the other. Third, incident response needs communications the intruder cannot read, such as out-of-band phones and accounts outside the main directory. It also needs a tested way to rebuild the directory from a known clean state.

Most US critical infrastructure is run by local governments, cooperatives and small companies with thin security staff. They cannot do everything at once. A sound order is to know every internet-facing device and retire the unsupported ones. Next, log administrative activity somewhere it will be read. Then write down how the most essential service would run for a week without the network. Then test one part of that plan each year.

With hindsight

The threat proved larger than first described. In January 2024 the Justice Department announced that the FBI had disrupted a network of hundreds of compromised, end-of-life routers that Volt Typhoon used to hide its activity. In February 2024 CISA, the NSA and the FBI said the group had kept access to some victim environments for at least five years. They said it was pre-positioning for disruptive or destructive attacks on communications, energy, transportation and water systems in a conflict. The same agencies published joint guidance on detecting living-off-the-land techniques. Later in 2024 a separate China-linked group, known as Salt Typhoon, was found inside major US telecommunications carriers. Edge device replacement has been the most concrete response and the easiest to fund. Behavioral detection and manual-mode planning remain harder, especially for the smallest operators.