The control system map most operators did not have

The week

The National Security Agency and the Cybersecurity and Infrastructure Security Agency issued a joint alert, dated July 23, 2020, recommending immediate actions to reduce exposure across operational technology and control systems. It was addressed to the Department of Defense, national security systems, the defense industrial base and US critical infrastructure operators. The alert said internet-accessible control system assets were increasing and being targeted, and that the security of legacy control systems had not kept pace. It described attacker techniques ranging from spearphishing and ransomware to direct manipulation of exposed devices. Recommendations included building an accurate map of control networks as actually operated, reducing internet exposure, having a resilience plan for operating without control systems, and exercising incident response.

Operational technology is usually seen as a concern for utilities and manufacturers. Banks, health systems and public agencies run it too: municipal water and wastewater treatment, hospital building automation, medical gas and clinical engineering systems, and the power and cooling behind every data center. The July 2020 alert set out a sequence of work that applies to all of them. First, map what is actually connected. Second, decide what should be connected and protect it. Third, be able to operate when the control systems cannot be trusted. Each step depends on the one before, and the first is where most operators were weakest.

Map what is actually connected

The alert asked operators to build an accurate map of control system networks as operated, and the distinction matters. Many organizations hold design drawings or vendor documents showing how systems were meant to be connected. Fewer know how they are connected today, after years of maintenance, remote support arrangements, temporary links and integrations feeding data to business systems.

The gaps take predictable forms. At a municipal water utility, it might be a remote access tool a contractor installed years ago. At a hospital, it might be a building management system reachable from the corporate network, or a vendor’s cellular modem attached to a chiller. For a bank, it might be the facilities systems at a colocation data center, run by the provider under a contract that says little about security.

Building the map takes a joint walk-through by IT and operations staff, cabinet by cabinet. Passive monitoring tools designed for control system protocols can then confirm what talks to what without disturbing fragile devices. Active scanning can crash older controllers, so it needs operations sign-off and a maintenance window. The output should record each device, its firmware version, its connections, who maintains it and how vendors reach it. A map is only useful if it stays current, so changes to control networks need to pass through a change process that updates it.

Decide what should be connected, and protect it

With a map in hand, the next step is reducing exposure. Control devices reachable directly from the internet should come off it. Remote access should go through one controlled, monitored path with strong authentication, rather than through tools each vendor installs for its own convenience. Connections between control and business networks should be limited to what is needed, with data flowing out through defined points rather than in both directions.

These choices involve trade-offs. Remote access lets small utilities and hospital facilities teams cover nights and weekends with few staff. Data from building systems supports energy management and regulatory reporting. Disconnecting everything is rarely realistic. Deciding deliberately is: record each connection’s purpose, owner and controls, and remove the ones nobody can justify.

Protection for what remains connected follows from the map. Segmenting control networks into zones by process, monitoring traffic at the boundaries, and alerting on new connections or unexpected commands give operators a chance to see an intrusion before it reaches equipment. Default passwords on controllers and human-machine interfaces should be changed, a basic step the later water utility incidents showed was still often missed.

Vendor contracts deserve attention here. Many control systems are maintained under agreements that predate any security expectations. Renewals are the moment to require named vendor accounts, multi-factor authentication, session logging and notice before changes. Where a vendor cannot meet those terms, the remote path can be wrapped in controls the operator owns, such as a jump host that records every session.

Be able to operate without trusted control systems

The alert asked operators to plan for running when control systems are unavailable or untrustworthy. That includes manual operation and restoring devices from known-good configurations. Many organizations are weakest here. Staff who once ran plants or building systems manually have retired. Procedures have not been rehearsed, and backups of controller logic and configurations may be missing or out of date.

Resilience at this step is as much about people as technology. It needs documented manual procedures, staff trained to follow them and exercises that test them under time pressure. It also needs current backups of controller programs and firmware, stored offline and tested by restoring to spare hardware. Above all, it needs clarity about when to shut a process down deliberately rather than run it on untrusted controls, and who has authority to decide. A water utility, a hospital and a data center will answer differently, which is why the decision has to be made in advance.

Across all three steps, ownership is the common gap. IT security teams often lack visibility of control systems and the expertise to work on them safely. Operations and facilities teams own the equipment but not its security. Vendors maintain much of it under thin contracts. Naming one owner for control system security, with authority across IT, operations, facilities and vendor management, is what allows the map, the exposure decisions and the resilience plan to happen at all.

With hindsight

Control system security moved up the agenda quickly. In February 2021, officials in Oldsmar, Florida, reported that someone had used remote access software to change chemical settings at the city’s water treatment plant, an incident examined later in this series. In 2023 the former city manager said it had been employee error, and the FBI said it had found no evidence of a cyberattack. In May 2021, a ransomware attack on Colonial Pipeline’s business systems led the company to shut down pipeline operations as a precaution. In late 2023, attackers linked to Iran compromised internet-exposed controllers at several US water utilities.

Federal agencies issued repeated warnings to water systems. The Environmental Protection Agency pressed states to assess cybersecurity in water system inspections, then withdrew its 2023 memorandum on the subject after legal challenges. The three steps in the 2020 alert, map, reduce exposure and plan to operate manually, remained the core of the guidance that followed.