Get Your Operational Technology Off the Internet

Twelve states reported hacked water systems this summer. What kept the damage contained was a worker turning a valve by hand.

Over the past several weeks, at least twelve states told the FBI that their water infrastructure had been breached, from treatment plants to pump stations. Hundreds of thousands of Americans were affected. No one was hurt.

That outcome was not the product of resilient design or top-of-the-line cybersecurity. It held because trained operators still had manual overrides and knew how to use them.

The Cybersecurity and Infrastructure Security Agency (CISA) had warned months earlier that geopolitical tension and aging domestic infrastructure made exactly this kind of attack likely (advisory). This was a prelude to more capable attacks to come, across energy, water, healthcare, and communications. The 2021 Texas grid failure killed several hundred people with no adversary involved at all; the same weaknesses in hostile hands are a different order of problem.

Much of the technology that would close these gaps has already been invented and paid for. Most of it still sits in national labs and early-stage companies that have spent years trying to reach a utility that can afford it.

What happened

By August 4, at least twelve states had reported attacks on water and wastewater systems (Axios). Minnesota alone counted more than 30 systems across 30 communities. Braham shut down its well and treatment plant and ran the town on backup water tower storage, resuming operations within two hours. Plymouth had two water towers and 14 sewer lift stations compromised; operators identified the attack vector, disconnected the cellular devices from the network, and ran the system manually.

In a more serious case, Clayton County, Georgia lost a pump station for roughly three hours, causing a loss of water pressure or access entirely. The county had to issue precautionary boil-water advisories to many of its 300,000 customers. Local reporting described “unauthorized cyber activity that may have caused or contributed to the disruption” (States Newsroom, WSB-TV).

How it happened

The failure point in most cases was the programmable logic controller. A PLC is a rugged industrial computer built to run a single deterministic control loop, continuously, for decades. In the simplest cases it was reachable from the public internet with no password or the manufacturer’s default. Nothing sat in front of it.

Where the controller was not directly exposed, access still had to come from somewhere. Remote sites are often served by a cellular modem wired straight to the PLC. If no private access point name (APN) is provisioned — usually a cost decision — the SIM gets an address reachable from the open internet, and the modem passes command logic directly to the controller. Modem logins are frequently left at factory defaults, so one working credential can open every site running the same vendor’s hardware.

The joint cybersecurity advisory reports that actors “deployed Dropbear Secure Shell (SSH) software on victim endpoints” (FBI IC3), giving them persistent remote access over port 22 that survived reboots. From there they changed controller IP addresses and passwords to lock operators out of their own equipment, and modified add-on instructions — the reusable code modules inside a PLC program — to disable safety shutdown and alarm logic.

This is still developing. On August 19, the NSA, CISA, FBI, Department of Energy, and EPA issued a joint advisory on an active threat to Siemens S7 series controllers (advisory). Among the attack vectors it names for operators to watch: AI-assisted code development, public code repositories, and internet-device scanning services. The tooling for finding and exploiting an exposed controller is getting easier to use, not harder.

The IT/OT gap

Enterprise IT protocols exist to firewall employee-facing computers. Operational technology, which governs the pumps and valves, gets a fraction of that attention — and cellular access to a modem bypasses the enterprise firewall entirely, trading control for exposure.

The immediate fixes are cheap. A private APN alone would have stopped most of this. On a public APN, a modem is discoverable by anyone scanning the carrier’s address space; no attacker needs prior knowledge that the equipment exists. A private APN isolates the SIMs behind corporate networks and VPNs, which also stops the device-to-device scanning that let one compromise become thirty. A site-to-site VPN removes the public listener. Multi-factor authentication and unique credentials cost nothing.

Acting CISA director Nick Andersen, speaking at Black Hat in early August, put it plainly: “We’re seeing things like [programmable logic controllers] that are open and accessible on the internet with either no password set or default password set. We’re not making ourselves hardened targets.” His prescription: “Get your operational technology off the internet, set a password” (Nextgov).

The structural gap runs deeper than passwords. IT devices last three to five years, patching is routine and reversible, authentication is a baseline expectation, and endpoint detection is near-universal. OT hardware lasts 20 to 40 years. Patching often means shutting down the plant. The protocols were designed without authentication, and the controllers are too lightweight to run detection software at all.

What we need to do

Beyond the simple OT hygiene, several technical interventions would help. Cryptographic attestation, where the logic running on a PLC is checked against a signed original, would let a controller prove it has not been rewritten. Physics-based anomaly detection could measure water pressure, flow, or current against a hydraulic model — a digital twin — to catch a manipulated PLC reporting false values. An inline device can add authentication to a controller without replacing the PLC at all.

In response to the campaign, the FBI proposed manual operation as a form of cyber mitigation. But there are several vectors of attack and defense worth considering for critical infrastructure, across both the cyber and physical dimensions and the local and system-wide ones.

Figure 2. Threats and available technical solutions, mapped across the cyber/physical and system/local dimensions. Read a threat block against the solution block in the same position.

An attack can be launched at the system-cyber level from anywhere in the world, while what prevented tragedy this time was a person intervening at the local-physical level. That asymmetry is the whole problem: cyber attacks scale far more rapidly than physical defenses.

Why commercialization is the bottleneck

The upstream moves are the right ones: secure-by-default hardware, private APNs out of the box, authenticated protocols. But recall the 20-to-40-year lifecycle. A controller specification written in 2026 does nothing for one purchased in 2004 by a town that cannot afford to replace it. Vendor mandates are necessary and slow.

The local-physical layer is the only one that can improve on the timescale of the threat, and the limit there is cost rather than invention.

Several companies are already building for it.

FedTech alumni company Cynalytica developed SerialGuard, a passive sensor that reads serial traffic between controllers and field devices without touching the control signal; it now runs a dedicated water and wastewater practice alongside a partnership with PwC.

Sentinel Devices, from our 2023 cohort, runs anomaly detection at the edge, fully air-gapped, with all data and processing held inside the facility; its partners include Oak Ridge National Laboratory and the Air Force.

SixMap, from our 2021 cohort, has raised nearly $14 million doing precisely what Andersen described — finding every internet-facing asset an organization owns — and has been evaluated by the Army under an SBIR.

The list goes on, and the lab side is just as far along. The National Renewable Energy Laboratory has patented a hybrid intrusion detector that cross-checks cyber logs against the physical behavior of the power system, paired with a network that isolates a compromised node and heals around it. Argonne has a Markov decision framework for risk-informed operations, already carried from nuclear plants into offshore oil and gas. Idaho National Laboratory has spent years mapping how failure in one sector cascades into the next. Each fills a block in Figure 2. None was built with a small water utility in mind.

The gap between a laboratory capability and a pump station in Braham, Minnesota is commercialization, not more research. Commercialization has to move faster than the hardware replacement cycle, and deployment has to move faster than the interval between attacks. We cannot out-deploy an attacker one utility, one plant, or one operator at a time — but we can commercialize and scale new layers of defense. It’s time to modernize and secure our infrastructure.