For Albuquerque’s Water Authority, a New Vision for Cybersecurity

An ongoing convergence of enterprise and operational technology is proving new opportunities for the utility to secure its operations.

Oldsmar, Florida, may not be the best-known Florida municipality. But after a concerning and potentially dangerous hacking incident in early 2021, it was front and center for all water utilities and municipal leaders across the globe.

During the incident, the attackers briefly used remote access systems in an attempt to increase the amount of caustic soda, or lye, used in the city’s water treatment system to potentially toxic levels. Luckily, technicians quickly detected the intrusion before harm could be inflicted. But it surely served as a wakeup call for water utilities and other critical infrastructure sectors. “Water utilities may not be thought of as being a huge target, but they really are,” says Kristen Sanders, CISO of the Albuquerque Bernalillo County Water Utility Authority. “It’s critical infrastructure that’s not as well regulated on the cybersecurity front as the electrical industry.”

Water utilities have many challenges. Most of the 54,000 drinking water systems in the U.S. serve fewer than 50,000 residents and are confronted by financial constraints and aging infrastructure. According to one security consultant, many also rely on potentially vulnerable remote access systems to monitor and control their facilities.

But the outlook is improving. The America’s Water Infrastructure Act of 2018 required almost all water systems to include cybersecurity considerations in their risk assessments and emergency response plans. Funding in the Biden Administration’s $2 trillion infrastructure proposal includes $111 billion for clean drinking water, including funds to upgrade and modernize systems and infrastructure. And as utilities modernize mission-critical systems, they are seeing the benefits of upgrading technology with security in mind.

“When I started in this job, cybersecurity was something that simply wasn’t talked about,” says Cody Stinson, CIO for Albuquerque’s Water Authority. “Now, it’s talked about every single day.”

IN ALBUQUERQUE, AN END TO THE ‘AIR GAP’

Serving more than 650,000 water users through 3,000 miles of water supply pipelines, the Water Authority serving Albuquerque and Bernalillo County is New Mexico’s largest water and wastewater utility. Its 635 employees oversee more than $5 billion in assets and seek to optimize operations in a desert region where water is a precious asset.

The utility’s modernization journey began several years ago when infrastructure at its groundwater site was nearing the end of its useful life. As part of an upgrade, the site’s existing SCADA (supervisory control and data acquisition) system, which gathers and analyzes real-time data from operational controls, was replaced with one running on a modern networking platform. The project also began a much larger convergence of enterprise and operational technology across the utility.

Along with the addition of smart metering, the transition has already helped the utility save more than $1 million and automate a range of tasks, including the same kinds of chemical adjustment automation that potentially imperiled the breached Florida water system.

At the same time, it revealed new challenges that needed to be addressed.

“Once we had visibility into the environments with Cisco Cyber Vision, we saw the lack of cybersecurity that existed,” says Stinson.

That’s not as dire as it sounds. Historically, utility operational systems relied on the so-called “air gap“— physically isolating mission-critical systems from other networks. While ransomware attacks have occurred on the business side of utilities, they were far less likely to impact operations for this reason.

“Now, with more of these smart water solutions coming out, you can’t just put them off on a little island and have them talk to nothing,” Sanders says. “You’re going to have to put some thought into cybersecurity because the air gap that was relied on forever is no longer an option.”

NEW VISIBILITY

The Water Authority’s ongoing convergence of enterprise and operational technology has provided new opportunities to secure its operations. The Water Authority had previously implemented cybersecurity standards promulgated by the National Institute of Standards and Technology (NIST) on the enterprise side, and it has since applied them to operations as part of network consolidation, according to Stinson.

Upgraded networking across the utility’s surface and groundwater systems also provided the opportunity to use modern cybersecurity tools that provide never-before-seen insights into network usage. “We have visibility; we have firewalls; we have logging on the water side we never had before,” Stinson says.

Importantly, the integrated tools provided a clearer picture of all the hardware connected to networks — including aging tools running operating systems that could no longer be upgraded with the latest security patches.

“In the past, we weren’t really 100 percent sure what they were running or how many servers were outdated,” Sanders says.

Modern cybersecurity tools also automate data collection and provide an intuitive user interface. “We’re actually able to see what these devices are doing — what protocols are running, who’s talking to who — in a way that actually makes sense to us as opposed to being like a spreadsheet,” Sanders says.

Automated monitoring also allows the authority to create baselines of network behaviors that can automatically trigger alerts — particularly helpful with “man in the middle” attacks in which normal network connections are surreptitiously intercepted by cyberattackers.

“Traditional solutions are not going to alert you of a strange command being sent to a PLC (programmable logic controller) because it’s not going to know any different,” says Sanders. “The big thing is that you can’t protect what you don’t know about, and if you’ve got [visibility] into all your devices, you have a very in-depth inventory of what’s on there and what is and not normal that you can use to identify [threats].”

WHAT’S NEXT

Over the next decade, the Water Authority plans to consolidate all its SCADA systems into a virtualized enterprise system that spans the entire water authority.

“It was about convincing the operational folks that the general enterprise can work within a SCADA environment, and we proved that it can,” Stinson says. “That’s the model going forward.”

Like most enterprises, the Water Authority had to respond quickly to the COVID-19 pandemic in spring 2020. As dispatchers and customer service representatives continue to work remotely a year later, leaders are looking to new cybersecurity practices, including zero-trust networking and multi-factor authorization, to ensure the growing number of endpoints created by remote operations remain secure.

“We could have visibility into exactly what’s going on — whether it’s on or off the network, we could still have a good idea that this person authenticated with this IP address at this time from this exact geolocation,” Sanders says.

The utility also is leveraging technology to improve physical security. As it looks to expand network connectivity to remote water tanks and reservoirs, plans are in the works to provide access controls and physical security components, according to Stinson.

“We really need to protect and ensure that the integrity of our water system remains in place,” he says.

For other utilities, the potential of federal infrastructure funding and the growing number of state mandates requiring agencies to implement new security policies and practices and/or improve preparedness represent an opportunity for leaders to shift their organizations to a preventative mindset. Along with securing systems, network-enabled approaches to cybersecurity can provide staff with the time and capacity to shift their focus to other priorities. “Anything we can do to automate and streamline is huge so we can put our focus on planning proactive maintenance,” Sanders says. “You can’t focus on what controls to put in place if everything is constantly breaking and you’re running around trying to fix it. And that’s when you end up with a breach.”