That’s one conclusion reached by Prescott, Ariz.'s IT Director Nate Keegan, whose city very nearly fell to a ransomware attack in 2020.
Hackers at the time took advantage of a weak password to a legacy remote access system, letting them enter the city network, Keegan recounted. Thanks to quick detection by the FBI and the employee whose device was compromised, the city was able to act in time and contain the intrusion before it became a full-blown event. But the city can’t rely on such good fortune in the future.
“I would consider that being us dodging a bullet — barely, but close enough,” Keegan told Government Technology. “We were pretty lucky.”
The attack came at a particularly inopportune time. IT staff were already busy with budget season and with migrating the city to a new Internet service provider (ISP) — a project that would have phased out the remote access system the hacker compromised. But that ISP migration was paused when winter storms flooded the IT team’s offices in the City Hall basement, forcing personnel to turn instead to moving equipment to drier environs.
“It also happened to be the same week that COVID hit Arizona,” Keegan said. The scramble to enable remote work added to the confusion.
Government IT staff often simply have too much to do — and when personnel are overburdened and distracted with other needs, that’s when things slip that can turn into exploited vulnerabilities down the line, Keegan said.
“It’s the little things that end up killing you,” he said.
In this case, a network engineer detected a suspicious login on his computer and changed his password. The FBI soon called to warn about a poster on the dark web who claimed to have access to an Arizona government network. The FBI believed the perpetrators intended to sell this access to ransomware attackers or use it themselves to launch their own extortion.
The warning prompted the city to shut off systems and unplug the accessed device. IT staff followed up by triggering password resets for all admin accounts and searching for other potentially accessed accounts to shut down. The team also replaced some systems that hackers could have accessed and monitored for any signs of yet-uncaught malware left behind.
A more successful attack could’ve disrupted a vast array of city operations, including those for “airport, finance, public safety, utilities, you name it,” Keegan said.
The organization is also part of an insurance risk pool with other agencies, meaning that its decisions over whether or not to pay ransomware could cause a price hike for other government entities in the state, too.
Preventing Round Two
Prescott cannot bet on getting that lucky again.
Keegan said the IT department has been adding new layers of security, including requiring multifactor authentication (MFA) on IT and other high-target accounts, adopting a new tool and raising awareness about the necessity of hard-to-crack passwords.
Prescott adopted a tool called Enzoic to run an automatic review of employee passwords. That tool revealed that 35 percent to 45 percent of the passwords were already compromised in data breaches, the same or very similar to other employees’ or very simple — think something like “password123,” said Keegan.
Those metrics helped IT push for a rule that all employees had to create new logins that met NIST’s criteria for strong passwords. Employees would confirm this by running their new credentials through the checking tool, which would flag easy-to-compromise ones.
IT also launched new efforts to raise awareness about the real — and serious — risks created by weak passwords. Employees often don't realize how the choices they make regarding their passwords can impact the entire state, Keegan said.
“I don't know that everybody necessarily equates ‘my individual account’ [with] somehow being the key to doom and all these services literally going offline,” Keegan said.
But cyber criminals often make use of compromised login details, with Verizon’s 2021 Data Breach Investigations Report finding that 61 percent of incidents impacting U.S. and Canadian public-sector agencies involved credentials.
That report recommends adopting certain CIS Controls, such as those related to managing account access and managing accounts’ privileges. That can include requiring MFA to access certain important systems.
Keegan said he hopes to also require MFA for all accounts, not just the most sensitive ones.
Automated Response, Detection
To stay ahead of future threats, Keegan also wants to adopt further automations that can increase the capacity of IT teams.
He aims to bring in security orchestration, automation and response (SOAR) and security information and event management (SIEM) tools this year. Attack attempts typically ramp up after 6 p.m., on weekends and holidays, when staff are out of office, Keegan said. These tools, however, could automatically send out threat alerts and launch certain responses, thus ensuring a level of detection and prevention continues even when staff are away or focused on other responsibilities.
Above all, Prescott’s 2020 brush with ransomware may have highlighted the necessity of keeping a tight focus on cybersecurity.
“Cybersecurity is important, but it’s easily pushed to the side by a lot of other things — and that’s really, really dangerous,” Keegan said.
