IT staff discovered the state court system’s cybersecurity software had detected unusual activity coming from a system administrator’s account at 2 a.m., well outside business hours. Also suspicious? That system admin was on vacation, said Patrick Brooks, director of IT services for Missouri State Courts, during the recent National Center for State Courts' (NCSC) Court Technology Conference.
What ensued was days of activity from court staff and a tech vendor working to contain the threat and recover. This included removing two accounts and a compromised virtual server as well as requiring more than 5,000 court personnel to reset passwords. They also blocked outbound data traffic to an increasing number of countries.
A picture emerged of an apparent reconnaissance campaign, conducted by threat actors who’d managed to exploit the Log4j vulnerability despite the court system adopting recommended mitigations. After three days working long hours, the courts’ response teams and the vendor were satisfied that the threat had been purged, enabling the courts to open as normal again.
The ordeal highlighted both opportunities to improve as well as strengths that helped stop the incident from becoming a full-blown crisis. For one, it showed just how fortunate Missouri courts were to have had continuity of operations and IT emergency response plans in place.
Courts need to know ahead of time what they’ll do during an incident to avoid wasting time and making decisions on the fly. It also means that if a key person is unavailable, someone else is ready, Brooks said.
This includes identifying all kinds of details, like who IT contacts first after discovering an incident, be it the chief justice, law enforcement or vendor. Missouri opted to first reach out to its vendor to ensure assistance was coming right away, skipping the extra step of having the justice tell them to call, Brooks said.
“It’s not any disrespect in any way, shape or form, but it’s better to get help on the way,” Brooks said. “No one's going to come and yell at me and say, ‘Why did you call Microsoft before you came to talk to me?’”
Plans also should outline details like which systems to prioritize restoring — for Missouri, payroll tops the list — and who’ll be on the emergency response team. It also means listing contact information for key groups. Missouri, for example, realized it hadn’t pre-identified anyone at the FBI, said retired Judge Gary Lynch, who chairs the Missouri Court Automation Committee. Instead, during the incident they’d resorted to contacting the FBI via a supreme court justice who had a friend in the bureau, a fortuitous event but not something to rely on going forward.
Lynch said plans should also make clear who’ll make decisions, whether government will pay during a ransomware incident and whether to prioritize restoring systems even if doing so would hinder investigations into culpability.
There were plenty of other lessons learned from this cyber incident, too.
On the technical side, bolstering defenses meant adopting new restrictions around data traffic. Previously, the court had blocked inbound traffic only. But as employees went through logs to understand the incident, they found indications of outbound traffic going to various other country IP addresses. After progressively blocking additional country IP addresses, the court decided to default to geoblocking all traffic directed outside North America, while allowing exceptions as needed should a judge be traveling, for example.
“What viruses do is that they tend to come in through an IP in the United States and then they call the mother ship … and so when we do the geoblocking out, it’s like they can’t call,” Brooks said.
Sophisticated attackers still could get around this by leveraging a U.S.-based IP address to receive the outbound traffic, but such a measure can help thwart less-dedicated attackers.
Geoblocking also isn’t a precaution everyone can take. State executive branches may need to avoid such measures to do business with overseas parties, Brooks said. But state judiciary branches generally deal with domestic court case participants.
There’s a human dynamic to incident response, too. Managing employee stress was important, and they did this by going on food runs and managing shifts to ensure no one was on for 24 hours, Brooks said. Working with a vendor with a large, global workforce also helped, because it meant the vendor had the staffing to continue working around the clock. Regular vendor progress reports also made the incident seem less bleak, Lynch said.
Finally, Brooks advised courts to make sure their contracts with primary tech vendors include cyber assistance options.
