A typical example of the problem can be found in a March 2019 audit of the Maryland Department of Information Technology (DoIT) by the state Office of Legislative Audits. The office found the department’s customer agencies’ workstations had not been updated with the latest releases for three application software products that are known to have ongoing security-related vulnerabilities. The report also pointed out that “workstations were not being regularly updated” with software patches, increasing the risk of security breaches “from attacks focused on the related vulnerable application software.”
The problem faced by Maryland’s DoIT is common throughout government and the private sector. In 2018, the Ponemon Institute, an international research organization, surveyed 3,000 cybersecurity professionals globally and found that more than half of the organizations suffered a data breach in the last two years. Of these, a majority attributed the breach to a vulnerability for which a patch was already available. The organizations that have been able to avoid a breach reported being able to detect a vulnerability quickly, and could patch vulnerabilities in a timely manner.
So, what is keeping government IT leaders from doing a better job of patching their systems and how are some IT organizations approaching the issue systematically?
The Hard Lessons of Decentralized IT
Nicholas Andersen, former chief information security officer of Vermont, pointed out that traditionally, governments have not thought of themselves as prime targets for nefarious online actors. Obviously, they can no longer think that now. “It is a mix of a lack of resources and starting to recognize that the risk to them is as significant as it is to larger government organizations,” he said.The CIO or CISO must be able to set patch management standards and enforce compliance, according to Andersen. “If they can’t do that, then they are not going to be able to succeed,” he said. “Decentralized IT in a lot of ways does not lead to enabling that CIO or CISO to be successful in this regard.”
Peter Romness, the cybersecurity solutions lead in the U.S. Public Sector CTO Office at Cisco Systems, noted that after a breach, you often hear people saying that if people would just patch and do basic hygiene, this wouldn’t have happened. “Then if you talk to people responsible for doing this work, they say, ‘That is true, and we do patch as much as we can, but it is not as easy as you make it sound,’” he said.
State and local governments are the most budget-constrained organizations in terms of cybersecurity that there are, Romness added. “When they need to do something on cybersecurity, it takes time to get the money budgeted for it,” he said. “They also have a hard time getting cybersecurity staff.”
In a recent blog post, Romness pointed out that busy IT execs may be managing thousands of computers that need periodic updates and, in some cases, they may not even know all the systems in their environment. “Managing updates on so many different pieces of software, on so many systems, can be overwhelming and effective updates may slip through the cracks,” he wrote. Of greater concern is that patches are only developed for known vulnerabilities. Once these zero-day vulnerabilities are found, it can take time to develop and distribute the patch. “But newer (or undiscovered) malware will slip right through the latest patch,” he said.
Romness argued for taking a holistic, architectural approach to cybersecurity. “We used to talk about defense in depth, but I have come to see that as throwing good money after bad and piling up solutions that become hard to manage. Now I like to talk about effective defense that makes sense,” he said. “That means you look at what you have, determine what is important to your organization and make appropriate defense decisions.”
The Speed Problem
One key issue around patching is speed. Hackers aren’t just attacking more aggressively, they are doing it at a much faster rate, according to Ponemon’s report, Patch Work Demands Attention, which was produced for ServiceNow, a cloud computing company. With the rise in AI-fueled attacks, the research organization believes the speed of hacking attacks will only increase. But many organizations rely on manual patching processes and have disconnected systems, making it harder to fight back in a cohesive way. That’s especially true in state and local government, but strategies are emerging.Vermont had a decentralized IT management approach until two years ago. When Gov. Phil Scott came into office, one of the first executive orders he signed created the Agency of Digital Services (ADS) as a cabinet-level department to support centralized IT management statewide. Andersen said that new centralized IT was key to how ADS approached patch management.
Previously, each agency or department had its own team to handle their own IT system deployment through the full life cycle, including patch management. Some agencies had only one or two IT people. They were focused on things that the agency needed (database administration or Web server management) and some didn’t have people with security expertise or enterprise architects or cloud architects. Others were better funded and had invested money in their IT infrastructure, but that wasn’t the case across the board.
“Now [Vermont] can consolidate all of that, not only to generate savings for the state, but also to provide greater visibility into its security posture and better governance of patch management status,” Andersen explained. Vermont now has an enterprise approach with a 30/60/90-day plan — 30 days for mitigating high vulnerabilities, 60 for medium vulnerabilities and 90 for low-risk vulnerabilities.
“Bringing everything under one agency allowed [Vermont] to build a shared services team to provide enterprise hosted options,” Andersen said. With a centralized patch management system, the shared services department can find high-level patches that need to be applied urgently. The department can respond quickly to alerts from a vendor such as Microsoft, the Department of Homeland Security, or the Multi-State Information Sharing and Analysis Center (MS-ISAC).
One important task is identifying legacy operating systems and applications and the agency business processes they support. That includes the network infrastructure too. “Every networking device you might use has its own vulnerabilities, and the same for mobile devices,” said Andersen.
Microsoft, in particular, does a good job of forecasting years in advance when they are going to end support for their systems, and when organizations are going to be able to purchase extended services to continue receiving critical patches even beyond their end-of-service date, according to Andersen.
An enterprise software solution provides patch management and patch status updates. A complementary vulnerability management system gives a comprehensive vulnerability management picture to ADS and department IT leaders.
If there is not a patch available, ADS wants to be able to see where an agency has a road map or plan to no longer use that end-of-service hardware or software and get it off the network. In the interim, if there is something the business says is critical to its mission, and that is not supported any longer by a vendor, IT will work to identify compensating controls — to keep it relatively isolated on the network and turn off nonessential services to reduce potential attacks and isolate those systems that are no longer supported.
“Having that vulnerability awareness platform in place allows [Vermont] to have a single person on staff who has visibility into everything in the enterprise,” Andersen said, adding that this visibility allows for conversations with IT leaders at the agency level about risks and needed updates.
Update that Operating System
One of the biggest complaints about information security — including patch management — is the need for more personnel to carry out the work. But more doesn’t always equal safer, according to the Ponemon Institute. Improving an organization’s security posture won’t happen with more personnel as long as the patching process remains broken. States with lean IT staff have begun to figure that out.In South Dakota, patch management is made somewhat easier by the fact that IT is highly centralized in the Bureau of Information and Telecommunications (BIT), where a small team of technology engineers oversees patch management. They use a tool from Ivanti for custom third-party patches, but also for patching Windows machines, according to Matt Guelde, technology engineer.
“Within the Ivanti endpoint management software, the patch definitions come to us with metadata that tells us how critical the patch is, both from Ivante’s perspective and the vendor’s perspective. We typically home in on those patches that have a critical or high vulnerability rate,” he said.
Michael Hanson, technology engineer, added that the state is on a fast track to making sure that all workstations are running Windows 10. “We are sitting at 91 percent now. By the first of the year, we should be at 100 percent.”
BIT sets policies for how agencies deal with software vendors and IT service providers, according to Deb Dufour, LAN service manager. “If it involves technology purchases, we have a review process the agencies have to go through,” she said. “Part of that review is they are required to have a maintenance agreement with the vendor. If a vendor is hosting a system for the state, we ask about how they do patch management.”
Building Risk into the Patching Equation
One of the first things Maria Thompson, chief risk officer at the North Carolina Department of Information Technology (DIT), noticed when she moved to state government from the federal sector was the lack of visibility. “There are always gaps in your environment you can’t see,” she said, “and you can’t protect what you can’t see.”Thompson uses a practice common in the federal sector called Continuous Diagnostics and Mitigation (CDM), which has enabled DIT to take a risk management approach to patching. “We are trying to gain better visibility across the landscape. We need to understand what our capabilities are and the gaps so that we can better protect ourselves.”
She described North Carolina state government IT as a hybrid model, with some agencies that are consolidated and some that are not. “We have to build relationships with decentralized organizations and help them understand that we are not trying to manage their environment,” she said. “We are trying to enable them to do their job but also feed that enterprise visibility so we can collectively help protect ourselves from any type of cyberincident.”
When she arrived, Thompson set out to expand the use of tools that scan for vulnerabilities. “What we have been working on is increasing our footprint to not just DIT but all the agencies that have a need for a solution similar to this and may not have anything similar in place.” The state uses an enterprise version of Tenable, a security monitoring solution for large organizations.
DIT has provisioned accounts for all the agencies and identified points of contact, such as security liaisons or application owners and system administrators. “They have access to get into the system to run reports as needed,” Thompson said. “We run a scan on a seven-day cycle.”
This helps the DIT team prioritize which things to target. The Tenable scanner is only one solution in a suite of tools that help complete this picture, she added. “We have other solutions to help monitor our external perimeter and give us a credit card score of how we look from an external perspective, and based on your score, there are points associated with the types of vulnerabilities and it helps you make a decision: If I make this change, which one will give me more bang for the buck and offer more protection? Prioritization is definitely key to this.”
