The discovery of easily exploitable weaknesses in Log4j, an open source piece of software used worldwide, has refocused attention on the risks inherent in today’s software environments. It has also spotlighted the challenges of relying on unpaid, all-volunteer forces to keep open source software updated and on trusting organizations to realize they need to adopt those patches.
Both proprietary and open source software frequently use open source components.
Senators sought to better understand the nature of the open source software space and assess the federal government’s role in fostering a more cyber-secure future. Sen. Rob Portman, R-Ohio, noted that this isn’t the only significant open source emergency — with Heartbleed compromised in 2014 and an unpatched Apache Struts deployment leading to the 2017 Equifax breach.
Sen. Gary Peters, D-Mich., who convened the hearing, also said he is concerned that Russia-based hackers may take advantage of Log4j to attack the U.S. in retaliation for supporting Ukraine.
Members of The Apache Software Foundation — the nonprofit behind the Log4j software — as well as technology firm Cisco Systems, cybersecurity company Palo Alto Networks and think tank The Atlantic Council testified during the hearing. They largely called for federal resources to support the open source community’s work, praised the collaboration enabled by CISA’s Joint Cyber Defense Collaborative (JCDC) and urged initiatives like software bills of material (SBOMs).
INSIDE LOG4J AND OPEN SOURCE SOFTWARE
Volunteers create and maintain open source software, which is then free for anyone to use as they wish. Most open source offerings solve common problems, thus giving developers ready-made “building blocks” they can use in their work and sparing them from having to each re-solve the same issues, said Apache Software Foundation President David Nalley.
“Most businesses, individuals, nonprofits and government agencies depend upon open source. It's an indispensable part of America's digital infrastructure,” Nalley said.
This model accelerates the pace of technological innovation, he said, and its usefulness has seen codes embedded in software worldwide. Log4j is used in everything from storage management software to Minecraft.
But the model also imposes certain cybersecurity challenges.
FINDING THE WEAKNESSES
For one, organizations like Apache do not require users to provide contact information, so they cannot directly reach out about warnings and patches.
Instead, Apache uses approaches like updating the National Vulnerability Database and creating patches that are machine-readable, to make it easy to automatically apply them, Nalley said.
Not everyone appears to even have learned of the vulnerabilities, however. As of mid-January, roughly 30 percent of downloads from Maven Central — a major repository of Java open source components — were for exploitable versions of Log4j, Nalley said.
“That’s roughly 10,000 downloads per hour … of the vulnerable version,” Nalley said.
Another challenge: Log4j has been incorporated deep into many tools that organizations may not realize use it. Even organizations capable of quickly applying patches may be unaware of the need to do so, unless they have detailed inventories of their software and its components, said Brad Arkin, Cisco Systems senior vice president and chief security and trust officer.
Sen. Maggie Hassan, D-N.H., noted that small developers are also unlikely to regularly check for updates to software components they’d used in their programs, so may miss making security fixes.
Such issues could cause Log4j vulnerabilities to linger for years on some systems. Hackers are eagerly looking to seize upon such oversights and have deployed botnets to automatically conduct mass Internet scans for any instances of the vulnerable code, said Jen Miller-Osborn, deputy director of threat intelligence, Unit 42 at Palo Alto Networks.
“The fact that it's been adopted by botnets as well serves to highlight that this vulnerability is never going to die. It is going to be scanned for years on the Internet,” Miller-Osborn said.
Adopting SBOMs can help clear up what software elements are used where, Arkin said, and Trey Herr — director of Cyber Statecraft Initiative for The Atlantic Council — said all federal contractors should have to provide SBOMs.
The JCDC can also help guide organizations that struggle to determine which of a seeming “infinite” number of threats or bugs to prioritize and provide details to better inform responses, Arkin said.
PROVIDING CONTEXT
Apache reviews all proposals to modify its codes. The compromised Log4j feature did not raise suspicions — it was submitted by an experienced software developer well-known to Apache and was vetted by a core member of the project management committee, Nalley said.
But developers creating open source code do not know all the different ways that users will employ it, and without such context, they are less able to design for maximum security, Nalley said.
“Some of these systems that we're talking about making use of in this particular feature were actually developed in the '90s, which was a very different place in the cybersecurity landscape. And so, I do think that there were some unintended consequences,” Nalley said. “It comes down to complex interoperation of multiple systems, because this requires three different systems to be in place to achieve this vulnerability.”
Valuable insight could be gained if more of the people who use open source software also participate in the communities creating it, he said.
FUNDING THE WORK
Open source groups rely on volunteered time and funding to enable their work.
Some, but not all, private firms that use the free offerings in their proprietary software contribute back, and Nalley said these companies likely recognize that this serves their interests, because firms’ products are only secure if the elements comprising them are, too.
But this is really a spot for the federal government — and its global allies who also use open source software — to step in to ensure volunteers have the resources they need, Herr said.
“The key for this body, and a watchword for policy efforts, then, is to improve the security of open source. It's to fund the mundane, provide the resources where industry might not or [where] public attention fades, to drive structural improvements in the security of open source software,” Herr said.
Efforts to learn from Log4j continue, and the newly formed Cyber Safety Review Board will investigate the incident before releasing a report this summer about takeaways for improving the nation’s cyber posture.
