Log4Shell was a search problem before it was a patching problem

The week

On December 9, 2021, a remote code execution vulnerability in Apache Log4j 2, a widely used open-source Java logging library, became public along with proof-of-concept exploit code. Tracked as CVE-2021-44228 and soon called Log4Shell, it let an attacker run code on a vulnerable system by getting it to log a specially crafted string. That string could arrive through ordinary inputs such as web requests, form fields or user-agent headers. The flaw received the maximum severity score, and mass scanning began within hours. The Apache Software Foundation released fixed versions, and related issues led to further releases over the following weeks. On December 17, CISA issued an emergency directive to federal civilian agencies, and vendors across the industry published advisories listing affected products.

Most critical vulnerabilities affect a product: a firewall, a mail server, an operating system. An organization checks whether it runs that product and acts. Log4Shell affected a component buried inside thousands of products, many of which did not advertise that they used it. Some applications included the library directly. Others included it inside another library, inside a vendor application, running on an appliance. The fix, once found, was usually an upgrade or a configuration change. Finding it was the crisis. The search ran differently in three places, each with a different owner and a different source of truth: software an organization builds, software it buys, and software it consumes as a service.

Software an organization builds

For applications developed in-house or by contractors, the question was answerable in principle. Build tools such as Maven and Gradle record which libraries a project pulls in. In practice, many organizations had no central view across their code repositories, and some older applications had been built by teams that had since moved on. Searching repositories and build files found direct dependencies within hours. Transitive dependencies, pulled in by other libraries, took longer. Compiled applications with bundled libraries, and “fat” archives that repackage dependencies, required scanning the deployed artifacts themselves, because the source tree no longer told the full story.

Organizations with dependency scanning in their pipelines and a record of what was deployed where produced a list within a day or two. Those without relied on developers’ memories and manual searches, which took weeks and were never complete.

The search also had to be repeated. The first fixed release was found to be incomplete in certain configurations, and further issues in related code paths led to additional releases within weeks. Interim mitigations, such as a configuration flag and removing one class from the library, were recommended and then superseded. Each new release reset part of the work. Teams with automated build and deployment pipelines could rebuild and redeploy every affected application several times. Teams deploying by hand could not keep pace, and some ran partly patched estates into January.

Software an organization buys

For commercial software, the organization depended on vendors. Each vendor had to determine whether its products were affected, publish an advisory and deliver a fix. Responses varied widely in speed and clarity. Some published comprehensive product and version lists within days. Others were slow or issued statements that were vague about specific versions. A statement that a product “is being investigated” could stand for weeks.

Banks, health systems and agencies spent much of December tracking vendor advisories in spreadsheets, matching them to their own inventories and chasing vendors that had not answered. The work exposed how little most buyers knew about the components inside products they owned. Medical devices, building management systems and specialized line-of-business applications were hardest, because their vendors were smaller and their update processes slower. Scanning helped at the margins. File-system scanners could find Log4j archives on servers the organization controlled, but not inside sealed appliances.

This is where the software bill of materials moved from niche idea to procurement topic. An SBOM is a machine-readable list of the components in a piece of software. If every product had shipped with an accurate one, the question of where Log4j sat could have been answered by a search rather than a survey. The May 2021 executive order on cybersecurity had already called for SBOMs in federal procurement, and Log4Shell supplied the example that made the case to everyone else. An SBOM does not fix anything by itself. Its value depends on keeping it current and on having a system that can match a new vulnerability against all SBOMs on file.

Software an organization consumes as a service

For cloud platforms and software as a service, the provider was responsible for fixing its own systems. That was a real advantage, because customers had nothing to patch. But customers still needed assurance that providers had acted, and some had to consider whether their data had been exposed before fixes went in. Many providers published status statements within days. Few told individual customers whether exploitation had been attempted against their tenant, or what logs the customer could review to check.

The boundary of responsibility also needed care. On infrastructure services, the provider patched the platform, but Java applications the customer ran on its own virtual machines or containers were still the customer’s to find and fix. Platform services sat in between: a managed runtime might be patched by the provider while libraries packaged inside the customer’s application were not. Teams that assumed “it’s in the cloud, so the provider handles it” missed workloads that were plainly their own. A shared-responsibility matrix, kept per service rather than per provider, answers that question before an incident rather than during one.

The gap was contractual as much as technical. Most service agreements committed providers to notify customers of confirmed breaches of customer data. Few committed them to tell customers, within a set time, whether a widely exploited vulnerability had affected the service and what had been done. Buyers who added that obligation at renewal, along with rights to relevant logs, had better answers the next time. The trade-off was negotiating capital. Large providers resist bespoke terms, so the request works best when buyers in a sector ask for the same thing.

With hindsight

Log4Shell did not disappear after December 2021. The federal Cyber Safety Review Board reported in July 2022 that vulnerable versions would remain in systems for years, and exploitation continued well beyond the first wave. The policy response was substantial. OMB guidance issued in 2022 required software producers selling to agencies to attest to secure development practices, with a common attestation form in use from 2024. Congress required medical device makers to include cybersecurity information, including a software bill of materials, in premarket submissions from March 2023. The EU’s Cyber Resilience Act, in force since December 2024, set security obligations for products with digital elements sold in Europe. By this point in 2026, SBOMs are routinely requested in regulated procurement, though their quality and usefulness still vary. Inventory that reaches below the product name remains the precondition for every fix.