Log4Shell is yet another example of why we simply don’t get security right, and it strongly suggests there is little hope for change. There are plenty of blogs and articles that do a great analysis of the vulnerability from the exploitation and impact angle of this vulnerability. There are a lot fewer that examine why the bug exists in the first place, and why it took so long to notice. I’ll briefly touch on that and then look at log4shell from a bigger picture in the context of vendor responses and how they show all the chinks in the armor, enough to demonstrate that we don’t really have a set of armor at all.
If you are familiar with Log4Shell, you may want to skip the next two paragraphs, but part of the abridged summary spells out some CVE assignment confusion and explains how this blog will refer to the various issues.
What is Log4Shell? (Abridged)
On November 30, 2021, a vulnerability was made public in the Apache Log4j library that affected version 2.x but not 1.x. This is largely because, despite the same name, Log4j 2.x represents a complete rewrite of the Log4j 1.x codebase. For ease, I will refer to the library as Log4j in this blog but specifically mean the 2.x version.
The gist of the vulnerability is that when the Log4j library receives user-supplied input, it not only logs the information but has the ability to perform lookup actions. This can result in the execution of arbitrary code. Since logging is frequently implemented in a way to receive user-supplied input, this can represent a significant issue and allow for a trivial compromise of the system. In large enterprises, these logs can be automatically forwarded and consolidated on logging hosts that make administrative work easier. So that simple remote code execution might impact a different system than the one you intended.
The vulnerability was originally assigned CVE-2021-44228 but the initial fix was found to be incomplete. This led to a second CVE assignment, CVE-2021-45046, that immediately started being confused with a very separate issue affecting Log4j 1.x only, and only if the attacker had privileged access to begin with (CVE-2021-4104). Red Hat assigned CVE-2021-4125 to represent the original vulnerability but as applied to their specific products, which is in violation of CVE Numbering Authority (CNA) rules. To add more confusion, the second attempt to fix the original vulnerability resulted in a distinct vulnerability being discovered that resulted in a denial of service and assigned CVE-2021-45105. Finally, yet another flaw was discovered but it too required privileged local access, despite being called “remote” by many, and it was assigned CVE-2021-44832.
This blog will generally talk about the original vulnerability (2021-44228) unless stated otherwise and will be referred to by its nickname “Log4Shell”. Note that while the vulnerability was technically made public on November 30 and the vendor patched it on December 6, it was not widely covered until December 10th.
Log4j History (Abridged)
First, let’s look at a slightly different history of high-risk log4j vulnerabilities, including this one. This will become important in subsequent sections. Note that the severity scores represented are CVSSv2.
There are three more Log4j vulnerabilities, all without a CVE ID, but scored below 6.9 CVSSv2. On top of CVSS scores, it is also important to note that the current Log4j vulnerability (2021-44228) has been scored CVSSv2 10.0 / CVSSv3 10.0 by NVD. This, along with many other scores, is not correct as they fall into many traps that all stem from not following the CVSS specifications. As VulnDB notes, based on RBS Chief Research Officer Carsten Eiram’s analysis:
The CVSS score for this entry has been updated to reflect adherence to the CVSS scoring guidelines. On the surface this appears to be a CVSSv2 10.0 / CVSSv3 10.0 issue, as reflected in NVD and briefly by RBS. In some scenarios this is an accurate score, in others they are not. However, scoring must be done based on Log4j and the context it runs in, not speculation on possible use-case scenarios. The base score reflects that, and CVSS environmental scores allow an organization to re-cast the score based on their specific environment. So if e.g. a web server is passing user-controlled content to the logs which are then forwarded to a logging host, that represents a scope change and would get scored MS:C. Since Log4j is a library, it would have to be configured / implemented in such a way as to allow user-controlled strings which warrants Access Complexity Medium (AC:M) for CVSSv2, leading to a 9.3 score. In this case, AC:M is used for just that and does not represent user interaction required, which is often what AC:M designates. As always for libraries, they can only be scored for that *one* product, the library itself. All the products bunding the library will be affected in a myriad of ways that result in different scores, and organizations will have to tweak those scores depending on their individual configurations. A re-based score may be derived for different business units or applications even, depending on the organization’s security workflow.
Carsten Eiram, Chief Research Officer, Risk Based Security
This same scoring issue can be seen with NVD and CVE-2019-17571 (v2 7.5 / v3 9.8) as well as NVD’s score for CVE-2017-5645 (v2 7.5 / v3 9.8). The discrepancy between the v2 and v3 scores immediately stand out since their v2 score assumes the library runs with lower privileges (C:P/I:P/A:P) while their v3 score reflects that it may run with higher privileges (C:C/I:C/A:C). NVD’s scoring of Log4j issues giving it a 10 / 10 score does not follow specifications and the v3 score is scored (scope changed) with specific scenarios in mind that are not default.
Regardless of these scores, 2021-44228, 2019-17571, and 2017-5645 should all be treated equally as serious until an organization determines how it impacts them. In reality? That isn’t the case and it doesn’t make sense, other than the fact that news, researchers, and vendors all cried out that this one is “critical” while the previous vulnerabilities were largely ignored. Very similar to the Struts-Shock vulnerability which made the news primarily because it was used to breach Equifax, while other Apache Struts vulnerabilities didn’t enjoy any attention. Perhaps if the other 21 remote code execution vulnerabilities in Struts had names they too would be better known?
Vendor Disclosure Disappointment
This Log4j vulnerability stands apart from prior issues in the library, even ones with the same severity. We see vendor’s publish advisories calling it a “global event” and that it is “industry-wide“, using language that seems to convey that being vulnerable “couldn’t be helped, it affects everyone!” Sure? While that may be true it begs the question why we see them publish an advisory for Log4Shell but not prior vulnerabilities. In other cases, a vendor may definitively (and ironically) say that it “affects all service providers using the Log4J Java library” which is also untrue while also saying it “affects certain applications and services” of theirs. This is in addition to a staggering number of vendors publishing advisories saying they are not impacted by the vulnerability, something we have never seen at this volume. Of course, I have to mention the vendors that are “quickly working to assess any potential impact to our clients and the company” … three weeks later.
[1/8/2022 Next two paragraphs updated, thanks to L.P. for the pointer on origin of “LogJam” confusion.]
Despite not being the researchers to discover this vulnerability, LunaSec was one of the first to write an easy-to-digest blog explaining the issue. This is valuable to many administrators that need to quickly understand the risk. However, LunaSec decided to name the vulnerability despite not finding it. They chose “logjam” for the name, oblivious to the fact that Logjam was the name given to the “Diffie–Hellman Key Exchange DHE_EXPORT Cipher Handling Downgrade MitM Weakness” vulnerability back in 2015. That vulnerability didn’t fly under the radar by any means; consider that VulnDB has over 1,000 references for it, meaning many hundreds of vendors wrote advisories and patched it in their software. LunaSec apparently had no knowledge of this vulnerability and didn’t do a simple search for the name before using it. This is amateur hour, extremely counter-productive, and introduces serious confusion around the vulnerability.
Vendors that don’t publish advisories frequently are also more prone to make simple mistakes that introduce confusion. Take Pandora FMS or Kiteworks for example, who conflated Log4Shell with Logjam, no doubt because they saw LunaSec call it that. The two issues are extremely different. One is in Apache’s Log4J library and the other in the Diffie–Hellman Key Exchange that requires a privileged network position to intercept traffic. But this mistake isn’t just the domain of those new to security advisories as Micro Focus has decades of sharing vulnerability information, typically in changelogs instead of formal advisories, made the same mistake. You might chalk this up to a simple error, but remember that Micro Focus wrote at least five advisories referencing LogJam. [1] [2] [3] [4] [5]
For those using CVE IDs to track this vulnerability it can get confusing. Log4Shell has three associated CVE IDs: the original (2021-44228), the incomplete fix (2021-45046), and a duplicate assignment from a CNA (2021-4125). Then we have waves of incorrect CVE IDs being used, typically transposed number typos. This may cause someone to think there is yet another Log4J vulnerability though, or cause them to miss an advisory as they are searching for the correct CVEs. We can see some examples from researchers (2021-4438), bloggers and universities (2021-44248), companies (2021-44248), security news (2021-44882), vendors (2021-45056), security companies (2021-44226) and likely more variations. There are more incorrect CVE IDs attributed to this vulnerability than correct ones.
In many cases, it is the first time we see an advisory from hundreds of companies, despite them “caring deeply” and “taking your security seriously“. Yes, they responded quickly to this vulnerability, but why not the rest? Why aren’t there advisories on any of the vulnerabilities specific to your code? Why no advisories about other third-party library vulnerabilities? It only proves they didn’t care about all the prior vulnerabilities because there simply wasn’t the same level of public attention. They don’t care enough to let you know of the risk of dozens or hundreds of other vulnerabilities, but somehow they “deeply care“.
Dozens of vendors are publishing advisories that are only available to paying customers. When a vendor publishes behind a paywall it makes it difficult for many teams to access that information. While Alice and Bob from the security team may have access, Christine, Dave, Eve, and Frank may not. In other cases the IT/networking team may have access to the account in order to download new firmware, but the security team does not. Those restricted advisories also stay out of reach of third parties that aggregate that information, many of which are retained to provide vulnerability intelligence to the mutual customer. These companies don’t want to check hundreds of vendors for security updates every day; they want someone else to.
This also applies to vendors that say they will contact customers directly who require updates or publish an anemic advisory saying you need to contact customer service or “contact our support department for product-specific questions“. In many cases those notifications are not going to the appropriate people. Some accounts are set up by different divisions or managers that handle vendor accounts. In other cases there is employee turnover so that vendors are contacting people who left the company.
There are vendors that may understand the issue, may respond accordingly, but turn around and do not communicate that well to customers. For example, L-Soft’s advisory says that one version of their software uses Log4j 2.15.0 “and is therefore not vulnerable to CVE-2021-44228 (Log4Shell) but may still be vulnerable to CVE-2021-45046 and CVE-2021-45105“. This is incorrect, as 2.15.0 is vulnerable to Log4Shell in many instances. If it doesn’t affect them due to configuration then the advisory should specifically say that.
How and where vendors choose to disclose can also prove to be problematic. Modern large organizations may literally use products and services from thousands of vendors. That is why they don’t want to try to monitor them all for security updates, and it is easy to understand why. In addition to monitoring vendor security advisories, knowledge bases, wikis, support forums, and email distributions, some vendors opt to disclose elsewhere. In the case of Log4Shell, we’re seeing that happen on Twitter and Reddit.
In other cases we are seeing “advisories” being shared on GitHub in an image (PNG), which doesn’t let you cut & paste the text. When the article isn’t in your native language, it’s basically worthless since you can’t trivially translate it. It reminds me of vendors that publish changelogs or advisories in PDF format, but have disabled the copy & paste functionality in it.
The researcher SwitHack has done an incredible job collecting vendor advisories related to Log4Shell, but in doing so, has created a similar situation where disclosures happen in comments to their Gist. This collection of references also highlights the problem of “linkrot”; when a link goes 404 and the content is moved or removed completely. It’s been less than one month and half a dozen references are already 404. Some vendors publish their advisory in a place that almost guarantees it will move or vanish in time.
For those who rely on the version of software to track vulnerabilities, this is typically the best approach. In some cases, vendors work against your efforts when they say “The following product IS affected: AppDev Pack versions 1.0.5 – 1.0.11” and then “As of this writing the latest release of AppDev Pack, version 1.0.11, updates the referenced version of log4j2 to 2.15.0.” In this case, not only do you have to download the same version of the software as of a given date, you are still not protected weeks later because it only updates Log4J to 2.15.0 instead of 2.17.0 which resolves all of the known issues.
There is another type of vendor that likes conveying information in tables. For some data that is great, but for vulnerability information it can often be more difficult to process. Solace decided to share their matrices in image format that are basically unreadable as is. You have to copy the image URL and load it in a new tab to read it, since they aren’t even clickable links. HCL Technologies gives us one table to decipher, a word I use with reason. This handy table will let you figure out if you are impacted:
When it comes to clarity in vulnerability disclosure it can be tricky in a case like Log4Shell, given it has three CVE IDs and additional vulnerabilities were discovered within days of the original disclosure. However, when a vendor references three CVE IDs for Log4J vulnerabilities and then decides to also include a 2018 vulnerability in Log4Net, which is not affected by the Log4Shell issue, it can introduce confusion. Specific to Log4J, some of the initial confusion came in the form of version 1.x versus 2.x. While they share the same name, Log4J, version 2.x was a complete rewrite of 1.x and doesn’t share much beyond the name. Given that several vulnerabilities do not affect 1.x while one vulnerability affects 1.x and not 2.x, it’s messy. So a vendor responding to the Log4Shell vulnerability (in 2.x) proudly saying something like “We’re not vulnerable, we use 1.x!” and then not mentioning if they are impacted by CVE-2021-4104 (in 1.x) is frustrating.
If all of this seems discouraging and you better appreciate some frustrations around vulnerability intelligence, consider this; not even the vendor, Apache, gets it right. Three weeks after Log4Shell was disclosed, the Apache blog aggregating information about their own products is missing information (Daffodil, HBase, ManifoldCF, etc.) and incorrect about some (Geode, Jena, etc.).If the vendor can’t aggregate accurate and timely information on their own product portfolio, what hope do the rest of us have?
Keeping Up
As you can see from the ‘Disclosure Disappointments’ above, trying to stay current with the Log4Shell vulnerability is a nightmare. VulnDB is currently cataloging over 1,100 vendor advisories in addition to vendor specific solution references, exploits, and a wide variety of additional references. In less than 30 days, VulnDB 275958 (Log4Shell) has more references than Spectre, Meltdown, SWEET32, SLOTH, FREAK, Shellshock, Heartbleed, and Logjam. The only vulnerability with more references right now is POODLE, but that is likely to change in the coming weeks as more vendors finally publish their advisories.
Every day the VulnDB team continues to scour vendor web sites, Twitter, GitHub, and other sources looking for more information. When a new advisory is found we have to determine if any information needs to be cataloged and update the entry accordingly. Consider adding one reference or one vendor/product/version combination or updating one field as a single data point, and imagine what that looks like. As of this blog, that is 9,749 points of data which is staggering for one vulnerability, especially for one published less than a month ago. Here’s what that looks like:
You can see when the news sank in and vendors really started responding to it. For the next half year or more, that entry will keep getting updated. For those who think that managing a vulnerability database is “just collecting public information“, you are right. But summiting Mount Everest is also just “walking up that hill“.
What can you do? The customers have the power here. If your vendor releases a poorly worded advisory, or this is their very first advisory, take them to task. You must hold them accountable and be ready to switch vendors, which is painful I know, so that your money can teach them a lesson. Demand not only better security practices from your vendors, but better communication about vulnerabilities that impact you. Unfortunately, that threat of switching vendors is typically an empty one, and vendors know it.
Based on everything above, enumerating many of the failures around disclosing a single vulnerability, it’s hard to imagine our industry has much hope. How can organizations really expect to maintain security in the face of vendors that say they care, but act otherwise? Vendors and the information security sector are expected to provide a level of defense that will repel attackers. Unfortunately, we’re our own worst enemy and the staggering number of data breaches are evidence of that. But hey, at least we put on a good show.
