Rants of a deranged squirrel.

VulnCon Day 2 Errata & Taking Ben Edwards to Task

jericho

[4/13/2025 Update: See very end, below last image, for an amusing update.]

Today was the second day of VulnCon 2025, a conference whose stated purpose is “to collaborate with various vulnerability management and cybersecurity professionals to develop forward leaning ideas that can be taken back to individual programs for action to benefit the vulnerability management ecosystem.” 

While the purpose is to develop “forward leaning” ideas, the reality is that sometimes conferences put forward ideas that are backward leaning at best. Conferences that have a formal Call for Papers (CFP) submission process with a review board are partially responsible in such cases. Conferences that accept talks without reviewing at least a bit of information are more culpable. Of course, the presenters are more culpable when this happens as they should be subject matter experts to a degree for their given topic.

Earlier today, two presentations had meaningful errors that I believe need to be brought to light. The first is a case where, while the presenter was wrong and arguably should have understood this, it also demonstrated an issue in a framework that has led to the same confusion in many people including some asking questions today. The second case is where someone made a statement that is demonstrably false, and worse, common sense and the most rudimentary understanding of the vulnerability disclosure ecosystem would lead you not to say.

If you are here with a bucket of popcorn, skip the first section.

Exploit Maturity & Shelby Cunningham

Predictably, there was more than one talk about CVSS given the nature of the conference. An advisory curator for the GitHub Advisory Database, Shelby Cunningham has considerable experience and a relatively unique view into the CVE ecosystem. Her team handles vulnerability advisories in a volume that few CVE Numbering Authorities (CNA) have seen. The gist of her talk was specifically around the ‘Exploit Maturity’ metric in the newly published Common Vulnerability Scoring System (CVSS) version 4 scoring framework.

Before I proceed, I want to point out that this issue has been a major point of contention by many people and a huge thorn in the side of the VulnDB team. We’ve been scoring every single vulnerability, regardless of CVE assignment, using the CVSS v2, v3, and v4 frameworks. Specifically, version four scoring has been done since August 1, 2024 meaning we have a considerable body of scores and experience with it. The following issue was immediately identified after the release of the specification and we saw it play out shortly after.

As Shelby’s talk began, I had already made a comment in the VulnCon discord pointing out that the Exploit Maturity metric had one value in particular, ‘Attacked’, that was a poor choice of terminology. Nick Leali, a CVSS SIG co-chair, was in the channel and asked me what would be a better term. I said that was not the right question to ask, rather, to ask if that metric should be abstracted. I pointed out that there are two criteria that can earn the ‘Attacked’ metric value:

These are very, very different things. The first criteria is what is more recently called KEV, or ‘Known Exploited Vulnerability’. It means that there has been relatively conclusive proof that a vulnerability has been exploited in the wild and used to compromise an organization. Therefore, KEV should be considered more serious and a higher risk. The second bullet point just means that a script or tool has been created to make exploitation easier. It does not speak to if that exploit has ever been used, let alone used to compromise a system. In the vulnerability world this is night and day.

A few minutes into Shelby’s talk was the first time I could point out to Nick the confusion the term ‘Attacked’ caused as Shelby conflated that metric value with a KEV. Not just “could be KEV”, but that time and repeatedly after it was a clear one-to-one correlation that scoring a vulnerability as ‘Attacked’ meant it was exploited in the wild. Someone from the audience asked a question where they too conflated ‘Attacked’ as KEV. In less than 30 minutes we had multiple people showing why the CVSS SIG should have either abstracted the ‘Attacked’ metric to separate the two criteria we see today, or named it differently.

This mistake has haunted my team for over a year, and it will continue to haunt us for the next decade. Why that long? Some companies are still using CVSSv2, released in April, 2005. So, this is a point of errata in Shelby’s talk, which I completely understand because of the CVSS SIG’s choice of words. That said, when an entire 30 minute talk is centered around one metric that has four values (three that have meaning), and each of them have two or three lines describing the criteria, I would hope that a lot of attention would be paid to those descriptions. More so when most of the talk centered around Exploit Maturity = ‘Attacked’, meaning understanding those two lines.

Nothing to Risk & Ben Edwards

While the last talk constitutes Errata and has a layered reasoning as to how something is misinterpreted through a framework that could have been better written, this one is much more egregious and I find personally insulting. So much so, I actually got a warning from FIRST due to “complaints” of my language in Discord during this talk. I took responsibility and acknowledged that yes, I broke the FIRST code of conduct (CoC) when I called Ben Edwards’ notion on one topic “the most ignorant thing yet“, and I also told them that I stand behind my statement. Since this is a personal blog and my opinion, we get to play by my rules where I get to point out just how much of a vulnerability tourist he is, and more. 

I first want to point out that yet again, the industry cares more about decorum than they do the truth and reality. In that email, I also told FIRST that the CoC should be amended to include a presenter knowingly lie to the community. If it isn’t a lie, then GOTO 10, ignorance. This is the second time he has made this stupid claim too, the first being on a LinkedIn post ten months ago. This is precisely the kind of absurd claim that I take personally because I know it to be demonstrably false, and that it consistently proves there are two types of “vulnerability analysts” in this story.

  1. Those that analyze vulnerabilities and/or vulnerability disclosures.
  2. Those that analyze the aggregated form of that data (e.g. CVE)

Edwards and another presenter (who I took a lot less exception with), gave a presentation titled “Nothing to Risk but Risk Itself: Expanding Vulnerability Risk with Internet-Scale Data“. The gist of the talk was that using EPSS you can better evaluate risk when evaluating vulnerabilities. While EPSS doesn’t even appear in the abstract for some reason, it was an integral part of the presentation. Through the talk, both Edwards and Vinbert used the term “CVE” interchangeably with the actual word, “vulnerability”. There are many reasons this is a fundamentally bad thing to do, and simply wrong. Not all CVEs are vulnerabilities. Not all vulnerabilities get a CVE ID. That simple; two reasons why it is wrong to do so, so stop please.

This led me to ask a question, via Discord since I am attending remotely, “How do you factor in the thousands of vulnerabilities that are publicly disclosed every year, that do not receive a CVE ID?” When a moderator asked in the room, Edwards smiled and chuckled a bit before saying he was going to have a “controversial opinion”. No sir, it is a dumb take. It’s CVE propaganda at best. It also betrays your experience in this field, as well as what you actually do with your so-called data science. I use that term because the data scientist I work with doesn’t have these absurd notions and he has seen first hand how the sausage is made. When you do, it opens your eyes to an incredible amount of bias that would otherwise seep into your work. Hey look, I used a data sciency word!

Edwards’ dumb take is, and was, and I paraphrase here, “that either there aren’t that many vulnerabilities without a CVE (see the LI post) or today that they aren’t serious ones”. He actually believes that everyone in the technical world that finds a vulnerability would go to MITRE or a CNA and request a CVE if serious. If this wasn’t so fucking naive, I would laugh. But it is too serious of a matter to let this ignorance be spread at a conference focused on vulnerabilities.

This is a factual reality, and I am going to say it again very clearly, and remind you that you are absolutely, inarguably wrong about this. Since 2004, I have worked on a vulnerability data set where we have aggregated over 112,000 vulnerabilities that do not have a CVE ID. That is a fact. Many of them are quite serious, many of them are junk vulnerabilities, just like the current CVE data set. That is a fact. The reality is, there is a very high probability that 200,000 to 500,000 vulnerabilities are out there, publicly disclosed in some fashion, that we haven’t aggregated either. That is a fact in my eyes, because I aggregated a majority of that 112,000 I mentioned and I know where to look for these.

So, let me unpack your abject stupidity from your remarks today as well as your LI post. No, I am not going to show you the first sign of respect, because you didn’t show me and my work the first sign of respect initially. Fair is fair. So I will give the ‘Bullshit Epiphany’ along with the ‘Simple Reality’:

I am going to give you some of that data you want and prove several of your bullshit epiphanies wrong, all in one shot. You sir, are going to process this one URL I give you, find all the vulnerabilities, and request CVE IDs for them. They are all from 2007 so make sure you get the correct ID, specifically a CVE-2007 prefix. I’ll even make this so much easier for you by telling you that this fix-list is published by a CNA (IBM), in software that is very much important in our world (IBM AIX), and that following CVSS specifications these are mostly CVSSv2 10.0 / CVSSv3 9.8 / CVSSv4 who cares, contains at least 37 vulnerabilities (I don’t discount that I may have missed one), and only two of those currently have a CVE ID. Further, you can’t downplay those CVSS assignments because IBM AIX has a history of CVSSv2 10 / v3 9.8 vulnerabilities, the potential is absolutely there.

So, let’s give you until the end of month to get all of those CVEs and I expect to see them open by then, fair enough? This is a single changelog, from a vendor that has an incredibly big software and hardware portfolio. If you can find 37 vulnerabilities in that single changelog, do some of your stats nerding to estimate how many more vulnerabilities are out there, publicly disclosed, without a CVE ID. In this case you have your basic math! One changelog, one vendor, 37 vulns, only 2 have a CVE ID (5.4% I think?!), and that is more than enough to work with. At the very least, it should be trivial to see how I, and a small group of people, have found 112,000 over the last 31 years. Is my math starting to sound more reasonable yet?

You are off to the races sir; “5300-07 Technology Level for AIX operating system“. That’s right, it isn’t even public any longer which reminds us, every single day we lose vulnerability intelligence due to link rot. And only through the saving grace of an angel that is the Internet Archive, do we sometimes get lucky and find a copy. [Seriously, please consider donating to them, they are a 501(c)(3) and can desperately use your help. Yes, I donate every year.]

So let’s recap and add some more, for your “they must not be serious” bullshit epiphany: 

In addition to your IBM homework and a minimum of 35 vulnerabilities there, you have one more from the Morris Worm, and you should be able to trivially track down three in WhatsApp that are named vulnerabilities that made the news as they were used by exploit-for-hire companies to target journalists and such. I wish you the best of luck getting IDs for all those, and if you can’t, I honestly understand. The only reason I think you might get them is so MITRE can try to spite me and say I am wrong in some fashion. Oh, I almost forgot to mention… that bulleted list above, those all speak to known exploited vulnerabilities. That serious enough for you?

With that, I stand behind my code-of-conduct violating remark that your notion on this topic is ignorant. Since it is my blog and my opinion, I will double down by invoking a beloved character to all of us, someone that called B.S. when he saw it. So, here’s to you Dr. Edwards! 🤪

4/13/2025 Update: Two days after Ben Edwards’ presentation, he was on a panel about CISA’s North Star Vision for the CVE program. During this, at one point he said “the lack of CVE for a vulnerability … makes it a 0-day”. This explicitly contradicts his prior answer to the question I asked as well as his LinkedIn post. Further, that statement is absurd and simply wrong. A 0-day vulnerability, regardless of your definition of the term. The unequivocal truth is a vulnerability that is exploited before the public or vendor has knowledge of it, at that point in time, has nothing to do with the CVE ecosystem. It is part of the broader vulnerability ecosystem. Once that vulnerability gets a CVE ID assigned, only then does it become part of the more narrow CVE ecosystem. As of this update, there are at least 663 vulnerabilities that have been exploited in the wild and do not have a CVE ID assigned. Of those, 418 were discovered in the wild (0-day). So again, Ben, please educate yourself more on this topic before spreading falsehoods.

4/20/2025 Update: Still no CVE IDs assigned for those issues 12 days later, by Ben or anyone else. With the recent mistrust in CVE after the defunding mess, there have been some amusing memes. Ironically, they are also a sharp jab at Ben’s backwards view of the CVE ecosystem. Here’s a couple to brighten his day!

Exit mobile version