I went into a LinkedIn post expecting to have to buy a new box of red sharpies to be honest, but I am pleasantly surprised at the conclusions regarding CVE / NVD, which I think are largely accurate. As grim a picture as is painted, they are still a bit too generous. I say that as someone who reads, quite literally, every new CVE published and have for coming up on 20 years. Pretty sure no one at MITRE does that.
Originally, I was replying to a post and quickly hit the limit in replies, which is ridiculous. I was 2,648 characters over before I realized it. No wonder LinkedIn has so little meaningful exchanges. Fortunately, Mark Curphey made his original post a formal blog so I can publicly reply to it w/o issue.
What I began replying to was simple enough, then I saw the link to the polished blog which added a lot more, and immediately grabbed a red sharpie. So, replying to the pre-blog version of it:
A few fun bits…
- “not verified effectively“? No, they do not verify at all. Not a single bit, and it is actually worse than that. If you are thinking “how could it be worse?” Trust me, it can and is. I cannot betray trust, but the number of MITRE folks that have told me things that are not Steve Christey or Lyger would shock them. [Both of them you and I might expect two, both have/had pure integrity and never crossed that line.]
- “can’t possibly deal with the rate of vuln ingestion“. True! Worse? That is only with 260 CNAs, many that have never published a vuln, and researchers coming to MITRE. The CVE staff do not go out and monitor any sources, and haven’t for ages. Of course there are big glaring gaps in coverage!
- “badge of honor and fluff” Oh… my… god… this cannot be overstated enough! Since MITRE does zero verification, it rewards the system. For half a decade or more, low-end researchers (mostly) will get a CVE and immediately Tweet/FB/LI it, “I JUST GOT CVE-2014-54321!!!” and then? NEVER publish details, literally a decade later. They just want that ID, which is assigned often times w/o any knowledge of the vuln itself. I routinely try to chase down these “public IDs” and ask and either get it is under embargo or they long since forgot and lost their notes.
- “doesn’t have the right data to reproduce a vulnerability” Eh, sometimes they do, but that doesn’t matter one bit. They simply do not read the proposed CVE description; they rubber stamp and publish. They don’t want or care to verify anything. So the presence of that information doesn’t matter at all.
- “doesn’t understand clones and forks” But this is complicated. First, you are correct. Second, this drastically over-simplifies the problem because even the best out there have problems too.
- “CNA scheme has become a way for vendors to hide issues” .. this one is fun! Because they were hiding them to begin with. The CNA title gives them agency to do it more formally. Worse, MITRE has never had any intention of policing CNAs. They had a “CNA transparency report” back when I was on the board and I railed at them hard because it showed zero transparency, instead, giving some trivial publication statistics instead. I yelled at them to do better and I got threatened to be removed from the board several times for it. Even when I tried to help coordinate a multi-CNA disclosure, using the … wait for it … CNA mail list. MITRE told me that my coordination attempts led some CNAs to feel “called out“, even when the mail was factual and professional.
- “The CPE namespace does not …” <- You really could have stopped there, because there really isn’t much it can do well. At least, not while in the hands of NVD, which is a blind servant to MITRE and that whole arrangement. It just forces other vendors that are ahead of the curve to get creative, follow CPE specs as best they can, and invariably run into problems when NVD decides to assign a CPE that is arguably (or not) correct, but different than what the other party used first.
- “lot of the world doesn’t trust the US government” <- See, this is another fun one for many reasons, especially when comparing to CNVD, RNVD, etc. Our government is ‘more transparent’ in a sense, by volume. But like their governments, our actual government doesn’t play nice with disclosing vulns they think valuable.
Now, the one big nit to pick with the original post is this: “Vulnerability databases don’t work for open source code.“
If you are basing that on CVE / NVD? Sure, I get it. They fail spectacularly at OSS, but it is their model that makes it so ineffective. You even explicitly said why… “Most developers don’t care about reporting vulns“. Some developers actively push back against assignments, often because of valid reasons, and a history of CVE ID assignments that were for invalid issues. Once assigned, getting them REJECTed is not easy in most cases and leaves a “permanent mark” on their record so-to-speak.
A properly run vulnerability database (VDB) that actually caters to OSS coverage? They can do it quite well. Many F50+ pay good money for that too, but it requires paying some money. CVE is free and if companies can find a way to cut a corner, they do. Look at the number of data breaches the past X years and you see the real cost of using that database in some cases.
Bottom line; there are two things at play. First, proper vulnerability intelligence, and we can both agree CVE / NVD isn’t it. I’ll go one step farther and say anyone using it is is just negligent at this point. Their deficiencies have been pointed out in many ways for a long time, but not in a way that reaches the right audiences. Second, how you use it. From a VDB perspective, that is somewhat a downstream problem as every organization is a unique and delicate flower. They have to try to cater to the entire garden when every plant wants different amounts of sunlight and water.
But I will assure you, without the first? The second is doomed to fail no matter how good the team is.
Bonus content! Replying to the formal blog here, which basically took each bullet from above and expanded it.
- “and even an IETF RFC for responsible disclosure.” Eh, no, lost me here. For the millionth time, do not use “responsible” disclosure. That is a loaded term created by vendors to work against researchers and it has been extremely effective since. That said, are you talking RFC 9116? If so that is just to aid in the format of the disclosure. Are you talking ISO/IEC 29147:2018 perhaps, centered around the higher-level concepts of disclosure? Yeah, organizations have to pay for that. You simply cannot compare either of those to something like Bugtraq, which you directly do.
- “The Quick History of CVE and NVD” – While this is a very quick history, it also glosses over many important facts that are relevant to the original blog and subsequent discussion. Let’s just take a single example and point out that the paper linked, specifically section 2, “Roadblocks to Interoperability” describe both why the project was created and what it hasn’t done in 15 or more years.
- “Today there is a total ecosystem around CVEs” – Yay! The word “ecosystem” is precisely correct and one I have been using for a long time, for many reasons you point out and more.
- “The reality though is … that if the vulnerability data you are relying on is garbage, then it just doesn’t matter. It’s garbage in, and garbage out.” – This sentence can not be given enough praise.
- “What we found was that all too often, the NVD entry described the wrong system or library and the vulnerability as described wasn’t the actual vulnerability in the first place.” – Right, and in context, back then it wasn’t even NVD doing that analysis. They outsourced it to junior analysts at Booze Allen. Since then they have gone in-house, but it doesn’t mean they are hiring qualified vulnerability analysts either.
- “… the team at CVE who were doing a stellar job with the limited resources they had… ” – Yeah, no, let me just stop you right there. The budget with which the CVE program operates is patently absurd. I know because I did FOIA requests. Both MITRE and NVD hate when I do that, both have actively fought it, and my last FOIA against DHS for current MITRE/CVE numbers got stonewalled hardcore this time, a trick out of NVD’s playbook but worse. Anyway, the CVE project enjoys “mountains of cocaine” money to operate a poor version of a vulnerability database. Much more mature databases exist that operate at a fraction of the cost, move data much faster, with a lot more robust metadata. Ask for an org chart of the CVE team, see if they give you one. Based on one I informally put together years ago, that will be the first hint as to the biggest waste of money.
- “I believe the time period is two years, but I can’t find the rules.” – Read the section for more context here first. The problem is much worse than that because a vendor can ignore you and stonewall for longer than two years before even assigning a CVE ID if they want. If they go dark you can go to MITRE directly to publish, but they will happily cite rules of “GOTO CNA” and ignore anything after. You publish the vulnerability, mail MITRE, and say “here it is, public”? They still won’t publish in CVE sometimes. That means work for them, which is not part of how they want to operate CVE.
- “I know CVE does not describe itself as a vulnerability database…” – Yep, that’s something I often point out and ridicule. They say that trying to evade any responsibility of being one, despite literally being the definition of a VDB.
With that, I bring the blog to a close with a direct message to Mark, the author. This is a great breakdown of the CVE/NVD ecosystem and problems, primarily looking at the root cause, CVE. While I had some nitpicks, you are largely spot on with your assessment and you clearly have identified many problems that have been there for a long time. I mean this so much so, that you will notice my blog did not come with the “Vulnerability Tourist” tag which I use all-too-often. That may seem tongue-in-cheek but for those that know me? That is one hell of a compliment. Kudos.