[Note that this was half-written on 2020/11/13 but never finished and published. Going back through old blog drafts, I am opting to post this as-is, and back date it to when it was started. Toward the end it is not clear from notes if I am quoting the Tweet or making a note on how to reply to it in the blog.]
———–
This blog started out as a much longer ‘FIN’ notice for the OSVDB project. Announcing the end of a 15-year old project is rough, especially when a handful of people spent an incredible effort keeping it limping along. After the announcement was made, we were shocked at how much positive feedback we received. The number of “thanks” and “RIP” messages was truly surprising, as most projects shutting down and not making high-quality free data available are met with scorn. We of course had some that were idiots or trolls, and there were some that came across as not understanding the topic, but certainly not mean-spirited. Others were curious as to what the project entailed. That prompted me to take the original blog, enhance it further, and add more to address some points that were brought up. First, a bit of backstory to make some facts crystal clear, as better explain some missing gaps in people’s understanding of the project.
Background: The Short Story
The Open Source Vulnerability Database was announced on August 1, 2002 at DEF CON 10 by H.D. Moore. About a year later, Jake Kouns and Chris Sullo took over the daily management of the project as the project was idle, and recruited me in December. On March 31, 2004, the project was officially announced as ‘live’ and ready for more volunteers and public contribution. Days later, the site was SlashDotted over the news and 18 minutes later, the server went down. That April, the soon-to-be 501(c)(3) Open Security Foundation (OSF) was incorporated to help ensure the long-term success of the project. In August, the OSVDB Vendor Dictionary was opened, a full ten years before the HackerOne dictionary that served the same purpose, albeit a bit more limited. April of 2005 saw this blog created, to give additional commentary to the vulnerability world, especially focusing on the long-debated ideas of vulnerability disclosure. At CanSecWest ’05, Jake and myself gave a presentation on OSVDB in which they challenged the industry to do better in regards to vulnerability databases, that they must evolve. Over time, OSVDB led the way in new features, new metadata to track, and new ideas that would be implemented down the road. In December of 2007, version 2.0 of the database went live and shortly after, was re-branded as the Open Sourced Vulnerability Database, to better clarify the intent (information is sourced openly, and that it was no longer following a popular open source license). This slight rename was after three years of getting virtually no real community support, no help from companies using our data, and no indication that the community would step up to help.
For those five years, over 95% of the vulnerability import and mangling was performed by Lyger and myself. Even with two dedicated volunteers, it was trivial to keep up with CVE output and continue to add additional vulnerabilities not found there, but impossible to keep up with maintaining a VDB with complete entries up to our own standards. The change in branding also introduced a simple license; the data was still entirely free to use for personal use, home networks, etc. Companies that wished to integrate the data into their products or services would have to license the data. This was the only way we saw to get a shred of funding to help keep the project alive. We had no interest in limiting individual use, even in a professional capacity, meaning a pen-tester could continue to use the data even. For the next four years, we limped along with no community support, and no companies looking to license the data.
In February of 2011, OSF partnered with Risk Based Security (RBS) to offer increased accessibility to data and full commercial support for data entry. This meant that RBS began to staff up and fund an entire team to manage data import, as the extremely limited volunteer effort hadn’t been cutting it for some time. By November of 2013, OSVDB had reached over 100,000 vulnerabilities; over 20,000 more than CVE/NVD. For more than four years, RBS continued to push the data back to OSVDB and make a significant amount of it free for OSVDB users, under their license. While the entries didn’t show all of the fields as before, that did not deter companies from scraping the data to resell as their own. Late in May of 2015, RBS stood up an entirely new data entry system that resolved long-time outstanding bugs, and added new features. By this point, they had invested considerable Developer resources to the upkeep of OSVDB and implementation of new features. With the cutover to the new data entry system, updates to the public OSVDB site were put on hold. Months later, we had not seen a single complaint, or been asked why the database wasn’t being updated. If the system was down, people noticed. But no new data? No problem it seemed.
OSVDB was built on the idea of crowdsourcing, before the term was coined, where interested InfoSec professionals would volunteer 15 minutes of time a week, to help create the ultimate vulnerability database. If a thousand or more professionals did this, keeping up with vulnerability disclosures was guaranteed. Even better, with that much expertise, individual entries could be enhanced with personal experience and additional details not seen anywhere else. This truly would create the more comprehensive database, while remaining completely open and free to use by all. Many years later, the reality was that a small handful of volunteers struggled to keep it running by dedicating incredible amounts of their time. Sponsorship was a difficult sell in the sort-of Catch-22 scenario where we needed more support for a better database, and companies didn’t want to sponsor a database that needed improvements. Even trying to fund via minimal ads, InfoSec types are simply not known for clicking those ads to read about companies they were already very familiar with, so we lost more than one sponsor due to “lack of click-through”. The rare times a company was interested in licensing the data, instead of scraping it, there were huge concerns over data standards and lack of any support. Without consistent funding, a lot of the hosting and developer resources came out of the personal pockets of OSF officers if the minimal coffers didn’t cover it.
By this point, commercial companies had long been scraping the site to use the data in their own products and offerings, without contributing back to the project. As we learned of such incidents where a company violated the disclaimer saying to contact us for commercial use, we contacted them asking them to contribute, license the data, or cease using the material. In one case, a company clearly making considerable money on the back of our work offered us a whole $20.00 when we asked. For years, the scraping problem was a common theme, while the rare volunteer would sign up to help only to vanish a week later. During this period, and the years after, OSVDB was the foundation of, or contributed heavily to dozens of security companies. Some of them went on to make seven figures, while we were offered nothing.
When we finally opted to restrict some of the data that was being made public, the cries of “what about all of the data from volunteers?” began. The notion of a crowdsourced project, is very misguided in many minds it seems. In a ten-year period, volunteers outside the few core people contributed a laughable amount of data. Their absurd idea that 0.001% of the generated data should somehow obligate the rest of the core member’s contributions was misplaced. Every volunteer, regardless of name or time spent working on the database, agreed that the contributions would become that of the OSF when they signed up. By this point, it was clear that the open source model worked for some projects (e.g. NMAP, Metasploit), while failing tens of thousands of others, including OSVDB. In the months leading up to the FIN announcement, we ran into software problems that OSVDB had no developer resources to fix. Then a round of hardware problems hit, making us realize that the hosting agreement was ridiculously overpriced compared to competition. This was the last two months in which we had extensive discussions about trying to keep it afloat longer, or finally kill the project.
Our decision to close this project should not come as a shock to anyone, at all, if they remotely followed our story all these years. We demanded better from public vulnerability databases, and we worked extensively to try to make that happen while keeping it as open as we could. Over time, as the industry failed to support us, we were in turn forced to find an alternative model. The first two years of using commercial support to maintain and build the database got it to where it was a viable option for organizations needing vulnerability intelligence. The next two years saw much needed developer attention, adding features that were wanted a decade prior, and writing extensive documentation on both the data standards as well as the technical mechanisms to access it. Now, it becomes obvious why that level of manpower and support was needed, as the data trivially shows that other efforts to aggregate vulnerabilities are not cutting it. For four and a half of those years, vulnerability data was still given back to OSVDB for the community to use, even to the point of the public OSVDB being considered competition during licensing discussions. Companies would actually ask bluntly, “why pay for a vulnerability feed when we can scrape it from OSVDB?” When confronted with the reality that the OSVDB data still required licensing, and that doing so was unethical, it didn’t change their mind too often.
Managing a vulnerability database is not a trivial task. To do it even half-assed requires a lot of time. As vulnerability disclosures steadily increased the last ten years, we’ve seen the toll it takes on those who do the work. As organizations devote more resources to it, they realize it becomes increasingly difficult to continue giving the information away. Secunia moved a bulk of their information behind a login and reduced what would be made available a couple years ago, while also scaling back their coverage of sources this year. SecurityFocus’ BID database has offered the most minimal of public information for ages, with several month+ long outages where public updates weren’t available; remember, the bulk of their data is only available to Symantec DeepSight customers. The last two weeks have seen increasing criticism of CVE, who enjoys over over 1.5 million dollars a year to run CVE and OVAL. Even with that absurd budget, over three times more than RBS utilizes, they struggle to aggregate half of the vulnerabilities RBS does. The time and costs associated with running these vulnerability databases explain why there is an increasing shortage of public vulnerability databases, as HD Moore observes.
In summary, we deeply regret not being able to provide OSVDB for the community. It was our goal for over a decade to keep that information free where appropriate, and only charge those who sought to profit off our hard work. We needed community support to do so, and it never manifested. In changing the model, we found a sustainable way to ensure the database is not only maintained, but can finally enjoy the evolution we loudly screamed for, and desperately wanted.
Post-announcement Thoughts
“Let someone take over!” Was equally predictable as it was annoying.
very sad and disappointed. cvedetails is now the best source of public vuln info? too bad you won’t let someone else take it over.
https://twitter.com/CyberStartsHere/status/717780301208883201
no! Are they looking for another charity to take it over? Such a great resource to let go in the community. #saveosvdb
sad to see @OSVDB go. Does anybody have a “good backup”? Would love to make that live again at http://isc.sans.edu .
Shame about @OSVDB. I got annoyed with how they did things sometimes, but was still a great resource. DB going to be made available?
Someone could share one of their last dump that was available at “http://osvdb.org/file/dumps ” ?
Probably not, @OSVDB loves to hoard their database and charge you for API access. It’s not open.
Even CVE is more open and have archivable database dumps.
[idiot of the year]
strawman argument
it’s weird in open source. some people don’t want to devote time when someone else will sell their output.
that is the go-to strawman argument in this case. simply isn’t valid re: OSVDB
https://twitter.com/n0x00/status/717744322725224449
Hahahahhahaha “OPEN SOURCE Vulnerability Database…” No that’s cute
Don’t get me wrong, I have nothing against @OSVDB, but it wasn’t “open source”. What happened today is just another company closing.
With regard to @OSVDB: it wasn’t really open source, it was a business.
https://twitter.com/n0x00/status/717611850276405248
be nice if there was an actual Open Source project, it’s shame all that community contributions hasn’t earned us a db dump
https://twitter.com/n0x00/status/717743526742794241
if we get the database we can look at the numbers of people committing time to submit to them…It’s where I sent my vulns
emailing a vuln isn’t contributing to maintaining the VDB. big disconnect there.
https://twitter.com/n0x00/status/717745340670164993
so if no one emailed in what exactly would need administrating ?
no offense meant, but you really don’t understand how a VDB works
https://twitter.com/n3tjunki3/status/717815390177533952
no offence meant but could u summarise how u understand a vdb works, the administration & work beyond email submissions
Even CVE is more open and have archivable database dumps.
CVE is open source, OSVDB (“Open Sourced” Vulnerability Database) is not. CVE provides full DB dumps, OSVDB has API with 10 query/day limit.
https://twitter.com/averagesecguy/status/717719423021400064
It wasn’t “open”, you had to pay to access the data.
PSA: *many* open infosec efforts battle poor contribution quality & unsustainably-low contributor/consumer ratio. Great fight, @OSVDB 🙁
also how long will the @OSVDB website stay up? Will we get a database dump?
Despite being sad about @OSVDB, I’m glad they announced that it is shutting down rather than pretend like they will bring it back up longer.
[we weren’t pretending. it was a tough call to shut down instead of bring it back up. we were considering both]
https://twitter.com/averagesecguy/status/717730356946927616
For those mourning @OSVDB, how were you using it? Were you including any of the data in your pentest reports? What were you paying for it?
https://twitter.com/tmanning/status/717726384311775233
Quite a loss. We integrated OSVDB IDs into the BreakingPoint products. Maybe we should have paid or something?
Did they ask for help/let people know they were being overwhelmed?
ouch, this hurts, with MITRE reducing the scope OSVDB going out looks like options to track vuln are thinning out.
several offers to figure out how to throw money at the project, now that it is gone
Unfortunately, this is the type of support we
needed over the past 10+ years and never manifested. As is, a third party
(Risk Based Security) has been providing the information to the project
for around four years, but we still face extensive license violations from
companies who in turn use it for their own products and services. In the
current InfoSec environment, that is something that is prohibitive to such
support, and still fosters no volunteer help.
will we get a DB dump? All of the user submitted advisory information came from the public domain, but now it’s gone.
slight flaw to the argument. “came from the public domain”. it’s still public elsewhere.
: I’d be interested is resurrecting the project for what it offers to the
: community, understanding that community contributions are difficult
: (i.e., impossible) to rely on and maintain. I’ve run some community
: projects in the past. It’s never been about glory or fame, but rather
: giving back to the community that’s served all of us.
That is what we tried, and the ‘community contributions’ do not even hit
0.01% of the database. Yet, we now have people demanding a full dump of
the entire database that they didn’t contribute to, or sent in a few vulns
to (while posting them elsewhere publicly, that we would have aggregated
anyway).