[This was originally published on the OSVDB blog.]
[2014-05-09 Update: We’d like to thank both McAfee and S21sec for promptly reaching out to work with us and to inform us that they are both investigating the incident, and taking steps to ensure that future access and data use complies with our license.]
Every day we get requests for an account on OSVDB, and every day we have to turn more and more people away. In many cases the intended use is clearly commercial, so we tell them they can license our data via our commercial partner Risk Based Security. While we were a fully open project for many years, the volunteer model we wanted didn’t work out. People wanted our data, but largely did not want to give their time or resources. A few years back we restricted exports and limited the API due to ongoing abuse from a variety of organizations. Our current model is designed to be free for individual, non-commercial use. Anything else requires a license and paying for the access and data usage. This is the only way we can keep the project going and continue to provide superior vulnerability intelligence.
As more and more organizations rely on automated scraping of our data in violation of our license, it has forced us to restrict some of the information we provide. As the systematic abuse rises, one of our only options is to further restrict the information while trying to find a balance of helping the end user, but crippling commercial (ab)use. We spend about half an hour a week looking at our logs to identify abusive behavior and block them from accessing the database to help curb those using our data without a proper license. In most cases we simply identify and block them, and move on. In other cases, it is a stark reminder of just how far security companies will go to to take our information. Today brought us two different cases which illustrate what we’re facing, and why their unethical actions ultimately hurt the community as we further restrict access to our information.
This is not new in the VDB world. Secunia has recently restricted almost all unauthenticated free access to their database while SecurityFocus’ BID database continues to have a fraction of the information they make available to paying customers. Quite simply, the price of aggregating and normalizing this data is high.
In the first case, we received a routine request for an account from a commercial security company, S21sec, that wanted to use our data to augment their services:
From: Marcos xxxxxx (xxxxxxx@s21sec.com)
To: moderators osvdb.org
Date: Thu, 16 May 2013 11:26:28 +0200
Subject: [OSVDB Mods] Request for account on OSVDB.orgHello,
I’m working on e-Crime and Malware Research for S21Sec (www.s21sec.com), a lead IT Security company from Spain. I would like to obtain an API key to use in research of phishing cases we need to investigate phishing and compromised sites. We want to use tools like “cms-explorer” and create our own internal tools.
Regards,
—
S21sec*Marcos xxxxxx*
/e-Crime///Tlf: +34 902 222 521
http://www.s21sec.com , blog.s21sec.com
As with most requests like this, they received a form letter reply indicating that our commercial partner would be in touch to figure out licensing:
From: Brian Martin (brian opensecurityfoundation.org)
To: Marcos xxxxxx (xxxxxxx@s21sec.com)
Cc: RBS Sales (sales riskbasedsecurity.com)
Date: Thu, 16 May 2013 15:26:04 -0500 (CDT)
Subject: Re: [OSVDB Mods] Request for account on OSVDB.orgMarcos,
The use you describe is considered commercial by the Open Security
Foundation (OSF).We have partnered with Risk Based Security (in the CC) to handle
commercial licensing. In addition to this, RBS provides a separate portal
with more robust features, including an expansive watch list capability,
as well as a considerably more powerful API and database export options.
The OSVDB API is very limited in the number of calls due to a wide variety
of abuse over the years, and also why the free exports are no longer
available. RBS also offers additional analysis of vulnerabilities
including more detailed technical notes on conditions for exploitation and
more.[..]
Thanks,
Brian Martin
OSF / OSVDB
He came back pretty quickly saying that he had no budget for this, and didn’t even wait to get a price quote or discuss options:
From: Marcos xxxxxx (xxxxxxx@s21sec.com)
Date: Mon, May 20, 2013 at 10:55 AM
Subject: Re: [OSVDB Mods] Request for account on OSVDB.org
To: Brian Martin (brian opensecurityfoundation.org)
Cc: RBS Sales (sales riskbasedsecurity.com)Thanks for the answer, but I have no budget to get the license.
We figured that was the end of it really. Instead, jump to today when we noticed someone scraping our data and trying to hide their tracks to a limited degree. Standard enumeration of our entries, but they were forging the user-agent:
88.84.65.5 – – [07/May/2014:09:37:06 -0500] “GET /show/osvdb/106231 HTTP/1.1” 200 20415 “-” “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0”
88.84.65.5 – – [07/May/2014:09:37:06 -0500] “GET /show/osvdb/106232 HTTP/1.1” 200 20489 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko”
88.84.65.5 – – [07/May/2014:09:37:07 -0500] “GET /show/osvdb/106233 HTTP/1.1” 200 20409 “-” “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)”
88.84.65.5 – – [07/May/2014:09:37:08 -0500] “GET /show/osvdb/106235 HTTP/1.1” 200 20463 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36”
Visiting that IP told us who it was:
So after requesting data, and hearing that it would require a commercial license, they figure they will just scrape the data and use it without paying. 3,600 accesses between 09:18:30 and 09:43:19.
In the second case, and substantially more offensive, is the case of security giant McAfee. They approached us last year about obtaining a commercial feed to our data that culminated in a one hour phone call with someone who ran an internal VDB there. On the call, we discussed our methodology and our data set. While we had superior numbers to any other solution, they were hung up on the fact that we weren’t fully automated. The fact that we did a lot of our process manually struck them as odd. In addition to that, we employed less people than they did to aggregate and maintain the data. McAfee couldn’t wrap their heads around this, saying there was “no way” we could maintain the data we do. We offered them a free 30 day trial to utilize our entire data set and to come back to us if they still thought it was lacking.
They didn’t even give it a try. Instead they walked away thinking our solution must be inferior. Jump to today…
161.69.163.20 – – [04/May/2014:07:22:14 -0500] “GET /90703 HTTP/1.1” 200 6042 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
161.69.163.20 – – [04/May/2014:07:22:16 -0500] “GET /90704 HTTP/1.1” 200 6040 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
161.69.163.20 – – [04/May/2014:07:22:18 -0500] “GET /90705 HTTP/1.1” 200 6039 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
161.69.163.20 – – [04/May/2014:07:22:20 -0500] “GET /90706 HTTP/1.1” 200 6052 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
They made 2,219 requests between 06:25:24 on May 4 and 21:18:26 on May 6. Excuse us, you clearly didn’t want to try our service back then. If you would like to give a shot then we kindly ask you to contact RBS so that you can do it using our API, customer portal, and/or exports as intended.
Overall, it is entirely frustrating and disappointing to see security companies who sell their services based on reputation and integrity, who claim to have ethics completely disregard them in favor of saving a buck.
