While reading “This Is How They Tell Me The World Ends“, early in the book I ran across a single line that made me double-take. I took a note to revisit it after a complete read since it was so early in the book. For those familiar with my blogs, I tend to write about vulnerability statistics and this one fits the bill. This blog is a bit different in that a single line provoked it, but re-reading the section for clarity still takes me down other rabbit holes. Ultimately, this is a good example of how one sentence can have a lot of interpretations depending on how you read it, what statistics you use, and the deeper context that the sentence is embedded in.
Below are some additional lines that offer the full context of the line in question:
The first shift in the wind was Bill Gates’s memo. In 2002, after a series of escalating attacks on Microsoft’s software and customers, Gates declared that security would become Microsoft’s top priority. (P35)
On January 15, 2002, just as iDefense was getting going, Gates fired off the cybersecurity equivalent of the “shot heard round the world.” From that point on, Gates said, security would be the company’s “highest priority”. (P37)
What the security community wrote off as a stunt became an economic force. Microsoft froze new products and dredged up existing ones, ripping its software apart and training nearly ten thousand developers to build it back up again with security principles at the core. For the first time, procedures were put in place to embrace the hacking community. Microsoft set up a customer service line for hackers, tracked each caller and even logged their psychological quirks, noting which hackers needed to be handled with kid gloves, which had rock-star status, and which were just trolls. It instituted a regular system for rolling out software patches, releasing them on the second Tuesday of every month – “Patch Tuesday” – and offered customers free security tools.
And while plenty of zero-day bugs were still discovered, the frequency and severity of Microsoft bugs started to dry up. (P38)
For those not familiar with the memo, titled “Trustworthy computing”, it can be read in full here. The question that came to mind was, did the frequency and/or severity of Microsoft bugs go down? Before we answer, remember that this is fairly broad since it encompasses all Microsoft bugs, not specific to Windows or Internet Explorer for example. It is also important to note that Perlroth says they started to dry up, but not for how long. On the back of the Gates memo it would be expected that some researchers may change their attitude toward disclosure if they could sell the exploits for a higher payout. Finally, all of what follows is kind of moot because Perlroth’s statement is made on the back of a known unknown. That is, we know there are zero-day bugs discovered, but by nature, they are only zero-days if not publicly known.
Perlroth says two more lines that essentially tips her hand, I believe, demonstrating that her comments were made in mindsight based on extrapolation, not fact. First, she qualifies that she joined the security beat eight years after this memo. Second, she says:
The ripple effect of Gates’s [sic] memo could be seen far from Redmond, in underground dark web forums and in hotel rooms at the big security conferences.
The dark web barely existed in 2002. Given that Tor was released in September of that year, the first hint of dark web sites would have been starting. Gates’ memo was published eight months before Tor was released in fact. It’s hard to imagine that there were already established well-known forums to trade or sell vulnerabilities that would have a noticeable change at that point. With all of that in mind, I think that the rest of this rabbit hole is academic at best but illustrates why we must be careful when describing vulnerabilities in such a manner.
The Stats
There was a significant drop in volume from 2002 to 2003 so it is easy to make this assessment in a very limited picture. But by 2004 it was back up quite a bit. Given what I outlined above about her tenure in the security beat along with questionable statements around the dark web as well as making statements based on unknown factors, the question here is how did she arrive at this conclusion. Further, the severity did not drop from 2002 to 2004 either.
The stats above are from VulnDB with the advantage of hindsight and a comprehensive collection of disclosures from that period. If someone made such a conclusion based on disclosures, it likely would have been based on CVE. Looking at only disclosures with a CVE ID, it does not change for the disclosure trends or severity.
We see a dip in disclosures from 2002 to 2003 for both Windows and MSIE, but both rebound to varying degrees in 2004. Then Windows shoots up higher in 2005 while MSIE drops in 2005, which could just have been the browser war with Firefox and Opera heating up. That leads us to one more section from page 38:
Finally, did the bugs dry up or did their perceived value go higher, so people were less likely to disclose or sell for lower prices? For a book that dives deep into the value of 0days I figured this would be the hot take. Oh wait, it is, right after saying the frequency/severity dried up, Perlroth says:
Then, in the shadows, a growing number of defense contractors, intelligence analysts, and cybercriminals started doling out higher rewards to hackers who promised to keep their bug discoveries secret. In these subterranean circles, people started assigning a far higher value to Microsoft zero-day exploits than what iDefense was paying.
So the fun part is go back to the charts and speculate. If the premise is that the Gates memo caused bugs to dry up because they were perceived more valuable, as outlined shortly after by Perlroth, why did the disclosures rebound in 2004? Did Microsoft suddenly stop caring about security a year later? Was 2003 just an abnormal, albeit coincidental, year for disclosures? Were there other factors at play?
There are a lot of questions that Perlroth nor the vulnerability statistics answer.