Rebuttal: Ponemon on Network Breaches [Richmond/Ponemon]

[This was originally published on This is a rebuttal piece to “Security Professionals Say Network Breaches Are Rampant” (2011-06-22) by Riva Richmond (@rivarichmond) of the New York Times.]

The Ponemon Institute does not command much respect in many InfoSec circles. Like other ‘research analysis’ firms (e.g., Gartner), their reports rarely provide any insight or information that hasn’t been known for years by security practicioners. To C-level executives that prefer to read the sports section before articles pertaining to security, their papers are undoubtedly “eye openers”. Peddling common knowledge to these executives for thousands of dollars does not seem ethical to me, but if people are willing to pay…

Case in point:

There has been a flood of news about hacker break-ins at companies. But how bad is the situation really?

Significantly worse than the headlines suggest, and getting worse still, a new study from the research firm Ponemon Institute suggests. The study says breaches are rampant and occurring much more often than is publicized.

This is not news to anyone following security, and really should not be news to anyone in the business sector either. For decades, it has been proven time and time again that many breaches do not go reported. Many only see the light of day after word slowly leaks out, an employee mentions it in passing or fraud occurs as a result of the breach. In recent years, data breach notification laws have forced more companies to fess up when a breach happens, giving us a better look at the frequency of such breaches. Of course, this only counts toward the breaches that are known. Bad guys frequently compromise a network and perform no activity that would warrant detection, do not exfiltrate data, or if they do copy data, don’t use it in a manner that would be detected (e.g., mass credit card purchases, identity theft, etc.).

Breaches have been rampant for years. Compromises that may or may not have involved the breach of sensitive data have been staggering for years. shows almost 50,000 incidents (mass defacements generally don’t count as separate intrusions) in the last decade. Does Ponemon consider this when making the statement above? Or would Richmond / Ponemon like to qualify what “publicized” means to them? Just because you don’t look at a given publication, doesn’t mean it wasn’t publicized.

The firm’s survey of 581 security professionals at large companies in the United States, Britain, France and Germany found that 90 percent of them had at least one breach in the last year and 59 percent had two or more. And the costs are mounting; 41 percent of break-ins cost more than half a million dollars.

41% of break-ins cost more than half a million dollars. This is a staggering number, one that certainly makes the reader pay attention. In reality, it is your duty to question that number. Where did the damage figure come from? Did the random security professionals at large companies claim this? Did they have any backing for the damage figures? Are there public records in SEC filings and court records that verify even a fraction of the 41%? Hey Ponemon, you are a ‘research’ center, you did research and fact-check this… right?

The topic of inflated damage figures for computer crime goes a long way back. As far back as 1999, I was ranting about the topic and showing how numbers appear to be arbitrary. In subsequent years, it was demonstrated that the damage figures in the Mitnick case were severely inflated, and that the companies and prosecutors making the claims could not back the figures when pressed to do so.

Given the diversity of a “break-in”, the notion that so many cost “over half a million” dollars sounds unreliable at best. If a web server is defaced, the incident response likely does not cost that much. If a large-scale intrusion into the network happened at a global organization, that number could easily be more than 1 million. There are simply too many unknowns for anyone to come up with accurate numbers, so many fudge them and lean on the side of “spectacular” rather than “cautious”. Since Ponemon does not disclose their survey methodology or publish their questionnaire, we cannot even see how these numbers may have been reached.

Indeed, hackers are increasingly staging targeted attacks aimed at stealing something specific, said Larry Ponemon, founder of the institute. They study the target, find an opening and then quietly get in and out. Most are mercenaries, members of criminal syndicates or representatives of unfriendly countries, he said, and their attacks “are much more stealthy and much more difficult to identify.”

Three words for you Larry, and get used to them: CITE YOUR SOURCES!

How can you say with any certainty that “most are mercenaries, members of criminal syndicates or representatives of unfriendly countries”, if they “quietly get in and out”? How can you say anything about their demographics if they were undetected? You can’t, so you make it up to scare people, all the while profiting heavily off your “research”.

About 60 percent of respondents said they were able to identify the source of at least some of the attacks suffered by their organizations. They traced 34 percent of them to China and 19 percent to the Russian Federation.

As you are an adjunct professor for ethics and privacy at Carnegie Mellon University’s CIO Institute, I find it ironic that you don’t feel compelled to qualify this statement more. I would go so far as to say that you are being unethical by omission. Organizations “traced” these attacks simply by observing the IP address of the attack. As anyone with a couple months in security knows, that does not necessarily tell you where the attack originated. Many criminals compromise systems in specific countries for a variety of reasons, including disinformation. An American criminal launching an attack against an American corporation via a compromised system in China is smart, and has absolutely nothing to do with the attack being from the Chinese.

Nearly half of the breached companies surveyed by Ponemon suffered a damaging loss of data, which “speaks volumes about the mindset of the attacker community,” said Karim Toubba, vice president of security strategy at Juniper, which sponsored the survey.

This is perhaps the most telling line in the entire article. Juniper, a company that offers a wide variety of security solutions, sponsored this survey. The higher the number of breaches, the higher the number of damages, the better it is for a security company looking to sell solutions. Notice that while Juniper links to the survey from their page linked above, they do not make it obvious that it was a sponsored survey? “Press Releases: Ponemon Institute Survey Finds 90 Percent of Businesses fell Victim to Cyber Security Breach at Least Once in the Past 12 Months”

For a sponsored survey like this one, it is that much more important that the methodology be published for peer review. Let the industry see what questions were asked, if they were leading or if they were unclear. If you don’t, then we can only assume this is one big circle jerk between Ponemon and Juniper to drum up business for both companies, not a meaningful survey to our industry.

There’s certainly little reason to talk when, as the Sony case has shown, news of your vulnerability might make hackers hit you harder, customers lose confidence in you and your stock price drop.

For a New York Times reporter, one has to question the author of the article’s professionalism. Saying that the recent Sony compromises lead to a stock price drop is a bold statement. More so when you don’t disclose that their stock had been steadily dropping since March 1, a full month before the first DDoS attack against Sony. While the GeoHot lawsuit, negative press coverage and resulting attacks likely did hurt them, there is strong evidence that more issues were at play. It is ironic that Richmond links to a New York Times article about Sony’s earnings that does not mention stock movement, but cites a reason for significant losses: the Tsunami that recently hit Japan.

Your own article links to evidence suggesting you are not aware of the whole story, and have not looked at the subject matter closely. Disappointing to say the least.

Leave a Reply

%d bloggers like this: