The Not-So-Scientific Process

[This was originally published on]

During a recent trip to New York to attend HOPE 2000, I was introduced to a new project underway to help “dispel the myths about hackers”. Founded by a four person team at the Laurentian University School of Commerce, they have devised a survey to help “further Hackerdom’s growth by enabling outsiders to better understand Hackerdom and focus on its positive contributions to society, now, and in the future..

During the group’s presentation at HOPE 2000, several points were brought up by the audience that casted doubt as to the validity of the survey, the ‘scientific’ nature of their work, the scope of their questions, and the honesty of their intentions. After the talk, I stopped by their booth to pick up additional information as well as a full copy of the survey they were asking ‘hackers’ to fill out. As I began to read the survey and digest their replies to audience questions, I had more and more doubts as to the survey’s use and ultimate goal. Rather than dispelling hacker myths as it claimed, it seemed to be a tool that helped perpetuate some myths, while ignoring others that need dispelling.

Matt and I were able to catch up to one of the team members at Defcon 8 and ask her some follow-up questions. Her replies fueled my frustration and lead me to wonder what this team was doing on such a project at all. Instead of explaining points that were unclear, she only brought more doubts and concerns as vague and dodgy answers came out.

The Hackerstudy Team and Contact Information:
Dr. John Dodge – Business Strategy and Ecommerce professor
Kevin Ellis – recent graduate of the MBA program
Jano Lehocky – forthcoming B.Comm graduate
Dr. Bernadette Schell – Human Resources professor

Project Web Site:
Team Bios:

Laurentian University School of Commerce & Administration
Attention: Hacker Study
Ramsey Lake Road
Sudbury, Ontario
Canada P3E 2C6

Voice: 705.675.1151 x 2123 ……………… Fax: 705-673-6518

The Fundamental Problem

One of the most plagueing problems for all things hacker related is defining the term ‘hacker’. With the wide variety of meanings attached to the world, most traditional journalists find themselves explaining their use of the word as a qualification to their article. To blindly use the word ‘hacker’ without qualifying yourself, you open your work up to arguments and errata as each person looks upon the word differently. To some, hacker is a badge of honor, bestowed upon those coming up with brilliant solutions to difficult problems. Others see it as a term to describe malicious computer criminals that break into networks illegally. With such varying meanings, it is impossible to blindly use such a term, especially in a ‘scientific’ study.

Another serious point of concern that went unanswered by the team at HOPE 2000 (despite being questioned), is how the team will qualify their participants as “hackers”. If this survey’s strength resides in answers coming from the hackers themselves, then how do they ensure that only hackers are answering the questions? From: About the Study:

Our study will prove useful to all in Hackerdom by revealing facts
about hackers, as derived from hackers’ responses to our survey items

Any random kid off the street has the ability to seriously skew the results of this survey unless some attempt is made to qualify participants as hackers.

I attempted to clear this up at Defcon 8 by asking Dr. Bernadette Schell a few questions about these concerns. As each question progressed, Schell’s voice got quieter and quieter until Matt and I were leaning over the table trying to hear the whisper of each answer.

Brian: “What is a hacker in your opinion?”
Schell: “Hackers are a number of things..”
(At this point, no definition or explanation was offered.)
Brian: “So how do you qualify that the people taking the survey are hackers?”
Schell: “We let everyone who declares themselves a hacker participate.”
Brian: “That doesn’t exactly seem scientific, how can you be sure you are
collecting valid data?”
Schell: “I would say the people filling it out are serious.”

Re-reading their literature I find myself stumbling on one sentence in particular, especially compared to Schell’s comments above:

It is our hope that from our study, science will be able to dispel some
of these myths and provide the public and organizations with a balanced view regarding Hackers in society

Science does not come in the form of a “collection of self assessments” from people they ‘believe’ are serious.

Sleight of Hand

To many hackers, their identity and privacy are the most important thing. They will not relinquish information that could identify them and demand their privacy be respected. That in mind, participants should be interested in a few key points regarding this survey. Comparing various quotes from their literature one begins to wonder why they make such a point of claiming participants will remain anonymous when all the evidence suggests otherwise.

To ensure anonymity and confidentiality, we will NOT ask for
your legal name, your company’s name, or any other identification
in the questionnaire

Please note that your identity and personal results will remain
strictly confidential

This is a cleverly worded sentence that might be a good way to divert attention from a serious issue. There are two problems here:

They DO ask for identifying information in the questionnaire.
Even if they did not, there are several other ways they can track the person taking the survey.

The first way is extremely obvious. At several points, they offer participants a “personalized analysis” of your survey. How do you get your personal analysis? Answer yes or no in the ‘Followup Feedback’ section at the end of the questionnaire. Follow-up feedback:

Would you like your personal profile?  Yes___  No___

If "yes," please tell us in the space below how to get the information to you (ie: E-mail address, P.O. Box, fax number).

THANK YOU so much for completing this survey. PLEASE e-mail ( your responses to us, or fax them to us (705-673-6518). You may also mail your completed survey to:

Laurentian University School of Commerce & Administration
Attention: Hacker Study
Ramsey Lake Road
Sudbury, Ontario
Canada  P3E 2C6

I think it is pretty apparent that each way of contacting them gives the team some way to identify you. When Dr. Schell was questioned about this:

Brian: “Doesn’t the ‘followup feedback’ identify the participant?”
Schell: “I won’t know who they are.”

That answer doesn’t adequately address the concerns at hand. It shows a complete lack of understanding of how technology works, or is an outright lie.

The Tip of the Iceberg

For those wishing to participate in the survey online, you can do so from the Hackerstudy web page. Clicking on the “Online Survey” you get a one pager stating the goal of the survey etc. Clicking on “Begin Survey” is a whole ‘nother story. Rather than go to their own survey hosted at the University, it redirects you to a third party site ( that is hosting it. Relying on a third party for such a confidential scientific study seems irresponsible.

Interestingly enough, to take the survey you must provide a login and password or sign up for an account on this web site. Since the site uses cookies for functionality, it offers the perfect tracking device for those willing to participate in the survey. Once you have concluded the survey, it might be of interest to click on’s privacy policy.
“We may share user information in order to provide you with a more
integrated and customized user experience within our site.”

Great, so if the Hackerstudy team doesn’t give out my information, will.

Details, details…

During the HOPE 2000 presentation I asked how many people would be participating in the survey. Two and a half minutes later, the team finished with “I hope that answers your question.” No, it sure didn’t. It took a second direct question at Defcon to finally ferret out the answer from Dr. Schell. When asked, she replied “hundreds”. Giving her the benefit of the doubt and adding a healthy amount on top of that, lets consider 500 people responding to this survey.

The notion that 500 self proclaimed hackers could adequately represent the hacker population is absurd. Thinking back to the simple fact that the term hacker has not even been defined for this survey or anything else is amusing. So now we have 500 people professing to be something that we can’t define, representing tens or hundreds of thousands of people around the world. Hrm, there is another interesting point, around the world. Since the presentation and booth occurred at HOPE 2000 and Defcon, with the project residing in Canada, this survey seems doomed to represent North American hackers only. The lack of foreign translations to accommodate hackers worldwide backs this notion. Oops. There goes the science again.

Changing Tunes

Brock Meeks with MSNBC was present during the HOPE 2000 panel and took a keen interest in the claims of the survey being ‘scientific’. After several unclear answers to his questions, he managed to establish that the Hackerstudy team would indeed put their ‘scientific’ study up for peer review. This has been a longstanding tradition among scientific studies, that peers and critics could examine your material looking for errors or searching for ways to improve the results.

At the conclusion of this study, the team declared it would be put up for peer review on, possibly printed in a journal, and that they would likely “write a book”. The only real scrutinization the material would receive early on is from the Laurentian ethics committee. At Defcon, Schell confirmed that a book would be a likely result of the project, but did not mention the ethics committee. Could profit from book sales be a driving motivation behind this study?

While you’re at it…

After reading the questions on the survey, it seems that there are many rumors left untouched. Even pretending the Laurentian Hackerstudy survey was to be successful, many stones would remain unturned. Myths surround hackers and how they meet friends, if they do, whether it was online or in person. Others think that hackers are shut ins, never leaving their dark basements and that they enjoy the lighting their monitor provides. Hackers never visit the sun lit swimming pool, rarely venture out into the public for movies or playing pool. There is a definite link between hackers and shooting guns, no link between hackers and dating, etc. Which are fact or fiction? The questions found in this survey won’t help clear that up. These questions are either vague, extremist, irrelevant, or flat out contradict their claims of protecting anonymity.

Questions on the Survey

To save you the time of getting to the questions, I’ve included a few below taken from a printed copy of the survey handed out at Defcon. My comments appear in [brackets].

#5  Circle one label that best describes your sexual and lifestyle preference:
    a) Monogamous heterosexual
    b) Monogamous homosexual
    c) Bisexual
    d) Polygamous
    e) Commune/group living
    f) Open marriage
    g) Abstinence is bliss

#7  My last year's annual personal income before taxes was: ______
#8  If employed, how many employees work there?
#9  My formal job (or student) title is: _____

    [Aren't these questions useful in identifying someone? Combine these answers with an IP address or login, then give it out to advertisers or the FBI...]

#10 On average, how many hours a week do you spend on related "hacking" activities?

    [Since the survey and the team never define 'hacker', how can they expect a fair or honest answer when it is not clear what 'hacking' activities are? Oh wait, they 'define' it in another question...]

#13 Given the time you spend on "hacking" activities, what percentage of your "hacking" time is spent on the following activities:
    a) Breaking into websites and changing them
    b) Cracking software releases
    c) Breaking communication codes
    d) Designing/Creating new software
    e) Designing/Creating new hardware
    f) Communicating with other hackers (ie: email, irc, etc)
    g) other

    [Oh, this is perfect. Brand these charlatans with the big 'H' for 'hypocrisy'. If 'hacking' activity can be lumped into these six things with a casual "other" for leeway, the Laurentian Hackerstudy team has already proven this survey worthless. They leave out some choice options that are dominant in the 'hacker' subculture I believe. Reading or writing about hacking/security?
     Breaking into computers with the owners permission? Maintaining a hacker/security WWW/FTP resource? Communicating with security professionals discussing hacking or security issues? That is certainly a lot to lump under 'other'.]

#15 Do you typically collaborate with other hackers on your hacking projects?
    a) No, i tend to work alone 
    b) Yes, I tend to collaborate with others

    [What, no 'c'? How about "Yes, with other NON hackers"? Their assumption that hackers can only be bad/evil/illegal/negative connotation is a contradictory statement to their own goals.]

#18 How do you typically, identify yourself on-line?
    a) I use my birth name
    b) I use a net handle
    c) I use a combination of my birth name and net handle

    [Identifying information?]

Part 2: Over the past two weeks, how often have you experienced the following health symptoms? Please use the following s cale for your responses:

    Not at all (0)      Littled (1)     Quite a Bit (2)      Extremely (3)

    1. Headaches
    3. Being unable to get rid of bad thoughts or ideas
    6. Feeling critical of others
    7. Bad dreams
    8. Difficulty in speaking when you are excited
    11. Feeling easily annoyed or irritated
    19. Poor appetite
    21. Feeling shy or uneasy with the opposite sex
    25. Constipation
    26. Blaming yourself for things
    29. Feeling lonely
    30. Feeling blue
    32. Feeling no interest in things
    34. Your feelings being hurt
    36. Feeling others do not understand you or are unsympathetic to you
    43. Loose bowel movements
    45. Wanting to be alone
    52. Feeling hopeless about the future

    [Looking beyond the duplicate questions (23/33, 23/37/48), considering the above list in the context of a two week period, what does this prove or disprove? If you happened to be sick the past week and then fill this survey out, you could potentially skew the results. If you answer honestly about many of these vague and unqualified questions, you are fueling more stereotypes and myths that can be applied to ANY group of people in the world.]

Part 2 B) Mind-Body Symptoms
    2. I have often felt "very down" or "depressed"
    3. I regularly blame myself for things that I have done or not done.
    9. When I find myself in "a very self-confident" or "a high" mood, I am sometimes easily annoyed or irritable
    11. When I find myself in "a very self-confident" or "a high" mood, I can recall doing foolish things with money.
    13. When I feel "very down" or "depressed" I sometimes feel very bad and do not know why.

    [In today's society, aren't most of these 'symptoms' seen in everyone, regardless of being a 'hacker'? Doesn't question 13 vaguely define "depressed" or "very down"?]

Part 3  Routine Behaviour
    3. I am mainly concerned with my own well being.
    16. Certain conditions or situations are the most important cause of my personal misfortunes.
    19. Reason, rather than emotion, guides my behaviour.
    35. Certain situations and states (eg, at my place of work) tend to make me unhappy, 
        but there is n othing I can do to alter things.

    [Isn't 'behaviour' 19 extremist? Do they not see the possibility of a mix of reason and emotion guiding behaviour? Who is qualified to give a self diagnosis to that degree?]

Their World is Collapsing

Sensing Dr. Schell’s hesitation and lack of solid answers, Matt jumped into the fray at Defcon. He began asking what special insight industrial psychology had in their survey. Rather than providing an explanation, Dr. Schell took on the look of a deer in headlights, as if dumbfounded that someone could or would ask these questions. Matt went on to explain that other projects and surveys had been conducted around the psychology and sociology of hackers. He questioned if the team had read this previous work, would cite it, and most importantly, build on it. Dr. Schell could not answer when asked to name an author that could be credited with their ‘approach’ or methodology.

Claims of a scientific study to help dispel the myths about hackers. A survey to be taken by ‘hackers’ to generate new findings and results about a community the Hackerstudy team has little knowledge about. One would argue that a lack of understanding about the way hackers operate might give them an unbiased view, an edge in guiding this study. At that point critics should be quick to point out that their qualifications in psychology, sociology and communication should be top notch. With no foundation or credible backing on their approach, their carefully worded and misleading assertions of anonymity, and a fundamental lack of communication skills when engaged in simple verbal dialogue… I would steer clear of this group.

Leave a Reply

%d bloggers like this: