First, this post is not sanctioned by DEF CON in any way. I am a member of the CFP team who decided to keep some rudimentary statistics on the submissions this year. I did this to give the team a feel for just how many submissions we got, how many talks we accepted, and primarily to track the way we voted. This greatly assists the powers that be (the amazing Nikita) to more quickly determine which talks are well-received. Every day that I kept up on the spreadsheet, the more ideas I had on tracking. Other team members said “you should track…”, and I typically did. So this blog is to give some insight into the entire CFP process, with a solid slant on statistics about the submissions.
First, a few basics:
- DEF CON 25 CFP opened on February 01, 2017
- DEF CON 25 CFP closed on May 01, 2017
- 17 talks were submitted after closing date and were considered for various reasons
- We received 536 submissions
- Three of the submissions were retracted by the end of CFP
- BlackHat received 1,007 submissions this year for comparison
Next, who are we? There were technically 31 DC CFP reviewers this year, and you can read their fun profiles now (mouse over stuff here and there, call it an Easter egg)! Ten of them are considered ‘specialty reviewers’, where they typically review talks on a very specific topic such as ‘social engineering’ or ‘legal’. These are generally topics where the submissions are either too numerous and potentially murky to figure out if they are worth accepting (social engineering), or a topic that most of InfoSec aren’t really experts on, even when some of us are the #1 armchair lawyer in InfoSec. The specialty reviewers are expected to review their topic only usually, while a few are open to review multiple topics. That means there are 21 reviewers who are expected to review ‘as many talks as you can’, understanding that we may DEFER on a given submission if we feel it is out of our wheelhouse, and remembering that this is extremely time-consuming and we all have day jobs. Some of us have night jobs, and some of us have social lives (not me).
Every year we come up short on reviewers who are truly qualified to give solid feedback on a given topic. This year DC CFP put out a call for more volunteers and we hit a bit of gold, getting several new reviewers who are quality and put in a crazy amount of time. Next year? We know there are topics we need help on, so if you are sharp, kind of special(ty), or the top of your game in a popular field… come join us. I can’t stress how important this is. Instead of just working on a talk or doing a thing, you have the ability to help influence the presentations given at a conference with some 20,000+ attendees. That is a lot of power, a lot of influence, and the potential to do a lot of good. Personally, that is why I still sacrifice the incredible time I do.
Shout outs! The only way to start this paragraph is to call out Nikita for handling almost all CFP submission related emails. Incoming submissions, replies saying “you didn’t follow directions”, second-attempts, replies saying “no really you ‘brilliant hacker’, you didn’t read our guidelines”, posting them to the CFP platform, watching for the CFP team to say “I have questions” and us largely forgetting to flag it back to her, her following-up with the submitter, repeating several times in some cases, posting their replies, looking for the CFP team to ask more questions… hopefully you get the picture. The amount of work she fields in a three-month span, just related to CFP, is insane. I say that as someone who has worked more than 80 hours a week in this industry for the last twenty years. Oh, did I mention that she also voted on 60% of the talks? While five ‘full’ reviewers voted on less talks than her.
A plea! If you didn’t see the numerous Tweets and requests to get your talks in early, I cannot emphasize how much it benefits you, more than us. When a talk comes in during the first few weeks, it gives us plenty of time to not only review and ask questions, but to give feedback in the way of suggestions. In some cases, one of the team will break away from the board and work with the submitter to improve their submission. This year, I did that once with someone who’s original two submissions garnered a single yes vote. After working with them and giving feedback on how to combine the talks and hone in on the areas of interest, the re-submission received 12 yes votes and zero no votes. In an ideal world, that would happen for every submission, but a significant number of talks are submitted the last two days.
Meaningless numbers! Because our industry loves to work with statistics that they don’t fully understand or have little meaning without serious caveat and disclaimer (PPT), let me throw out a few. For the 536 submissions we received, the CFP team voted yes 1,223 times, no 3,555 times, maybe 186 times, deferred 945 times, and abstained 54 times. Again, we defer if we feel that a topic is not one we can fairly judge based on our expertise and rely on the rest of the team to review. We abstain when there is a potential conflict of interest: if we work with the submitter, we contributed to the submission, or have a negative personal past with the submitter.
Meaningful numbers! We requested feedback from the submitter 125 times and changed our votes 61 times. Working with us to answer our questions, willingness to accept our feedback, and work with us to build a better presentation benefits everyone. As Nikita tweeted, more than 60 of the accepted talks were from first-time DEF CON speakers. Given there were ~ 110 accepted talks (and 422 rejected), that is quite a lot. It is encouraging to see this many new speakers given some of the past submissions from egotistical industry veterans that felt they deserved a speaking slot on the back of a weak submission, simply because of “do you know who I am?!”
More meaningful numbers! Of the 536 submissions, 185 (34.77%) said they would release a new tool. Only 56 (10.53%) of those submissions said they would release a new exploit, and some of those claims were questionable. It is common for people submitting to DEF CON to also submit to BlackHat and/or BSidesLV. This year, 218 (40.98%) of those submissions were also submitted to BlackHat and 65 (12.22%) of them were also submitted to BSidesLV. For various reasons, often around the ability to get to Las Vegas, some submitting to BlackHat will submit to DEF CON but say that acceptable at DEF CON is contingent upon acceptance at BlackHat. This year, 36 (6.77%) talks were submitted to us with that caveat. In a somewhat arbitrary categorization, overall I felt that 200 (37.31%) of the talks were ‘red’ (offensive), 88 (16.41%) were ‘blue’ (defensive), and 38 (7.09%) were ‘black’. By ‘black’, I mean that the topic really had little merit or benefit for red-teaming and were really in the realm of criminals.
Even more meaningful numbers! Some of the most basic stats that can be generated for your ocular pleasure. First, these are arbitrary categories that were developed as we received submissions. Nothing formal and some talks were hard to classify:
From there, I broke it down further by some topics that aren’t necessarily specific to the red or blue domain. Again, kind of arbitrary and based on seeing the submissions as they came in and note that one talk may have been flagged as more than one topic:
When building a schedule over four days and across five tracks, while considering if it is better to suggest a talk for a village or alternative venue (e.g. Skytalks), Nikita has to play Tetris of sorts based on the accepted talks, the requested time, and the schedule. This is what she had to work with:
One of the more popular questions this year after an increased awareness and public discussion around diversity in InfoSec, is the gender breakdown for submissions:
Finally, a general picture of the submissions by month. Recall what it looked like for the April breakdown above and you once again get a good idea why we would like more submissions earlier in the process:
Finally, a quick note on a common perception for InfoSec conferences and talks in general. Given the drastic rise in the number of conferences popping up, there is a saturation that demands more submissions to fill the schedules. That means that veteran speakers can typically shop their talks around or be selective in where they submit based on the venue they find appealing. That also means more new speakers are submitting which results in a wide range of topic and quality of submissions. That led me to argue this Tweet and remind people that a conference can only work with what is submitted. Personally, I feel that the overall quality of submissions to DEF CON (and a couple other conferences I review for) have gone down this year and last. That means that DEF CON ended up accepting some talks that I personally did not care for.
Bottom line? If you are researching a cool topic, submit a talk on it. Have a unique perspective or done more digging on something? Share your work. Never submitted before? Submit early and let us work with you if you need it. If a security conference is lacking, it is due to the community as much as anything else.