≡ Menu

The NFL Draft and the Wisdom of Crowds

[Chase note: Take a look at the name at the top of this post. Our good friend Andrew continues to desire to post here, and we thank him for that.]

Not the focus of Galton's experiment.

Not the focus of Galton's experiment.

In 1906, Sir Francis Galton probably wasn’t thinking about the NFL draft when he asked almost 800 fair goers to guess the weight of an ox. No one person accurately guessed its weight, and the guesses were all over the map, but the mean of all the guesses (1197 lbs) was within one pound of the actual weight of the ox. As I looked through endless mock drafts leading up to last Thursday night, I wondered if there was anything to be gained by looking at the wisdom of the crowds. Could we do a better job of predicting the NFL draft if we took all the knowledge and tried to put it together?

And the answer appears to be yes… to an extent. The NFL draft is not exactly a place where we’d expect the wisdom of crowds to be particularly strong. The power of the wisdom of crowds comes from lots of people bringing their own independent information to the table. For example, prediction markets appear to do a great job of predicting events like a president’s chances of being reelected. Sports prediction markets (a.k.a sportsbooks) similarly succeed in predicting game outcomes. And the stock market often reveals companies’ true values. In each case, every individual transaction represents a piece of information which gets reflected in the price.

Of course, the crowd is not always so wise. Stock markets can go haywire. Betting lines can be affected by people’s biases. The wisdom of crowds can break down when groupthink occurs and people stop having independent opinions. The NFL draft certainly looks like such a case. All the mock drafts are out there and the experts have the implicit pressure to not be too different.1 In those circumstances, we could lose in a haze of groupthink much of the original information that people have.

And it looks like there may be some of that going on, too. Looking across a bunch of mock drafts, it simultaneously looks like the crowd both does better than all but a couple predictions and that there is some important information in those draft boards that look different from the rest.

The Data

I collected 27 mock first round drafts from the internet on May 8. These were the final draft predictions for each individual I collected. I took many of the top ranked forecasters from the last five years on The Huddle Report. I also included a few more of the particularly well-known football media people such as Peter King, Mike Mayock, and Mike Florio. I included some of these guys in part to try to get some heterogeneous opinions.

To construct the Wisdom of Crowds (WOC) draft, I assign each draft pick a total point score. A player gets 32 points for someone predicting them first overall, 31 points for second, and 1 point if someone was predicted to get last in the first round. The idea here is basically to take the mean prediction across all the predictors. So Jadeveon Clowney is the predicted #1 pick because he collected 855 points for his 23 first place votes, 3 third place votes, and 1 fourth place vote.

The Wisdom of Crowds Draft

Here’s the WOC draft. Each player is listed with their total number of points. In parentheses is the player’s actual draft position.

PickPlayerTotal points
1Jadeveon Clowney (1)855
2Greg Robinson (2)818
3Khalil Mack (5)794
4Sammy Watkins (4)788
5Jake Matthews (6)732
6Mike Evans (7)707
7Johnny Manziel (22)666
8Taylor Lewan (11)646
9Aaron Donald (13)589
10Zack Martin (16)566
11Justin Gilbert (8)546
12Anthony Barr (9)496
13Odell Beckham Jr. (12)485
14Eric Ebron (10)481
15Ha Ha Clinton-Dix (21)471
16Darqueze Dennard (24)465
17Blake Bortles (3)439
18Calvin Pryor (18)379
19Kyle Fuller (14)362
20C.J. Mosley (17)361
21Brandin Cooks (20)302
22Ryan Shazier (15)283
23Marqise Lee (39)260
24Derek Carr (36)212
25Jason Verrett (25)199
26Bradley Roby (31)194
27Teddy Bridgewater (32)150
28Cyrus Kouandjio (44)138
29Timmy Jernigan (48)95
30Morgan Moses (66)86
31Ja'Wuan James (19)75
32Cody Latimer (56)70

One way to assess the accuracy of the WOC draft is to compute the distance of each of the actual players drafted from their predicted position. For example, Johnny Manziel has a prediction error of 15 because the WOC prediction was for him to go at #7, when he actually went at #22. Ja’Wuan James has a prediction error of 12 because the WOC had him at #31, while he went at #19.

To assign a distance score for the players who were not predicted to go in the first round and for all the first round mock drafts, I use the difference between 33 and where the player was selected, adding a 3 point penalty for not picking the player in the first round. It doesn’t really matter what penalty is picked when we compare the WOC to the individual mocks.

The total distance between the WOC draft and the actual players picked in the first round is 162. Looking at all the other mocks, the WOC finishes tied for first out of the 28 entrants.

PrognisticatorTotal distance
Wisdom Of Crowds162
Luchene and Mecino162
Christensen163
Mayock165
Kadar166
Brinson168
Lawlor172
Phlipot176
Clearwater181
Rang184
Kirwan184
Boris186
Brugler187
Fitzgerald188
Spencer and Engle189
McGinn190
Kiper193
Fronczak193
Zalis193
Martz194
King196
Farmer201
Fairburn203
McCrystal206
Welser208
Prisco211
Salem218
Florio230

So the WOC draft does better than most drafts at estimating where players will end up. There is some important value to be gained in adding up all the information contained across the different mock drafts. But the WOC draft, like the mocks on which it was based, also had some notable prediction failures. The data offer some hints as to why those failures might have happened.

Groupthink and Prediction Failures

When Galton was estimating the weight of the ox, he actually cared about the median guess first. That also turned out to be close to the actual weight, but not quite as close as the mean. There are some reasons to think the median might do better. If somebody guessed 100,000 lbs, that would cause the mean to be off, but the median would still be OK. At the same time, if someone has some valuable information that’s different from the crowd, the mean will do a better job of picking that up.

For the NFL draft, is there any evidence that people who made unusual predictions may have been on to something that the rest of the crowd missed? There are a few examples that suggest this might be the case. Blake Bortles is maybe one example. His WOC prediction was 17 and the only reason it’s that low is because a few people picked him to go pretty early. Three predictions had him going at #8, one at #4, and one #1. These outliers wound up being closer to the truth than the majority of the forecasters.2

Another case is Ja’Wuan James. Four people actually picked him exactly right at #19, while 21 had him out of the first round altogether. It seems likely that the four who had him at the right spot had some valuable information that the rest of the crowd missed. Most likely, those mock drafts recognized that (a) Miami really would want to draft an offensive tackle, and (b) the top tier tackles would all be off the board, and the Dolphins would go with need over best player available.

Finally, consider the graph below for Darqueze Dennard, who went 24th overall to the Bengals. Twenty of the 27 predictions had him going in the top 15. But there was a small cluster of other people who had him pegged almost correctly. Three people pegged him at #25, just one spot off where he landed. It is possible that the crowd herded on earlier spots in the round, while this smaller group had some valuable information about where Dennard would actually go.

Dar Denn

Conclusion

The Wisdom of Crowds (WOC) draft performed much better than most mock drafts in terms of putting players near where they were eventually selected. Since the experts are putting those mock drafts out there semi-constantly, those opinions are far from independent. Still, the wisdom of crowds mostly works, subject to some clear limitations.3

Of course, predicting the draft could mean something different. For example, the Huddle Report’s prediction contest gives points for predicting people to the correct team and getting them in the first round. We could think about creating a smarter wisdom of crowds prediction machine that would attempt to achieve the objective of assigning players to teams, too. Here, the prediction machine is pretty simple, just picking the highest remaining player regardless of position. That will still tend to account for much of the variation, but it can put someone at 17 who is actually equally likely to be selected at 14 and 20, for example, with the teams in between interested in other positions.

One of the reasons the WOC draft works somewhat better than the individual drafts themselves is that it incorporates outliers who may have some independent information. It does better than most mocks with players like Ja’Wuan James and Darqueze Dennard where most people have herded on a certain option, but a minority has a very different opinion. Note, too, that the cases of James and Dominique Easley illustrate that we may particularly want to pay attention to people with localized knowledge. Local mock drafts from Miami and New England got those two surprise picks exactly right.

  1. In some cases, there may be incentives to stand out from the crowd with an original prediction, too. Overall, there are incentives that can make predictions depend on those made by others. []
  2. Chase’s mock draft also wisely put Bortles towards the top at #6. Of course, he also had Bridgewater at #3, which was slightly less accurate. []
  3. It seems almost certain that people shade their mock drafts to match the crowd. This kind of shading of results can happen even with polling data, where it’s just a little bit more nefarious and important. []
{ 11 comments }
  • wiesengrund May 13, 2014, 2:42 am

    I also think that thi method captures a different aspect than Huddel Report, since it just looks at pick number, not at the team. Predicten wheren in the draft a player might go is easier than predicten WHO might take him because of trades.

    That is also a reason why we see cluster like Dennard. This is clearly a case of people thinking about certain teams and projecting needs. If the Bengals traded up to the ate teens to get him, noone would have been close in guessing that position, but that cluster in the early 20s probabyl was close to it “in spirit” if they thought about it as a Bengals-pick.

    Maybe there is a way to incorporate a second distance-measure, that measures how far away the teams are that were projected to make the pick? I think for instance, if you pegged Vikings to take Bridgewater at 8, it’s ok to say you utterly failed in the “at what spot wil Bridgewater go” question, but you still got a major part of the process right (predicting that the Vikigns will take him, and that no other team before the Vikings will take him, and that no other team behind the Viking will trade up before the Vikings to take him etc…)

    I also felt no grading system could adequatley refelct both aspects of distance.

    Reply
    • Richie May 13, 2014, 1:48 pm

      Raiderjoe?

      Reply
    • Andrew Healy May 13, 2014, 5:53 pm

      I think that’s right that it’s hard to both capture both distance of the player from the projected spot and the team projected to make the pick. But I do think you could come up with another wisdom-of-crowds sort of estimator that would aggregate all the opinions people had about who would make the picks. The simplest thing would be to take the team predicted most often to select each player (making some adjustment to try to account for too many players ending up with one team or something like that). Certainly doable. My guess is it would probably work less well than just trying to minimize the distance, but I may be wrong.

      I think you’re right about the positional stuff for Dennard, but it’s also true that a quarter of the projections had the Jets taking a corner. The cluster separate from the rest appears more clearly for Dennard than it does for other corners like Fuller, Gilbert, and Verrett, who were picked closer to where people expected. But we are definitely talking small samples on this stuff.

      Reply
  • Topher Doll May 13, 2014, 10:33 am

    I actually did the exact same thing except for big boards instead of mock drafts. I always find that using a larger group tends to lend itself to more accurate results. Now my goal wasn’t to more accurately predict the draft but to measure which players will be better (which is what a bit board is anyways, ranking players by ability) and then in 3 and 5 years compare their AV to the average for their position taken to see if they were above or below the average and then compare where they were supposed to be taken.

    Here is the short version of my combined big board from 2014: http://www.milehighreport.com/2014/5/8/5694044/nfl-draft-2014-the-ultimate-big-board

    Some changes were made by another writer to make it more Bronco centric (the site I write for is a Broncos site) but the data itself is good.

    Thanks for doing this, it was interesting to read your data and see that WOC works for mocks as well as big boards.

    Reply
    • Andrew Healy May 13, 2014, 5:22 pm

      Neat stuff. I wanted to check that out, too. If you’d be willing to share your data (on the big boards from earlier years, which it sounds like you’ve collected if I’m understanding), I’d be interested to do some analysis with it. Definitely seems like the wisdom of crowds should apply to predicting future success, and there might also be some groupthink going on for those predictions, too.

      Reply
      • Topher Doll May 13, 2014, 5:37 pm

        I’m going through a move but I’d be willing to share it with you when I get settled in.

        Reply
  • Richie May 13, 2014, 1:49 pm

    Excellent post.

    Reply
  • Trey May 14, 2014, 8:55 am

    Great post, Andrew.

    Reply

Leave a Comment