≡ Menu

Why Aren’t Teams Better At Drafting Now?

The NFL Draft has emerged from an afterthought to the center of the sports world every April spring. A cottage industry of draftniks has emerged. Teams spend more time, money, and other resources on scouting than ever before. Scouting departments have grown exponentially in both size and sophistication. The Draft used to be much less important, as evidenced by the way teams happily traded away future first round picks like they were fringe benefits. Over the last 45 years, teams should have become a lot better at drafting. But have they?

Measuring how well teams draft in the aggregate is not easy.  But suppose teams were perfect at drafting. In that case, the first pick would always turn out to be better than the second pick, the second pick would always turn out to be better than the third pick, and so on.  Right?

Well, maybe not. Nobody quite knows the ratio, but player development is a crucial part of the drafting process. Prospects do not come to the NFL as finished products, and it’s up to the team (and the player) to turn that college athlete into an NFL player. Making a selection on draft day is just step one, not the final step. When a player busts, is it the fault or the person in charge of the draft or the person in charge of the development?  When a player booms, is it because of the GM or the coach? I don’t know.  You don’t know. Nobody knows.

But we do know that player development is an important variable, so even in a perfect drafting world, we probably wouldn’t expect each player to turn out to have a better career than the player drafted after him. But comparing draft status to player production seems like the most basic and obvious way to measure draft efficiency. Frankly, I don’t know even how else one would measure draft efficiency than by comparing draft slot to player production.  I’m open to other ideas in the comments, but here’s what I did.

The top 25 selections

I looked at every draft from 1970 to 2008, and initially focused on the first 25 picks.1 Based on my Draft Value Chart, roughly 35% of all the draft value of the top 2222 picks in every draft is captured in just the first 25 selections. Have teams become more efficient when it comes to the top 25 picks? The graph below shows the percentage of Career Approximate Value grades from Pro-Football-Reference.com for the top 25 picks (as a percentage of the total career AV of the top 222 picks) in each year from 1970 to 2008:

draft top 25

As you can see from that black trend line, teams aren’t extracting more value on a relative basis from the top 25 picks.  In other words, there are still a lot of busts inside the top 25. That’s not a shocking conclusion to anyone, of course, but why haven’t teams improved when it comes to drafting first round picks over the last four decades? In 1978, the Bills traded an over-the-hill O.J. Simpson to San Francisco for the 49ers first round pick in 1979. That turned out to be the number one pick in the draft, which the Bills used on Tom Cousineau… who promptly went to Canada over Buffalo. That seems more like the leather helmet era than something approximating the modern NFL.

So shouldn’t teams be performing better? Shouldn’t there be fewer busts in the first round?

Making the matter more complicated, one could argue that better scouting could make it even harder to evaluate the draft.  Maybe a Jimmy Graham — a basketball player who caught 17 passes in college — wouldn’t have been drafted 20 years ago, but in the modern era became a third round pick. So it’s a credit to scouting that a player like Graham was taken. As a result, if teams are finding more steals later in the draft, that would make the players selected in the first round look less valuable on a relative basis, which could explain what we see in this chart.

On the other hand, Graham looks to be one of the five or ten best players from the 2010 draft, so it’s easy to say that he’s an example of draft error. I mean, in retrospect, he would be a top ten pick, so should the NFL as a whole be punished or praised for Graham going at 95?  That’s hard to say, and unfortunately, I have no answers. Four decades ago, it’s possible that Graham wouldn’t have even been drafted. But if teams are so good at finding steals now, what explains the myriad of first round disasters?

An alternative theory, and perhaps a desirable one, is that player development is more important than pure talent. A player’s work ethic should already be hard-wired into his draft status, so this would really be a reflection of each team. Can your wide receivers coach get the most out of the raw but talented rookie? Will your offensive line coach be able to improve on the technique of your 3rd round pick? On the other hand, teams don’t show a consistent ability to get more out of players than other teams, either.

I looked at one other study that did seem to show some improvement by NFL teams. I measured the correlation coefficient between draft value and player production (using PFR’s AV) at the top 222 selections in each draft from 1970 to 2008. For draft value, I used the draft values from my draft pick calculator, calculated on a percentage basis of all 222 draft picks. For example, the 1st overall pick is worth 2.41% of all draft picks in each year, while the 53rd pick is worth 0.65%. For each player, his grade is a percentage of his Career AV divided by the total Career AV of the top 222 selections. So in 1970, Terry Bradshaw gets 2.41% of all draft pick value and 3.0% of all career AV; meanwhile, Mel Blount used only 0.65% of all draft capital but produced 2.9% of all career AV.

The correlation coefficient measures how closely the two variables are related. The results here are interesting: there is a moderately strong correlation (as we would expect) each year between draft value and career value. I then plotted the CC from each season over the period 1970 to 2008:

draft cc

The black line shows that yes, drafting does appear to have gotten slightly better, at least since the early ’70s. But the early ’80s were a pretty efficient time for drafting, and the 2005 and 2006 drafts were not very “accurate” for our era. In ’05, Trent Cole, Frank Gore, Chris Myers, and Justin Tuck were later round picks who fared very well, while the end of the first round (Aaron Rodgers, Logan Mankins, Roddy White) were were better than many of the earlier picks (Troy Williamson, David Pollack, Mike Williams, Erasmus James, Cadillac Williams).

In 2006, Jahri Evans and Brandon Marshall were fourth round steals, Maurice Jones-Drew was a late second, and Kyle Williams, Cortland Finnegan, Charlie Johnson, Antoine Bethea, and Jeromey Clary were all successful end-of-draft selections. Meanwhile, Matt Leinart, Tye Hill, Vince Young, and Bobby Carpenter were first round picks who left a lot to be desired.

Perhaps the best question is how good can teams get at drafting? Are we already at the optimal level? That seems hard to believe, but I also can’t imagine that we’re ever going to get to a place where there are only a few busts in every first round. Let me know your thoughts in the comments, along with any other ideas on how to measure draft efficiency on a league-wide basis.

  1. Why 25? If I used top 32, I was concerned I might bias the results by giving extra weight to bad teams with multiple picks in earlier years. []
  2. Why 222? The 1994 Draft had just 222 picks.  As we’ll see in a minute, I also analyzed the entire production from each draft, but limited myself to just the top 222 picks in each draft so as to compare apples to apples. []
{ 19 comments }
  • scott April 29, 2014, 12:20 am

    Could it be possible to measure the length of time a player has had in the NFL vs. draft spot. Like a first rd player is always talked about as a high level contributor for 10+ years. While the 7th rd pick a special teams player or developmental pick. So if you have a 1st rd player drafted that plays 10+ years you got your value out of him. While if you take a player in with 222nd pick and you get 10+ years out of him you have a weighted value or if that player is cut before he even plays that’s a negative value.

    Reply
  • chaz April 29, 2014, 8:40 am

    Chase-

    great article (and site!) top of head ideas:
    1. remove players who suffered a career ending injury in say, their first three years. even a perfect pick and great development efforts can’t protect against this. removing these players would remove the injury penalty (if any; no idea how often this really hapens) in this analysis.
    2. account for changes in coaching/front office staffs. a player drafted by a 4-3 team whose coaches are fired and replaced by a 3-4 scheme the next season may struggle to produce and underperform his draft slot. on the one hand, it seems fair to punish the team for failing the player’s development in this regard. but on the other, we’d expect a player drafted into a stable regime to progress faster/further than one drafted into a tumultuous organization, right? so maybe, “what is the AV per draft slot of players who played in the same system for X number of seasons”?

    Reply
  • Richie April 29, 2014, 3:26 pm

    In the first chart, I assume the x-axis is showing “year #”? 1=1970, 2-1971, etc?

    Reply
    • Chase Stuart April 29, 2014, 3:28 pm

      Yeah that was pretty bad; don’t know how that happened.

      Reply
    • Richie April 29, 2014, 3:28 pm

      If that’s what I’m looking at, it seems like there has been a reduction in the variance from year-to-year. Is it a significant reduction? Does that tell us anything?

      Reply
      • Chase Stuart April 29, 2014, 3:32 pm

        I’m not sure. Could be worth trying to investigate.

        How would you go about measuring whether teams are getting better at drafting?

        Reply
        • Richie April 29, 2014, 3:54 pm

          I think you did a pretty good job of estimating things. Situation is so critical. Tom Brady will always be a famous example. If Bledsoe didn’t get hurt in 2001, would Brady have ever gotten a chance? How long would it have taken? What if Bledsoe got hurt early in 2000 instead, and Brady was given a shot (looks like John Friesz probably would have been called instead), but wasn’t as prepared and looked like Case Keenum?

          I have to think that if teams really were getting better at drafting that the top 25 out of 222 would be a pretty good approximation. But like you mention, a guy like Jimmy Graham makes draft skill look worse in 2010. What about gross AV for those top-25 picks each year? Does that show a trend? That would reduce the effect of Brady or Graham “hurting” the top-25 efficiency.

          I think you did a draft analysis before where you were only looking at the first 5 years of a player’s career. Deion Sanders kept playing 4 more seasons after Barry Sanders retired and almost caught him in career AV, but I don’t think that’s really representative of draft skill.

          Side note/observation:
          Looks like the ’91 draft must be the low point on your chart above. Of the top-25 career AV from that season, only 4 (Ted Washington, Herman Moore, Russell Maryland and Todd Lyght) were drafted in the first 25 picks. This is the draft with guys like Favre, Aeneas Williams, Rick Watters, Mo Lewis, Keenan McCardell.

          Reply
        • Richie April 29, 2014, 4:15 pm

          hmmm….another thought:

          What if you use a binary success/fail with each pick? Take the top-25 picks from each draft, and define each pick as a success or a fail (based on some AV level), and look at percentage of success. In the extreme, what if a 1st overall pick scored 40% of the career AV for a draft, this would make it look like team really got that draft right. But maybe the 24 teams all busted their picks.

          Even the “years as primary starter” from PFR might be a good approximation. At a glance, it looks like players with 5+ years as a starter look like good picks.

          Maybe do buckets.
          First 25 picks – 5+ years as a starter
          26-50 – 4+ years as a starter
          51-75 – 3+ years as a starter
          etc.

          1993 would have 8 busts in the top-25:
          Leonard Renfro DT
          Patrick Bates DB
          Dan Williams DE
          Ernest Dye T
          Deon Figures DB
          Eric Curry DE
          Darrien Gordon DB
          Wayne Simmons LB

          And 17 successes:
          Rick Mirer QB
          Irv Smith TE
          O.J. McDuffie WR
          Steve Everitt C
          Tom Carter DB
          Lester Holmes G
          Robert Smith RB
          Garrison Hearst RB
          John Copeland DE
          Sean Dawkins WR
          Marvin Jones LB
          Lincoln Kennedy T
          Curtis Conway WR
          Drew Bledsoe QB
          Willie Roaf HOF T
          Jerome Bettis RB
          Brad Hopkins T

          For a 68% success rate.

          Reply
          • Robin April 30, 2014, 2:51 pm

            Years is not a fine enough statistic or your buckets are not appropriately divided. Any measure that concludes that the selection of Rick Mirer was a success is generating some bad data (even if the majority of the other names on the list look appropriately divided). In fact many high pick QB “busts” seem to hang around for years as backups.

            Reply
            • Richie April 30, 2014, 3:07 pm

              “backup QB” does have some value. But that wouldn’t be picked up in my threshold, because being backup doesn’t qualify as a “year as starter”.

              But I agree that Mirer probably shouldn’t be looked at as a successful draft pick – if he was taken anywhere in the top-25 picks. I didn’t like seeing him on my list. But I was just throwing out an idea that wasn’t thoroughly researched.

              Just for info, here is a list of all QB’s drafted in the top-25 between 1986 and 2005:
              Player St CarAV
              Peyton Manning 15 155
              Vinny Testaverde 15 97
              Drew Bledsoe 12 103
              Troy Aikman HOF 12 97
              Donovan McNabb 11 107
              Kerry Collins 11 81
              Ben Roethlisberger 10 82
              Steve McNair 10 100
              Jim Everett 10 87
              Eli Manning 9 85
              Carson Palmer 9 79
              Jeff George 9 65
              Alex Smith 8 34
              Philip Rivers 8 95
              Michael Vick 8 88
              Trent Dilfer 7 51
              Chris Miller 7 52
              Aaron Rodgers 6 81
              Joey Harrington 6 30
              David Carr 5 44
              Chad Pennington 5 55
              Daunte Culpepper 5 86
              Rick Mirer 5 32
              Jason Campbell 4 45
              Kyle Boller 4 17
              Byron Leftwich 3 33
              Rex Grossman 3 23
              Tim Couch 3 30
              Heath Shuler 2 6
              Tommy Maddox 2 23
              J.P. Losman 1 19
              Akili Smith 1 1
              Cade McNown 1 7
              Ryan Leaf 1 1
              David Klingler 1 10
              Chuck Long 1 11
              Dan McGwire 0 2
              Todd Marinovich 0 3
              Andre Ware 0 5
              Kelly Stouffer 0 2

              It kind of looks like the threshold needs to be 5+ seasons as starter, and 50+ career AV, although that would define Alex Smith as a “bust” which I don’t think is accurate.

              It might be interesting to see if highly-drafted QB’s get more “second chances” to prove themselves than other positions. How many Rick Mirers or Trent Dilfers are there at the Linebacker position?

              Reply
              • Robin April 30, 2014, 4:30 pm

                I understand that you were spit balling and I agree that 1) there is value in a backup QB, and 2) high pick QBs almost certainly are more likely to get second chances that most other positions. That said, Trent Dilfer is much further above Rick Mirer than I think this list suggests.

                Your “St” stat seems wonky too. I assume it means years starting? Mirer was benched 3/4 into his 3rd year as a starter and was a backup ever afterward (though for a twice for extended stints). Even totaling his career starts and dividing by 16 (equivalent years starting) only gets 4.25, so maybe “St” is something else?

                Reply
                • Richie April 30, 2014, 4:41 pm

                  It’s straight from PFR.
                  St — Number of years as primary starter for his team at his position

                  I assume it means 8+ games listed as starter.

                  Reply
                  • Robin April 30, 2014, 4:50 pm

                    Fair enough, though that makes much more sense to apply to most other positions than QB. Just another PFR stat I find misleading. The “starter” label is also pretty antiquated in this day and age, especially on the defensive side… but that digression would take us way off base. Thanks for clarifying the stat for me.

                    Cheers.

                    Reply
  • Richie April 29, 2014, 3:36 pm

    I also can’t imagine that we’re ever going to get to a place where there are only a few busts in every first round.

    I think there will always be an unpredictable psychological component that can never be predicted, or “fixed”. But, I have no idea how often it happens.

    I think back to when I was 21 or 22. I was (and still can be) pretty lazy, and made bad decisions. If somebody gave me a guaranteed $10M contract, I am not confident that I would have worked my ass off to earn that money. I think of Jamarcus Russell as a guy who may have had this problem.

    Likewise, there are the guys who felt disrespected that they fell in the draft, and worked extra hard to prove people wrong. I think Aaron Rodgers and Maurice Jones-Drew are guys who may fall into this camp. (I think I’ve heard these sorts of comments from them, but I could be mistaken.)

    Reply
  • Ed May 8, 2014, 2:10 pm

    In your second chart, I’d be willing to bet that your error bars due to yearly variance would eliminate any statistically significant trending that seems to exist. At least since the 90s, when scouting/drafting theory really started to heat up.

    Speaking of the chart, I still don’t quite get what you did. I understand you calculating a players value from a draft year, but then what are you doing with that number and your draft pick calculator values? And how is that giving you a fraction that is about .5? Seriously dude, add some graph axis titles in your posts. Thanks.

    Reply
  • Steve August 11, 2014, 7:24 pm

    Chase,

    Kevin Meers used your CAV to make a new draft chart over at Harvard Sports Analytics as I’m sure you know. He added a second article using values from that chart to point to over and under achievers. In that second article, he makes mention of how the GB Packers had four of the ten highest over achievers. They were drafted in 1996 through 2000 with only the 97 draft not producing a boomer.
    Then in 2001, although not mentioned by Kevin, there is a top ten bust.

    Is it any surprise to see ownership moved the coach in to the added position of GM?

    Reply

Leave a Comment