≡ Menu

Are certain teams better at drafting than others?

A.J. Smith knows a perfect throwing motion when he sees one.

A.J. Smith knows a perfect throwing motion when he sees one.

On the surface, this seems like an obvious question. Yes, certain teams are good at drafting and certain teams are bad at drafting. It’s easy to think of the Matt Millen Lions or the Raiders at the end of the Al Davis era as terrible drafting teams. With the benefit of hindsight, they were terrible at making draft selections. But is it easy to identify going forward which teams will be good or bad in the draft?

Think back to April 2007. A.J. Smith and Bill Polian were widely considered the two best draft minds in the NFL. At the time, Smith’s last three draft classes had been outstanding. He added Philip Rivers (via the Eli Manning trade), Igor Olshansky, Nick Hardwick, Shaun Phillips, Michael Turner, and Nate Kaeding in 2004, and followed that up with Shawne Merriman, Luis Castillo, Vincent Jackson, and Darren Sproles in 2005 and another strong class (Antonio Cromartie, Marcus McNeill, and Jeromey Clary) in 2006.

The defending Super Bowl champions were the Indianapolis Colts, a team that Polian had built from scratch. From 1996 to 2003, Polian’s eight first round picks were spent on Marvin Harrison, Tarik Glenn, Peyton Manning, Edgerrin James, Rob Morris, Reggie Wayne, Dwight Freeney, and Dallas Clark. His most recent first round pick was Joseph Addai, who had 1400 yards from scrimmage as a rookie and 143 yards in the Super Bowl victory over Chicago.

Fast forward six years later, and both Smith and Polian have been fired. From 2007 to 2011, Polian’s first round picks brought Anthony Gonzalez, Donald Brown, Jerry Hughes, and Anthony Castonzo to town. The Colts were without a first round pick in 2008, because Polian packaged his 2007 fourth rounder (Dashon Goldson) and his 2008 first rounder to the 49ers for the the 42nd pick in the draft, which was spent on …. Tony Ugoh. Indianapolis’ second- and third- round picks were even worse over that five year stretch: in addition to Ugoh, the Colts drafted Mike Pollak, Pat Angerer, Fili Moala, Ben Ijalana, Dante Hughes, Philip Wheeler, Jerraud Powers, Drake Nevis, Kevin Thomas, and Quinn Pitcock in those rounds. After Peyton Manning missed the entire 2011 season, and Polian’s combination of Kerry Collins, Curtis Painter, and Dan Orlovsky predictably failed, Polian was fired. He went from scouting genius to draft failure overnight.

Meanwhile, A.J. Smith went from masterful talent evaluator to the most hated man in San Diego in roughly the same amount of time. The Chargers’ first round picks starting in 2007: Craig Davis, Antoine Cason, Larry English, Ryan Mathews, Corey Liuget, and Melvin Ingram. After fielding perhaps the most talented roster in the NFL in 2006, San Diego failed to restock as veterans moved on, and Smith left Rivers with an anemic supporting cast. The Chargers drafted 40 players from 2007 to 2012, and only one — Eric Weddle — has made a Pro Bowl.

There are teams that are good at drafting just like there are players who are clutch and captains who can correctly call the coin toss. The problem is, we recognize that someone who correctly calls the coin toss is just lucky while we label “good drafters” as oracles capable of separating the draft wheat from the chaff. Being a great drafter is simply another example of What Are the Odds of That:

You’ve undoubtedly heard of Wyatt Earp, who is famous precisely because he survived a large number of duels. What are the odds of that? Well, it depends on your perspective. The odds that one person would survive a large number of duels? Given enough time, it becomes a statistical certainty that someone would do just that. Think back to the famous Warren Buffett debate on the efficient market hypothesis. Suppose that 225 million Americans partake in a single elimination national coin-flipping contest, with one coin flip per day. After 20 days, we would expect 215 people to successfully call their coin flips 20 times out of 20. But that doesn’t mean those 215 people are any better at calling coins than you or I am. The Wyatt Earp Effect, the National Coin Flipping Example, and my Splits Happen post all illustrate the same principle. Asking “what are the odds of that?” is often meaningless in retrospect.

It would have been meaningless six years ago to ask what the odds were that A.J. Smith was actually an average (or worse) drafter who just happened to have three excellent drafts in a row. The odds would be very low, but that doesn’t answer the question we really want to answer. Smith was excellent in the draft right up until the moment he stopped being excellent and instead became terrible. If you think of the draft as an efficient market — and I believe it is — then there is no reason to think any one team/GM/scouting department is going to be better than another.

All of that is theory, of course. What about the data? That’s pretty easy. In this post, I graded every draft class from 2000 to 2007. I credited each team with the marginal Approximate Value generated by each player selected subject to two caveats: one, I limited the production of each player to only his AV in his first five years, and two, only production that came with the team that drafted him was included. Finally, I then assigned each team credit for the amount of marginal Approximate Value generated relative to what the expectation was (based solely on draft pick). This way, teams with higher picks did not have an advantage.

Using those same methods, I went back and graded every draft since 1970 (Friend-of-the-Program Danny Tuccitto just did something similar.) That enabled me to compare over 1000 pairs of team seasons and determine how likely teams were to sustain their good or bad draft fortune in consecutive years. As it turns out, the correlation coefficient between a team’s draft grade in one year and draft grade in the next year was just 0.07. This means that there is essentially no relationship between how well a team does relative to expectation in the draft this year to how they did the prior year.

The 1975 Bears had one of the most amazing draft classes in history. With the 4th pick in the draft, they selected Walter Payton, but the depth of the class is the real story. The Bears used a 12th round pick on Doug Plank, who was a starter for seven years. With a 17th round pick, Chicago added Roland Harper, Payton’s starting fullback for five years. The second round pick was used on Mike Hartenstine, a defensive end who played for 12 years with the Bears, most of them as a starter (and he started 8 games on the ’85 team). The fourth round pick was Virgil Livers, a starting corner and returner for half a decade. Two sixth round picks yielded Bob Avellini (started 50 games at quarterback) and Tom Hicks (four-year starter at middle linebacker), while fifth-round guard Revie Sorey started 77 games for Chicago. On a pick-for-pick basis, I don’t know if a team has ever had a higher batting average in a draft (Chicago did not have a third round pick that year), and the Bears received tremendous value from their late round picks.

But guess what? In 1976 and 1977, the Bears were below average. They were the proverbial coin that went “heads” on 9 out of 10 tosses but then came up heads on just 9 of the next 20 flips.

What about the famous Steelers draft class of 1974? With Jack Lambert, Lynn Swann, Mike Webster and John Stallworth, that was obviously an excellent draft. But the season before, their second round pick, Ken Phares, never played a game (to be fair, this was injury-related), while their third and fourth round picks Roger Bernhardt and Gail Clark provided almost no value. Their first round pick wasn’t a bust (J.T. Thomas), but the class as a whole came in far below expectation. And in 1975, it was even worse: not one of their 21 picks ever started a game for the Steelers. Pittsburgh had picks in each of their first six rounds: those players combined to play in just 24 games for Pittsburgh. The Steelers surrounded the greatest draft of all time with two duds.

Some of the low value provided by the ’73 and ’75 classes were no doubt in part because the Steelers were so talented that roster spots weren’t plentiful. Admittedly that’s a small problem, and I don’t know what you can do about that. In general, though, there appears to be almost no relationship between draft years. I looked at the top 50 draft classes from 1970 to 2007. On average, those classes produced 133 points of marginal AV against 56 points of marginal AV, meaning those teams outproduced expectation by 77 points. In the following year, those 50 teams, on average, outproduced expectations by only 4 points of AV. In the prior year, they outproduced by just 3 points of AV.

In retrospect, there are good and bad drafting teams. But in retrospect, there are people who make lots of money picking the stock market and flipping houses, and there are people who lose just as much money on the same endeavors. The true question of whether something is skill or luck is if it is repeatable. I’m not saying the door is closed on the issue, but there appears to be no real evidence that picking winners in the draft is a repeatable skill. If you have any other suggestions for how to measure whether “picking winners” in the draft is a skill, I have the data, so leave a note in the comments. (I’ll also note that I’ve conflated the issues of “drafting well” and “developing players” here, although I don’t know if there’s any way to untangle them.)

That’s not a knock on NFL GMs. In fact, it’s a pat on their collective backs. It’s not that all GMs are stupid or blindly lucky, it’s that scouting is so good that there are no “steals” left to find. If you threw 31 random fans and a GM into a draft, I’m sure that one GM would do very well most years. But graded against 31 other GMs who are focused on the exact same goal, consistently beating the pack is an unrealistic expectation.

If you want to take a look at my draft class grades from 1970 to 2007, they are all available here.

[Update: Some good comments from Brian Burke this morning on the same issue.]

{ 46 comments }
  • Danny Tuccitto April 22, 2013, 12:15 am

    For reasons I won’t get into here (but maybe I’ll post an XP on FO about it sometime in the future), not sure that I trust the 0.07 correlation very much. Not necessarily saying it’s too low, just think this particular application is only giving you part of the true picture. That said, I do think your larger point is right, and can be generally expressed as, “Just because someone exhibits behavior X doesn’t mean they have the attribute of being an X-er, especially when there’s a lot of random variation in X.” The argument surrounding Kobe’s “clutchness” comes to mind.

    Reply
    • Chase Stuart April 22, 2013, 9:17 am

      I won’t argue that the CC is the end-all, be-all statistic here. But if “drafting well or poorly” was a sticky metric, the CC would be a lot higher than 0.07.

      Reply
  • Sunrise089 April 22, 2013, 1:23 am

    Loving the multi-post days recently!

    I feel you stole a little of your own thunder at the end of the post with the ‘all 32 GMs are doing a great job’ stuff, but if you we’re trying to run a win maximizing football team without unlimited funds what would you want to pay your GM, and what sort of tasks would you ask them to do?

    IMHO the evidence you gave doesn’t lead me to believe all GMs have an equal level of great skill.

    Reply
    • Chase Stuart April 22, 2013, 9:19 am

      Thanks Sunrise…. a lot of posts have been comments-free lately, so I’m glad to know people are still reading them :)

      Do you remember Doug’s old post about quitting scouting? http://www.pro-football-reference.com/blog/?p=295

      Reply
  • Alex April 22, 2013, 1:39 am

    Very nice. I’d like to know if any GMs are better at free agency/giving out contracts to their own as well. If not then it really is just a coin flip role

    Reply
    • Chase Stuart April 22, 2013, 9:21 am

      My guess there would be ‘no’ as well, as you would think that’s just as efficient a market. I’m sure there are small places where you can make gains on the marginals (maybe teams generally underpay position X), but you wouldn’t find that in a league-wide study over dozens of years.

      Ryan Grigson appears to be a good test case: the reigning GM of the Year made a lot of questionable decisions this offseason.

      Reply
  • Scott Tanner April 22, 2013, 2:15 am

    This seems…so counter-intuitive, and yet it’s hard to find any obvious reason why it’s wrong. I think you could argue that some GM’s are better at the “draft process” so to speak; they understand how to move picks around and where value is. The Jaguars taking Tyson Alualu like 10th or 11th overall jumps to mind. He turned out to be an average player, but even if he was great it still would have been dumb to take him there when he wasn’t leaving the board for another 20-30 picks. So while it still may be a crapshoot, I think some teams understand how to give themselves more shots at hitting a pick. Although if this were actually important, one would think the data would bear it out, which it doesn’t sound like it does.

    In light of your findings, what does it make you think of someone like Belichik’s drafting style? Ignoring the sub-par results (especially on receivers and secondary) and focusing on process instead, does it make sense to do what he does? Maybe it’s hard to say in a vacuum and it really just comes down to getting lucky with a given player. That just seems very frustrating and unsatisfying.

    I guess my question would then be this: if you were an NFL owner, would you not put any stock into your GM’s drafting prowess when evaluating their employment?

    Reply
    • Chase Stuart April 22, 2013, 9:39 am

      To your final question, the answer is “very little.”

      Vernon Gholston was an enormous draft bust, but all the reports indicated that if the Jets passed on him at 6, the Patriots were going to take him at 7. It’s possible those reports were false, but my guess is that Belichick also loved Gholston, and had he selected him, he also would have been a bust in NE. Sometimes, you just get lucky (or unlucky).

      Let’s say Russell Wilson retired tomorrow. Would you have faith in the Seattle front office to figure out which quarterback to draft this year?

      To be fair, there are great GMs, and poor GMs. It’s possible that Seattle and other teams have found some draft inefficiencies, but the likelihood is that (1) the more likely answer is that good drafting teams are lucky, and (2) to the extent they found a draft inefficiency, the window is likely to close within a year or two on that particular trait.

      Reply
  • Nate Dunlevy April 22, 2013, 6:48 am

    Link to the draft grades is down.

    The problem with using AV as the measure here is that the draft class after a fantastic draft class has a lesser chance to make an impact on the whole.

    AV is heavily influenced by playing time, and one great draft class can dramatically reduce opportunities for playing time for the class that follows it. It also forces GMs into a box. If you draft a great QB and running back in one year, the next year, that’s two fewer positions that you can choose from if you want to have maximum impact with your picks.

    Reply
    • Chase Stuart April 22, 2013, 8:41 am

      Thanks for the heads up – the link is now correct.

      I agree that AV does have a small drawback there. Can you think of any ways around this issue?

      Keep in mind that if teams draft well, they usually end up with good records. Good records mean lower picks, which means lower expected AV. So I’m not sure the effect is huge.

      Reply
  • Isaac Shalev April 22, 2013, 12:34 pm

    Are there other dimensions to being good at the draft besides choosing players who turn out to be good? Do some GMs exhibit less variance in future player performance (ie their busts tend to be smaller)? How about maximizing potential value through good trading? Or spending high value picks on high value positions? It seems to me that you’re largely correct that there is an efficient information market (nowadays) around potential draftees, but there is not an efficient market around selecting players!

    Reply
  • Brian Solomon April 22, 2013, 12:42 pm

    What if you narrowed the sample to only teams that kept the same front office year over year? Since 5-10 teams are working with new coaches and/or GMs every year, one would expect much lower correlation for those teams’ drafts. The question is whether one front office staff is able to replicate draft outcomes — not whether a franchise can replicate the outcomes of a previous front office staff.

    Reply
    • Chase Stuart April 22, 2013, 12:45 pm

      Yes, this would be ideal. Unfortunately, we don’t have historical front office information kept for very long. In addition, it’s tough on many teams to figure out who is making the calls (owner/GM/HC).

      Reply
      • Brian Solomon April 22, 2013, 12:58 pm

        You certainly would introduce new complicating factors — most notably not knowing who exactly has final say. But it seems to me that without doing so the conclusions can only go so far.

        Reply
  • Scott Tanner April 22, 2013, 12:57 pm

    In terms of disentangling drafting from and developing, there may be a way to tease at least some of that out. So far as I can tell, AV is just based on the career of the player. It would be interesting to see the value of players drafted just while they’re playing for the team that drafted them. If we see large gaps over time between total AV and AV while playing for the team that drafted, we might be able to make some inferences. Obviously if the AV for the team is much lower, it might suggest players are blossoming elsewhere and the players drafted were a poor scheme or personality fit for a given franchise. If the opposite is true, the team might be especially good at getting players they know they can develop and get the most out of. Don’t know if that’s possible, but just it’s just a thought that occurred to me.

    Reply
  • mrh April 22, 2013, 1:14 pm

    Building on Brian Solomon’s comment. “Teams” are not a unified set of decision-makers and GMs are not dictators. Or perhaps when they are, they fail. Take the Chargers – there is an obvious correlation between Smith’s change in drafting ability/luck after he fired Schottenheimer. Maybe that’s player development, maybe it’s a changed decision-making process, but it’s hard to believe that Marty (or Norv) didn’t have some say in the draft and that had an effect on the outcome. Similarly, the departure of Dungy from Indy may have played a role. And that’s ignoring all the other people in every drafting decision. Grading drafts by GM may be like assigning wins and losses to QBs.

    Second, even if a franchise has a constant draft-decision team, that “team” will age and stick to out-moded team-building strategies even as the league evolves. It’s pretty clear Don Shula wasn’t as good a coach in 1995 as he was in 1970 or that Tom Landry wasn’t as good in 1987 as in 1975. Gibbs I vs. Gibbs II. And on and on. It may be that a “team” one or two good ideas and then runs out of them while the league adjusts – I think this echos the concluding comment in the post.

    Third, I’m not sure there is any use in comparing drafting in the ’70s to drafting in the 21st century. The number of rounds are far different and free agency and salary caps have completely changed the draft dynamic. The creation of the draft combine also changed things. A year-to-year comparison gets around this to some extent but to some extent, every year is a slightly different environment. The assumption in this analysis has to be that every year is just like the previous year but that assumption may not be valid.

    Fourth, Bob Avellini is an example of how bad the Bears were at drafting, not how good they were. There is some underlying flaw in a methodology that gives the Bears points for that pick.

    Despite all those criticisms, I think you’re right there is a lot more luck and randomness in drafting than is generally acknowledged. But I think there is skill involved and that some “teams” are better. But each draft may be too small a sample to reveal that (and each “team” is together for too short a time) in this kind of analysis. In baseball a .600 team is almost certainly better than a .400 team. But you may not see that in a short series or 10-game span. The gap between good NFL drafters may be closer to that spread (.600 vs. .400) than between a 12-4 and 4-12 spread that we think of in football contexts.

    Reply
    • Chase Stuart April 22, 2013, 1:23 pm

      Thanks mrh.

      Do you have any suggestions on another way to measure things? I do think using over 1,000 pairs of seasons tends to eliminate a lot of the smaller issues. That’s a very big sample size.

      Reply
      • mrh April 22, 2013, 3:13 pm

        Let’s say I have a skill – I can flip a coin so that it comes up heads 60% of the time. I run 1000 sets of trials. In each set I flip a coin 10-20 times at first, gradually diminishing to 5-10 times at the end of the trials (declining number of draft picks). In each set I record the results but have a measurement error of 20% (AV is only approximate, pick the appropriate error rate and error distribution). Given the small number of flips in each set, the measurement error, and the rather small advantage my skill brings over a random coin-flipper, would a pairwise comparison of each set to the next be able to identify my skill?

        I don’t have the math to answer my own question. But now complicate it further by dividing the sets into groups of 32 (franchises), with a varying number of sets in each group (differing lengths of franchise existence) and changing coin flippers every 3 to 20 sets within each group (assumed amount of time a GM holds a job) but without knowing the skill level of each flipper (some may be better or worse than random, some may only be as good as random) or identifying when the flipper has changed. Is 1000 really a big enough sample?

        Again, I don’t know.

        Reply
        • Chase Stuart April 22, 2013, 3:21 pm

          I wouldn’t argue if someone was to say with such a small number of draft picks by each team, it’s impossible to really tell whether a GM is good or bad.

          The Seahawks did a great job in rounds 1-2-3 in the draft last year, but would anyone be surprised if they had a bad draft this year?

          Reply
  • Chase Stuart April 22, 2013, 1:25 pm

    To put it another way, suppose you think drafting well (or poorly) *is* a skill. How would you go about proving that?

    Reply
    • mrh April 22, 2013, 2:49 pm

      A fair question. I don’t know off the top of my head. But just as some things we think are skill (stock-picking) may be random, some things we think are random actually can be skills, or at least have their randomness affected by skill.

      For example, blackjack. Card-counting can affect what appears to be random (and was thought to be random for many years). A card counter will still lose many hands; I’m not sure the percentages but the card counter’s edge is not normally huge. But many of the player’s losses – and wins – would be due to randomness. How many paired observations (or series of observations) would it take to prove that a blackjack player was counting cards? Or good at it?

      Now take football. Besides the human element, there are numerous other variables beyond a card deck and numerous skills involved beyond counting cards. Mike Shanahan appears to be good at picking RBs and bad at picking CBs. Assuming for a moment that his picking of players at every other position is random, how would you identify that skill or lack thereof? That is, how would you prove a hypothesis that Shanahan is good at picking RBs and bad at picking CBs? During his time in Denver and Washington, assuming he’s the final decision-maker there unlike his brief stint working for Al Davis, he’s had 138 picks, 23 spent on DBs (22 plus 1 CB) and 19 on RBs (18 plus 1 FB). How would you separate the signal on those 42 picks, and remember we’re testing a hypothesis that he’s good at one thing and bad at another, from the noise of the 96 other picks? Especially since we haven’t even separated the DBs into safeties and CBs and the true running backs from the guys picked to be FBs or kick.punt returners? I’m pretty sure a pair-wise comparison of year-to-year success drafting wouldn’t do it. And throw into the mix that the QBs affecting the AV of the RBs range from John Elway to John Beck.

      Reply
      • James April 22, 2013, 4:15 pm

        Then again, if Shanahan is better than average at drafting RBs and average at all other positions, how much value is he providing to the team? RBs only represent ~6% of the roster/starters, so while there may be real skill there it’s so small that it’s washed away by the noise, and if it’s washed away by the noise how useful is that skill?

        Reply
      • James April 22, 2013, 4:19 pm

        To expand on my post: compare Shanahan’s drafting ability to a kicker that is better than average at 50+ yard field goals. If the kicker only attempts 4-7 50+ yard field goals each year (1) how are we going to detect that skill from randomness, and (2) does it really matter if the impact is so small it’s undetectable?

        Even if you could identify the skill correctly, its impact is so little that having that knowledge is essentially meaningless.

        Reply
        • Chase Stuart April 22, 2013, 4:43 pm

          What would be nice is if we had every GM’s draft board. From there, we could probably make some progress. But looking at just seven picks each year makes it difficult, especially when most of those 7 picks are considered reasonable.

          Reply
    • Brian April 23, 2013, 1:17 am

      Maybe another way to “prove” skill (both good and bad skill) is to run a no-skill bootstrap simulation and compare to the real data. Basically you would take the observed standard deviations of AV at each position, and simulate x years of this y times, thereby obtaining a nice distribution of what an all-luck world would look like. Then you would compare to what the actual distribution looks liked, and anyone vastly different on each tail would be called skilled.

      Note that this is what Eugene Fama and Ken French did to show that they is very little positive alpha in the mutual fund industry.

      Reply
  • Tim Truemper April 22, 2013, 1:51 pm

    Call Nate Silver.

    Reply
  • Jeffrey April 22, 2013, 5:42 pm

    A quick hypothetical, if you don’t mind:

    Let’s say Franchise X is led by Peyton Manning, whose QB ability is so incredible that it almost doesn’t matter who he is playing with. He is so feared as a passer that opposing defenses rarely blitz, resulting in the offensive line (collectively and/or individually) looking better than they really are. His accuracy and precision is so good that his receivers get open even when they aren’t open. His ability to stretch the field vertically and horizontally forces defenses to play pass first at all times, resulting in open holes for running backs. In the end, Manning’s oversees a record-setting offense; one that leads to lots of winning. Because the offense is so dominant, Franchise X almost always plays with a lead, which makes the defense seem better than it is, too.

    Now let’s say Franchise Y is led by Tim Couch, whose QB ability is just the opposite. He isn’t accurate, he isn’t mobile, defenses don’t fear him, and so on. Instead of making teammates look better than they really are, he makes them worse. Because he hangs onto the ball too long in the pocket, and has a high sack rate, the linemen are perceived to be below average pass protectors. Because teams stack the box against the run, ball carriers can’t find lanes. Receivers are open, maybe, but the ball never finds them or is too late getting there. The defense, in turn, never plays with a lead.

    Let’s make two assumptions:

    1. Franchise X has a winning season, and Franchise Y has a losing season.
    2. Outside of the QBs, both teams have the exact same players.

    It is very likely that Franchise X will try to keep its roster intact, while Franchise Y will be making several roster moves.

    This despite having the exact same players.

    Reply
    • Scott Kacsmar April 23, 2013, 3:51 pm

      Good comment. We can apply some real-life numbers to this.

      Based on AV in the common draft era:

      *Reggie Wayne is a top 5 WR
      *Dallas Clark is a top 15 TE
      *Ryan Diem is a top 40 OT

      The Colts drafted all three of those players in 2001-03. I’m pretty sure if the Colts had an average QB instead of Peyton Manning, those three AV facts would not be true today.

      Finding the franchise QB makes life so much easier on a GM, inflating his ability to find players because of how the QB can cover up for weaknesses.

      Reply
    • Fred March 11, 2014, 10:34 am

      There is an issue as well with backup quarterbacks. The Patriots used a 3rd round pick on Ryan Mallett, who has an AV of 0. Mallett has a value to the Patriots as an insurance policy even if the value can’t be “approximated”. So was that a good pick or a bad pick? I don’t know.

      Reply
  • Steve April 22, 2013, 5:57 pm

    I would be really interested to see if a similar methodology to evaluating portfolio managers could be of use here. In Michael Mauboussin’s book The Success Equation he mentions that his son and a colleague have done some research on the occurrence of streaks in mutual funds beating the market. Although overall serial correlations are similarly low, the number of streaks is far higher than would be expected by chance. The article can be found here (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1664031). A quick analysis would be to look at team-based streaks, but if someone has more time it would be really interesting to look at streaks by different team decision makers to account for the lack of continuity within a team around regime changes.

    Given a similar information environment (basically equal between teams) I would expect the luck/skill breakdown to be similar to investing with a lot of luck but a still-discernable skill component. The “streak” analysis would also probably show more than the null, completely random, model. As Mauboussin says in an interview “you look at streaks, not just in investing but in any endeavor, almost by definition they combine skill and luck. You have to have above-average skill and above-average luck to have a streak. If you look at it in the realm of sports, all the streaks are held by the most skillful players, although not all skillful players have streaks.”

    Reply
    • Steve April 23, 2013, 12:13 am

      So… I had some free time. Here is my version of the “quick” method I described above: http://www.sportsplusnumbers.com/2013/04/luck-vs-skill-nfl-draft-performance.html

      Enjoy

      Reply
      • Brian April 23, 2013, 1:21 am

        Are streaks really an appropriate way to look at it though if they hold no predictive power? What I am saying is, even if streaks occur at a rate higher than chance, if they can’t be identified ex-ante than what is the point? Note that streaks have been studied in baseball and they have been shown to have almost no predictive power, which has led most sabermetricians to conclude that hot and cold streaks are not worth considering much.

        Reply
  • JeremyDe April 23, 2013, 12:13 pm

    Saw a Washington Post blog talking about a recent ESPN the Magazine article rating NFL franchises for their draft performance over the past 20 years. I was amused by the following quote “And so, ESPN the Magazine’s draft preview edition, which ranks NFL teams by a variety of draft categories. The currency used here is “AV,” a player’s “Approximate Value,” as calculated by Pro Football Reference, which essentially divides up the amount a unit was better or worse than the league average into player-specific component.”

    Haven’t read the article myself yet, and I’m not sure if the article or their methodology is worth a read, but considering this topic and the number of comments it generated, I thought it was worth a mention in case anyone wants to hunt it down to check it out.

    There’s not much to the blog post, but it is at the following address if anyone is interested.
    http://www.washingtonpost.com/blogs/dc-sports-bog/wp/2013/04/23/redskins-nfls-fifth-worst-drafting-team-says-espn-the-magazine/

    Reply
  • Richie April 29, 2013, 4:48 pm

    I was thinking – do you have good data on draft pick trades?

    I would be curious to see if teams who trade up during the draft have a higher percentage of successful players? For instance – the Rams traded up to get Tavon Austin. Apparently they had their eye on him and really thought he would be a good player. They made more of an effort to acquire him than teams that just take the best available talent. Does this indicate higher success rates?

    Reply
  • Eric Barry April 30, 2013, 1:03 pm

    It seems to me that if we are measuring drafting ability, we should focus on the first few rounds. Teams expend a lot more energy on say the first three rounds, and expect their starters in general to come from the first three rounds. The success of later picks probably has a higher correlation to team stability and quality than those of the higher picks. As a Redskins fan I’ve watched absolutely marginal talents earn a fair bit of Career AV only because the team was so thin. For instance Reed Doughty has 23 career AV as a sixth round pick, which is a nice return for a sixth rounder. But it’s more a measure of the fact that the team has been devoid of healthy safety talent for his whole career, so the fact that he has stuck is not very impressive. In addition, only picks in the first three rounds will as a matter of course be given an honest chance to show what they can do and to fail some before they succeed. Finally, given that team success (particularly before the current labor agreement) is highly leveraged to the success of early picks, they should count more, and arguably be leveraged by rookie salary.

    My second tweak would be not to discard player achievements not made with the drafting team. Failing to maximize the value of talent is a different skill than drafting. I’d be inclined to go with four highest AV seasons over a whole career if we’re merely trying to tell how talented the players drafted were. That also allows some correction for injury, which seems largely unpredictable and arbitrary.

    My third tweak would be to add an adjustment for position. Teams often draft on need, and while that may not be the best long term strategy, it is in reality the best short term strategy for many front offices. Team goes into draft knowing they need, say, a safety. How did they do in ranking the ones available in their slot? Pennington is a lot better draft pick in a year with no other reliable QB starters than Philip Rivers in a draft when all the high end picks saw success.

    Another avenue of interest would looking at team variance from consensus of media draft boards and from day after grades. It perhaps shows more talent to go against consensus and succeed than to choose who everyone thinks is best. It would also be useful to know if teams would be better just following the wisdom of crowds.

    Reply
    • Chase Stuart April 30, 2013, 1:12 pm

      Good points, Eric.

      Reply
  • Steve May 10, 2013, 4:15 pm

    I came at this again from a different perspective (individual picks as opposed to entire drafts) and tried to see if a team making a “good” pick – one that exceeds expected value regardless of whether it’s with the drafting team – is more likely to make a good pick in their subsequent selection. The short version of the results is no, they are not. Read on for the long version: http://www.sportsplusnumbers.com/2013/05/luck-vs-skill-in-nfl-draft.html

    Reply

Leave a Comment