Previous post:

Drafting Diamonds in the Rough

by on May 7, 2014

in

Guest blogger Andrew Healy, an economics professor at Loyola Marymount University, is back and the author of today’s post. As a reminder, there a tag at the site where you can find all of his great work.

Small school defender takes down big school quarterback.

Asante Samuel. Jahri Evans. Robert Mathis. These three players share something in common that offers a hint to finding steals in the middle rounds of the draft. All three eventually made Pro Bowls. Each was drafted in Round 4 or later. And each played for a notable football powerhouse in college: Central Florida1, Bloomsburg, Alabama A&M.

The success of these smaller college players relative to their marquee school competitors turns out to be a much more general phenomenon. In the middle rounds of the NFL draft, players from outside the traditional power conferences have been more than twice as likely to eventually make the Pro Bowl as players from the most famous programs. On defense, small school players have been even more likely to make the Pro Bowl than their major school counterparts.

Let’s use the 2003 draft as an example. Only 5% (6 out of 116) of the major college players selected in round 4 or later eventually made it to a Pro Bowl. At the same time, 12% (6 out of 50) of small college players would eventually be selected for Hawaii. At the very least, if you were watching the draft and wanting to know what the chances are that your team drafted a future star, those chances increased in the middle of the draft when your team picked a player from a school like Bloomsburg than when it picked another player from the SEC.

In fact, it’s hard to think of anything else that can match the impact of simply picking small-school players as a way to find stars in the middle rounds. The data suggest that this logic has even applied at the top of the draft for comparisons such as those between Buffalo’s Khalil Mack and South Carolina’s Jadeveon Clowney. But the big gains from focusing on smaller football schools have come from finding the gems that the draft buzz mostly bypasses. Consistently, general managers have wasted picks on players from major conferences, missing chances to find difference-makers―particularly on defense―from schools such as Northern Colorado and Idaho State.2

The Data

I look at all players drafted from 1998-2007, stopping at the later year to give players time to make a Pro Bowl. The measure of excellence is making a Pro Bowl, but I’ll also look at All-Pro selections. I ignore players listed at special teams positions (P, K, and KR), although it’s possible you could make a Pro Bowl as a special teamer after being drafted at an offensive or defensive position. I also did not include fullbacks because it became so easy to make the Pro Bowl at that spot.

Major conferences are defined according to the traditional BCS definitions: Big East/American, Big 10, Big 12, Pac 10, SEC, and ACC. Notre Dame is also included with these bigger (in terms of football) schools. A school such as Wake Forest gets defined as a big football school by this measure and it probably shouldn’t be, but adjustments from this definition would be judgment calls and so this simple rule seems best.

Note that almost none of the middle-round small-school Pro Bowlers during this time period come from schools such as Boise State that were big football schools at the time. The two possible exceptions are Brett Keisel of BYU (drafted in 2002) and Paul Soliai of Utah, who was drafted in 2007 before Utah joined the Pac-12.

Comparing Average Success Across Schools

Small school players get drafted later than big school players, so we need to control for draft position to get a fair comparison between them. Later, I’ll use regression to do that. Here, I’m just going to break down results according to ranges of draft position. The chance of making the Pro Bowl is much higher in the early parts of the draft, so I’ll break things down there according to selection number rather than just the round.

The table below looks at the first three rounds of the draft. Overall, the chances of drafting a Pro Bowler tend to be higher for small school players in the first three rounds. The small school samples are limited in the first round, but the share of small school players who make a Pro Bowl is higher throughout than for big schools. Out of all the rounds, the 2nd round is the only one where we see a small trend the other way.

Small schools
Big schools
Round# of selections% Pro Bowlers# of selections% Pro Bowlers
1 (Pick 1-10)771.4%9355.9%
1 (Pick 11-20)850.0%9141.8%
1 (Pick 21-32)1145.5%9929.3%
2 (Pick 33-48)2623.1%13124.4%
2 (Pick 49-64)3511.4%11215.2%
38310.8%2449.0%

The largest differences, and the clearest benefit from drafting players from smaller schools, come in the middle rounds. The table below shows the differences in rounds 4-7. In round 4 over the ten-year period, teams have been about three times more likely to draft a Pro Bowler when picking from a small school rather than a big one. 12.9% of small school draftees in Round 4 have made the Pro Bowl, compared to just 4.1% of big school players.

Small schools
Big schools
Round# of selections% Pro Bowlers# of selections% Pro Bowlers
49312.9%2474.1%
51088.3%2283.5%
61353.7%2203.6%
71572.6%2941.7%

In round 5, we see a similarly large difference. Round 5 players from small schools have been more than twice as likely as big school players to make a Pro Bowl. Altogether, across rounds 4 and 5, despite 475 non-special teams players being drafted from big schools, just 18 (3.8%) have made a Pro Bowl. On the other hand, out of just 201 players drafted in those rounds from small schools, 21 (10.5%) made a Pro Bowl. If you wanted to find a future star in rounds 4 or 5, you would have increased your chances by more than double by looking at the Northern Colorados and Alabama A&Ms of college football rather than the USCs and Alabamas.

[Chase note: It is at this point that I decided I needed to stop reading the article.  I trust Andrew, but found his claims too remarkable to just blindly accept. So I decided to open up my database to confirm. I removed punters and kickers but kept everyone else in the database.  To my amazement, the numbers not only seem legit, but perhaps even under-reported.  The average player selected from the 4th or 5th round from a Big School made 0.06 Pro Bowls, compared to 0.22 Pro Bowls for players from non-major schools!]

Regression Results: Controlling for Draft Position in a Flexible Way

To figure out the average bonus small school players offer compared to large school players, we can use linear regression to control for draft position. In the regressions, I predict whether a player became a Pro Bowler with a cubic polynomial in draft position and whether the player went to a major school. The regression results indicate that, looking across rounds and controlling for draft position, players from small schools are about 3 percentage points more likely to become Pro Bowlers. ((We get almost the same result if we include higher powers of the pick number. We also get similar results if we use a logit instead of a linear regression. The standard error for the estimate is in parentheses.))

All rounds ( N = 2427 (0.014)):

$ProBowl=f(Pick,Pick^2,Pick^3)+0.030*SmallSchool$

The three percentage point bump for small school players is a substantial boost. Across all rounds of the draft, about 11.8% of the main position players made a Pro Bowl. Compared to this baseline, teams increase their chances of drafting a Pro Bowler by about 20% by drafting a small school player.

We can see more of this pattern by breaking things down according to the early and later rounds. If we look at rounds 1-3, nothing statistically significant emerges. The point estimate follows the overall pattern, but the result is not clear, in part due to the relatively small number of small school players drafted in the first three rounds.

Rounds 1-3 (N = 947 (0.034)):

$ProBowl=f(Pick,Pick^2,Pick^3)+0.021*SmallSchool$

On the other hand, in rounds 4-7, we get a very clear impact of picking small school players, an effect that is even more striking given the much smaller share of players who make the Pro Bowl in those rounds compared to earlier ones.

Round 4-7 (N = 1480 (0.011)):

$ProBowl=f(Pick,Pick^2,Pick^3)+0.033*SmallSchool$

We see that, controlling for the selection, small school players are 3.3 percentage points more likely to make the Pro Bowl.3 This represents about a doubling of the chance that a major school player makes the Pro Bowl. Just 3.1% of major school players drafted in Rounds 4-7 at the main positions made the Pro Bowl. The model predicts that around 6.4% of small school players drafted in those same positions would have made the Pro Bowl.

All-Pro Appearances

So focusing on small school players offers a much better way to draft a future star according to Pro Bowl appearances. And it doesn’t look like this is just about Pro Bowls. Instead, it’s pretty clear that small school players perform better more generally than major school players, once we control for draft position, with these differences primarily driven by the middle rounds, particularly 4 and 5.

Small school players drafted in rounds 4-7 are also about twice as likely to appear on an All-Pro team as their major school counterparts. Controlling for draft position, small school players are about 1.3 percentage points more likely to make an All-Pro team, relative to a baseline where 1.5% of major school players made an All-Pro team.

All-Pro (N = 1480 (0.008)):

$All-Pro=f(Pick,Pick^2,Pick^3)+0.013*SmallSchool$

Particularly given the relatively small number of players who made an All-Pro team, we can look at this another way by considering the number of appearances a player made on an All-Pro team. Controlling for draft position, players drafted in the middle rounds from small schools have an average of .036 more All-Pro selections than major school players. The mean number of All-Pro selections for major school players is .022, so small school players are predicted to have more than twice the number of All-Pro selections as their major school counterparts. ((The small school players drafted in rounds 4-7 who made an All-Pro team are (with the number of appearances in parentheses): Adalius Thomas (2), Asante Samuel (3), Brandon Marshall (1), Cortland Finnegan (1), Jahri Evans (5), Jared Allen (4), Jerry Azumah (1), Lance Schulters (1), Matt Birk (2), Michael Turner (2), Robert Mathis (1), Terrence McGee (2), and Trent Cole (1). Of these, McGee made it as a special teams player. Amongst major college players drafted in rounds 4-7, Dante Hall and Leon Washington made All-Pro teams as special teamers during this time.))

Number of All-Pro Appearances (N = 1480 (0.015)):

$All-ProAppearances=f(Pick,Pick^2,Pick^3)+0.036*SmallSchool$

The Best Defense Comes from Small Schools

One other interesting pattern in the data is the offense/defense breakdown. All of the above effects are driven by the defense. If we look just at offense, there’s basically no difference between big and small schools, which mimics what Chase found using a different methodology last year.  However, there are large gaps for defensive players.

Take the regression from before for rounds 4-7. Now let’s break it down separately for offense and defense:

Round 4-7, Offense only (N = 749 (0.016)):

$ProBowl=f(Pick,Pick^2,Pick^3)+0.003*SmallSchool$

Round 4-7, Defense only (N = 731 (0.015))::

$ProBowl=f(Pick,Pick^2,Pick^3)+0.060*SmallSchool$

The last gap is pretty enormous. Even if we don’t control for the spot the player is selected―which works against small school players since they get drafted later―we see the huge differences between small and large school defensive players. Out of 499 defensive players drafted in rounds 4-7 from major conference schools between 1998 and 2007, 10 (2.0%) made the Pro Bowl. On the other hand, out of 231 small school players drafted in those same rounds, 18 (7.8%) made the Pro Bowl. The gap for all-pro appearances is similarly large. There were a total of 10 all-pro appearances for the 499 large-school defensive players (.020 per player) and 17 all-pro appearances for the 232 small-school players (.073 per player) drafted in rounds 4-7 during this period.

Even though we have fewer than half as many draftees to pick from compared to major school players, look at the starting 11 we can field from small school players mostly picked in round 4 or later, with two round 3 draftees to fill in a couple of holes:

Note that if you go back a few more years, you can substitute La’Roi Glover (5th round, 1996, San Diego St.) for Soliai and Rodney Harrison (5th round, 1994, Western Illinois) in at SS, sliding Bethea in for Rhodes at FS. That is a pretty sweet defense, all built on middle-to-late round picks from small schools.

Conclusion

The data show that picks in the middle rounds of the draft have been substantially more productive when spent on players from smaller schools. Despite picking major-school players more than twice as frequently, teams have found as many stars from the smaller schools. On defense, they have actually found substantially more stars from schools such as New Hampshire than ones such as LSU. A defensive player taken in round 4 or later has been almost four times more likely to eventually make a Pro Bowl when that player comes from a school outside the traditional power conferences. Stars such as Jared Allen, Asante Samuel, and Robert Mathis are part of a larger pattern. Teams have found those essential mid-round steals by drafting players from smaller schools.

Why has there been this opportunity to do better by picking small school players? One possibility is that there was less information out there about those players, a gap that would have been decreasing as film and televised college games have become ubiquitous. That explanation makes some sense since the benefit to smaller school players emerges in the middle rounds, long after the Brian Urlachers (New Mexico) and Joe Greenes (North Texas) who were impossible to miss had been selected. However, with the sample going from 1998-2007, this explanation seems unsatisfying since teams have had relatively easy access to information about any college player.

The explanation that I think could make more sense is some kind of risk aversion, kind of like the bias that leads to punts on fourth down. Maybe teams in the middle rounds, not seeing clear standouts, felt that it’s safer to pick the player from Alabama instead of the one from Idaho State. Even though it’s anything but safer, general managers can say to themselves that they’re getting a player who’s a known quantity due to the college program he comes from. Picking the major school player might even be the kind of move that’s harder to criticize, putting the general manager in a similar position to the coach facing 4th and 3 at midfield, where the best choice for the team may not be optimal for the decision maker. Whatever the reason, the bias towards major school players in the middle rounds has left available potential stars to the teams that have chosen players from overlooked schools.

However, this potential opportunity may already be gone. Since 2008, six defensive players have made Pro Bowls and were drafted after round three. All six were actually from major schools: Kam Chancellor (Virginia Tech) and Richard Sherman (Stanford) in Seattle, Geno Atkins (Georgia), Henry Melton (Texas), Alterraun Verner (UCLA), and Greg Hardy (Mississippi). Across offense and defense, it’s eight Pro Bowlers for large schools (adding Carl Nicks and Jordan Cameron) versus four for small schools (Alfred Morris, True Receiving Yards champ Antonio Brown, Josh Sitton, and Julius Thomas, and not counting Jerome Felton, who plays FB), about the same ratio as players drafted altogether. Still, the biggest stars here are clearly the big school players.

Even though we need more years of data on all the players in these drafts, it is possible that the previous trend has shifted. Assuming that’s right, why might that have happened? One possibility is that ever more schools are getting national media attention, meaning that small schools aren’t so small anymore.5    Another possibility that seems even more plausible to me is that the increasing information on high school players means that great players are now less likely to be at small schools in the first place. Even though there will always be some great players who end up at small schools (see Watt, J.J.), maybe Jared Allen would have been recruited more heavily if he played now. There may now be fewer diamonds in the rough than there used to be. That idea suggests there might have been even more diamonds in the rough if we look at earlier years. And that looks like it might be exactly the case. Just looking at rounds 4 and later in some of these earlier drafts is kind of incredible. In 1989, there were five (non-kicker) Pro Bowlers from small schools and only one from a large school. In 1990, there were nine small school Pro Bowlers (including HOFer Shannon Sharpe) compared to just four from major schools. In 1991, it was eight small school Pro Bowlers compared to just two major school players.6 All of this appears even though substantially more large school players are drafted in rounds 4-8. While the chance to find a small school steal was just on defense from 1998-2007, it seems like the opportunities may have been all over the field in earlier years.

1. In 2003, Central Florida went 3-9 in the MAC. While the Blake Bortles Knights may not be a football powerhouse, either, the 2013 UCF team that went 12-1 in the American Conference bears little resemblance to where the program was a decade ago. []
2. Bonus points for getting those players. Aaron Smith was a 4th round pick out of Northern Colorado in 1999 and Jared Allen was a 4th round pick out of Idaho State in 2004. []
3. The t-statistic is 3.04 and the p-value is .002. []
4. Rhodes has actually never made a Pro Bowl, but he was second-team All-Pro in 2006. He did not count in the players from small schools who have made a Pro Bowl. []
5. Another possibility is that NFL teams have changed their behavior. There has been almost no change over time, though, in the share of small school defensive players selected at certain points in the draft. []
6. Some of the late round diamonds in the rough may have become undrafted free agents in later years. For example, James Harrison (Kent State) and London Fletcher (John Carroll) are two small school UDFAs who made Pro Bowls. []

Kibbles May 7, 2014 at 12:47 am

Vincent Jackson also hails from Northern Colorado and has three pro bowls to his name, although the 2nd doesn’t qualify as a “middle round”. I mostly mention it because I was so excited that I actually knew which pro bowler from Northern Colorado you were referring to… and then so disappointed when I found out there was another, too.

Also, another possible explanation for the closing gap between big school and small school players in recent years is that the NFL draft is a relatively efficient market, and during the course of self-analysis the decision makers discovered their tendency to underrate small school prospects and began correcting for it.

Andrew Healy May 7, 2014 at 1:34 am

I think that makes a lot of sense. It was sort of my first guess, too, until I looked at the trends over time in picking small school vs. big school players. Not much seems to have changed, but I haven’t looked at this comprehensively. And that non-pattern is only suggestive on this point, too.

BTW, my favorite small school with multiple Pro Bowlers during the period: Bethune-Cookman (Rashean Mathis (1) and Nick Collins (3)).

Shattenjager May 11, 2014 at 12:46 pm

My mother went to Northern Colorado (which is called UNC here, confusing out-of-staters) and makes fun of my school (Colorado) for being “a football school.” For half of life, I have responded by saying, “Hey, your school has the best football player of either of our schools in the NFL while mine has produced the most astronauts of any non-military school!” referencing Aaron Smith for a while and then eventually Vincent Jackson.

If the argument continues, I just say, “Robert Redford” and metaphorically drop the mic. Luckily, she cannot remember Rae Carruth’s name.

Tanner May 7, 2014 at 8:33 am

To add to your last footnote Tony Romo went to a small school and was an UDFA and has made 3 pro bowls.

Richie May 7, 2014 at 4:06 pm

This is one of those tricky things. If teams start overvaluing these small school players and bump them up in the ratings because small school players have success, and end up taking them too soon, then they start losing the value.

In regards to the offense/defense splits, how do offensive linemen fare? I have a hypothesis that offensive linemen have a higher success rate; like the defensive players. QB’s, WR’s and RB’s have the big stats that can shift their ratings. So if a RB from a small school rushes for 1,700 yards, NFL teams might give undue weight to the production stats. But defensive players and offensive linemen would be more closely scouted based on the visual perception of their performance/ability.

Andrew Healy May 7, 2014 at 4:57 pm

Great point. There actually aren’t that many Pro Bowl offensive linemen taken in rd 4 or later during this period. You do see a similar gap, though, on the o-line. 4.7% (4 out of 86) of small school players in those rounds made a Pro Bowl, compared to 2.5% (5 out of 198). And you see no such gap for QB/RB/WR (5.0% vs 5.8%). QB has actually looked a little better for major schools (e.g., Brady, Bulger, and Hasselbeck). So you may be exactly right.

Quinton May 7, 2014 at 4:45 pm

This must kill Gene Smith, the former Jags GM. He got killed for drafting small school guys even though he essentially only did it in the later rounds. He had some other draft flaws, mainly he traded up too much (including for Blaine Gabbert! Ouch!) but his underlying philosophy on this was solid. He had some nice hits (Cecil Shorts, Rashad Jennings) but no home runs. As Chase has shown though, drafting the wrong QB gets you fired and he drafted the most wrong QB of all time. Great article, loved this!

Andrew Healy May 7, 2014 at 5:12 pm

Thanks, Quinton. Yeah, missing on the QB can nullify everything else and hitting/having the right guy fall in your lap can make up for all sins (see Grigson, Ryan).

And Jacksonville is indeed the team that’s taken the most small-school players since 1998 in rounds 4 and later. They’ve used 49% of their middle-to-late round picks on small school players. Also them for the most overall picks from small schools (42%). Their biggest problem has been missing in the 1st round. They average .65 Pro Bowls per 1st round pick, compared to 1.22 for other teams. Of course, Blaine Gabbert is part of the reason for that. At least he’s a real American, though.

Richie May 8, 2014 at 1:44 pm

And Jacksonville is indeed the team that’s taken the most small-school players since 1998 in rounds 4 and later. They’ve used 49% of their middle-to-late round picks on small school players.

Interesting. I don’t remember the Jacksonville GM situation, but that covers many different GM’s. How much more often does Jacksonville pick small-school players in late rounds than other teams? If they are doing it significantly more often, it makes me think that maybe small-school value is something that the franchise picked up on a long time ago, and maybe the owner was even aware of the information and encouraged all of his GM’s to do it?

I was just listening to the Advanced Football Stats podcast with Cade Massey. He was discussing how trading up for draft picks is a strategy that might be in the GM’s best interest (win now), but not in the owner’s best interest (win over the long term). Seeing a team with a consistent strategy over ~15 years seems like an owner-encouraged strategy.

Andrew Healy May 8, 2014 at 2:19 pm

It’s a pretty small difference. It looks less like a clear strategy and more like a slight preference relative to other teams. Green Bay was second in picking 45% of its middle-to-late rounders from small schools. Across all teams, it’s 35% for small schools.

BTW, Chase wrote on recent traded future 1st round picks and found not so much on the principal-agent problem: http://www.footballperspective.com/analyzing-the-last-ten-years-of-trading-away-future-first-round-picks/

Trading up is another place to look for this and free agent binges (like Miami’s last year) are another. Many of the recent trade-ups don’t seem to fit that story (e.g., ATL for Jones, NO for Ingram), but others might.

Richie May 8, 2014 at 2:54 pm

BTW, Chase wrote on recent traded future 1st round picks and found not so much on the principal-agent problem: http://www.footballperspective.com/analyzing-the-last-ten-years-of-trading-away-future-first-round-picks/

This is interesting. There are some differences between Chase’s analysis and Cade Massey’s analysis.

Chase was looking only at first round picks that were traded, while Massey was looking at all picks.
Chase was including trades that included players, while Massey was only looking at trades involving draft picks.

Chase concluded that GM’s were trading away future picks because they were overconfident in their ability to identify specific players that they needed.

I think the combination of Chase and Massey’s conclusions is that GM’s may also not quite realize that they are overpaying for the privilege to move up.

A GM thinks, “I want to move up”, and the rule of thumb is that you pay 1 round per year that you move up. (You pay a 1st rounder next year for a 2nd rounder this year.) But Massey is saying that historically teams are paying 136% discount rate when they trade future picks.

But all this is completely off the topic of Diamonds in the Rough.

Andrew Healy May 7, 2014 at 4:56 pm

Great point. There actually aren’t that many Pro Bowl offensive linemen taken in rd 4 or later during this period. You do see a similar gap, though, on the o-line. 4.7% (4 out of 86) of small school players in those rounds made a Pro Bowl, compared to 2.5% (5 out of 198). And you see no such gap for QB/RB/WR (5.0% vs 5.8%). QB has actually looked a little better for major schools (e.g., Brady, Bulger, and Hasselbeck). So you may be exactly right.