≡ Menu

Not Tim Couch

Not Tim Couch.

The preseason is meaningless, right? Well, as it turns out, it might give us a window into quarterback development, despite what you might think. The threshold for whether the preseason is useful is whether including that information tells us anything about a quarterback’s potential that we don’t already know from his draft position (or perhaps certain analytics). I have been putting together data from preseason box scores going back to 1997. The data show that, for some quarterbacks, the preseason is not quite meaningless.

Neil Paine showed some interesting evidence relating to this idea on Friday. Looking at team performance since 2009 for teams with new quarterbacks, Neil showed that preseason passing efficiency helps predict regular season passing efficiency. It’s important to note that part of this result may have been pretty predictable even before we watched those preseason games. The 2012 Redskins replaced Rex Grossman and John Beck with the #2 pick in the draft who would have been #1 in an average year. So we would expect a big improvement to come just by way of moving from Grossman to a healthy RGIII.

Still, maybe the preseason does offer hints as to whether a quarterback is likely to succeed or fail even controlling for what we would have expected. Were there hints in the preseason that someone like Russell Wilson would outperform his draft status? 1 Does Blake Bortles’ impressive preseason performance so far tell us something about his chances of succeeding in the NFL even for someone drafted so highly?2

The answers to these questions are subtle. I think the answers look like yes and no, respectively. Let’s take this to the data.

Historical Performance of Early-Round QBs in their First Preseason

By going back to 1997, we can get enough data to look at how new quarterbacks perform during the preseason according to draft position. I collected these data from John Troan’s database.3 It’s wonderful that these data are even available, and they are mostly in good shape. I have checked them and fixed some coding errors, but it is important to emphasize that there may still be a few mistakes.4 I don’t think there are many.

John’s database reports box scores that include completions, attempts, yards, and touchdowns going back to 1997. Sacks are not broken down by quarterback and interceptions are only available since 2007, so I’m going to look at just Y/A.5

The table below lists the quarerbacks selected in the draft who threw at least 25 passes in the preseason the year they were drafted. The table is sorted as yards over average, with the average Y/A of these quarterbacks checking in at 6.12.6 The last column in the table is value above expectation (VAE), or how much more Approximate Value (AV) the player produced in his first five years relative to the expectation for other quarterbacks with his draft position.7 For example, Kyle Orton checks in as the rookie quarterback with the most impressive preseason since 1997. He averaged 10.2 Y/A on his 46 attempts, and wound up producing 23 points of AV during his first five seasons. Since Orton was the 106th pick in the draft, 23 points of AV represents 17.4 points more than expected. That means Orton represents one piece of evidence that yes, preseason statistics can provide some insight into a quarterback’s career.

QuarterbackTeamYearDraft SelectionAttemptsY/AYd Ov AvgAV First Five YrsVAE
Kyle OrtonCHI20051064610.21882317.4
Jay CutlerDEN200611629.051825219.2
Nick FolesPHI201288638.78168
Russell WilsonSEA201275638.51151
Bruce GradkowskiTB2006194618.3813883
Matt SchaubATL200490867.621292215.2
Peyton ManningIND19981648.131297737.5
Mark SanchezNYJ20095379.3812128-8.7
Carson PalmerCIN20031548.351205616.5
Andrew LuckIND20121667.91118
Kirk CousinsWAS2012102737.67113
Tim TebowDEN201025398.82105
John NavarreARZ2004202279.811000-4.5
Colt BrennanWAS2008186537.75860-5.3
Luke McCownCLE2004106428.148560.4
Ryan LeafSD19982817.16841-37.8
Andrew WalterOAK200569448831-9
Cody PickettSF2004217418.12821-2
Joey HarringtonDET200231126.848127-11.1
Randy FasaniCAR2002137437.98801-4.1
Pat BarnesKC1997110418.07800-5.4
Ben RoethlisbergerPIT200411437.79725825.2
Omar JacobsPIT2006164348.09670-5.4
Brian GrieseDEN199891507.44664336.3
Chad PenningtonNYJ200018367.8964290.3
Trent EdwardsBUF200792617.0859169.4
Tom BradyNE2000199327.94585247.3
Matthew StaffordDET20091557.0752477.5
Drew BreesSD200132716.77464927.3
Charlie FryeCLE200567506.96427-3.4
Tyrod TaylorBAL2011180606.841
Tarvaris JacksonMIN200664347.3241131.9
Kyle BollerBAL200319686.714019-9.1
Brooks BollingerNYJ2003200297.243361.4
David GarrardJAC2002108686.6331812.5
Danny WuerffelNO199799377334-2
Byron LeftwichJAC20037297.12833-2.4
Eli ManningNYG20041496.6727466.5
Matt LeinartARZ200610476.682612-21.4
Brady QuinnCLE200722456.67253-23.5
Dan OrlovskyDET2005145546.59254-1.2
Jeff SmokerSTL2004201426.71250-4.6
T.J. YatesHOU2011152266.9622
Jonathan QuinnJAC199886296.83213-4.3
Curtis PainterIND2009201596.46202-2.6
Mike McMahonDET2001149566.482060.8
Chris WeinkeCAR2001106726.381960.4
Tom BrandstaterDEN2009174466.48170-5.5
Jason CampbellWAS200525376.59174520
Jake LockerTEN20118496.4516
Nate DavisSF2009171496.41140-5.5
Craig NallGB2002164456.44141-4.4
David CarrHOU20021706.313455.5
Patrick RamseyWAS200232346.441114-7.7
Vince YoungTEN20063576.26833-5.1
John BeckMIA200740486.2564-14.4
Ricky StanziKC2011135346.265
Chris RedmanBAL200075626.1943-5.9
Robert GriffinWAS20122316.233
Chris SimmsTB200397406.2381.8
Sam BradfordSTL20101556.152
Brandon WeedenCLE201222496.06-3
Rusty SmithTEN2010176336-4
Colt McCoyCLE201085395.95-7
Shaun KingTB199950445.95-7172.1
Christian PonderMIN201112495.92-10
Tim CouchCLE19991695.94-1232-7.5
Mike TeelSEA2009178415.8-130-5.4
Michael VickATL20011285.64-135313.5
Kevin DaftTEN1999151425.76-150-5.3
Philip RiversSD20044305.47-194810.6
Jonathan CromptonSD2010168365.53-21
Keith NullSTL2009196285.36-211-3.9
Charlie BatchDET199860625.76-223018
Tim RattaySF2000212395.54-2395.4
Ryan LindleyARZ2012185925.85-25
Greg McElroyNYJ2011208595.68-26
Jimmy ClausenCAR201048425.5-26
Jeff RoweCIN2007151305.23-270-5.3
Ryan MallettNE201174635.67-28
Kevin KolbPHI2007361065.82-3211-9
Chad HenneMIA200857635.57-35229.2
Seneca WallaceSEA2003110435.3-35126.6
Andy DaltonCIN201135605.47-39
Todd HusakWAS2000202635.48-400-4.5
Joe GermaineSTL1999101364.86-451-4.9
Jake PlummerARZ199742435.07-454022.4
Cade McNownCHI199912505.2-467-25.2
Troy SmithBAL2007174374.86-4760.5
Matt FlynnGB2008209424.98-483-0.9
John David BootyMIN2008137414.95-480-5.1
Colin KaepernickSF201136505.14-49
Cam NewtonCAR20111575.26-49
Dennis DixonPIT2008156334.58-512-3.3
Levi BrownBUF2010209304.33-54
Josh FreemanTB200917474.98-54377.8
Aaron RodgersGB200524374.65-54359.5
Rex GrossmanCHI200322565.13-5516-10.5
Kevin O'ConnellNE200894324.38-560-6.4
Kellen ClemensNYJ200649294.07-593-12.2
Josh HarrisBAL2004187314.19-600-5.2
Brian St. PierrePIT2003163394.56-610-5.4
Brock OsweilerDEN201257334.21-63
Ryan TannehillMIA20128785.31-63
Blaine GabbertJAC201110705.21-64
Matt RyanATL20083594.98-677334.9
Adrian McPhersonNO2005152313.87-700-5.3
Sage RosenfelsWAS2001109494.69-703-2.5
Jim SorgiIND2004193303.7-7350
Craig KrenzelCHI2004148354.03-731-4.2
Spergon WynnCLE2000183574.74-792-3.3
Jesse PalmerNYG2001125393.92-861-4.1
Charlie WhitehurstSD200681474.21-902-5.9
Alex SmithSF20051444.07-9014-25.5
Mike KafkaPHI2010122634.65-93
Brian BrohmGB200856423.69-1020-13.1
Joe FlaccoBAL200818684.38-1186132.3
Quincy CarterDAL200153674.19-129173
Donovan McNabbPHI19992663.52-1725718.2
Marques TuiasosopoOAK200159451.4-2121-11.3
Josh McCownARZ2002811432.87-465157.1

Looking at the table, most of the top preseason performers did pretty well in their first five regular seasons. Wilson had the 7th-highest preseason Y/A since 1997 and he doesn’t factor into any of the calculations for AV because he hasn’t played his fifth season yet. Peyton Manning and Tom Brady are both in the top 20 for rookie preseason Y/A. Jay Cutler, Nick Foles, Andrew Luck, Carson Palmer, and Ben Roethlisberger also are in the top 20.

But so are Mark Sanchez and Tim Tebow. And look for Aaron Rodgers, way down at 4.65 Y/A. Donovan McNabb is at 3.52, two spots from the bottom. So preseason performance clearly doesn’t tell us everything about how a quarterback will perform. Does it tell us something, though?

For first round passers, I think the answer is not so much. The picture below has the first round quarterbacks from ’97 to ’10 along with a best-fit line to predict their VAE as a function of preseason Y/A.

healy fig1

On the other hand, for quarterbacks drafted in later rounds, preseason performance — at least in this data set — does tell us something useful about how they will do in the regular season. The graph below looks just at QBs drafted in rounds 2-4 from 1997 to 2010.8

healy fig2

For every additional one yard per attempt a quarterback produced, that adds about 2.4 points of AV in the first five years for a quarterback drafted in rounds two through four. The estimated effect is 2.0 points of AV added if we look from rounds 2-7.

Overall, it looks like preseason performance is useful for unearthing diamonds in the rough. It can help identify middle-round signal callers who the draft process undervalued, such as Wilson and Orton. What the preseason can’t do is separate QBs who the draft process already highly valued. So I’m not sure it’s going to tell us much about someone like Blake Bortles and perhaps not Teddy Bridgewater, either.

And maybe even the information about middle-round quarterbacks is something we could have found in the player’s college performance. For example, Marques Tuiasosopo may have been drafted in the second round, but his college numbers did not project well. Russell Wilson was a famous favorite of both Matt Waldman and  Football Outsiders leading up to the 2012 Draft. So maybe preseason performance is picking up stuff that we could have already seen in college stats.

I do think college stats can tell a big part of the story. College stats pointed towards players such as Kyle Orton and Matt Schaub outperforming their draft position, so that maybe we didn’t need the preseason to know they were probably going to be pretty good for where they were picked. Still, there are some players with high college completion percentages (era-adjusted) whose poor preseason stats portended future doom. Examples include Brian Brohm and Kellen Clemens. This issue is an open question, but I lean towards the preseason probably mattering a little for rookie passers even controlling for college stats. I could easily be wrong, though, and it would be great to check this out.

At this point, where I think the preseason might predict future success is not so much for 1st round QBs such as Bortles and Teddy Bridgewater, but for players like Jimmy Garoppolo, who got less attention leading up to the draft. And it’s not just that Garoppolo is a second-round passers. We have less information to go on with him based on his college performances coming against inferior competition. Based on the results above, I think it probably does tell us a little something that Garoppolo’s averaging 9.2 Y/A so far.9 Of course, full disclosure that I am invested in the success of the Pats latest GQ-worthy quarterback.

  1. Mike Tanier also looked at this topic last week, and lauded Wilson’s 2012 preseason performance. []
  2. His numbers on Thursday were inflated by a meaningless pass at the end of the first half, but he also had some very pretty passes that I wasn’t expecting. Notable were the pass right before the 1st half two-minute warning and the pass with 10:00 left in the third quarter. Both are here for now. []
  3. I’m grateful that John made these data available. []
  4. For example, I thought JaMarcus Russell had an incredible 2008 preseason, until I saw that the box score gave him 751 yards for the second game. []
  5. Looking at Y/A misses some of the fairly hilarious Leaf-to-Manning comparisons that Chase referenced in his post. While Leaf was thought to have had the better preseason in 1998, Manning actually had the better Y/A (8.13 vs. 7.16). Manning had a lower completion percentage and threw at least a couple more picks. []
  6. Note that we are not endorsing this as the preferred method to rank the quarterbacks; it just makes more sense than using pure Y/A given the varying number of attempts. []
  7. To calculate this, I regressed the AV in the first five years on a cubic in draft position for all QBs. The difference between actual first-five AV and the predicted value from that regression is VAE. []
  8. The line looks similar if we go all the way up to round 7, but the image gets very cluttered. In addition, we have to exclude quarterbacks like Wilson who have not played for five seasons yet. []
  9. The Football Outsiders LCF forecast was bullish on Garoppolo, too. []
  • Chase Stuart

    Fun post as always, Andrew.

  • Nick Bradley

    Andrew, Chase,

    I did a regression with draft position and Y/A, and I see a strong correlation, r^2 = .435, with tiny p-values (below). Threw out QBs without 5 years in league, of course. Mark Sanchez, Carson Palmer, Jay Cutler, Peyton Manning, and Ryan Leaf had the best preseasons for rookies. At the bottom you see the list and the delta between actual AV-5 and expected. Sanchez is a pretty big outlier.

    = -0.178*(Draft Pos.) + 2.156*(Y/A) + 19.764

    intercept = 0.004
    draft pos = 4.44138E-12
    Y/A = 0.034

    Quarterback Team Exp AV-5 AV-5 over expected
    Mark Sanchez NYJ 39.10 -11.10
    Carson Palmer CIN 37.59 18.41
    Jay Cutler DEN 37.32 14.68
    Peyton Manning IND 37.11 39.89
    Ryan Leaf SD 34.84 -33.84
    Matthew Stafford DET 34.83 12.17
    Ben Roethlisberger PIT 34.60 23.40
    Joey Harrington DET 33.98 -6.98
    Eli Manning NYG 33.97 12.03
    Byron Leftwich JAC 33.83 -0.83
    Chad Pennington NYJ 33.57 -4.57
    David Carr HOU 33.17 11.83
    Vince Young TEN 32.73 0.27
    Tim Couch CLE 32.39 -0.39
    Matt Leinart ARZ 32.39 -20.39
    Michael Vick ATL 31.75 21.25
    Kyle Boller BAL 30.85 -11.85
    Philip Rivers SD 30.85 17.15
    Brady Quinn CLE 30.23 -27.23
    Matt Ryan ATL 29.97 43.03
    Jason Campbell WAS 29.52 15.48
    Cade McNown CHI 28.84 -21.84
    Drew Brees SD 28.66 20.34
    Alex Smith SF 28.36 -14.36
    Patrick Ramsey WAS 27.95 -13.95
    Josh Freeman TB 27.47 9.53
    Donovan McNabb PHI 27.00 30.00
    Rex Grossman CHI 26.91 -10.91
    John Beck MIA 26.12 -22.12
    Joe Flacco BAL 26.00 35.00
    Kevin Kolb PHI 25.90 -14.90
    Aaron Rodgers GB 25.52 9.48
    Andrew Walter OAK 24.73 -23.73
    Tarvaris Jackson MIN 24.15 -11.15
    Shaun King TB 23.69 -6.69
    Jake Plummer ARZ 23.22 16.78
    Kyle Orton CHI 22.89 0.11
    Charlie Frye CLE 22.84 -15.84
    Chad Henne MIA 21.63 0.37
    Charlie Batch DET 21.50 8.50
    Matt Schaub ATL 20.17 1.83
    Kellen Clemens NYJ 19.82 -16.82
    Chris Redman BAL 19.76 -16.76
    Brian Griese DEN 19.61 23.39
    Quincy Carter DAL 19.36 -2.36
    Jonathan Quinn JAC 19.18 -16.18
    Trent Edwards BUF 18.65 -2.65
    Luke McCown CLE 18.45 -12.45
    Brian Brohm GB 17.75 -17.75
    Pat Barnes KC 17.58 -17.58
    Danny Wuerffel NO 17.23 -13.23
    Chris Simms TB 15.87 -7.87
    David Garrard JAC 14.77 3.23
    Chris Weinke CAR 14.65 -8.65
    Charlie Whitehurst SD 14.42 -12.42
    Randy Fasani CAR 12.58 -11.58
    Kevin O’Connell NE 12.48 -12.48
    Marques Tuiasosopo OAK 12.28 -11.28
    Joe Germaine STL 12.26 -11.26
    Seneca Wallace SEA 11.61 0.39
    Josh McCown ARZ 11.53 3.47
    Sage Rosenfels WAS 10.47 -7.47
    Dan Orlovsky DET 8.16 -4.16
    Omar Jacobs PIT 8.01 -8.01
    Mike McMahon DET 7.21 -1.21
    John David Booty MIN 6.05 -6.05
    Jesse Palmer NYG 5.97 -4.97
    Kevin Daft TEN 5.30 -5.30
    John Navarre ARZ 4.96 -4.96
    Craig Nall GB 4.46 -3.46
    Jeff Rowe CIN 4.16 -4.16
    Colt Brennan WAS 3.37 -3.37
    Bruce Gradkowski TB 3.30 4.70
    Nate Davis SF 3.15 -3.15
    Tom Brandstater DEN 2.76 -2.76
    Craig Krenzel CHI 2.11 -1.11
    Dennis Dixon PIT 1.87 0.13
    Tom Brady NE 1.46 50.54
    Adrian McPherson NO 1.05 -1.05
    Mike Teel SEA 0.58 -0.58
    Brian St. Pierre PIT 0.58 -0.58
    Brooks Bollinger NYJ -0.23 6.23
    Troy Smith BAL -0.73 6.73
    Cody Pickett SF -1.36 2.36
    Jeff Smoker STL -1.55 1.55
    Curtis Painter IND -2.09 4.09
    Spergon Wynn CLE -2.59 4.59
    Keith Null STL -3.57 4.57
    Todd Husak WAS -4.38 4.38
    Josh Harris BAL -4.49 4.49
    Tim Rattay SF -6.03 15.03
    Jim Sorgi IND -6.61 11.61
    Matt Flynn GB -6.70 9.70

    • Nick Bradley

      Forgot to mention that Tom Brady is the biggest positive outlier.

    • Chase Stuart

      Andrew emailed me this response:

      “Cool stuff, Nick. I actually almost wrote about Brady b/c he’s a case of the theory looking pretty good. Low pick with almost 8 Y/A his first preseason. And then I had a caveat about Mark Sanchez being the outlier who didn’t fit. So I agree all around.

      I think our numbers for Y/A might be a little different for at least some players if I’m understanding correctly (I think I had Ryan Leaf at good, but not great, for example at 7.16 Y/A).”

      • Nick Bradley

        Thanks for the relay — I took the Y/A numbers from the table above

        • Oh, I guess we agree on the Y/A then 🙂

          And my mistake: I misunderstood your regression. I read that as Ryan Leaf having the fifth-best preseason when it’s Leaf having the fifth-best projection according to draft position and preseason Y/A. Draft position certainly does dominate. Even about 8 Y/A can’t save Brady from a low projection for example.

          • Nick Bradley

            correct; but even being drafted #1 overall couldn’t save Alex smith from his 4.0 ANY/A.

            Speaking of Alex Smith, has any top draft pick ever had a career like his? everyone assumed he’s a total bust, but the team sticks with him for lack of better options, and he ends up being really good with his original team…then traded away and makes a pro bowl?

            Jim Plunkett is the closest example, but he did nothing in NE after being taken #1 overall. but he didn’t have a winning season until he was in his 30s and his third team.

  • I like how Brian Griese is off on an island by himself in that last graph.

    I remember Cutler’s first preseason. Or, rather, I thought I did. Bronco fans were absolutely going nuts about him because they were so anti-Plummer*= that anybody who could take a snap without falling down would be hailed as the next Elway, so I didn’t remember that he had actually performed really well in that preseason.

    • No idea what *= is doing there or means. Maybe the creepy smiley face really is Hyde!

      • Yeah, Griese kind of benefits from the five-year window. He had an AV of 43 in years 2-5 and then 18 the rest of his career. His stat line in 2000 (19 TDs, 4 INTs) was a little Folesian, or poor man’s Folesian.

  • Andrew

    The thing about all this that players can’t pick their teams so theoretically the best players go to the worst teams. Orton, whose first season was horrible but he didn’t have much help. Chicago had the fifth pick in the draft took Benson, a running back. So Chicago’s management might have been sub-par, players could have been injured, etc. Player evaluation is always a subject of contention though. That is why I hesitated in call players busts because of the context and unpredictability.