## Peyton Manning, Matt Ryan, and Andrew Luck are ESPN QBR Stars

Luck's rushing ability makes him a QBR star.

A few weeks ago, I put ESPN’s Total QBR under the microscope. Today, I want to look at the quarterbacks whose passing statistics most differ from their QBR grades.

Total QBR grades go back to 2006, so to start, I ran a regression using Adjusted Net Yards per Attempt to predict Total QBR. The best-fit formula was:

Total QBR = -13.5 + 11.23 * ANY/A

For those curious, the R^2 was 0.80, indicating a very strong relationship between ANY/A and Total QBR. What this formula tells us is that a passer needs to average 5.65 ANY/A to be “projected” to have a QBR of 50; from there, every additional adjusted net yard per attempt is worth 11.2 points of QBR. Last year, Peyton Manning averaged 8.87 ANY/A, which projects to a QBR of 86.2. In reality, Manning had a QBR of “only” 82.9; this means Manning’s QBR says he wasn’t quite as amazing as his excellent efficiency numbers would indicate (to say nothing of his otherworldly gross numbers). One likely reason for this result is that Manning ranked 29th in average pass length in the air (according to NFLGSIS) and 6th in yards after the catch per completion; this matters because ESPN gives more credit to quarterbacks on the yards they accumulate through the air. (Throughout this post, we will be forced to deal with educated guesses, because Total QBR is a proprietary formula.)

As it turns out, Manning rating higher in actual QBR than projected QBR is a stark departure from prior years. In 2012, he finished 7.2 points higher in actual QBR than projected QBR, but that’s nothing compared to his time with the Colts. In five years in Indianapolis during the Total QBR era, Manning finished at least 10 points higher in actual QBR each season.

Along with Manning, Matt Ryan and Andrew Luck are the two quarterbacks who are most likely to over-perform relative to their “projected” ratings. Let’s be careful about exactly what this means: whatever the ingredients that go into the QBR formula that don’t go into the ANY/A formula, Manning, Ryan, and Luck seem to have a lot of them.

Luck is a fascinating case. In 2012, he ranked just 20th in ANY/A, but 11th in QBR. I wrote several articles during Luck’s rookie season about how his QBR ratings surpassed his standard stats.1 Last year, he ranked 16th in ANY/A and 9th in QBR. Does this make Luck the quarterback most underrated (if you buy into QBR) by his traditional passing numbers (if you buy into ANY/A)?

Yes and no. He’s the most underrated among active quarterbacks, but one other quarterback (minimum 700 action plays since 2006) has him beat. Here’s what I did:

1) Calculate a career projected QBR grade, based on the projected QBR grade in each season (using the regression formula above) and weighting the rating in each season by number of action plays.

2) Calculate a career actual QBR grade, based on a weighted average (for number of action plays) of each season’s QBR grade. This is not the way ESPN comes up with their career QBR grades (assuming they even come up with them), but I am not too concerned about about the trivial distinction.2

3) Subtract the result from step 1 from the result in step 2.

As it turns out, the quarterback most harmed by using traditional stats is… Vince Young. It’s easy to snicker and say this is a drawback of QBR if we’re saying Young is underrated, but I don’t think that’s fair. We have to remember what the baseline is: out of the 55 quarterbacks with 700 action plays since 2006, Young ranks 45th in projected QBR, which puts him behind end-of-career Marc Bulger and Trent Edwards. He’s just slightly ahead of Tarvaris Jackson and Chad Henne.

Now if you’re speaking to a Longhorns homer who thinks Young was an above-average quarterback because of his career 30-17 record, well, then he’s not underrated. We already know that Young’s record is way out of whack with his passing stats. But seeing him atop the list here indicates that Young deserves slightly more credit than you would normally assign him, and there’s a pretty obvious reason why: ANY/A ignores rushing contributions, and Young was an outstanding runner.

Here’s how to read the table below: Young had three qualifying seasons since 20063 on 1,360 action plays in those seasons. He has a projected career QBR grade of 42.3 but an actual grade of 53.8, which means he’s outperformed the projection by 11.5 points.

QuarterbackYearsAct PlaysProj QBRAct QBRDiff
Vince Young3136042.353.811.5
Andrew Luck2149852.263.711.4
Matt Ryan6394658.468.19.7
Peyton Manning7471371.580.59.1
Christian Ponder3130439.545.35.8
Kurt Warner3181661.567.25.8
Colin Kaepernick290764.769.95.1
Kerry Collins3106648.2534.8
Jay Cutler739255256.44.4
Ryan Tannehill2133843.947.94.1
Michael Vick4216052.956.73.9
Eli Manning8507653.156.43.3
David Garrard5262352.555.63.2
Jeff Garcia286161.964.22.3
Shaun Hill285649.251.42.2
Ryan Fitzpatrick6310443.645.72.1
Colt McCoy288139.741.31.6
Drew Brees8574468.369.41.1
Jake Delhomme3137446.747.71
Joe Flacco6390650.451.10.7
Ben Roethlisberger8463557.558.20.7
Matt Hasselbeck7337544.445.10.7
Matt Schaub7353960.961.30.4
Aaron Rodgers6375072.973.20.4
Josh Freeman4235949.549.70.2
Carson Palmer7437953.853.90.1
Matthew Stafford4282553.353.40.1
Joey Harrington288036.536.60
Russell Wilson2119465.765.2-0.5
Tarvaris Jackson297341.941.3-0.6
Cam Newton3209256.255.5-0.7
Marc Bulger4204943.142.2-0.9
Matt Cassel6285246.645.6-1
Jon Kitna3186645.944.9-1
Andy Dalton3203552.651.6-1
Jason Campbell6288248.146.7-1.5
Tony Romo7433264.963.4-1.5
Derek Anderson3136946.244.6-1.6
Brett Favre5303953.451.7-1.7
Robert Griffin2121058.956.2-2.7
Kyle Orton4209953.350.1-3.2
Rex Grossman3140640.837.5-3.3
Trent Edwards3104843.139.6-3.5
Mark Sanchez4233240.236.1-4.1
Philip Rivers8495465.761.4-4.3
Blaine Gabbert289432.527.4-5.1
Donovan McNabb5279057.952.3-5.7
Alex Smith6299850.344.4-6
JaMarcus Russell279432.223.4-8.8
Nick Foles276069.857.7-12.1
Brandon Weeden296040.626.2-14.4

With Young, it’s easy to identify why QBR loved him. What about the three active passers atop the list?

• In 2012, Luck received a lot of credit for two areas ignored by ANY/A: rushing production and benefits gained via penalty. Luck ranked 3rd in rushing EPA among quarterbacks, behind only Cam Newton and Russell Wilson. And he ranked 1st in expected points added due to penalties. In 2012, the Colts offense led the league by a large margin in both penalties drawn and penalty yards gained (who knew?). In other words, ESPN QBR gives Luck credit for a 40 yard gain that comes via pass interference, or for drawing a player offsides, while ANY/A ignores both plays. Last year, Luck led all quarterbacks in rushing EPA, and he ranked third in penalty EPA. Luck ranks 4th among all players and (a very distant) 2nd among quarterbacks in rushing first downs picked up on 3rd/4th down over the past two seasons. While Luck’s traditional passing numbers are not overwhelming, I’m very inclined to side with Total QBR when it comes to the Colts passer. He seems to be an appropriate outlier.
• Ryan also excels at adding value via penalty yards. And Ryan, like Luck, has done very well in the fourth quarter of games. Even though ESPN has removed the clutch bonus associated with QBR, Ryan might still be getting some extra credit for all the comebacks and game-winning drives he’s led (without knowing the formula, one can’t quite say how Ryan’s QBR grade was derived). And, for whatever reason, Ryan’s passing EPA has outstripped his ANY/A, particularly in the 2010 season (perhaps related to the fact that the Falcons ranked 4th in passing first downs that season). Before running the numbers, I thought his great supporting cast of Roddy White, Tony Gonzalez, and Julio Jones (albeit not in 2010) meant Ryan would look better in ANY/A than QBR, because Total QBR tries to only reward a quarterback for his individual contributions (so a great catch by Jones won’t be worth as much to Ryan as it would in ANY/A). Ultimately, I can’t quite explain why Ryan stars here (the simple answer could just be the Ryan is better than his stats indicate, although I’m not even sure what that means in this context.)
• With Manning, it’s never surprising to see him “overachieve” in some metric, although it is kind of odd to see an argument that ANY/A underrates him. Manning tends to have prolific offenses and is excellent on third downs, so presumably that would help him here. And since he passes the eye test with flying colors, my guess is that all the little things that Manning does so well gets rewarded in QBR, even if they may not all get picked up in ANY/A.

• Brandon Weeden is the anti-Luck (in more ways than one). Weeden doesn’t have very good standard numbers, but he’s downright dreadful in QBR. There are lots of potential explanations — Weeden struggles on third down, has issues with fumbling, and it’s easy to envision QBR’s analysts assigning Josh Gordon a good chunk of the production on the few good plays Weeden makes [Please see the update below]. I will note that Weeden actually led the NFL last year in air yards per completion (Norv!), so I wouldn’t have guessed that Weeden would fare so badly (relative to his already bad statistics) in this analysis.
• Nick Foles led the NFL in ANY/A but ranked just fifth in Total QBR. In ESPN’s season in review of QBR, no justification was given to explain this difference. Eagles running backs easily led the league in yards per reception: it’s possible (and appropriate) that Foles’ QBR grade is lower because ANY/A overvalues him when LeSean McCoy takes a short pass for 30 yards. But otherwise, I can’t quite think of why Foles’ QBR was so low: perhaps more credit was given to the Eagles offensive line than other quarterbacks? I do know that Foles threw a number of potential interceptions that were dropped, but as I understand it, QBR wouldn’t penalize him for that. While I like Total QBR, this is a good reason why it’s heavily criticized: it’s silly that we don’t know what Foles ranks only 5th in this metric.
• Alex Smith is another player who does not excel in Total QBR. It’s easy to assume that screen passes to Jamaal Charles and a general aversion to deep throws are the reasons, but Smith “only” undershot his QBR projection by 3.8 points in 2013. It was actually his 2006 and 2010 seasons that gave him such a low place on the list.

Update: Alok Pattani of ESPN Stats and Information was kind enough to email me this morning to discuss the post. He noted, among other things, that there is no subjective adjustment on “Weeden to Gordon” type of passes. He noted that ESPN’s video trackers do not have the ability to “decide” how much credit the quarterback or receiver should get on each play.

I asked him how QBR treats the following three plays, which are obviously treated the exact same way using conventional metrics:

1) QB throws a perfect pass to WR, who makes an easy catch for 25 yards.

2) QB throws a terrible pass, but Elite WR makes an incredible diving catch for 25 yards.

3) QB throws a terrible pass, which hits a DB in the hands, but bounces into the WR’s hands, who records a 25 yard catch.

Alok noted that currently, all three plays will likely be looked at the same way from a division of credit standpoint based on the data ESPN records (i.e. if the air yards, pass direction, etc. are all the same). ESPN does not look at the “quality” of a completed pass, which would venture into more subjective territory (on the other hand, overthrow/underthrow/drop data is recorded for incomplete passes).4 In addition, he noted:

All the trackers can do is enter in “video-based” data – whether the QB was under duress or not, which direction the pass went, how many air yards/YAC there was, whether an incompletion was a drop or an overthrow, etc. The QBR algorithm looks at that data and then applies certain rules (based on analysis of past data) to divide credit on the play – e.g. if there were more air yards, relatively more credit to the QB.

So this is not really subjective – the division of credit is based on data relevant to what Weeden and Gordon did on the play. The trackers can’t say something like “Josh Gordon deserves 60% credit for the EPA on that play” and enter that – they just input the data they see on the video.

It looks like several of the guys you mention as QBR-ANY/A outliers are those with higher/lower sack rates. I know ANY/A takes into account sacks and sack yardage, but my thought is that QBR gives a higher penalty on sacks, so that’s another source of explanation for some of those QBs with discrepancies. Peyton Manning, Matt Ryan, Andrew Luck and even Vince Young have relatively low sack rates. Nick Foles, Alex Smith, and JaMarcus Russell have relatively high ones. Makes a big difference in QBR.

As for Weeden in particular, Alok noted that three possible explanations (for anyone who really wants to know more about Brandon Weeden and has made it this far, I salute you):

• He’s worse on 3rd downs than other downs (where he is still not good). This is going to be emphasized in a system based on EPA more than any stats that don’t look at the context of each play.
• While he didn’t rely on a lot of YAC in 2013, he was one of the league leaders in YAC/completion in 2012. So he got less credit for his completions than the average passer in his rookie year.
• In 2012, his interceptions were really penalizing. So his interceptions may have been worse, on average, than others (usually pick-sixes or other shorter throws where returns are more attributable to the QB).

I’d like to thank Alok and ESPN reaching out to help inform Football Perspective and its readers.

1. Although now I can’t recall if his 2012 ratings were inflated because of his 4th quarter comebacks.  And I can’t check, because once ESPN decided to cap the clutch weight associated with each play, they retroactively applied the current formula across past years. []
2. As explained in the footnotes to the earlier post, the scaling function that gives the “final” QBR on a 0-100 scale is nonlinear; as a result, you can’t just calculate a weighted average of the individual game QBR values to get season QBR (and presumably you then can’t just calculated a weighted average of the individual season QBR grades to get a career grade). Instead, you need to have the “points per play”-like value that’s behind QBR and calculate the weighted average of that (and weight based on the capped clutch weights, not even the action plays), then re-apply the scaling function to get it back on the 0-100 scale. So while I’m recreating QBR, I’m not recreating it the way ESPN would. That disclaimer aside, I don’t think my method will bias these results. []
3. Note that because Young posted an excellent ANY/A average in 2010 but on only 156 attempts, Young’s actual career ANY/A rating is better than you would think based on his projected QBR. When calculating his projected QBR, we were limited to his ANY/A ratings in the 2006, 2007, and 2009 seasons. For those curious, 2006 was the main reason for Young’s appearance at the top of this list: he outperformed his expected QBR by an incredible 19.1 points, nearly all of which is due to his 31.2 points of rushing EPA from that season. []
4. However, it’s possible such concepts, if ESPN could get comfortable with the data and history, could be made in the future. While subjective, I personally think it makes sense to treat these three plays differently, so I would welcome such a tweak to QBR. []
• Red

In 2010, Matt Ryan didn’t benefit from any big catch-and-run plays by his receivers. In 571 attempts, his longest completion was only 45 yards. Moreover, Ryan gained 61.8% of his passing yards via air yards, while the league average in 2010 was only 52.8%. That probably explains why QBR rates him much higher than ANY/A.

Agreed about Luck being an outlier. Is there anyway you could incorporate rushing and/or first downs into an ANYA-like formula? Individual first down data goes back to 1991, and FO has rushing data with kneeldowns excluded back to 1989.

I wonder if QBR is somehow penalizing Foles because of Chip Kelly’s offense. He benefitted from a lot of YAC (51.3% of his yards) and often seemed to have receivers WIDE open that didn’t require precise throws. His 7 TD game vs. the Raiders in particular appeared to be a schematic mismatch more than a great game by Foles.

• Chase Stuart

Thanks, Red. That’s a good point about Ryan’s 2010 season.

And yes, I think your guess on Foles is a good one. But I frankly have no clue whether that’s the reason or not.

• Regarding footnote 1, I see on ESPN that the QBR numbers for 2013 all match the ones in PFR. However, prior to that, they do not match. For example, PFR lists 2012 Peyton Manning as 84.11 compared to 82.4 on ESPN. 2012 Drew Brees is 67.86 on PFR and 66.5 on ESPN.

Given the differences, it may be that PFR hasn’t applied the same retroactive updates. It may be worth looking in to if you care.

• Chase Stuart

Very ingenious of you! However, Andrew Luck had an old QBR of 64.99 in 2012 and now has a QBR of 65.2 for that season. Which is really weird, unless ESPN actually removed the clutch weight two years ago. Otherwise, there’s no way Luck could have gone down, right?

Luck was 9th in ANS’ EPA ranking in 2012, and 4th in WPA – http://wp.advancednflstats.com/playerstats.php?year=2012&pos=QB&season=reg – presumably any other measure that rewards “clutch” play would show similar results.

• That is curious. I’m not sure what the discrepancy is then, but ESPN and PFR seem to differ from 2008-2012. Looking at Peyton Manning

ESPN
2006 – 87.2
2007 – 78.4
2008 – 79.3
2009 – 82.8
2010 – 71.7
2012 – 82.4

PFR
2006 – 87.22
2007 – 78.4
2008 – 79.77
2009 – 82.94
2010 – 68.61
2012 – 84.11

Looking back, I should have picked a guy who played in 2011 too, but I have already crossed the Rubicon.

• How are you calculating “action plays”? It doesn’t seem to be attempts + sacks + rushes, that would give you 698 for Foles. Foles’s actual play count is 1156, so it’s not that either.

• Chase Stuart

I took the data from ESPN, which calculates action plays. This includes false starts and penalty plays, so I think that explains the difference.

• Every time someone looks into QBR, I get reminded about how awesome some of ESPN’s data-gathering (some of which goes into QBR) is. As much as ESPN is a target of so many fans, that’s something they can do that probably no one else (except the NFL itself) can do, and I look forward to seeing what they do with it as they have that information over a longer stretch of time.

Have you yet started working on jokes for next off-season when you point out that Matt Cassel surprisingly led the NFL in yards per completion?

• Red

QBR is not adjusted for the offensive environment of each season, which along with the lack of SoS adjustment, seem like major oversights in such a complicated formula.

Here is the league average QBR for every season it’s been calculated:
2006 – 47.73
2007 – 49.57
2008 – 52.07
2009 – 49.98
2010 – 50.09
2011 – 51.50
2012 – 54.12
2013 – 51.32

I find it interesting that QBR was noticeably higher in 2012 than either 2011 or 2013, given the league average ANY/A was 5.9 in all three seasons. Perhaps the read option and inflated QB rushing numbers is a factor.

• Michael Terry

I have reservations about Luck getting credit for drawn PI calls. When I watched Indy last year, it appeared that Colts receivers are instructed to flop if they don’t have a chance at a catch, a brilliant strategy that other teams should emulate.