Luck's rushing ability makes him a QBR star.

A few weeks ago, I put ESPN’s Total QBR under the microscope. Today, I want to look at the quarterbacks whose passing statistics most differ from their QBR grades.

Total QBR grades go back to 2006, so to start, I ran a regression using Adjusted Net Yards per Attempt to predict Total QBR. The best-fit formula was:

Total QBR = -13.5 + 11.23 * ANY/A

For those curious, the R^2 was 0.80, indicating a very strong relationship between ANY/A and Total QBR. What this formula tells us is that a passer needs to average 5.65 ANY/A to be “projected” to have a QBR of 50; from there, every additional adjusted net yard per attempt is worth 11.2 points of QBR. Last year, Peyton Manning averaged 8.87 ANY/A, which projects to a QBR of 86.2. In reality, Manning had a QBR of “only” 82.9; this means Manning’s QBR says he wasn’t quite as amazing as his excellent efficiency numbers would indicate (to say nothing of his otherworldly gross numbers). One likely reason for this result is that Manning ranked 29th in average pass length in the air (according to NFLGSIS) and 6th in yards after the catch per completion; this matters because ESPN gives more credit to quarterbacks on the yards they accumulate through the air. (Throughout this post, we will be forced to deal with educated guesses, because Total QBR is a proprietary formula.)

As it turns out, Manning rating higher in actual QBR than projected QBR is a stark departure from prior years. In 2012, he finished 7.2 points higher in actual QBR than projected QBR, but that’s nothing compared to his time with the Colts. In five years in Indianapolis during the Total QBR era, Manning finished at least 10 points higher in actual QBR each season.

Along with Manning, Matt Ryan and Andrew Luck are the two quarterbacks who are most likely to over-perform relative to their “projected” ratings. Let’s be careful about exactly what this means: whatever the ingredients that go into the QBR formula that don’t go into the ANY/A formula, Manning, Ryan, and Luck seem to have a lot of them.

Luck is a fascinating case. In 2012, he ranked just 20th in ANY/A, but 11th in QBR. I wrote several articles during Luck’s rookie season about how his QBR ratings surpassed his standard stats. ^[1]Although now I can’t recall if his 2012 ratings were inflated because of his 4th quarter comebacks.  And I can’t check, because once ESPN decided to cap the clutch weight associated with … Continue reading Last year, he ranked 16th in ANY/A and 9th in QBR. Does this make Luck the quarterback most underrated (if you buy into QBR) by his traditional passing numbers (if you buy into ANY/A)?

Yes and no. He’s the most underrated among active quarterbacks, but one other quarterback (minimum 700 action plays since 2006) has him beat. Here’s what I did:

1) Calculate a career projected QBR grade, based on the projected QBR grade in each season (using the regression formula above) and weighting the rating in each season by number of action plays.

2) Calculate a career actual QBR grade, based on a weighted average (for number of action plays) of each season’s QBR grade. This is not the way ESPN comes up with their career QBR grades (assuming they even come up with them), but I am not too concerned about about the trivial distinction. ^[2]As explained in the footnotes to the earlier post, the scaling function that gives the “final” QBR on a 0-100 scale is nonlinear; as a result, you can’t just calculate a weighted average of the … Continue reading

3) Subtract the result from step 1 from the result in step 2.

As it turns out, the quarterback most harmed by using traditional stats is… Vince Young. It’s easy to snicker and say this is a drawback of QBR if we’re saying Young is underrated, but I don’t think that’s fair. We have to remember what the baseline is: out of the 55 quarterbacks with 700 action plays since 2006, Young ranks 45th in projected QBR, which puts him behind end-of-career Marc Bulger and Trent Edwards. He’s just slightly ahead of Tarvaris Jackson and Chad Henne.

Now if you’re speaking to a Longhorns homer who thinks Young was an above-average quarterback because of his career 30-17 record, well, then he’s not underrated. We already know that Young’s record is way out of whack with his passing stats. But seeing him atop the list here indicates that Young deserves slightly more credit than you would normally assign him, and there’s a pretty obvious reason why: ANY/A ignores rushing contributions, and Young was an outstanding runner.

Here’s how to read the table below: Young had three qualifying seasons since 2006 ^[3]Note that because Young posted an excellent ANY/A average in 2010 but on only 156 attempts, Young’s actual career ANY/A rating is better than you would think based on his projected QBR. When … Continue reading on 1,360 action plays in those seasons. He has a projected career QBR grade of 42.3 but an actual grade of 53.8, which means he’s outperformed the projection by 11.5 points.

With Young, it’s easy to identify why QBR loved him. What about the three active passers atop the list?

• In 2012, Luck received a lot of credit for two areas ignored by ANY/A: rushing production and benefits gained via penalty. Luck ranked 3rd in rushing EPA among quarterbacks, behind only Cam Newton and Russell Wilson. And he ranked 1st in expected points added due to penalties. In 2012, the Colts offense led the league by a large margin in both penalties drawn and penalty yards gained (who knew?). In other words, ESPN QBR gives Luck credit for a 40 yard gain that comes via pass interference, or for drawing a player offsides, while ANY/A ignores both plays. Last year, Luck led all quarterbacks in rushing EPA, and he ranked third in penalty EPA. Luck ranks 4th among all players and (a very distant) 2nd among quarterbacks in rushing first downs picked up on 3rd/4th down over the past two seasons. While Luck’s traditional passing numbers are not overwhelming, I’m very inclined to side with Total QBR when it comes to the Colts passer. He seems to be an appropriate outlier.

• Ryan also excels at adding value via penalty yards. And Ryan, like Luck, has done very well in the fourth quarter of games. Even though ESPN has removed the clutch bonus associated with QBR, Ryan might still be getting some extra credit for all the comebacks and game-winning drives he’s led (without knowing the formula, one can’t quite say how Ryan’s QBR grade was derived). And, for whatever reason, Ryan’s passing EPA has outstripped his ANY/A, particularly in the 2010 season (perhaps related to the fact that the Falcons ranked 4th in passing first downs that season). Before running the numbers, I thought his great supporting cast of Roddy White, Tony Gonzalez, and Julio Jones (albeit not in 2010) meant Ryan would look better in ANY/A than QBR, because Total QBR tries to only reward a quarterback for his individual contributions (so a great catch by Jones won’t be worth as much to Ryan as it would in ANY/A). Ultimately, I can’t quite explain why Ryan stars here (the simple answer could just be the Ryan is better than his stats indicate, although I’m not even sure what that means in this context.)

• With Manning, it’s never surprising to see him “overachieve” in some metric, although it is kind of odd to see an argument that ANY/A underrates him. Manning tends to have prolific offenses and is excellent on third downs, so presumably that would help him here. And since he passes the eye test with flying colors, my guess is that all the little things that Manning does so well gets rewarded in QBR, even if they may not all get picked up in ANY/A.

What about the reverse?

• Brandon Weeden is the anti-Luck (in more ways than one). Weeden doesn’t have very good standard numbers, but he’s downright dreadful in QBR. There are lots of potential explanations — Weeden struggles on third down, has issues with fumbling, and it’s easy to envision QBR’s analysts assigning Josh Gordon a good chunk of the production on the few good plays Weeden makes [Please see the update below]. I will note that Weeden actually led the NFL last year in air yards per completion (Norv!), so I wouldn’t have guessed that Weeden would fare so badly (relative to his already bad statistics) in this analysis.

• Nick Foles led the NFL in ANY/A but ranked just fifth in Total QBR. In ESPN’s season in review of QBR, no justification was given to explain this difference. Eagles running backs easily led the league in yards per reception: it’s possible (and appropriate) that Foles’ QBR grade is lower because ANY/A overvalues him when LeSean McCoy takes a short pass for 30 yards. But otherwise, I can’t quite think of why Foles’ QBR was so low: perhaps more credit was given to the Eagles offensive line than other quarterbacks? I do know that Foles threw a number of potential interceptions that were dropped, but as I understand it, QBR wouldn’t penalize him for that. While I like Total QBR, this is a good reason why it’s heavily criticized: it’s silly that we don’t know what Foles ranks only 5th in this metric.

• Alex Smith is another player who does not excel in Total QBR. It’s easy to assume that screen passes to Jamaal Charles and a general aversion to deep throws are the reasons, but Smith “only” undershot his QBR projection by 3.8 points in 2013. It was actually his 2006 and 2010 seasons that gave him such a low place on the list.

Update: Alok Pattani of ESPN Stats and Information was kind enough to email me this morning to discuss the post. He noted, among other things, that there is no subjective adjustment on “Weeden to Gordon” type of passes. He noted that ESPN’s video trackers do not have the ability to “decide” how much credit the quarterback or receiver should get on each play.

I asked him how QBR treats the following three plays, which are obviously treated the exact same way using conventional metrics:

1) QB throws a perfect pass to WR, who makes an easy catch for 25 yards.

2) QB throws a terrible pass, but Elite WR makes an incredible diving catch for 25 yards.

3) QB throws a terrible pass, which hits a DB in the hands, but bounces into the WR’s hands, who records a 25 yard catch.

Alok noted that currently, all three plays will likely be looked at the same way from a division of credit standpoint based on the data ESPN records (i.e. if the air yards, pass direction, etc. are all the same). ESPN does not look at the “quality” of a completed pass, which would venture into more subjective territory (on the other hand, overthrow/underthrow/drop data is recorded for incomplete passes). ^[4]However, it’s possible such concepts, if ESPN could get comfortable with the data and history, could be made in the future. While subjective, I personally think it makes sense to treat these … Continue reading In addition, he noted:

All the trackers can do is enter in “video-based” data – whether the QB was under duress or not, which direction the pass went, how many air yards/YAC there was, whether an incompletion was a drop or an overthrow, etc. The QBR algorithm looks at that data and then applies certain rules (based on analysis of past data) to divide credit on the play – e.g. if there were more air yards, relatively more credit to the QB.

So this is not really subjective – the division of credit is based on data relevant to what Weeden and Gordon did on the play. The trackers can’t say something like “Josh Gordon deserves 60% credit for the EPA on that play” and enter that – they just input the data they see on the video.

It looks like several of the guys you mention as QBR-ANY/A outliers are those with higher/lower sack rates. I know ANY/A takes into account sacks and sack yardage, but my thought is that QBR gives a higher penalty on sacks, so that’s another source of explanation for some of those QBs with discrepancies. Peyton Manning, Matt Ryan, Andrew Luck and even Vince Young have relatively low sack rates. Nick Foles, Alex Smith, and JaMarcus Russell have relatively high ones. Makes a big difference in QBR.

As for Weeden in particular, Alok noted that three possible explanations (for anyone who really wants to know more about Brandon Weeden and has made it this far, I salute you):

• He’s worse on 3rd downs than other downs (where he is still not good). This is going to be emphasized in a system based on EPA more than any stats that don’t look at the context of each play.
• While he didn’t rely on a lot of YAC in 2013, he was one of the league leaders in YAC/completion in 2012. So he got less credit for his completions than the average passer in his rookie year.
• In 2012, his interceptions were really penalizing. So his interceptions may have been worse, on average, than others (usually pick-sixes or other shorter throws where returns are more attributable to the QB).

I’d like to thank Alok and ESPN reaching out to help inform Football Perspective and its readers.

Russell Wilson is too awesome for snide comments.

For example, have you ever considered how rookie quarterbacks performed compared to how their teams passed in the prior year? David Carr, Tim Couch, and Kerry Collins took over expansion teams, but we can compare the passing stats of the other 45 rookie quarterbacks to the team stats from the prior season. To compare across eras, I am grading each individual and team relative to the league average each season.

Let’s start with Net Yards per Attempt. Ben Roethlisberger averaged 7.41 NY/A in 2004 when the league average was 6.14; therefore, Roethlisberger was at 121% of league average. Meanwhile, the 2003 Steelers under Tommy Maddox were at 99% of league average. For each of the 45 rookie quarterbacks, I plotted them in the graph below. The Y-axis shows how the quarterback performed as a rookie, while the X-axis shows how his team performed in the prior season. Because it makes sense to think of “up and to the right” as positive, the X-axis goes in reverse order. Take a look – I have an abbreviation for each quarterback next to his data point:

Remember, the higher (larger NY/A) and farther to the right (taking over a worse team), the more impressive (at least in theory) the rookie season. Obviously someone like Cam Newton (replacing Jimmy Clausen) has an advantage over a Jeff Garcia (replacing Steve Young) or Mark Sanchez (replacing Brett Favre), but the chart does help visualize how the rookie performed and the type of situation he entered.

Cam Newton stands out as excpetion, while even someone like Kyle Orton goes on the positive side of the ledger. Matt Ryan and Ben Roethlisberger both had excellent rookie years, but Ryan took over a worse team.

If we look at the rookie quarterbacks as measured by NY/A, Robert Griffin III ranked ahead of Russell Wilson who was ahead of Andrew Luck who was ahead of Brandon Weeden: however, the 2011 Redskins were ahead of the 2011 Seahawks who were ahead of the 2011 Colts who were ahead of the 2011 Browns. The fifth rookie quarterback, Ryan Tannehill, is a little tough to see, but he’s just a little bit to the left of Weeden on the graph: he was at 94% of the league average in ANY/A, a few tenths of a percentage point ahead of where the Matt Moore’s Dolphins were in 2011.

The worst quarterbacks? Blaine Gabbert and Ken Dorsey are the farther from the arrow, meaning they were the farthest (on the negative side) compared to the prior year’s performance. For Dorsey, he was replacing a very good quarterback in Jeff Garcia, while Gabbert failed to meet even the average standards set by David Garrard. Grading a rookie quarterback by how he performed in one statistic to the team’s prior quarterback is far too simplistic, but this is just another data point in the argument that it’s time for Jacksonville to give up on Gabbert. Some have leaped praise on Eagles rookie Nick Foles, but it’s worth remembering that his performance was still far below where the team was in 2011 (of course, the same could be said for Michael Vick).

What if we do the same thing with touchdown rate instead of NY/A?

A picture can say a thousand words, or just three: Russell Wilson – damn. The 2011 Seahawks threw touchdowns on just 2.9% of their passes, ranking 26th in the league. Russell Wilson had a 6.6% touchdown rate, ranking him second in the NFL behind Aaron Rodgers. Touchdown rate isn’t subject to the same inflation as completion percentage — in fact, touchdown rate is lower now than it was in the pre-merger NFL. That makes this stat pretty impressive: Since 1970, Dan Marino (6.8%) is the only true rookie ^[1]Mark Rypien had an 8.7% touchdown rate in his rookie season in 1988, but it comes with a couple of asterisks. He threw only 208 passes, below the qualifying amount, and he was drafted in 1986 and … Continue reading with a higher touchdown percentage than Wilson.

One more chart, this time looking at interception rates. Note that as in the prior charts, above 100% is better, which means fewer interceptions than average. Because of how the interception rate is skewed, I’ve had to modify the proportion of the X-axis relative to the Y-axis to make it easier to view:

Robert Griffin III’s interception rate was absurdly good — it is essentially ‘off the charts’ good — and comes a year after Washington ranked in the bottom quarter of the league in that metric. However, because interception rates are notoriously random and since a lot of really good quarterbacks struggled with interceptions as rookies, I hesitate to put too much emphasis on the metric. Interception rate is also an area where Nick Foles excelled, but so did Charlie Batch and Neil O’Donnell.

I love the graphs, but sometimes you just want to look at the numbers. The table below shows all 45 rookies in this study, and lists the team’s NY/A, Touchdown Rate, and Interception Rate in the year prior to the rookie quarterback starting, along with those same metrics during the player’s rookie season. As above, all the stats are listed as percentages of league average. The final three columns show the differentials. It is sorted by the NY/A differential. Since Newton was 9% above average in NY/A and the 2010 Panthers were 30% below average in NY/A, Newton is credited with a 39 in the NY/A differential column.