Which stats should be used to analyze quarterback play? That question has mystified the NFL for at least the last 80 years. In the 1930s, the NFL first used total yards gained and later completion percentage to determine the league’s top passer. Various systems emerged over the next three decades, but none of them were capable of separating the best quarterbacks from the merely very good. Finally, a special committee, headed by Don Smith of the Pro Football Hall of Fame, came up with the most complicated formula yet to grade the passers. Adopted in 1973, the NFL has used passer rating ever since to crown its ‘passing’ champion.
Nearly all football fans have issues with passer rating. Some argue that it’s hopelessly confusing; others simply think it just doesn’t work. But there are some who believe in the power of passer rating, like Cold Hard Football Facts founder Kerry Byrne. A recent post on a Cowboys fan site talked about Dallas’ need to improve their passer rating differential. Passer rating will always have supporters for one reason: it has been, is, and always will be correlated with winning. It is easy to test how closely correlated two variables are; in this case, passer rating (or any other statistic) and wins. The correlation coefficient is a measure of the linear relationship between two variables on a scale from -1 to 1. Essentially, if two variables move in the same direction, their correlation coefficient them will be close to 1. If two variables move with each other but in opposite directions (say, the temperature outside and the amount of your heating bill), the CC will be closer to -1. If the two variables have no relationship at all, the CC will be close to zero.
The table below measures the correlation coefficient of certain statistics with wins. The data consists of all quarterbacks who started at least 14 games in a season from 1990 to 2011:
As you can see, passer rating is indeed correlated with wins; a correlation coefficient of 0.51 indicates a moderately strong relationship; the two variables (passer rating and wins) are clearly correlated to some degree. Interception rate is also correlated with wins; there is a ‘-’ sign next to the correlation coefficient because of the negative relationship, but that says nothing about the strength of the relationship. As we would suspect, as interception rate increases, wins decrease. On the other hand, passing yards bears almost no relationships with wins — this is exactly what Alex Smith was talking about last month:
“I could absolutely [not] care less on yards per game. I think that is a totally overblown stat because if you’re losing games in the second half, guess what, you’re like the Carolina Panthers and you’re going no-huddle the entire second half. Yeah, Cam Newton threw for a lot of 300-yard games. That’s great. You’re not winning, though.”
If nothing else, Smith is correct that passing yards are not very correlated with wins. However, as you’ve surely heard before, correlation does not imply causation. Even if it did, such an implication would not tell you which way the causal arrow points: If you see a group of fireman putting out a fire, and you have never seen a fire not in proximity to fireman, that doesn’t mean that firemen cause fires. Often another variable is driving both forces, which explains the correlation. Falling asleep with your shoes on and waking up with a headache are surely correlated. But “drinking heavily” is the driving force behind both results; both variables (falling asleep with your shoes on; waking up with a headache) are a result of being drunk, and neither variable causes the other.
Being ahead late in games is strongly correlated with winning games, of course. And think what that means: quarterbacks who are ahead late in games play conservatively, which increases their completion percentage and decreases their interception rate. As it turns out, passer rating significantly overvalues those two statistics, which compounds the problem. Conversely, teams trailing in games are likely to lose, and also likely to throw riskier passes, lowering completion percentages and increasing interception rates. As a result, the driving factor behind the correlation between passer rating and wins is a third factor that causes both: leading late in games.
Of course, that’s just a theory. So how do we test it? If having a high passer rating is the driving factor behind winning games, than such a variable would manifest itself in all games, not just the current one. As before, I looked at all quarterbacks from 1990 to 2011 and noted those quarterbacks who started at least 14 games in a season. Then, I randomly3 divided each quarterback season into two half-seasons. I calculated each quarterback’s rating in each category and measured the correlation between a quarterback’s rating in each half-season with their number of wins in the other half-season. The results:
Wins in Same Half-Year
Wins in Other Half-Year
The variable ‘interceptions per attempt’ drops to irrelevance. All of the statistics become less correlated, which makes sense: predicting the future is really difficult. So what can we take away from this?
Passer rating is made up of four variables: completion percentage, yards per attempt, interception rate and touchdown rate. Another way to think of passer rating is as follows4:
2.0833 + 0.8333*CMP% + 4.1667*Y/A – 4.1667*INT_Rate + 3.3333*TD_Rate
That might look just to you, so just focus on the coefficients. The passer rating formula says one more interception per 100 pass attempts is equally as bad as 100 fewer yards on 100 pass attempts; both will decrease your rating by 4.1 points. Take a look at three hypothetical passers, all with the same passer rating:
The first row represents the league average from the 2011 season. If a quarterback mirrored those numbers, but instead averaged an incredible 9.2 yards per attempt — but also threw 2 more interceptions per 100 passes — he’d have the same rating. The converse holds as well: all three sets of numbers are equal, according to passer rating.
From an explanatory standpoint, that’s not as bad (although, of course, still bad). As we saw in the first table, interceptions are correlated with winning. But because interceptions are both random and heavily impacted by game situation, they’re a terrible statistic to use for predictive purposes. Yards per attempt is much more predictive, and it is absurd to equate an interception to 100 yards of offense.
Another thing to note: the interception variable is 5 times as large as the completion variable. What does that mean? Giving a quarterback five more completions per 100 passes — while not changing his amount of yards, touchdowns or interceptions — is equivalent to giving him 100 extra yards or one fewer interception. According to passer rating, a completion, by itself, is worth 20 yards in the formula.
Since passer rating is just an average of four statistics, there’s a better way to analyze the four inputs. You can run a multiple regression analysis to see how much weight should be placed on each variable, with future wins (i.e., wins in the other half-season) as the output. The P-values on the completion percentage and interception rate variables were not significant at the 1%, 5% or 10% levels. Essentially, this means that for predictive purposes, two of the four inputs in passer rating are meaningless.
Of course, a careful analysis of the earlier table would make that clear. The CC between each stat and future wins was 0.26 for yards per attempt, 0.25 for touchdowns per attempt, 0.20 for completion percentage, and -0.08 for interception rate. But the CC between passer rating and wins was only 0.26, the same as it was for yards per attempt. Adding in the interception and completion percentage variables does nothing to make the formula more predictive.
What does make the formula predictive? Using net yards per attempt — which deducts sacks from a passer’s production — is the simplest and best way to predict future performance. That’s why when looking at which quarterback will perform the best in the future, NY/A is my favorite statistic. When analyzing past quarterbacks, I prefer Adjusted Net Yards per Attempt, which gives a 45-yard penalty for interceptions and a 20-yard bonus for touchdowns. That’s more useful as an explanatory statistic than NY/A, but is not as helpful in predicting the future.
Passer rating? To the extent it is based around yards per attempt (and to some extent, touchdown rate), it is useful. On the margins, it certainly does the job, and when comparing quarterbacks with similar interception rates and completion percentages, it can be effective. But with a significant 100-yard penalty on interceptions and a 20-yard bonus for completions, passer rating only really works when you stack the deck: and that’s precisely why passer rating will always be correlated with wins. That doesn’t make it a useful stat, though.
- Adjusted Net Yards per Attempt, calculated as follows: (Passing Yards + 20*Passing Touchdowns - 45*Interceptions - Sack Yards Lost) / (Pass Attempts + Sacks) [↩]
- Net Yards per attempt, which includes sack yards lost in the numerator and sacks in the denominator. [↩]
- I mean randomly: I did not use home/road, first half/second half, or even/odd to split the seasons, but rather assigned random numbers to each game. [↩]
- This formula will work on the season level and nearly all the time with smaller samples. However, passer rating does have minimum and maximum limits, which are not incorporated into the formula, which could lead to discrepancies when examining passers with a small number of attempts [↩]