Which stats should be used to analyze quarterback play? That question has mystified the NFL for at least the last 80 years. In the 1930s, the NFL first used total yards gained and later completion percentage to determine the league’s top passer. Various systems emerged over the next three decades, but none of them were capable of separating the best quarterbacks from the merely very good. Finally, a special committee, headed by Don Smith of the Pro Football Hall of Fame, came up with the most complicated formula yet to grade the passers. Adopted in 1973, the NFL has used passer rating ever since to crown its ‘passing’ champion.

Nearly all football fans have issues with passer rating. Some argue that it’s hopelessly confusing; others simply think it just doesn’t work. But there are some who believe in the power of passer rating, like Cold Hard Football Facts founder Kerry Byrne. A recent post on a Cowboys fan site talked about Dallas’ need to improve their passer rating differential. Passer rating will always have supporters for one reason: it has been, is, and always will be correlated with winning. It is easy to test how closely correlated two variables are; in this case, passer rating (or any other statistic) and wins. The correlation coefficient is a measure of the linear relationship between two variables on a scale from -1 to 1. Essentially, if two variables move in the same direction, their correlation coefficient them will be close to 1. If two variables move with each other but in opposite directions (say, the temperature outside and the amount of your heating bill), the CC will be closer to -1. If the two variables have no relationship at all, the CC will be close to zero.

The table below measures the correlation coefficient of certain statistics with wins. The data consists of all quarterbacks who started at least 14 games in a season from 1990 to 2011:

Category | Correlation |
---|---|

ANY/A^{1} | 0.55 |

Passer Rating | 0.51 |

NY/A^{2} | 0.50 |

Touchdown/Attempt | 0.44 |

Yards/Att | 0.43 |

Comp % | 0.32 |

Interceptions/Att | -0.31 |

Sack Rate | -0.28 |

Passing Yards | 0.16 |

Attempts | -0.14 |

As you can see, passer rating is indeed correlated with wins; a correlation coefficient of 0.51 indicates a moderately strong relationship; the two variables (passer rating and wins) are clearly correlated to some degree. Interception rate is also correlated with wins; there is a ‘-’ sign next to the correlation coefficient because of the negative relationship, but that says nothing about the strength of the relationship. As we would suspect, as interception rate increases, wins decrease. On the other hand, passing yards bears almost no relationships with wins — this is exactly what Alex Smith was talking about last month:

“I could absolutely [not] care less on yards per game. I think that is a totally overblown stat because if you’re losing games in the second half, guess what, you’re like the Carolina Panthers and you’re going no-huddle the entire second half. Yeah, Cam Newton threw for a lot of 300-yard games. That’s great. You’re not winning, though.”

If nothing else, Smith is correct that passing yards are not very correlated with wins. However, as you’ve surely heard before, **correlation does not imply causation**. Even if it did, such an implication would not tell you which way the causal arrow points: If you see a group of fireman putting out a fire, and you have never seen a fire not in proximity to fireman, that doesn’t mean that firemen cause fires. Often another variable is driving both forces, which explains the correlation. Falling asleep with your shoes on and waking up with a headache are surely correlated. But “drinking heavily” is the driving force behind both results; both variables (falling asleep with your shoes on; waking up with a headache) are a result of being drunk, and neither variable causes the other.

Being ahead late in games is strongly correlated with winning games, of course. And think what that means: quarterbacks who are ahead late in games play conservatively, which increases their completion percentage and decreases their interception rate. As it turns out, passer rating significantly overvalues those two statistics, which compounds the problem. Conversely, teams trailing in games are likely to lose, and also likely to throw riskier passes, lowering completion percentages and increasing interception rates. As a result, the driving factor behind the correlation between passer rating and wins is a third factor that causes both: leading late in games.

Of course, that’s just a theory. So how do we test it? If having a high passer rating is the driving factor behind winning games, than such a variable would manifest itself in all games, not just the current one. As before, I looked at all quarterbacks from 1990 to 2011 and noted those quarterbacks who started at least 14 games in a season. Then, I randomly^{3} divided each quarterback season into two half-seasons. I calculated each quarterback’s rating in each category and measured the correlation between a quarterback’s rating in each half-season with their number of wins in the other half-season. The results:

Category | Wins in Same Half-Year | Wins in Other Half-Year |
---|---|---|

ANY/A | 0.55 | 0.28 |

Passer Rating | 0.51 | 0.26 |

NY/A | 0.50 | 0.28 |

Touchdowns/Attempt | 0.44 | 0.25 |

Yards/Attempt | 0.43 | 0.26 |

Comp % | 0.32 | 0.20 |

Interceptions/Attempt | -0.31 | -0.08 |

Sack Rate | -0.28 | -0.14 |

Passing Yards | 0.16 | 0.15 |

Attempts | -0.14 | -0.01 |

The variable ‘interceptions per attempt’ drops to irrelevance. All of the statistics become less correlated, which makes sense: predicting the future is really difficult. So what can we take away from this?

Passer rating is made up of four variables: completion percentage, yards per attempt, interception rate and touchdown rate. Another way to think of passer rating is as follows^{4}:

2.0833 + 0.8333*CMP% + 4.1667*Y/A – 4.1667*INT_Rate + 3.3333*TD_Rate

That might look just to you, so just focus on the coefficients. The passer rating formula says one more interception per 100 pass attempts is equally as bad as 100 fewer yards on 100 pass attempts; both will decrease your rating by 4.1 points. Take a look at three hypothetical passers, all with the same passer rating:

Comp% | Y/A | INT Rate | TD Rate | Rating |
---|---|---|---|---|

60.1 | 7.2 | 2.9 | 4.3 | 84.3 |

60.1 | 9.2 | 4.9 | 4.3 | 84.3 |

60.1 | 5.2 | 0.9 | 4.3 | 84.3 |

The first row represents the league average from the 2011 season. If a quarterback mirrored those numbers, but instead averaged an incredible 9.2 yards per attempt — but also threw 2 more interceptions per 100 passes — he’d have the same rating. The converse holds as well: all three sets of numbers are equal, according to passer rating.

From an explanatory standpoint, that’s not as bad (although, of course, still bad). As we saw in the first table, interceptions are correlated with winning. But because interceptions are both random and heavily impacted by game situation, they’re a terrible statistic to use for predictive purposes. Yards per attempt is much more predictive, and it is absurd to equate an interception to 100 yards of offense.

Another thing to note: the interception variable is 5 times as large as the completion variable. What does that mean? Giving a quarterback five more completions per 100 passes — while not changing his amount of yards, touchdowns or interceptions — is equivalent to giving him 100 extra yards or one fewer interception. According to passer rating, a completion, by itself, is worth 20 yards in the formula.

Since passer rating is just an average of four statistics, there’s a better way to analyze the four inputs. You can run a multiple regression analysis to see how much weight should be placed on each variable, with future wins (i.e., wins in the other half-season) as the output. The P-values on the completion percentage and interception rate variables were *not* significant at the 1%, 5% or 10% levels. Essentially, this means that **for predictive purposes, two of the four inputs in passer rating are meaningless.**

Of course, a careful analysis of the earlier table would make that clear. The CC between each stat and future wins was 0.26 for yards per attempt, 0.25 for touchdowns per attempt, 0.20 for completion percentage, and -0.08 for interception rate. But the CC between passer rating and wins was only 0.26, the same as it was for yards per attempt. Adding in the interception and completion percentage variables does nothing to make the formula more predictive.

What does make the formula predictive? Using net yards per attempt — which deducts sacks from a passer’s production — is the simplest and best way to predict future performance. That’s why when looking at which quarterback will perform the best in the future, NY/A is my favorite statistic. When analyzing past quarterbacks, I prefer Adjusted Net Yards per Attempt, which gives a 45-yard penalty for interceptions and a 20-yard bonus for touchdowns. That’s more useful as an explanatory statistic than NY/A, but is not as helpful in predicting the future.

Passer rating? To the extent it is based around yards per attempt (and to some extent, touchdown rate), it is useful. On the margins, it certainly does the job, and when comparing quarterbacks with similar interception rates and completion percentages, it can be effective. But with a significant 100-yard penalty on interceptions and a 20-yard bonus for completions, passer rating only really works when you stack the deck: and that’s precisely why passer rating will always be correlated with wins. That doesn’t make it a useful stat, though.

- Adjusted Net Yards per Attempt, calculated as follows: (Passing Yards + 20*Passing Touchdowns - 45*Interceptions - Sack Yards Lost) / (Pass Attempts + Sacks) [↩]
- Net Yards per attempt, which includes sack yards lost in the numerator and sacks in the denominator. [↩]
- I mean randomly: I did not use home/road, first half/second half, or even/odd to split the seasons, but rather assigned random numbers to each game. [↩]
- This formula will work on the season level and nearly all the time with smaller samples. However, passer rating does have minimum and maximum limits, which are not incorporated into the formula, which could lead to discrepancies when examining passers with a small number of attempts [↩]

{ 19 comments… read them below or add one }

Good explanations and well thought out. I really enjoyed the comparisons between the coefficients and working through the whole formula piece by piece. I also think you meant to say “correlation does not imply causation” when you wrote the same phrase without the ‘not’ in bold.

Thanks, Matt. Yes, that was an important ‘not’ to not leave out. I’ve fixed the post.

So can we use this information to deduce anything about the contribution a QB’s performance actually makes towards wins and losses?

The correlations in “wins in other half-year” all look fairly low to me. Does that mean that maybe a QB’s ability doesn’t have as much to do with wins as we often think? What it kind of looks like to me is that generally, a QB’s ability doesn’t lead to a lot of wins, but if a QB is able to avoid sacks and interceptions in a game, that is the biggest factor towards a win or loss.

Interceptions have long been discussed as huge factors in wins and losses. But all the statistical analysis I’ve been reading about the NFL over the past 5 years or so leads me to believe that sacks are a much larger factor in success than conventional wisdom leads to believe.

The correlations are low, Richie. But remember that a team’s passing offense is just one part of a game; rushing offense, pass defense, rush defense and special teams all count, too. And, of course, sometimes good quarterbacks play poorly and bad quarterbacks play well, which will lower the relationship between a quarterback’s ability and future wins.

As far as how can I QB maximize his team’s odds of winning? I’d still say it’s by achieving as high a ANY/A ratio as you can (putting aside more obviously things like scoring a touchdown on every play). Remember, a QB’s ability to avoid INTs is strongly tied to what the score of the game is.

To be sure, INTs are huge factors in wins and losses. But so are 45-yard pass plays. I don’t know if you can say sacks are more important than interceptions, but I think the numbers are clear that sack rates are much more consistent than interception rates.

Just in case someone is still paying attention to this old article…

Someone has started doing some research on sacks and how they relate to victory. I haven’t completed another seasonal breakdown since I wrote this, but I expect I will be able to start correlating pressures to interceptions watching games this year, and that will help my data for sure.

http://www.coldhardfootballfacts.com/content/quarterback-attack-the-impact-sacks-nfl-games/23501/

For some all-time great squads that lived by the sack, look at my pieces on the ’91 Skins and mid-’80s Bears.

Oh, and if there’s a way to turn on “notify of replies to post” here I’d love to be able to.

I’m not a web designer by any means, so I’m always open to advice. I think I just activated a plug-in to do just that, Richie. You’ll have to let me know.

No, I still don’t think it’s working for me.

Test.

Hi Chase: Great article, I enjoyed the read. Out of curiosity, what was the R^2 (Goodness of Fit) on the regression? Thanks (and apologizes if the R^2 was in the article and I missed it)!

Thanks for the comment, Camden. The R^2 was 0.28. It’s difficult to come up with a model that ignores much of the game and get any better than that.

Does yards after the catch factor into this? A QB can only get the ball to his supporting cast, but talented players who can break tackles and make people miss can really help a QB’s stats.

Yards after the catch are counted in the same way as yards before the catch (or yards in the air) in passer rating, and most passing metrics. Brian Burke does calculate a statistic he calls ‘Air Yards’ which you might be interested in: http://www.advancednflstats.com/2012/01/air-yards-2011.html

Curious if Chase or anyone else has used NY/A to predict team performance. Something like using a team’s NY/A differential (off NY/A – def NY/A) to predict how well they’ll do later in a season.

I would like to see the information you speak of on sacks, Richie. I am not disagreeing based upon me having any facts that say you are wrong, I just find sacks can be a highly overrated stat. That is what makes me curious about what you said about sacks having to be more important an indicator for wins than thought (or something to that effect).

I have always compared a sack to the stolen base in baseball. While both can be very effective and help a team immensely, sometimes it seems, they have no value. For instance, If I steal a base and then three outs occur while I remain at second (or third, whichever I stole, not counting home obviously), did I help my team in any way?

If I sack the quarterback and the team goes on to get a first down on that set of downs, did I help my team really? God forbid you hurt the quarterback, but other than that, if the team gets a first down on that set of downs, isn’t the sack irrelevant.

Some could argue the mind set of a team getting to your quarterback over and over could be debilitating mentally. However, an easier argument would come in favor of the team who continually overcomes the sacks by getting first downs.

I have always thought that should be tracked (levels of sacks). It is one thing to sack a quarterback on third down and knock them out of field goal range and force a punt as compared to having a team connect on a 40 yard td pass on third and 15 after getting sacked on first down.

You can say this about a lot of things, though. Imagine a RB-carry on first down. He’s hit in the bakfield but because he’s good he fights his way through it and turn that potential loss of 2 into a gain of 8. I think we can both agree that he helped his team immensely, no matter if the next three plays are two incomplete passes and a punt. Future events doesn’t affect the value of present production.

I think what Richie means with sacks being somehow a stronger statistic, is that he thinks it’s a stronger PREDICTOR. Certainly interceptions are more valuable than sacks – no argument here. However, if you present me with two defenses Defense A who leads the league in interceptions and Defense B who leads the league in sacks, Richie (and I) conjecture that Defense B is more likely to help it’s team win in the future, even though Defense A have been more valuable so far.

Enjoyed the article Chase.

Very nice read some great statistical info. There is no question interceptions are far more valuable than sacks and they are predictable. if you see a passer who has a low ypa then defences whom im sure have this info can begin to sit on routes and come up with int’s this is now just theory but im gonna test it on sunday ints drive t.o.p. and number of plays which hugley impact games t.o.p. winner wins most games since the sack forces change of posession so rarley (would love if someone could find that stat) I.N.T. remains king

Would it be fair to force maximum correlation out of passer rating (replacing Y/A with ANY/A) by adjusting weighting of the 4 inputs until correlation is maximized? Would the end result of that be that everything gets weighted at zero besides ANY/A because that is already the strongest indicator, or can some combination of the four produce better results? Or is the whole idea of forcing correlation some kind of statistical no-no? I can imagine if you opened up all passer stats to a computer and asked it to find the best combination it would simply return with “wins correlates 1.0 to wins.”

{ 29 trackbacks }