≡ Menu

One of the very first posts at Football Perspective measured how various passing stats were correlated with wins.  One of the main conclusions from that post was that passer rating, because of its heavy emphasis on completion percentage and interception rate, was not the ideal way to measure quarterback play. But what about ESPN’s Total QBR, a statistic invented specifically to improve on — and supersede — traditional passer rating?

As a reminder, we can’t simply correlate a statistic with wins to determine the utility of that metric. The simplest way to remember this is that 4th quarter kneeldowns are highly correlated with wins. Just because you notice it’s raining when the ground is wet doesn’t mean a wet ground causes rain; i.e., just because two variables are correlated doesn’t mean variable A leads to variable B (alternatively, variable B could lead to variable A, variable C could lead to both variable A and B, or the sample size could be too small to determine any legitimate causal relationship). That said, it at least makes sense to begin with a look at how various statistics have correlate with wins.

The Sample Set

Throughout this post, I will be looking at a set of quarterback data consisting of the 152 quarterback seasons from 2006 to 2013 where the player had at least 14 games with 20+ action plays. Games where the quarterback had fewer than 20 plays were excluded, but the quarterback was still included if he otherwise had 14 such games.

The next step was to sum the weekly quarterback data on various metrics, including wins, and create season data.1 This allowed me to measure the correlation between a quarterback’s statistics over those 14+ games with that player’s winning percentage in those games.

As it turns out, ESPN’s Total QBR is very highly correlated with wins, with a 0.68 correlation coefficient.2 This is to be expected; after all, Total QBR is based off Expected Points Added on the team level, which generally tracks wins and losses. The second most correlated statistic with wins was Adjusted Net Yards per Attempt, my favorite non-proprietary quarterback metric. After ANY/A, both traditional passer rating and touchdowns per attempt were the next most correlated statistics with wins (after all, this is only a step or two away from saying scoring points is correlated with wins). In another unsurprising result, passing yards had almost no correlation with wins, while pass attempts had a slight negative correlation (as any Game Scripts observer would know).  Take a look:

Stat
CC
ESPN QBR0.68
ANY/A0.57
Passer Rating0.56
TD/Att0.54
NY/A0.46
Yd/Att0.45
INT/Att-0.43
Cmp%0.33
Sack Rate-0.21
Pass Yds0.16
Attempts-0.10

When ESPN first introduced QBR, I wrote that I was intrigued by the possibility of this metric, but frustrated that the specific details of the formula remained confidential. At the time, a clutch weight feature was included in the calculations, which made the metric more of a retrodictive statistic than a predictive one. Since then, ESPN has tweaked the formula several times, and the clutch weight has been capped.3 ESPN is not engaged in academia, so I understand why they have not published all the fine print; as a researcher, I’m still frustrated by that decision. Still, with 8 years of QBR data now publicly available, we can answer two questions: does Total QBR predict wins and how sticky is Total QBR?

We know that a high Total QBR is correlated with winning games, but we also know that there’s limited value to such a statement. If having a high Total QBR was one of the driving factor behind winning games, than such a variable would manifest itself in all games, not just the current one. So with my sample of 152 quarterbacks, I used a random number generator to divide each quarterback season into two half-seasons. Then I calculated each quarterback’s average in several different categories and measured the correlation between a quarterback’s average in such category in each half-season with his winning percentage in the other half-season.4 The results:

Stat
CC
ESPN QBR0.31
Wins0.28
ANY/A0.25
Passer Rating0.25
TD/Att0.24
NY/A0.22
Yd/Att0.20
Cmp%0.17
Pass Yds0.16
INT/Att0.15
Sack Rate0.14
Attempts0.06

As you would expect, all of our correlations are now smaller. But ESPN’s quarterback rating metric remains the best measure to predict wins. Perhaps even more impressively, Total QBR is more correlated with future wins than past wins. That’s pretty interesting. Another interesting result is that passer rating fares pretty well here, although much of the same issues as before remain with using correlation to derive causal direction.5

One other concept to remember is that our sample of quarterbacks consists of players who were heavily involved in at least 14 games. That makes sure Peyton Manning, Tom Brady, and Drew Brees are involved, while filtering out some Christian Ponder, Blaine Gabbert, and Brandon Weeden seasons. In other words, the data set contains more above-average quarterbacks than a random sample would, so we may not be able to justify certain conclusions from this study.

The other important question is whether Total QBR is predictive of itself; i.e., how “sticky” is this metric over different time periods. We know that interceptions are very random, and knowing a quarterback’s prior interception rate is not all that helpful in predicting his future interception rate. Where does Total QBR fall along those lines?

Stat
CC
Pass Yds0.69
Attempts0.66
Sack Rate0.56
Cmp%0.49
Passer Rating0.49
ESPN QBR0.47
ANY/A0.46
NY/A0.45
TD/Att0.43
Yd/Att0.42
Wins0.28
INT/Att0.2

The most “sticky” stats were passing yards and pass attempts, which in retrospect isn’t too surprising. These reflect the style of the offense, the talent of the quarterback, and the quality of the defense, so they should be easier to predict. The second-least sticky metric was wins, which also makes sense. After that, ESPN’s Total QBR fits in a narrow tier with most of our other metrics as being somewhat predictable.

Conclusion

The numbers here indicate that Total QBR is worth examining.  It may be a proprietary measure of quarterback play, but it’s not a subjective one with no basis in reality.  It does seem to be the “best” measure of quarterback play, although whether the tradeoff in accuracy for transparency is worth it remains up to each individual reader. One of the drawbacks I see in Total QBR is the failure to incorporate strength of schedule. And while no other traditional passer metric does, either, it’s also easy enough to make those adjustments. Hopefully, an SOS-adjusted Total QBR measure will be released soon (I’ll note that the college football version does include a strength-of-schedule adjustment).  My sense is that Total QBR is underutilized because (1) ESPN haters hate it because it’s an ESPN statistic, (2) it’s proprietary, and (3) analytics types disliked it because of the (now-eliminated) clutch rating.  While I would not suggest making it the only tool at your disposal, it does appear to deserve a prominent place in your toolbox.

  1. For ESPN’s QBR, I took a weighted average of the weekly QBR data. I should note that this is not the way ESPN calculates QBR. As explained to me via email, the scaling function that gives the “final” QBR on a 0-100 scale is nonlinear; as a result, you can’t just calculate a weighted average of the individual game QBR values to get season QBR. Instead, you need to have the “points per play”-like value that’s behind QBR and calculate the weighted average of that (and weight based on the capped clutch weights, not even the action plays), then re-apply the scaling function to get it back on the 0-100 scale. So while I’m recreating QBR, I’m not recreating it the way ESPN would. That disclaimer aside, I don’t think my method will bias these results. []
  2. As a reminder, the correlation coefficient is a measure of the linear relationship between two variables on a scale from -1 to 1. If two variables move in the same direction, their correlation coefficient will be close to 1. If two variables move with each other but in opposite directions (say, the number of hours you spend watching football and your significant other’s happiness level), then the CC will be closer to -1. If the two variables have no relationship at all, the CC will be close to zero. []
  3. When Dean Oliver was on the Advanced NFL Stats podcast, he noted that the formula was tweaked in 2013 so that the “clutch index” part of the formula was essentially capped. He added (beginning at 13:45): “The most clutch plays are ending up counting essentially the same as all other plays. [What] we ended up deciding is that for games that are out of reach, when quarterbacks are putting up meaningless statistics because they are playing against a defense that is not trying as hard because they know that the game is essentially over – so that you can get your yards but we’re just trying to run out the clock – so we still keep in a clutch weight reduction effectively, associated with garbage time. But there isn’t the increase in clutch weight associated with clutch plays.” []
  4. Then I did the entire process again, using a new set of random numbers, and averaged the results. []
  5. For example, because passer rating is biased towards high completion percentage and low interception rates, quarterbacks who play with the lead tend to produce strong passer ratings; well, playing with the lead is pretty highly correlated with winning, and winning is also correlated with future wins. []
{ 35 comments }

Wilson scrambles and gets credit for it.

Wilson scrambles and gets credit for it.

I hate passer rating. So do you. Everyone does, except for Kerry Byrne. Passer rating is stupid because it gives a 20-yard bonus for each completion, a 100-yard penalty for each interception, and an 80-yard bonus for each touchdown. In reality, there should be no (or a very small) weight on completions, a 45-yard weight on interceptions, and a 20-yard weight on touchdowns.

But let’s ignore those issues today. Reading Mike Tanier’s recent article inspired me to make see what passer rating would look like if we make three tweaks. I’m not going to change any of the weights in the formula, but just redefine the variables.

1) There’s no reason to exclude sack data from passer rating. I’ve stopped writing about how sacks are just as much (if not more) on the quarterback than other passing metrics, because I think that horse has been pretty well beaten by Jason Lisk and me.

2) Scrambles should be treated like completed passes. If Russell Wilson is about to be sacked, but escapes and run for 7 yards, why should that be treated any differently than if Peyton Manning is about to be sacked, but throws a seven-yard pass at the last second?

3) Lost Fumbles should be counted with interceptions. One could make a few advanced arguments here — we should use all fumbles instead of lost fumbles, or fumbles should be given an even stronger weight than interceptions (although consider that in light of this post), or that we should limit ourselves to just fumbles lost on passing plays. I’m going to play the simple card here, and just use lost fumbles data on the season level.

Passer rating consists of four metrics, all weighted equally: completions per attempt, yards per attempt, touchdowns per attempt, and interceptions per attempt. I will use the same formula with the same weights and the same variables, but redefine what those variables are. Here are the new definitions, with the additions in blue.

Completion percentage is now (Completions plus Scrambles) / (Pass Attempts plus Sacks plus Scrambles)

Yards per Attempt is now (Passing Yards plus Yards on Scrambles minus Sack Yards Lost) / (Pass Attempts plus Sacks plus Scrambles)

Touchdown Rate is now (Passing Touchdowns plus Touchdowns on Scrambles) / (Pass Attempts plus Sacks plus Scrambles)

Turnover Rate will replace Interception Rate in the formula, and is calculated as (Interceptions plus Fumbles Lost) / (Pass Attempts plus Sacks plus Scrambles)

The table below lists all of those metrics for the 32 quarterbacks who had enough pass attempts to qualify for the passer rating crown, along with Alex Smith and Colin Kaepernick, who just missed qualifying. Let’s look at the Robert Griffin III line.

He completed 258 of 393 pass attempts for 3200 yards, with 20 touchdowns and five interceptions. Those are the standard stats that make up passer rating, but he also took 30 sacks and lost 217 yards on those sacks. That makes Griffin’s numbers worse, but he also had 38 scrambles for 302 yards (which gets recorded as 38 completed passes for 302 yards), with no scramble touchdowns. Finally, he lost two fumbles. His new completion percentage is 64.2%, his new yards per attempt is 7.13, his new touchdown rate is 4.3%, and his turnover rate (which includes fumbles) is 1.5%. The final two columns show each quarterback’s passer rating under the normal system and their passer rating using these metrics, which I’ll call the FPPR for short.
[click to continue…]

{ 17 comments }

Correlating passing stats with wins

Which stats should be used to analyze quarterback play? That question has mystified the NFL for at least the last 80 years. In the 1930s, the NFL first used total yards gained and later completion percentage to determine the league’s top passer. Various systems emerged over the next three decades, but none of them were capable of separating the best quarterbacks from the merely very good. Finally, a special committee, headed by Don Smith of the Pro Football Hall of Fame, came up with the most complicated formula yet to grade the passers. Adopted in 1973, the NFL has used passer rating ever since to crown its ‘passing’ champion.

Nearly all football fans have issues with passer rating. Some argue that it’s hopelessly confusing; others simply think it just doesn’t work. But there are some who believe in the power of passer rating, like Cold Hard Football Facts founder Kerry Byrne. A recent post on a Cowboys fan site talked about Dallas’ need to improve their passer rating differential. Passer rating will always have supporters for one reason: it has been, is, and always will be correlated with winning. It is easy to test how closely correlated two variables are; in this case, passer rating (or any other statistic) and wins. The correlation coefficient is a measure of the linear relationship between two variables on a scale from -1 to 1. Essentially, if two variables move in the same direction, their correlation coefficient them will be close to 1. If two variables move with each other but in opposite directions (say, the temperature outside and the amount of your heating bill), the CC will be closer to -1. If the two variables have no relationship at all, the CC will be close to zero.

The table below measures the correlation coefficient of certain statistics with wins. The data consists of all quarterbacks who started at least 14 games in a season from 1990 to 2011:

Category
Correlation
ANY/A10.55
Passer Rating0.51
NY/A20.50
Touchdown/Attempt0.44
Yards/Att0.43
Comp %0.32
Interceptions/Att-0.31
Sack Rate-0.28
Passing Yards0.16
Attempts-0.14

As you can see, passer rating is indeed correlated with wins; a correlation coefficient of 0.51 indicates a moderately strong relationship; the two variables (passer rating and wins) are clearly correlated to some degree. Interception rate is also correlated with wins; there is a ‘-‘ sign next to the correlation coefficient because of the negative relationship, but that says nothing about the strength of the relationship. As we would suspect, as interception rate increases, wins decrease. On the other hand, passing yards bears almost no relationships with wins — this is exactly what Alex Smith was talking about last month:
[click to continue…]

  1. Adjusted Net Yards per Attempt, calculated as follows: (Passing Yards + 20*Passing Touchdowns - 45*Interceptions - Sack Yards Lost) / (Pass Attempts + Sacks) []
  2. Net Yards per attempt, which includes sack yards lost in the numerator and sacks in the denominator. []
{ 52 comments }