Adam Steele is back for another guest post. You can view all of Adam’s posts here. As always, we thank him for contributing.

---

When Does ANY/A Get It Wrong?

In Chase’s review of week one passing stats, I commented that the league’s passing efficiency was inflated by ANY/A in comparison to expected points added (EPA). Today’s post takes a deeper dive into the discrepancy between ANY/A and EPA and which quarterbacks look better in each metric.

While ANY/A is a good metric for quick and dirty analysis, it ignores a number of important variables for accurately measuring a quarterback’s passing efficiency. These variables include: first downs, failed completions, air yard / YAC splits, dropped passes, fumbles, the context of interceptions, and garbage time adjustments. My metric of choice to solve these issues is ESPN’s model of expected points (the primary component of Total QBR). I prefer ESPN’s version in particular because it attempts to isolate the quarterback’s share of credit for every play; the EPA numbers found at Pro Football Reference and Advanced Football Analytics hold the quarterback fully responsible for his team’s pass plays, which, in my opinion, is not much better than just using ANY/A.

In order to compare EPA to ANY/A, I divided pass EPA by dropbacks then converted EPA/A into an index stat using the same formula for ANY/A+. For those not familiar, index stats are scaled so a score of 100 is average and 15 points represents one standard deviation above or below that average. EPA data goes back to 2006 which gives us 439 qualifying seasons to compare. As you would suspect, these two variables are closely correlated (R^2 of 0.74) in the aggregate, but there will be many individual outliers. In the graph below, the X-Axis shows the ANY/A+ for each quarterback, while the Y-Axis shows the EPA/Attempt+ for that quarterback.

Below is the same information but with the full data. The table is sorted by the difference in index scores between the quarterbacks’ EPA/A+ and ANY/A+:

If someone asked you to rattle off QB seasons most underrated by ANY/A, I doubt Ryan Mallett’s name would ever pop up. But a closer look reveals the primary reason for Mallett topping this list. In recent years, three franchises have been allergic to YAC regardless of who’s playing QB: Houston, Baltimore, and Tampa Bay. In 2015 Ryan Mallett had the misfortune of playing for two of those teams! Not coincidentally, Brock Osweiler, Joe Flacco, and Jameis Winston also show up near the top because their poor Y/A was symptomatic of their receiver’s inability to create yards after the catch. (See also Chase’s old article on where Y/A gets it wrong.)

Peyton Manning and Matt Ryan appear to be systematically underrated by ANY/A. In Manning’s case, it’s because ANY/A ignores his dominant ability to pick up first downs, avoid sacks and fumbles, and complete passes downfield without the aid of YAC. Matt Ryan is a bit of a mystery, as he profiles as the type of QB who would be viewed neutrally by both advanced and boxscore stats, but EPA sees something special in him. Note that Ryan’s only two seasons scoring higher in ANY/A+ were his 2016 MVP campaign aided by a huge schematic edge and his 2018 season which contained considerable garbage time production.

If we sort the table by most overrated ANY/A seasons, a whole new cast of characters appears. Appropriately, Nick Foles’ fluke 2013 season stands alone at the top. I think the term “system QB” gets thrown around too much, but Foles’ dominant ANY/A in 2013 really was a product of Chip Kelly’s scheme. That was Kelly’s first year in the NFL and his offense proved to be an unsolvable riddle for defensive coordinators. As a result, Foles had the pleasure of throwing to more wide open receivers than I’ve ever seen, highlighted by his seven TD performance in Oakland where it looked like the Raiders defense was playing with nine men on the field. EPA recognizes Foles’ limited role in producing these preposterous numbers. To underscore the importance of scheme, Foles’ 2014 season is one of the most underrated ANY/A seasons in the study! Just one year later, after coaches had a full offseason to figure out Kelly’s madness, Foles’ traditional stats fell off a cliff despite him actually improving according to EPA/A.

There has been a lot of talk about Aaron Rodgers’ decline and whether he’s overrated by pundits who still name him as one of the league’s best quarterbacks. Well, according to EPA/A his entire career has been vastly overrated. Rodgers’ EPA/A+ has lagged behind his ANY/A+ in ten of his eleven seasons as the Green Bay starter; in five of those seasons he rated as a full standard deviation worse by EPA/A+. What would cause a superstar QB to have such divergent scores in these two metrics? I’d venture to guess that the biggest culprit is sacks. The analytics community has done a plethora of research on which players are responsible for a QB getting sacked, and the results overwhelmingly point the finger at the QB himself. I don’t know exactly how much blame ESPN’s model puts on the QB for taking a sack, but I do know that the penalty is steep (and rightfully so). Basically, EPA says that Rodgers’ drive killing sacks are more than enough to cancel out his historically low INT rate.

Finally, let’s take a step back and think about the NFL as a whole. It seems to be an accepted fact that 2018 was the most prolific passing season in NFL history; yards, completions, TD passes, passer rating, and ANY/A all reached historical levels last season. However, EPA/A paints a very different picture. 2018 was only the fifth most efficient passing season since 2006, trailing 2016, 2015, 2014, and 2009. These numbers support my long held theory that scheming and strategy have been the primary driver of the recent passing boom rather than improvement in quarterback skill. None of this is necessarily provable or falsifiable, but here’s my theory. Receivers have never been more open and able to gain yards after the catch. More passes are thrown behind the line of scrimmage than ever before. Two minute offense seems to be schematically way ahead of two minute defense. Prevent defenses during garbage time are softer than ever. Passing efficiency in low leverage situations has increased disproportionately compared to high leverage scenarios (Y/A in minus territory is way up while third down and red zone efficiency are static). It all adds up to an explosion of junk passing yards that boxscore metrics like ANY/A aren’t able to filter out. And no, this isn’t my first time questioning ANY/A.