## Guest Post: Questioning ANY/A

Adam Steele is back for another guest post. And, as always, we thank him for that. You can view all of Adam’s posts here.

Within the analytics community, we seem to have reached a consensus that ANY/A is the best box score metric for measuring passing efficiency. Over at the Intentional Rounding blog, Danny Tuccitto tested the validity of ANY/A using a technique called Confirmatory Factor Analysis. You can read his three part analysis here, here, and here. Essentially, he discovers that Y/A and TD % are valid statistics for measuring QB quality, while sack % and INT % are not. At first I was skeptical, but after some pondering I came up with a half-baked theory of why this might be true:

As we evaluate the potential for an athlete to succeed in professional sports, there are two kinds of statistics: Qualifying and Disqualifying. In the case of quarterbacks, I define a qualifying statistic as a minimum threshold the player must meet to even be considered NFL worthy. If we deconstruct ANY/A into its four components, Y/A and TD % emerge as qualifying statistics. In today’s NFL, I estimate that a QB must possess a true talent level of at least 6.0 Y/A and 2.5 TD % to deserve a roster spot. There are very few people in the world who can reach those thresholds against NFL caliber defenses (my best guess is around 100). With these two simple statistics, we’ve already weeded out the vast majority of quarterbacks from ever playing in the NFL.

Next, we turn to sack % and INT %, which are disqualifying statistics. By themselves, neither of these skills qualify a QB to play in the NFL. Anybody can avoid sacks or interceptions if they’re not worried about gaining yards. However, the inability to avoid sacks or interceptions will disqualify a QB from the NFL, regardless of how high his Y/A and TD % might be. I estimate these limits as roughly a true talent 12% sack rate and 4.5% INT rate. The population of quarterbacks who can stay under these limits AND perform above the minimum Y/A and TD % is very small. In most years, there aren’t enough of these QB’s to fill the 32 NFL starting spots. Among quarterbacks who receive significant NFL playing time, there is a strong survivorship bias for the disqualifying statistics of sack % and INT %, as the quarterbacks who make too many negative plays have already been weeded out of the sample. Given that Y/A and TD % are far rarer skills with no upper limits, these two statistics are the true measuring stick at the NFL level.

To test this theory, I created a very simple metric called Positive Yards Per Attempt (PY/A). It’s just passing yards plus a 20 yard bonus for touchdowns, divided by pass attempts (which does not sacks). I then converted PY/A into a value metric by measuring it relative to league average (RPY/A)1 and VALUE above average by multiplying RPY/A by attempts. We already have these variations of ANY/A (that is, RANY/A and VALUE), so comparing the two metrics is very straightforward. Since the merger, there have been 1,423 QB seasons of with least 200 dropbacks. This table lists the top 100 seasons of PY/A VALUE, as well as the ANY/A VALUE and rankings for these players. The “Diff” column signifies the gap in ranking between the the two metrics, with a positive number indicating a QB who is favored by PY/A and negative number favoring ANY/A.

RankQuarterbackTeamYearDpbkRPY/AVALUERANY/AVALUERankDiff
1Peyton ManningIND20045103.2916374.27217621
2Dan MarinoMIA19845772.8315984.0923591-1
3Aaron RodgersGNB20115383.0815443.6193652
4Kurt WarnerSTL20015842.714722.2913392521
5Kurt WarnerSTL19995282.914483.22170161
7Peyton ManningDEN20136772.0513483.1121043-4
8Kurt WarnerSTL20003673.6512652.8210357668
9Lynn DickeyGNB19835242.5512321.71898112103
10Steve YoungSFO19944922.6712302.961454155
11Steve YoungSFO19934932.6412182.5212413524
12Ken StablerOAK19763103.912123.4811504533
13Daunte CulpepperMIN20045942.1611812.471468130
14Boomer EsiasonCIN19884183.0411782.8511924026
15Chris ChandlerATL19983723.5511612.35876119104
16Drew BreesNOR20116811.7511472.4216507-9
17Randall CunninghamMIN19984452.6911433.32147912-5
19Bert JonesBAL19763723.0811283.8415259-10
21Drew BreesNOR20095342.1110842.74146514-7
22Daunte CulpepperMIN20005082.2410612.1310836240
25Joe MontanaSFO19894192.589963.161322272
26Tony RomoDAL20075441.849551.79279973
27Aaron RodgersGNB20145481.839512.59142118-9
28Mark RypienWAS19914282.249443.25139120-8
29Steve YoungSFO19985651.829412.0511564314
30Steve YoungSFO19924312.39253.33143617-13
31Jim KellyBUF19915051.959231.8392510069
32Ben RoethlisbergerPIT20095561.829211.5284513098
33Nick FolesPHI20133452.869083.371162429
34Peyton ManningIND20054701.999042.76130029-5
35Brett FavreGNB19956031.568911.911474611
36Tony RomoDAL20144652.028781.9891910670
37Drew BreesNOR20086481.388761.92124234-3
38Steve BeuerleinCAR19996211.538741.6410197941
39Dan FoutsSDG19823422.038573.07134223-16
40Roger StaubachDAL19733292.778462.09735169129
41Aaron RodgersGNB20095911.548321.8811115918
42Ken AndersonCIN19754092.058253.04132526-16
43Matt SchaubHOU20096081.418221.861130529
44Dan FoutsSDG19854481.918192.229938440
45Ken StablerOAK19743282.458093.231128549
46Jeff GeorgeMIN19993572.458081.88672193147
47Boomer EsiasonCIN19864951.728072.1410606922
48Peyton ManningIND20005911.418052.08123236-12
49Dan MarinoMIA19866401.298022.12135522-27
50Peyton ManningDEN20126041.378012.02122237-13
51Jim EverettRAM19895471.547971.9810826312
52Eli ManningNYG20116171.357961.69868735
53Warren MoonHOU19906201.367952.08128732-21
54Donovan McNabbPHI20045011.687902.3115444-10
55Peyton ManningIND20065711.417872.63150310-45
57Joe NamathNYJ19723352.237712.2179114689
59Vinny TestaverdeBAL19965831.397651.27743163104
60Steve YoungSFO19973912.157642.3692210444
61Drew BreesNOR20136871.177631.7116541-20
62Aaron RodgersGNB20105061.617631.8291910745
63Joe MontanaSFO19844541.767593.02137021-42
64Brett FavreGNB19975381.477551.7292310238
65Drew BreesNOR20126961.127511.2989511449
66Steve YoungSFO19912922.677463.1692210337
67Steve McNairTEN20034191.857412.67111958-9
68Trent GreenKAN20045881.337371.4585612658
69Brett FavreMIN20095651.387352.03114447-22
71Drew BreesNOR20065721.327292.28130428-43
72Tony RomoDAL20095841.327281.95114049-23
73Brett FavreGNB20015321.437281.881003818
74Neil LomaxSTL19846091.297251.76107166-8
75Tony RomoDAL20063582.147231.85662200125
76Peyton ManningIND20095811.267221.93112057-19
77Aaron RodgersGNB20126031.37191.4487112144
78Peyton ManningIND20075361.397151.839798911
79Ben RoethlisbergerPIT20052912.647082.22647208129
80Ken AndersonCIN19743642.027072.459519414
81Dan FoutsSDG19816281.157022.37148611-70
82Dan FoutsSDG19806211.197011.69104871-11
83Jeff GarciaSFO20005851.257012.21129031-52
84Ben RoethlisbergerPIT20074511.736991.06476299215
85Trent GreenKAN20024961.476931.7285412742
86Peyton ManningDEN20146141.156881.59979882
87Ben RoethlisbergerPIT20146411.126841.75112156-31
88Craig MortonDEN19814301.86751.06455318230
89Peyton ManningIND20035841.176642.22129430-59
90Trent GreenKAN20035431.256541.95105670-20
91Peyton ManningIND19995471.226511.94106267-24
92Brett FavreGNB19965831.196481.5288911624
93Dan FoutsSDG19833541.96452.3884313239
94Greg LandryDET19712902.316442.13660201107
95Carson PalmerCIN20055281.266441.97104174-21
96Roger StaubachDAL19712342.866433.9799185-11
97Carson PalmerCIN20065561.236421.4681214043
98Joe MontanaSFO19874201.41640295893-5
100Dan FoutsSDG19784031.676362.1285212828

This list makes a strong case for the validity of PY/A. It’s populated by the greatest QB seasons of all time at the top, and filled out by a number of other notably great and very good seasons. There are a few head scratchers (most notably Lynn Dickey at #9), but for the most part it’s a very credible list that closely mirrors the ANY/A rankings. That’s the point, really. When we remove sacks and interceptions from ANY/A, it doesn’t lose much accuracy, if any. At first glance, I was concerned that PY/A systematically overrates certain quarterbacks and underrates others. That’s probably true to a certain degree. However, I would argue that ANY/A has the same issue, except it’s a different set of quarterbacks who are over- and underrated by it. The true balance almost certainly lies somewhere in between the two metrics. FWIW, the correlation between RPY/A and RANY/A is a robust 0.877, with an r-squared of 0.769.

Now lets look at the other end of the spectrum – the 100 worst PY/A VALUE seasons since 1970.

Rank TeamYearDpbkRPY/AVALUERANY/AVALUERankDiff
1423Derek CarrOAK2014623-2.02-1209-1.36-8481395-28
1422Drew BledsoeNWE1995659-1.71-1086-1.09-7161366-56
1421Jon KitnaCIN2001606-1.67-972-1.48-8981408-13
1420Chris WeinkeCAR2001566-1.79-964-1.5-8481396-24
1419Joey HarringtonDET2003563-1.67-928-1.38-7791380-39
1418Kyle BollerBAL2004499-1.93-894-1.51-7551374-44
1417Blaine GabbertJAX2011453-2.16-894-2.28-103214192
1416Jack TrudeauIND1986446-2.14-893-1.96-8741405-11
1415Vince EvansCHI1981459-2.02-883-1.78-8181391-24
1414Ryan FitzpatrickCIN2008410-2.23-828-2.18-8921407-7
1413Archie ManningNOR1975387-2.23-803-2.76-113914229
1411Mark RypienWAS1993335-2.47-788-2-6711354-57
1410Bobby HoyingPHI1998259-3.46-775-3.94-102014188
1409Kordell StewartPIT1998491-1.68-769-1.78-8731404-5
1408Kyle OrtonCHI2005398-2.05-753-2.19-8721403-5
1407Jimmy ClausenCAR2010332-2.51-749-2.8-93014136
1406Blake BortlesJAX2014530-1.57-745-2.39-1268142317
1405Colt McCoyCLE2011495-1.59-736-1.19-5911329-76
1404Mark MalonePIT1987354-1.91-734-2.24-90714106
1403A.J. FeeleyMIA2004379-2.06-732-2.25-8511399-4
1402Joey HarringtonDET2002437-1.66-711-1.4-6131337-65
1401Akili SmithCIN2000303-2.65-708-2.44-7381371-30
1399Jake PlummerARI2002566-1.27-676-1.54-87014023
1398Rusty HilgerDET1988337-2.19-672-2.38-8021386-12
1397Gary MarangiBUF1976254-2.62-649-3.14-85214014
1396Joe FlaccoBAL2013662-1.05-648-1.42-942141620
1395Matt CasselKAN2009535-1.27-627-1.43-7631378-17
1394Dan PastoriniHOU1973320-2.02-626-2.39-8161390-4
1393Steve SpurrierTAM1976343-1.84-610-1.3-4771265-128
1392Joe FergusonBUF1983535-1.2-609-1.06-5701321-71
1391Jeff GeorgeIND1991541-1.25-608-1.37-7431372-19
1389Jake PlummerARI1999408-1.57-600-2.65-1079142031
1388Joe NamathNYJ1976246-2.44-598-2.91-7631377-11
1387John FrieszSDG1991519-1.22-596-0.89-4601254-133
1386Mark MalonePIT1986438-1.38-588-0.78-3441159-227
1385Mike PhippsCLE1975341-1.76-587-2.01-7311368-17
1384JaMarcus RussellOAK2009279-2.37-584-3.39-945141733
1383David CarrHOU2002520-1.31-583-2.17-1127142138
1381Bernie KosarCLE1990460-1.37-580-1.27-5851327-54
1380Ryan LeafSDG1998267-2.31-566-3.44-918141131
1379Phil SimmsNYG1980438-1.41-565-1.41-6161338-41
1378Mark BrunellWAS2004252-2.35-558-1.85-4671261-117
1377Steve DeBergSFO1978319-1.84-554-2.25-7191367-10
1376Christian PonderMIN2012515-1.14-551-0.97-4991285-91
1375Browning NagleNYJ1992414-1.42-549-1.45-5981332-43
1374Rick MirerSEA1993533-1.13-547-1.27-6761356-18
1373Joe KappBOS1970246-2.34-546-3.55-933141441
1372Josh FreemanTAM2011580-0.99-543-1.18-6851359-13
1370Steve DilsMIN1983481-1.22-542-0.68-3281142-228
1369Alex SmithSFO2007210-2.79-539-2.43-5111290-79
1368Chuck LongDET1987433-1.13-535-0.93-4611257-111
1367Todd BlackledgeKAN1984308-1.82-535-1.14-3521169-198
1366Joe FergusonBUF1984379-1.55-532-2.13-807138721
1365Donovan McNabbPHI1999244-2.45-530-2.8-6841358-7
1364Stan GelbaughSEA1992289-2.08-529-2.63-759137511
1363Jack ConcannonCHI1970409-1.29-528-0.68-2951104-259
1362Jim HartSTL1979403-1.4-528-1.39-5591315-47
1361Josh McCownARI2004439-1.29-525-1.06-4671260-101
1360Tommy KramerMIN1979602-0.93-524-0.43-2581062-298
1359Jim ZornSEA1976464-1.11-521-1.09-5411305-54
1358Mike LivingstonKAN1978308-1.79-520-1.08-3321144-214
1357Kerry CollinsCAR1997408-1.36-518-2.28-930141255
1356Rick MirerSEA1994408-1.35-515-0.71-2911100-256
1355Trent DilferTAM1996510-1.07-515-1.18-6011333-22
1354Donovan McNabbPHI2000614-0.9-514-0.44-2681071-283
1353Craig WhelihanSDG1998335-1.6-511-2.38-798138532
1351Boomer EsiasonCIN1992297-1.82-505-2.23-66213521
1350Dan PastoriniHOU1972336-1.57-502-1.35-4831270-80
1349Joey HarringtonMIA2006403-1.29-501-1.23-4941279-70
1348Kordell StewartPIT1999297-1.82-501-1.83-5451308-40
1347Doug PedersonCLE2000227-2.38-500-2.55-5781325-22
1346David KlinglerCIN1993383-1.45-499-1.37-5241297-49
1345Kelly StoufferSEA1992216-2.62-497-3.39-732136924
1343Cleo LemonMIA2007334-1.6-496-1.17-3921212-131
1342Vince FerragamoBUF1985306-1.71-491-2.04-62313431
1341Danny KanellNYG1998321-1.63-488-1.6-5131291-50
1340Steve FullerKAN1979307-1.8-485-2.28-701136323
1339Steve DeBergSFO1979595-0.83-4800.32193537-802
1338Tony BanksSTL1998449-1.17-479-1.32-5921330-8
1337Bobby DouglassCHI1971255-1.98-474-2.6-708136427
1336Joey HarringtonDET2004525-0.97-474-0.52-2741075-261
1335Marc BulgerSTL2008478-1.07-472-1.36-652135015
1334Steve BonoKAN1996460-1.07-470-0.66-3051120-214
1333Mark SanchezNYJ2012487-1.02-462-1.61-786138350
1332David WoodleyMIA1980344-1.4-459-1.3-4461244-88
1331Matt HasselbeckSEA2009520-0.94-458-1.07-5581314-17
1330Mike PagelBAL1982237-1.62-458-0.83-2511055-275
1329Brett FavreGNB2006634-0.74-456-0.18-114901-428
1328Roman GabrielPHI1974373-1.26-456-0.59-2341036-292
1327Neil LomaxSTL1986473-1.08-454-0.92-4361241-86
1326Ken DorseySFO2004239-2-451-2.1-5011286-40
1325Brandon WeedenCLE2012545-0.87-449-0.98-5361301-24
1324Mike BorylaPHI1976275-1.71-448-2.07-608133511

I actually find the Worst list even more validating of PY/A than the Best list. When we think of bad quarterbacks, most us reflexively focus on quarterbacks who make a lot of mistakes and sink their teams in obvious and memorable ways. And this list is filled with conventionally terrible quarterbacks. But remember, nearly all of their negative plays have been removed, so it’s not their mistakes putting them on the list. It’s their impotence. These guys couldn’t make plays or move the ball down the field, killing their teams slowly and agonizingly. At the very top (err, bottom), we find Derek Carr’s rookie year. A lot of fans and pundits classify Carr as a budding franchise QB who showed “flashes of potential”. Actually no, he showed the exact opposite. While the younger Carr avoided sacks and interceptions at a reasonable rate, his Y/A was absolutely pathetic. Even accounting for his lousy supporting cast, that is a major red flag. It’s much easier for a young QB to reign in his mistakes than it is for him to suddenly learn how to make positive plays down the field. Blake Bortles fits precisely the same troubling profile, so I don’t have much hope for the class of 2014.

Does this change your feelings about ANY/A? Do you think Danny and I are wasting our time? If anyone else has created their own passing metric using basic stats, I’d love to hear about it.

1. Note that in calculating league average, I excluded the player in question from the league average totals. So each player is compared to a slightly different definition of league average. []
• Dan

The question that Danny is asking is different from the ones that we usually ask.

Normally we ask questions like “How much did this QB’s performance contribute to his team’s success?” and “Which stats best reflect a QB’s underlying abilities? How strong are this QB’s underlying abilities, as evidenced by his performance?”

By choosing to use factor analysis, Danny is asking something like “If we assume that there is a single core ability underlying much of a QB’s performance, which stats best reflect that one ability? And how good is each QB at that one core ability, as evidenced by his performance?” A quarterback may also have other abilities, besides that one core ability, which are stable and which help his team succeed, but those are left out in this analysis.

For example, in the NBA “being good at rebounding” may be a stable ability which some point guards have, which helps their team win, but which is unrelated to most of the attributes that PGs are usually judged on. So rebounding would show up in a factor analysis as “not a valid measure of PG quality”, where “PG quality” refers to the one core ability which underlies the largest chunk of a PG’s performance. But, all else equal, I’d still rather have a good rebounding PG on my team because his rebounding is a stable trait which helps the team win (even if it is unrelated to his more purely pointguardy abilities). That is basically what Danny’s analysis is showing about sacks & INTs.

• Wolverine

Excellent point. In fact I wouldn’t be surprised if PY/A has a much weaker correlation to winning since it eliminates turnovers (which, while usually random, correlate strongly to W/L).

• Dr__P

Rightfully so as winning/losing is a TEAM measure not an individual stat.

Yet this is an attempt to get an understanding of the INDIVIDUAL performance

Thanks for bringing this up. As with any basic metric, this one breaks down at the margins. With peripheral stats like sacks, interceptions, fumbles, and QB rushing, the 80% of QBs clustered around the center of the bell curve are not really affected. Any differences mostly represent noise. However, at the extremes these measurements do matter. Think Marino’s sack rate, Gabriel’s INT rate, Warner’s fumbles, Vick’s rushing, Tarkenton in all four categories. So I agree that PY/A does not capture the skillset of all quarterbacks, but I think it does a good job in the majority of cases.

I like the NBA analogy. Using your example, an exceptional rebounding guard like Rajon Rondo will always be underrated by traditional PG measures. That’s why there is no such thing as an all encompassing metric that measures everything a player does. Thanks for commenting.

• Clint

Still can’t wrap my head around the calculations for these stats, but I do like it. I like the idea of QBs being measured by their own incompetency, rather than just their mistakes.

As a Browns fan, I always felt like Colt McCoy was just about the worst QB I had seen. A complete inability to move the ball down the field, he quickly gave up on plays and was far too cautious to win games. This stat totally validates me, as it has his ’11 season even lower than Brady Quinn’s ’09 season and THE BIG D Doug Pederson in ’00. No idea how Derek Anderson’s ’09 season didn’t make this list though. Apprx. 3 tds 11ints and I don’t think he completed half of his passes.Had the infamous 2/17 game that resulted in a win.

http://www.pro-football-reference.com/boxscores/200910110buf.htm

Yes, cases like McCoy are precisely whom this stat is designed to highlight (or lowlight). I’m glad it resonates with you. The only reason Derek Anderson’s ghastly ’09 season doesn’t make the list is because he didn’t have enough attempts. I don’t have my spreadsheets with me but later today I’ll be happy to look that season up for you.

Are you having difficulty wrapping your head around Danny’s numbers or mine? I’ll be the first to admit that I don’t entirely understand the complex modeling processes he uses, but I trust that he’s doing it right and therefore take his results seriously.

• Wolverine

Great analysis of a fascinating premise. As a Lions fan, this resonates with me after watching Joey Harrington timidly checkdown over and over again. He didn’t throw a particularly high number of INTs, nor did he standout as taking too many sacks. He just could not move the team an inch, and almost never took any risks. You got this sense of progressive, frustrating, hopelessness when watching him.

I would much rather have a quarterback who takes some risks to try to overcome his team’s weaknesses (Brett Favre comes immediately to mind), than someone who has no balls.

• Tom

Wolverine – totally agree, see my post about Bradshaw below. The guy throws three picks in the SB, but also continues to launch the ball down field. Yes, we’d be singing a different song if they lost, and it’s true he’s got Stallworth and Swann, but the point remains – the guy was ballsy (I think he was calling those plays as well). Favre is that way, and Flacco too.

BTW, this is great “You got this sense of progressive, frustrating, hopelessness when watching him.” You’re perfectly encapsulating the way a lot of fans probably feel about their own QB’s.

• sacramento gold miners

Yes, a QB can definitely be too efficient, or robotic, and that can be a losing effort. In some cases, you’re playing right into the defense’s hands, and it’s not conducive to coming back from behind. Kenny Anderson was a terrific QB, but would often throw that 3rd down pass for eight yards, when the Bengals needed 11, for example. Late in games, Cincinnati was pretty much toast when they fell behind, Anderson was a poor comeback QB. In fact, he’s tied with Russell Wilson for fourth quarter comebacks.

Yes, Bradshaw was calling plays then, and was doing so when many other QBs in the 70s and early 80s weren’t doing so.

• Wolverine

I never liked TNT’s NFL coverage, but I remember a game sometime in the mid-90’s where the Jeff Blake-led Bengals came into Pittsburgh and whupped the Neil O’Donnel-quarterbacked Steelers. Blake was taking deep shots over and over again, while O’Donnell was playing very conservatively, despite trailing by 2+ scores for much of the game. One of the TNT announcers made a comment (it’s stuck with me because the TNT announcers rarely said anything intelligent): “The fact that Neil O’Donnell has zero interceptions in this game is totally irrelevant. He’s not giving his team a chance to win.”

• That’s a great quote. Any idea who it was?

• Wolverine

It was the color commentator, and perusing Wikipedia, it looks like TNT’s color commentator was Pat Haden.

• It was October 10, 1995. Blake had a perfect passer rating. And you were right, the commentator was Pat Haden.

Thanks, Wolverine. Joey Harrington is THE poster boy for impotent quarterbacking. He was the master of going 18/30 for 120 yards. While turnovers are more dramatic, a parade of 3-and-outs is just as damaging and even more exasperating to watch. I also got this hopeless feeling from Kyle Boller, Jason Campbell, Alex Smith, and Sam Bradford as you mentioned.

• Clint

The PYA calculation. I think I get thrown off by the RPY/A and Value. Just not familiar with it. That’s probably all it is.

PY/A is just yards per attempt with a 20 yard TD bonus.

RPY/A is PY/A compared to league average on a per attempt basis.

VALUE is RPY/A multiplied by the number of attempts. It’s meant to balance efficiency and volume.

League average PY/A generally hovers between 7.5 and 8.0, although last season it was 8.1. Hope that clarifies things.

Since the Browns rejoined the league, they’ve had 39 QB seasons with at least 10 dropbacks. Of those, 32 provided negative value. Ouch.

Their best season by far was Anderson’s ’07 at +318, followed by Kelly Holcomb’s ’04 at +187 and Holcomb’s ’02 at +154. Couch in ’00, Dilfer in ’05, and Hoyer in ’14 posted barely positive seasons, but that’s it.

Anderson’s dismal ’09 season rates at -479 on just 188 attempts. Yikes! His ’08 wasn’t much better at -398. Talk about a one season wonder.

In 2013, Brandon Weeden posted a -231 and Jason Campbell registered a -309…on the same team.
In 2000, Spergon Wynn posted a -240 on only 54 attempts. Jake Delhomme flubbed his way to -264 in 2010.

In 2008, Ken Dorsey was worth -335 on just 91 attempts, while Quinn and Gradkowski chipped in -130 and -136 for a team total of -602.

Of course 2011 takes the cake. Colt McCoy’s dizzying -736 is the headliner, but Seneca Wallace added -264 for an even -1000 as a team!

• Have you looked at correlation between PY/A in years n and n+1? How about following Chase’s lead and looking at its correlation to wins? I like this “potency” metric, and I’d like to see it examined inside and out like ANY/A has. You may be way ahead of me on this, I dunno.

Thanks, Bryan. I was thinking about running some correlations, but haven’t gotten around to it yet. I think both of your suggestions would be useful in determining the true value of PY/A. If I had to guess, I bet ANY/A correlates better with past wins because it includes interceptions, but PY/A might correlate better with future wins because more of the noise is filtered out.

A follow up post may be in order 🙂

As a Packers fan growing up in the 1980s, I have to say that if you take away interceptions and sacks, Lynn Dickey was about the best QB in the league. Big arm, accurate on downfield throws, and completely immobile.

• I think sack% tells us something if the sample size is big enough. Players like Marino and Manning obviously meet the Y/A and TD% threshold. But they’re also incredible at avoiding sacks. So it’s not like they’re just being overly cautious at the expense of gaining yards. Though I saw something on 538.com (I think) that suggested that Aaron Rodgers’ low INT% could be a result of him being overly cautious late in close games. So yeah, sack% and INT% on their own aren’t great indicators.

• Wolverine

If I were an Eagles fan, the frequency with which Sam Bradford shows up on the second list would give me major palpitations.

I know, right? Given his high salary, I can hardly think of a worse choice to be your starting QB. Maybe Chip Kelly is going to run the ball 60% of the time, who knows with him…

• Tom

I like the idea of removing the negative stuff, keeping the positive stuff, and seeing how things shake out (I’m assuming that’s the general idea of what you’ve done; honestly, Danny’s blog post was waaaaay over my head).

Couldn’t help quickly running some Super Bowl numbers:

Top 5 ANY/A Super Bowl Performances (not adjusted for era or opponent)
1. Plunkett, 1980, 14.5
2. Montana, 1989, 12.6
3. Simms, 1986, 12.4
4. Aikman, 1992, 11.3
5. Williams, 1987, 11.2

Top 5 PY/A Super Bowl Performances (again, not adjusted for era or opponent), number in parentheses is ANY/A rank
2. Plunkett, 1980, 15.2 (#1)
3. Williams, 1987, 13.4 (#5)
4. Wilson, 2014, 13.0 (#16)
5. Montana, 1989, 13.0 (#2)

Two big shifts – Bradshaw’s performance against the Rams in 1979 gets a HUGE boost by us forgetting that he threw 3 picks in that game, Russel Wilson gets a similar boost by ignoring his three sacks and one INT (of course, if we adjusted this for era, he’d probably be a bit lower).

Of course, this leads us down the road of maybe just tweaking the interception yards penalty for ANY/A, etc.

In any event, great post and definitely not a waste of time. Thanks Adam!

• Adam, I hope you’ll forgive me a very long response. This is an interesting and important idea. Danny’s work was tough for me to follow, but the summary is straightforward enough.

Broadly speaking, I think there’s something to the findings. I’ve always remembered a quote, and to my great frustration I think the source for this is gone to the winds of time (though I may have read it in a Peter King MMQB from ’02 or so) … around that time, Mike Holmgren told Matt Hasselbeck to “throw some interceptions.” Hasselbeck was too worried about making mistakes, and Holmgren needed him to open things up and take some chances. He did, and the Seahawks became a perennial playoff team. Along the same lines, Frank Gifford was quoted as saying, “All venturesome running backs fumble.” Creating positive plays is more important than avoiding negative ones. I entirely agree with that. You and I are very much on the same page about the young quarterbacks in the league today. The tone of that piece is very sympathetic to the ideas you and Danny were working with.

Reading this, the player who immediately came to my mind was Jason Campbell. I lived in Washington for part of his tenure there, and Campbell was so conservative, so afraid of making a mistake, that he rarely accomplished anything. Bill Walsh had that famous quote about Steve DeBerg, that he was just good enough to get you beat. DeBerg’s comp% was high, but his yds/att was low. His sack% was good and his INT% was average, but his TD% was low. DeBerg’s yards/comp was the lowest of that era — by a lot, half a yard. The link doesn’t include Y/C, but DeBerg checks in at 11.3. The mean was 12.7, the median was 12.8, and the standard deviation was 0.7 (n=25). DeBerg is two standard deviations below the middle of the group, and about 2/3 standard deviation beneath the next-lowest (Tommy Kramer, 11.8).

One of my low-priority projects is convincing the world that yards per completion is a valuable stat. Show me a quarterback who completes a five-yard pass on 3rd-and-7, and I’ll show you an overrated player. Yards per completion is a degree-of-difficulty metric. Any QB on an NFL roster can complete a bunch of four- and five-yard passes if he doesn’t care about generating first downs. But a player who generates big plays is creating something for his team. I would add Y/C to your list of qualifying criteria. Obviously yds/att and yds/comp are highly correlated, but a guy who completes a high percentage of very short passes can fool the stats. QBs with good ANY/A and low Y/C are usually “system players” with middling true talent levels.

With good people around him, a mediocre QB can throw short passes, limit his risk, and put up decent efficiency numbers just by completing a high % of passes and avoiding INTs. Players who don’t stretch the field vertically can’t lead big comebacks, and they can’t beat good defenses. They pad their stats in low-leverage situations and leave us wondering why a guy with such good numbers has so little success. I don’t believe the reverse is true: a QB with above-average stats and high Y/C is almost invariably a pretty good player.

So I think you and Danny have a good basic idea, that there’s not enough emphasis on positive plays/qualifying criteria. But I don’t agree with lumping sacks and interceptions together. Sacks and INTs often balance out — a player who eats sacks under pressure usually doesn’t rush passes into coverage and get picked, while a player who unloads the ball under pressure seldom takes a lot of sacks — but I think that hides a fundamental flaw in this approach.

Danny’s idea that AY/A might be a better stat than ANY/A seems totally crazy to me. Maybe his research revealed something that I just don’t understand — my football knowledge is much greater than my math skill — but that strikes me as ridiculous. I would love a layman’s explanation of why sack data isn’t terribly important; convincing evidence would force me to radically re-evaluate my approach to quarterback analysis.

I think a lot of analytic fans put too much faith in ANY/A. Some people treat the stat as perfect, the end of the story. That’s foolish, but even beyond that, ANY/A can be fooled, and in particular it can be fooled in the way you discuss: by rewarding mistake-avoidance without enough emphasis on positive production.

That said … PY/A is an interesting idea, but I don’t understand how omitting data will improve ANY/A. Sacks and interceptions are important, and a quarterback does have some control over them. Any research that finds otherwise is missing something. Ideally, we’d have more context for these events. A sack on third down is seldom a big deal. A long interception on third or fourth down probably won’t cost your team the game. But a sack or a turnover on first down matters a great deal. Getting intercepted on a Hail Mary at the end of the first half shouldn’t meaningfully affect our evaluation of a passer, and so on. If we could break down the data with appropriate context, it would be extremely valuable, a necessary component of informed analysis.

Absent an easy way to put all our data into that context, it’s tempting to filter out the noise by excluding it altogether. I think that’s a mistake. Granting that I don’t understand Danny’s process, I do understand that as a general rule, good quarterbacks make fewer negative plays than bad quarterbacks. They take fewer sacks, and they throw fewer interceptions. Including that data, and working around the imperfections, seems like a better idea to me than excluding those data. I think you have a good idea, and moreover, a really interesting good idea. But I think ANY/A is an superior statistic to PY/A.

• Clint

“But a player who generates big plays is creating something for his team.” Btw, Brian Hoyer was 1st in the league in Y/C

I’ve been too distracted reading comments on your articles so I somehow missed this one until now…

Let me start by saying Danny’s conclusions don’t entirely make sense to me, either. Taking sacks and throwing interceptions are obviously bad in a general sense, so it doesn’t compute that ignoring them completely would make ANY/A more accurate. But at the same time I believe that ANY/A systematically overrates pedestrian passers who don’t really contribute to winning. Fitting you mention Jason Campbell, because he was my inspiration for exploring this idea in the first place.

I agree that negative plays are begging for context, and in recent years I think ESPN’s QBR has done the best job of incorporating said context. Unfortunately for people interested in historical comparisons like you and I, precise data simply doesn’t exist for most of NFL history.

In that vein, my goal is to create a better version of ANY/A, and this article was essentially a brainstorming session in that direction. Perhaps ANY/A would be improved by lowering the INT penalty without discarding it entirely, or making the formula nonlinear in the way the components are weighted. I wholeheartedly agree with you on the merits of Y/C, so I would give an exponential penalty for Y/C below a certain threshold. This would bring down the ratings for the checkdown artists like Campbell no matter how well they did in other areas. Similarly I would give an exponential bonus for QB’s above a threshold for Y/C provided they maintain at least an average comp %. Another idea regarding INT’s…penalize them only above or below one standard deviation from league average. Everyone within one SD would fall into the randomness zone and rate the same.

I may tinker with some of these ideas and write a follow up article. Thanks for your input and willingness to explore out-of-the-box ideas, it’s more fun that way!

• bubqr

Great piece – It does tie in with the belief
many people have, including myself, that the chances an over aggressive QB will
learn how to cut down on his mistakes are higher than the ones that an
overly cautious, captain checkdown type of QB will learn to take the necessary
risks/air-it-out.

Now as an Eagles fan, I go back to my daily