## The Greatest QB of All-Time IV, Part I (Methodology)

In 2006, I took a stab at ranking every quarterback in NFL history. Two years later, I acquired more data and made enough improvements to merit publishing an updated and more accurate list of the best quarterbacks the league has ever seen. In 2009, I tweaked the formula again, and published a set of career rankings, along with a set of strength of schedule, era and weather adjustments, and finally career rankings which include those adjustments and playoff performances.

If nothing else, that was three years ago, so the series was due for an update. I’ve also acquired more data, enabling me to tweak the formula to better reflect player performance. But let’s start today with an explanation of the methodology I’m using. To rank a group of players, you need to decide which metric you’re ordering the list by. I’ll get to all of the criteria I’m not using in a little bit, but the formula does use each of the following: pass attempts, passing touchdowns, passing yards, interceptions, sacks, sack yards lost, fumbles, fumbles recovered, rush attempts, rushing yards and rushing touchdowns. Most importantly, the formula is adjusted for era and league.

Two of the best quarterbacks ever.

So where do we begin? We start with plain old yards per attempt. I then incorporate sack data by removing sack yards from the numerator and adding sacks to the denominator1. To include touchdowns and pass attempts, I gave a quarterback 20 yards for each passing touchdown and subtracted 45 yards for each interception. This calculation — (Pass Yards + 20 * PTD – 45 * INT – Sack Yards Lost) / (Sacks + Pass Attempts) forms the basis for Adjusted Net Yards per Attempt, one of the key metrics I use to evaluate quarterbacks.

For purposes of this study, I did some further tweaking. I’m including rushing touchdowns, because our goal is to measure quarterbacks as players. There’s no reason to separate rushing and passing touchdowns from a value standpoint, so all passing and rushing touchdowns are worth 20 yards and are calculated in the numerator of Adjusted Net Yards per Attempt. To be consistent, I also include rushing touchdowns in the denominator of the equation. This won’t change anything for most quarterbacks, but feels right to me. A touchdown is a touchdown.

Fumbles are an overlooked aspect of quarterback play. The book “The Hidden Game of Football” values a fumble at -50 yards, making them slightly more costly than interceptions (THGOF also first derived the -45 yard penalty for interceptions). Now I don’t have ‘fumbles lost’ data but I do have information on “fumbles” and “fumbles recovered” going back to 1945; the difference between those two could be characterized as net fumbles. However, just because a quarterback fumbles and does not recover the ball does not mean his team loses possession (and remember, he is already being penalized for the sack and the sack yards lost).

For those of you who haven’t given a bunch of thought to fumbles data, here are some numbers from 2000 to 2011:

• 52% of all fumbles are by quarterbacks
• 25% of all quarterback fumbles are ultimately recovered by the quarterback, and an additional 3% bounce out of bounds, making them relatively harmless.
• Who generally recovers the other 72%? A defensive player recovers roughly 44% of all fumbles, leaving the remaining to be recovered by the offense (offensive lineman are responsible for recovering 15% of quarterback fumbles, while 13% are grabbed by an offensive skill position player). This means that roughly 60% of all quarterback fumbles not recovered by the quarterback are recovered by the defense.

Therefore the value of “net fumbles” — the number of fumbles by a quarterback minus the number of fumbles he recovered — is -30 yards, since, on average, a fumble lost is worth -50 yards. So we can update the formula to:

[(PYD + 20*(PTD + RTD) – 45*INT – SKYDLST – 30*(FUM-FumRec)) / (ATT + SK +RTD)]

That calculates each QB’s value per play; we then compare that number to the league average, and multiply the difference by his total number of plays (i.e., ATT + SK + RTD) to get each QB’s value added over average. The last step is to add a rushing component. I still haven’t figured out a very good way to handle quarterback rushing, but what I’ve done in the past is to add all QB rushing yards over 4.0 yards per carry. It still doesn’t feel very scientific, but the results have been noncontroversial, so I’ll continue to use it until I think of something better. I’m going to call final per-play measure of quarterback value added “converted yards per play” or CYP. Once you have each quarterback’s CYP, you then compare it to the league average. I take the CYP for each quarterback, subtract it by the average baseline for that season, and multiply the difference by the number of plays that quarterback had. This does a nice job of balancing the trade-off between compilers and guys that excelled for a short period of time. I have also added in adjustments for shortened seasons, for non-NFL leagues, and for the wartime era from 1943 to 1945.

But enough about methodology for now: let’s take a look at the 2011 results:

QBTmAttYdTD/INTSk/YdRsh-Yd-TDFum/FRCYPVAL
Aaron RodgersGNB502464345/636/21960-257-35-29.32063
Drew BreesNOR657547646/1424/15821-86-11-08.21838
Tony RomoDAL522418431/1036/22722-46-17-67.41009
Eli ManningNYG589493329/1628/19935-15-19-37.2997
Matthew StaffordDET663503841/1636/25722-78-05-16.8869
Cam NewtonCAR517405121/1735/260126-706-145-36.5705
Matt RyanATL566417729/1226/17337-84-25-26.7680
Matt SchaubHOU292247915/616/9815-9-23-17.7661
Michael VickPHI423330318/1423/12676-589-110-46.1528
Ben RoethlisbergerPIT513407721/1440/26931-70-09-36.2323
Carson PalmerOAK328275313/1617/11916-20-12-16.3225
Alex SmithSFO445314417/544/26352-179-27-46194
Jay CutlerCHI314231913/723/15918-55-17-46140
Matt HasselbeckTEN518357118/1419/15320-52-04-15.744
Kyle Orton2tm25217589/910/5411-13-04-05.2-114
Andy DaltonCIN516339820/1324/16037-152-15-05.4-114
Joe FlaccoBAL542361020/1231/20339-88-111-45.4-151
Matt MooreMIA347249716/936/22932-65-214-34.9-274
Kevin KolbARI25319559/830/21917-65-08-04.7-277
Dan OrlovskyIND19312016/414/846-5-07-14.2-288
Ryan FitzpatrickBUF569383224/2322/14856-215-07-45.1-289
Tim TebowDEN271172912/633/225122-660-614-34.1-313
John SkeltonARI275191311/1423/16228-128-04-24.3-383
Curtis PainterIND24315416/916/10417-107-05-14-389
Matt CasselKAN269171310/922/12025-99-05-04.3-402
Tarvaris JacksonSEA450309114/1342/29340-108-19-44.8-416
Rex GrossmanWAS458315116/2025/20120-11-18-24.6-521
Christian PonderMIN291185313/1330/16428-219-06-03.7-522
Mark SanchezNYJ543347426/1839/24337-103-610-14.7-527
Colt McCoyCLE463273314/1132/17361-212-011-54.4-631
Josh FreemanTAM551359216/2229/16455-238-49-24.5-654
Blaine GabbertJAX413221412/1140/29348-98-014-43-1207

Those who overrate counting stats were likely to put Drew Brees ahead of Aaron Rodgers last year, but there was no contest as to which quarterback produced the most dominant statistics. You might be surprised to see Tony Romo 4th on the list — I know I was — but he had better numbers than Eli Manning in both touchdowns, interceptions and fumbles, which was enough to beat out Eli Manning’s edge in yards per attempt. Romo is a polarizing figure, but how many know that he had 31 TDs and just 10 INTs last year? Few remember how efficient Matt Schaub was last season: on a per-play basis, he came in only behind the big three (he also ranked 4th in ANY/A in 2011).

Of course, the point of this system is to be able to compare players across eras. More on this tomorrow, but for now, here’s a look at the top 100 single-season performances:

YearTeamQBAttPydTDINTSkSkydRshRydRTDNetFumCYPVALUE
2004INDPeyton Manning49745574910131012538029.72314
2011GNBAaron Rodgers50246434563621960257339.32063
1953CLEOtto Graham25827221192620943143638.11879
2011NORDrew Brees65754764614241582186118.21838
2006INDPeyton Manning557439731914862336427.91740
1943CHISid Luckman2022194281222-4010111739
1992SFOSteve Young40234652572915276537467.81727
1994SFOSteve Young461396935103116358293738.21706
1982SDGDan Fouts330288317111294981-17.81694
1976BALBert Jones34331042492928438214227.71675
1999STLKurt Warner49943534113292012392197.81631
1998MINRandall Cunningham425370434102013232132128.41585
2000SFOJeff Garcia561427831102415572414437.31570
1970SFOJohn Brodie37829412410867929217.51548
2004MINDaunte Culpepper548471739114623888406297.61506
1984SFOJoe Montana432363028102213839118227.91498
1981CINKen Anderson479375429102514046320137.31489
1981SDGDan Fouts60948023317191342256076.91471
2003INDPeyton Manning56642672910181072826027.21459
1967WASSonny Jurgensen50837473116181511546256.41455
1964BALJohnny Unitas30528241963424137162217.91453
1942GNBCecil Isbell2682021241436831071450
2009SDGPhilip Rivers48642542892516726501581413
2009NORDrew Brees51443883411201352233287.91412
1993SFOSteve Young462402329163116069407267.21400
1953RAMNorm Van Brocklin286239319141194811026.71395
1975CINKen Anderson377316921113224749188236.81385
1991WASMark Rypien42135642811759156167.91385
1983WASJoe Theismann459371429113424237234117.21378
1960CLEMilt Plum25022972153528317-24227.61373
2005INDPeyton Manning4533747281017813345057.71366
1998SFOSteve Young5174170361248234704546871364
1963NYGY.A. Tittle36731453614302681899217.51363
1988CINBoomer Esiason388357228143024543248117.71356
2008NORDrew Brees63550693417139222-1047.41341
1989SFOJoe Montana386352126833198492273681338
1967NYJJoe Namath4914007262827270614005.81335
2006NORDrew Brees55444182611181054232077.21323
2001STLKurt Warner546483036223823328600871312
2000INDPeyton Manning5714413331520131371161471309
1949CHIJohnny Lujack3122658232213101864216.21290
1959BALJohnny Unitas367289932142318129145256.71286
2000MINDaunte Culpepper474393733163418189470767.11280
1968OAKDaryle Lamonica41632452515262161998136.31275
2009GNBAaron Rodgers54144343075030658316567.31267
1989RAMJim Everett51843102917292142531107.21260
2009MINBrett Favre53142023373424797027.51256
2009HOUMatt Schaub58347702915251494857027.31255
1961PHISonny Jurgensen41637233224262122027046.71251
1980CLEBrian Sipe55441323014232172055126.71241
1947WASSammy Baugh354293825152547287.21241
1979DALRoger Staubach461358627113624037172026.71237
2009INDPeyton Manning57145003316107419-13027.41237
2004PHIDonovan McNabb46938753183219241220377.51235
2010SDGPhilip Rivers54147103013382272952057.51221
1975MINFran Tarkenton4252994251327245161082-26.11220
2009DALTony Romo55044832693419635105147.41211
1968NYJJoe Namath3803147151716118511216.51198
1973PHIRoman Gabriel46032192312312191210145.71196
2008SDGPhilip Rivers47840093411251513184067.71195
1974CINKen Anderson328266718103629243314226.21192
1969OAKDaryle Lamonica42633023425111001336116.31191
1974OAKKen Stabler310246926121814112-2126.91188
1967BALJohnny Unitas43634282016241892289016.31188
1991SFOSteve Young2792517178137966415428.21185
2005CINCarson Palmer50938363212191053441147.11182
1995GNBBrett Favre570441338133321739181386.91175
1966DALDon Meredith344280524123525738242546.41170
1966KANLen Dawson284252726102321324167057.31169
1999INDPeyton Manning53341352615141163573246.91169
2002OAKRich Gannon618468926103621450156366.71154
2003KANTrent Green52340392412201302683256.91144
2007INDPeyton Manning515404031142112420-5337.21140
1983SFOJoe Montana515391026123322461284236.61138
1962NYGY.A. Tittle375322433201612717108247.11132
1976OAKKen Stabler29127372717192037-2147.11129
1976MINFran Tarkenton4122961178252212745126.11129
2002KANTrent Green470369026132614131225107.11127
2010GNBAaron Rodgers475392228113119364356437.41125
1971DALRoger Staubach21118821542317541343257.31124
1961HOUGeorge Blanda3623330362210128712037.61122
1967OAKDaryle Lamonica425322830203732322110425.61121
1998NYJVinny Testaverde42132562971914024104147.41120
1990HOUWarren Moon5844689331336252552152146.61115
1980PHIRon Jaworski45135292712272132795126.81114
1948PHITommy Thompson246196525111246108.11113
1950RAMNorm Van Brocklin2332061181412931522146.51112
1984STLNeil Lomax560461428164937735184396.31096
1990KANSteve DeBerg44434442342219121-5067.21092
1978DALRoger Staubach4133190251632219421821361085
1977BALBert Jones393268617112622128146215.51084
1966GNBBart Starr25122571432418121104277.41084
1977DALRoger Staubach36126201893021951171355.81083

And now, the huge caveat: It’s important to remember that this is just a measure of each team’s passing game, assigned to the quarterback on the field for those plays. Obviously the quality of the offensive line, the ability of the receivers, the versatility of the tight ends and running backs, the philosophy of the coaches, the strength of the schedule, and good old randomness have a significant impact on the above numbers. The reason for these posts is to accurately measure quarterback statistics, and nothing else. Once we have strong measures of QB performance, we can then judge QBs based on how much of their success (or lack thereof) we want to assign to the QB and how much to other people/factors.

And, of course, even if supporting casts are the same, numbers don’t tell the full story. The point here is simply to get the most out of the numbers we have.

1. I have individual sack data for every quarterback since 1969. For seasons before then, I have team sack data going back to 1949. For seasons before 1950, I ignored sacks; for seasons between 1950 and 1969, I gave each quarterback an approximate number of sacks, giving him the pro-rated portion of sacks allowed by the percentage of pass attempts he threw for the team. While imperfect, I thought this “fix” to be better than to ignore the data completely, especially for years where one quarterback was responsible for the vast majority of his team’s pass attempts. []
• If my math is right, eight of the 100 “most valuable” seasons belong to Peyton Manning. Nobody else has more than five (Steve Young).

• Shattenjager

In one month I get both a new Sight & Sound Greatest Films of All Time poll AND a new entry in Chase Stuart’s The Greatest QB of All-Time series?! O frabjous day (well, month)!

I think that’s enough to make up for the bar exam sucking away a few months out of my life forever before that.

• Chase Stuart

Appreciate the love, Shattenjager.

• Tim Truemper

Not to be persnickety, but I see sack data for Johnny Lujack in 1949. Which is great to have (as estimated) but occurs outside of the range of 1950 to 1969 in which you estimated individual sack data where it was not precisely available.

Great list. If my eyes don’t decive me, Joe Namanth’s 1967 season is the only one in which the ration of Td’s to Int.’s is below 1.0 (TD as numerator, Int. as Denominator).

• Chase Stuart

You know, now that I think about it, I think I did have 1949 sack data (not in front of my database now) but maybe not complete. As always, you guys are keeping me on my toes. If it was anyway but you, Tim, it would be persnickety, but I’ll give you a pass.

My eyes tell me you missed one more season: Namath’s 1968.

• Andrew

For the 2011 results, something is wrong with the the ranking by TD/INT ratio. Matt Cassel’s 10/9 sits at the top, followed by Skelton’s 11/14. Just thought you should know something was off with it.

• Chase Stuart

Yeah, that’s a table sorting plugin issue; not sure what I can do about that. I will look into it, though.

• Andrew

The number of these that have happened recently is kind of staggering. By my count, 19 (or 19% of the 100 best seasons) of these have happened during or after the 2000 season, despite that time period only representing about…oh, about 17% of the time period represented by those seasons covered by this list (every season since 1942, the oldest of these top 100 seasons).

• Andrew

Oops, disregard that last post. I was only looking at the top 50. The real numbers are 33% of the top 100 greatest seasons ever compiled in a mere 17% of the timeline in question. That’s more the profound statement I was looking for.

• Chase Stuart

Yeah, on the surface, that 33% number does seem a bit high. I will note that even though from the years 1942 to 2011, the 2000s represents just 17% of the years, it represents 23% of the team-seasons. But the bigger reason for the discrepancy is the increase in pass attempts. Being great on 500 attempts is better than being great on 300 attempts. Even with the pro-rating for fewer than 16 game seasons, the increase in pass attempts is a big part of the disproportionate treatment; personally, I’m okay with that.

• Andrew

It all just highlights how ridiculously hard it is to truly compare QBs across eras, I suppose. I’m betting that we will never really have a definitive statistical analysis AND consensus on who was the greatest QB ever (though my vote goes to Montana, every time). There are simply too many factors playing into success. For instance, can you imagine the numbers Dan Marino would have put up were he the one throwing to jerry Rice and John Taylor? Or if he had Emmitt Smith running the ball. Or if he faced one fewer elite pass defences a season. Or if…this could really go on forever. It’s just too much to account for, and that is frustrating.

• sacramento gold miners

Agree about Joe Montana as the best ever QB of all time, and we’ll never have a statistical formula to clearly rank the elite at every position. I would say Dan Marino in 1984 had a great pair of receivers in Mark Duper and Mark Clayton to throw to, and center Dwight Stephenson was better than anyone on the 49ers offensive line. Marino with an elite RB like Roger Craig would have been scary.

• Pingback: Trivia of the Day – Saturday, September 1st()

• Nathan

Way, way late to this, but I just stumbled across it, so why not? Part of the reason for the skew toward the later era (despite the era adjustment via baseline) could very well be the number of games per season. Multiplying by a factor of 16/14 for 60s through 77 and by a larger corresponding factor for earlier years would disperse the best seasons across eras better…if you haven’t accounted for this already. It would also make Fouts’ strike-shortened 82 year look ridiculous…and there’s the question of how you deal w/ 87.

Other reasons for the concentration of good seasons on the last decade or so would be the pass-heavy nature of the game. Even w/ normalization for schedule length, any system that uses attempts/plays will favor recent players. You can’t adjust for this in a linear manner because there was too much variance in terms of how pass heavy teams were in earlier eras. The penalty for lower QB attempts/game early on is somewhat mitigated by another form of variance anyway: efficiency. If you’re comparing against the league avg. for a given year, players weren’t as tightly bunched, so that gives guys like Luckman and Graham an opportunity to stand out regardless.

One other thing: it’s been brought up by others before, but Manning’s 2004 season looks even more statistically freakish when you consider the time he sat on the bench in blowouts (7 quarters) compared to Brady (3 quarters) and Marino (roughly 2, if I remember correctly). The final regular season game v DEN in particular where Manning played a token series really put a dent in his numbers. If we were to assume that he would have shredded DEN the same way he did the following week in the playoffs had he been unleashed in a meaningless final regular season game, he could have been looking at a season value of roughly 2630.

• Pingback: Quarterback Age Curves()