≡ Menu

Adam Steele is back to recap his Wisdom of Crowds work. As always, we thank him for that. Football Perspective wouldn’t be what it is without contributions like this from folks like Adam.


I’d like to thank everyone who voted in this year’s Wisdom of the Crowds, and I also appreciate your patience in waiting for the long overdue recap article. I’m not much for small talk, so let’s get right to it.

Originally, my plan was to simply tally the scores and use the totals for the QB ranking. However, it quickly became evident that this wasn’t going to work, as we had very large discrepancies in how voters allocated their points. Some people awarded 25 points to their pick for best ever, while others didn’t give any QB more than six points. It would be just plain wrong for one voter’s GOAT to be weighted four times more than the next voter. My solution (helmet knock to commenter hscer1, since he came up with it) is to tabulate points in proportion to the highest score on each ballot. Thus, a QB who scores five points on a ballot with a 25 maximum receives 0.2 ranking points, while a five-pointer on a ballot with a maximum of six is awarded 0.83 ranking points. This levels the playing field for all ballots, and in my opinion yields a far more honest result than the simple tally method. Since the abstract concept of ranking points is tough to put in proper context, I’ve translated them into Share %, which is the percentage of possible points earned. We had 51 legal ballots submitted this year, so Share % = ranking points / 51.

Results

In order to qualify for a WOC ranking, a quarterback had to be listed on a minimum of three ballots, leaving us with 36 qualifying QB’s. The table below lists the quarterbacks’ Share %, ballot appearances, “pantheon” appearances (ballots where he received at least 0.5 ranking points), and ballots where he received the highest score (including ties). I also included the ranking each QB earned in the 2015 edition of this exercise, as well as the number of positions gained or lost from 2015 to 2017.

#QuarterbackShareBallotsPantheonGOAT2015 RkChange
1Tom Brady88.4%50473032
2Peyton Manning83.8%5147211-1
3Joe Montana81.9%5147122-1
4Johnny Unitas66.5%4941540
5Dan Marino61.9%4936350
6Otto Graham51.7%4330493
7Steve Young47.1%442406-1
8Brett Favre42.6%462007-1
9John Elway39.3%481408-1
10Aaron Rodgers37.4%41170122
11Roger Staubach34.5%4310010-1
12Sammy Baugh33.3%33162142
13Fran Tarkenton33.2%3715011-2
14Drew Brees32.9%4012013-1
15Bart Starr18.3%2850150
16Dan Fouts14.3%2220160
17Sid Luckman13.7%2060214
18Norm Van Brocklin12.5%1820257
19Sonny Jurgensen11.7%1520278
20Terry Bradshaw10.5%163018-2
21Warren Moon8.5%140020-1
22Troy Aikman6.9%1400220
23Len Dawson6%1200241
24Y.A. Tittle5.8%1100284
25Ben Roethlisberger5.7%1210261
26Ken Anderson5%100023-3
27Bobby Layne4.9%1100314
28Kurt Warner4.2%100017-11
29Philip Rivers3.9%1000301
30Jim Kelly3.5%80019-11
31Bob Griese2.2%500332
32Joe Namath1.8%50029-3
33Tony Romo1.8%50032-1
34Donovan McNabb1.1%3004410
35John Brodie0.9%30034-1
36Ken Stabler0.7%300360

In a result that should surprise absolutely no one, Tom Brady is declared the 2017 Greatest QB of All Time. Brady had plenty of GOAT support even before the 2016 season, and winning his fifth ring in dramatic fashion surely added to his cadre of proponents. Peyton Manning and Joe Montana round out the top three, with a major drop off between them and fourth place Johnny Unitas. Interestingly, Brady, Manning, and Montana all received exactly 47 pantheon votes; Brady distanced himself by getting almost as many GOAT votes as PFM and Joe Cool combined.

Check out the large gap between 14th place Drew Brees and 15th place Bart Starr – this makes for an obvious demarcation point between the upper-tier HoF’ers and everybody else. Fascinatingly, these same 14 quarterbacks also made up the top 14 spots two years ago. This result is remarkable considering only 16% of the voters from 2015 also submitted a ballot in 2017, and the scoring method was completely different. I’m quite encouraged by this, as it proves that the rankings are robust regardless of who is voting and how we tally the scores (at least among FP quality readership; the alternative will be discussed later).

The other noteworthy development from 2017 is the systematic rise of pre-merger quarterbacks, which I consider this a heartwarming turn of events. Otto Graham jumps three spots, Luckman and Tittle gain four, while Van Brocklin and Jurgensen make huge leaps of seven and eight positions, respectively. I can’t point to any obvious reason why the old timers have gained esteem in the eyes of FP readers, but I do feel safe in saying that these guys are now closer to where they belong. Of course, these jumps had to come at someone’s expense. For reasons beyond my comprehension, Kurt Warner and Jim Kelly each freefall 11 positions in the rankings. This is especially puzzling  for Warner, since he was just inducted into the Hall of Fame (I happen to agree with the lower rankings for Warner and Kelly, but that’s just one man’s opinion).

Comparing Methodologies

I enjoy running these WOC experiments for many reasons, but one of the principle purposes is to provide a baseline for comparison. In this table, I once again list the 2017 and 2015 WOC rankings. Additionally, you’ll see the rankings from an entirely different WOC format, an ongoing poll hosted at ranker.com. We also need a statistically derived baseline, so I included Bryan Frye’s VALUE metric (based on TAY/P above average), and Brad Oremland’s QB TSP (based on career value above replacement). The designation “-” is shorthand for Missed Cut, which I assigned to quarterbacks who didn’t qualify for ranking in any of the respective methods (in the case of VALUE, I consider any QB with negative career value to have missed the cut). The three QB’s with a dash mark under TSP are too old to be eligible for ranking. The table includes every QB who cracked the top 50 in any of the methods.

Table B2017 WC2015 WCRankerVALUETSP
Tom Brady13333
Peyton Manning21211
Joe Montana32158
Johnny Unitas447124
Dan Marino55422
Otto Graham691787
Steve Young768610
Brett Favre876159
John Elway9852514
Aaron Rodgers101213917
Roger Staubach111091418
Sammy Baugh1214-17-
Fran Tarkenton13111676
Drew Brees14131245
Bart Starr1515113848
Dan Fouts1616201111
Sid Luckman17213621-
Norm Van Brocklin1825331615
Sonny Jurgensen1927291313
Terry Bradshaw2018104943
Warren Moon2120184119
Troy Aikman2222144539
Len Dawson2324312242
Y.A. Tittle2428242416
Ben Roethlisberger2526221830
Ken Anderson2623441012
Bobby Layne2731425426
Kurt Warner2817152328
Philip Rivers2930431921
Jim Kelly3019193125
Bob Griese3133253968
Joe Namath3229213331
Tony Romo3332492035
Donovan McNabb3444383038
John Brodie3534632720
Ken Stabler3636237852
Randall Cunningham-35277134
Eli Manning-37269762
Boomer Esiason-38324022
Steve McNair-39393650
Phil Simms-40308861
Daryle Lamonica-41-2936
Drew Bledsoe-4235-55
Vinny Testaverde-4359-49
George Blanda-454113277
Matt Ryan--532632
Jeff Garcia--732840
Trent Green--943241
Bert Jones--823433
John Hadl--793524
Rich Gannon--604237
Carson Palmer--474345
Jim Hart---4429
Roman Gabriel--504623
Mark Brunell--654754
Jim Everett--814827
Dutch Clark---50-
Russell Wilson--405178
Joe Theismann--285246
Daunte Culpepper--686144
Bernie Kosar--486758
Doug Flutie--4584-
Andrew Luck--4694100
Norm Snead----47
Jim Plunkett--34-101
Archie Manning--37-92

As I alluded to earlier, the ranker.com list represents a sharp departure from the rankings generated by FP readers. Their scoring method is much simpler, using a thumbs up / thumbs down system. And more importantly, their voters appear to be far less informed than the football savvy folks we have at FP. I don’t what the voting demographics look like at Ranker, but my guess is a heavy dose of casual fans, homers, and people who are relatively new to football viewing. I’ll give them credit – the top of the Ranker list is quite reasonable. But beyond that, their rankings follow a trail through ring counting territory, then descend into the abyss of recency bias and blatant fanboyism. I mean, Tim Tebow makes the top 100, while Sammy Baugh misses the cut entirely. As boneheaded as that may sound to the FP crowd, I have a feeling the Ranker list is a fairly accurate portrayal of the average football fan – the kind who tend to make arguments based on emotion rather than facts.

Returning the to sanity of VALUE and TSP, we see a number of QB’s whose ranking diverges wildly from their standing in the WOC lists. This is not surprising, since we “know” that some QB’s are misrepresented by their regular season statistics, both for better and for worse. Bart Starr fares much better in WOC than he does in the cold world of stats, as do John Elway, Troy Aikman, and Terry Bradshaw. Conversely, FP readers believe that Drew Brees and Ken Anderson are overrated by their statistics, as are Matt Ryan, Trent Green, and John Hadl (and I think they’re right). The beauty of sports analysis is the fact that there is no right answer, that is takes a multitude of different approaches to inch us closer and closer to the truth.

  1. I highly encourage you to check out hscer’s collection of Sporcle quizzes. []
  • Given the way TablePress operates, the bottom table cannot sort properly with MCs in the columns. I would suggest something like 999 (or something more elegant) just to put them at the bottom.

    • As always, a good suggestion. Fixed!

      • Adam

        For the sake of aesthetics, would you mind changing the 999’s to dashes?

    • Adam

      Whoa that’s weird, thanks for pointing that out. I’ll have Chase fix it.

  • Ryan

    Great stuff Adam, looking forward to the ball catcher rankings.

    • Adam

      Thanks, Ryan. These things are always fun!

  • Here’s a look at the average “age” of everyone’s ballots, going by first career year of the players.

    https://pbs.twimg.com/media/C77nuEPXgAIm1_l.jpg

    • Richie

      Is age the average birth year of the ballots or average start of career?

      • I multiplied each player’s first year by points allocated on the ballot. Then I divided the sum of those products by the sum of points to find a weighted average ballot age.

  • Great write-up. I certainly agree with the last sentence. The final table is really interesting, too. Seeing names like Theismann and Archie Manning in the top 40 on Ranker makes me think that name recognition, whether through TV or offspring or whatever, is another large component of casual fan opinion.

    Glad I could help in my own little way, and thanks for the Sporcle bump.

    • Adam

      Thanks, hscer. There’s no doubt that name recognition and media exposure have influence on the perceptions of casual fans. It’s a great example of the availability heuristic – when people don’t know much about a given topic, they default to whatever morsels they have heard of. Why isn’t Fran Tarkenton regarded very highly among casuals? Probably because he’s been out of the limelight for a long time and isn’t part of a famous family.

      • Richie

        If only “That’s Incredible!” was still on the air.

        • Adam

          I had no idea that show even existed. Pretty cool! Here’s a clip of Tark interviewing 5 year old Tiger Woods:

          https://www.youtube.com/watch?v=A3y5vOFxnEY

          • Tom

            Adam, you’ve never heard of That’s Incredible? Well, it’s a pretty old show…man, I used to watch it all the time…Cathy Lee Crosby, Fran, some other dude…John Davidson!

            • Adam

              I was born in the mid 80’s so I’m probably too young to remember the show’s heyday. Besides I’m generally out of the pop culture loop anyway haha.

  • Tom

    I’ll probably have more comments, but from the get go, this is pretty darn cool, and I agree with your methodology (as far as the weights go, etc.). Also, I think it’s *very* encouraging, as you noted, that you’re getting the same results with different people submitting and different methodology.

    • Adam

      Thanks, Tom. I was blown away that the top 14 not only remained the same from two years ago, but maintained such a gap on the rest of the field. I feel comfortable saying that there’s a good chance these really are the 14 greatest quarterbacks to ever play.

      It doesn’t look like this is Fearsome 14 is going to change for a very long time, either. Of the current QB’s, I guess Philip Rivers has an outside shot, but he appears to be in a decline and plays for a dysfunctional franchise. Tony Romo could return from injury and play like Peyton did, but that’s highly unlikely. Ben Roethlisberger doesn’t have much time left, and barring something miraculous it feels like his legacy is already written. Perhaps Matt Ryan is the strongest candidate, but he probably needs to spend another five years being among the best QB’s in football. Russell Wilson? Maybe he cracks the top 25 is his current play holds into his mid 30’s, but that’s pretty much his ceiling IMO. Cam is too inconsistent and has already taken a huge physical pounding. Am I forgetting anyone?

      • Richie

        Andrew Luck?

        • Adam

          If he were playing for a team other than the Colts, I’d give him a shot. But I have zero faith in Jim Irsay to make good decisions for his franchise, and expect Indianapolis to be a dysfunctional mess for the remainder of Luck’s tenure as their QB. I hope he decides to leave after his latest contract runs out.

          • Richie

            Lots of variables to go right or wrong for him. But after 5 seasons, he seems on pace to be amongst the 15 best QB’s ever.

            • Adam

              Is he, though? In five seasons he has yet to produce an All Pro caliber season, and he’s already been beaten to smitherines. I don’t see him winning multiple MVP’s and/or rings, especially if he stays in Indy. I have a feeling Luck will end up with a similar career to Donovan McNabb.

              • Richie

                Just a quick comparison to McNabb, Luck has three 100+ ANY/A+ seasons in his first 5 years (116, 109, 101, 97, 82). His worst was 82 in 2015 when he only played 7 games, and was pretty hurt in a couple of those games.

                McNabb had 2 such seasons (106, 103, 99, 95, 69). So Luck is a bit ahead, and he has done it in 500+ more attempts than McNabb.

                Over those first 5 seasons, McNabb was 15th in total passing yards. Luck was 9th in yards. McNabb had pretty good efficiency, but didn’t have the volume of some of the other good QB’s of his time.

                But, my main point was that Luck at least has 5 pretty good seasons under his belt. If he can turn it up a notch over the next 5 seasons (like most great QB’s do in the age 28-33 range) then he has a shot to join the conversation.

                • Ryan

                  Good point, but mcnabb may close the gap or pull ahead with his rushing value.

                  • Richie

                    Perhaps. Luck has also been a decent runner, but not quite up to McNabb’s early years. But I don’t think many people consider rushing much when thinking about which QB’s are great.

                • Adam

                  The reason I compared Luck to McNabb is because both suffered from poor supporting casts in their early years. Unfortunately for McNabb, 2004 was the only season he got to play alongside elite talent. Unless the Colts do a better job of surrounding Luck with good teammates, I have a feeling his statistical profile will look like McNabb’s – generally above average ANY/A, but never the string of truly great seasons.

                  Luck is ahead of McNabb through five years; he’s probably the superior quarterback. But I don’t think it’s accurate to say that Luck has had five pretty good seasons under his belt. His rookie year was uneven and seemed better than it really was because of all the late comebacks. He played poorly in 2015 even before the injury, and I think his career numbers would actually look worse had he stayed healthy for that whole season.

                  He definitely has a shot to join the NFL’s all time elite, I just don’t think it’s a very good shot.

          • Tom

            Agree 100%. Luck is great, but that franchise is just too shaky…something isn’t right over there.

        • Tom

          That was my first thought as well…

      • sacramento gold miners

        Agree about Roethlisberger not having much time left, he’s taken a ton of punishment over the years. However, this isn’t a Romo-type situation, with a brittle QB. And the Steelers should be contenders over the next few years, the defense has some intriguing young players.

        • Tom

          Totally separate topic, but I agree that the Steelers have a promising defense…got a brother who is a stone-cold fan and he questions the dynamics in the locker room and Tomlin’s leadership. But they’ll continue to be contenders, that’s for sure…

        • Adam

          I could see Roethlisberger having a couple more very good years. But even if he adds a few more Pro Bowls and wins another SB, how much will that really help his legacy? He still won’t reach the level of Brees, Rodgers, Staubach, Tarkenton. In my opinion, Ben’s legacy is relatively set as ending between 15th and 25th all time. He’s never been a dominating force like the QB’s in the top 10.

      • Tom

        Totally agree with you on this…there just doesn’t seem to be a guy playing today who is on that “road” to greatness. I suppose Andrew Luck might be able to move in there, but he’s been up and down. Russell Wilson has a shot; besides being really good, the Seahawks are a good franchise, and we all know that helps. Big Ben, Romo, Rivers, yeah, the window is closing for those guys…I don’t think they’ll be moving up.

  • WR

    Thanks for doing the legwork on this, Adam. I’ve been eager to see these results.

    I’m also surprised that Kelly and Warner dropped so far since the last list, although I forgot to include Kelly on my own ballot, so I’m sure that didn’t help. And I’m a McNabb fan, so I’m pleased to see him ranked where I think he belongs. The fact that Brady and Rodgers have each moved up two spots makes intuitive sense. It’s surprising to me, given the love some people have for Tarkenton, that he’s as low as he is.

    And what about poor Drew Brees? I can’t figure out what the guy has done in the last 2 seasons to actually move DOWN the list.

    • Agree about Brees. Among the QBs in the top-20, I think he’s the most underrated. I have him in the top-10.

      • Adam

        I think Brees and simultaneously overrated and underrated. He’s overrated by his numbers, which place him in the top five all time. But he’s underrated by his teams’ success (or lack thereof), and some people use this to claim he lacks the intangibles / winnersauce / clutchiness of the true greats. If he plays at his current pace through his 40th birthday, I don’t see how anyone will be able to exclude Brees from their top ten.

  • sacramento gold miners

    Norm Snead has been mentioned before, and he produced one of the strangest stat lines in a game during his final season with the Giants. Snead earned a 0.00 passer rating in a 12-9 win over the Redskins, 3-14, for 26 yards.

  • garymrosen

    Considering who is ahead of Brees, and that there is a huge gap from him to the next, I don’t think he’s doing too badly. He is only 1.6 pts from overtaking three legends – Tarkenton, Baugh and Staubach which would put him at #11. As for his stats, *all* modern day QBs are “overrated” by their stats. I noted in another thread that there were only 4 seasons in the entire 80s with QBR > 100, three by Montana and one by Marino, while now there may be that many in a year. That doesn’t mean that Brady, Manning, Rodgers and Brees don’t belong with the others, you just have to adjust.

    • WR

      By era-adjusted Passer Rating+, Montana and Rodgers are at 123, Brady and Manning at 118, Brees 116, and Marino 113. It’s worth remembering that Manning, Brees, Brady, and Marino have the most pass attempts. I had Brees 9th on my list, and that, if anything, underrates him.

      • garymrosen

        Not only is the gap after Brees remarkable, but the next 6 are all pre-Montana until you get to Warren Moon who is way way behind the leaders. Taking the 14 elite QBs by era you get:

        old school – Baugh, Graham
        60s-70s – Unitas, Tarkenton, Staubach
        80s – Montana, Marino, Elway
        90s – Young, Favre
        2000s – Brady, Manning, Rodgers, Brees

        That’s when it gets interesting, arguing both within those groups and across eras :^).

        • Richie

          Not a bad distribution. And remember, the 2000’s are nearly 2 decades (17 seasons) now.

          So Brady, Manning and Brees are basically 2000’s players while Rodgers is a 2010’s player.

          • Tom

            For some reason, I’m still thinking of the 2000’s as “current”. That was an entirely separate decade…

        • WR

          The one guy I might add to the list is Dan Fouts, who is much underrated in my opinion. Like Brees, he almost never got to play with good defenses, but his era-adjusted numbers are very impressive. I always thought he was overshadowed by guys on more successful teams, and his career began 10 years before Marino and Elway, so he started out in the dead ball era. Bryan and Brad’s metrics both rate Fouts in the top 11, which is closer to where I think he belongs.

          Warren Moon would also make the list if you include his career in Canada.

        • Ryan

          I would add old guys van brocklin and luckman, and possibly younger guys jurgensen and fouts to the top tiers, or remove favre and elway for a group of 12.

          • Adam

            If I had to make a subjective switch among the elite 14, I’d swap Elway for Van Brocklin. I believe The Flying Dutchmen should be up there with Graham, and Elway is a bit overrated (though still great).

          • garymrosen

            Agreed that Fouts and van Brocklin at least are worthy of the top tier. Van Brocklin was the only QB to win championships with two different teams, right?

            • Peyton Manning won a championship with the Colts and then had the Broncos win a championship with him.

              • garymrosen

                D’oh! :^)

  • Mike

    Will there be another edition of QBGOAT series to compare to these rankings. I feel like Brees will be much higher in that ranking compared to the wisdom of crowds unless there is a weather adjustment in the rankings.

    • WR

      If you’re talking about Chase’s GOAT rankings, then yes there is a weather adjustment. It would be interesting to see how that system treats Brees.

    • Adam

      Check out the right hand columns in the second chart – they represent a pair of statistically based career rankings. Brees ranks 4th and 5th by these methods.

  • Trepur

    Out of interest, If you just added it all up, how different would the list have been?

  • Pingback: Statistically Speaking: Sammy Baugh - The GridFe()