Which Passers Were Most Effective on Third Down in in 2014?

In 2012 and 2013, I looked at which passers were most effective on third and fourth downs; today, we examine those numbers for 2014. Throughout this article, when I refer to “third downs” or “third down performance”, note that such language is just shorthand for third and fourth downs.

To grade third down performance, I included sacks but discarded rushing data (in the interest of time, not because I thought that to be the better approach). The first step in evaluating third down performance is to calculate the league average conversion rate on third downs for each distance. Here were the conversion rates in 2014, along with the smoothed (linear) best-fit rates:

To GoPassesFirst DownsRateSmoothed
125815158.5%55.8%
238921655.5%53.1%
347723148.4%50.4%
450224849.4%47.8%
552323745.3%45.1%
646420844.8%42.4%
743618742.9%39.7%
846117437.7%37%
936310729.5%34.4%
1059017629.8%31.7%
112165224.1%29%
121795027.9%26.3%
131312317.6%23.6%
141262116.7%21%
151181815.3%18.3%
16841416.7%15.6%
1768913.2%12.9%
1864812.5%10.2%
195048%7.6%
2056712.5%4.9%

Once we know the expected conversion rate for each distance, itâ€™s easy to grade the quarterbacks.1 The next table is a bit complicated, so let me just walk you through the best third down quarterback in the league last year. Tony Romo had 141 third down passing plays (either pass attempts or sacks); the average distance “to go” on those plays was 7.0 yards, and the expected conversion rate (based on the smoothed rates in the table above for each play) was 39.8%; that means Romo would be expected, if he was an average quarterback, to convert 56.1 first downs. In reality, he converted 70 first downs, a 49.6% conversion rate. This means, despite being a choker, Romo produced 13.9 more first downs than expected, the best in the NFL (and the category by which the table is sorted). He also had a 1st down rate over expectation of 9.8%, the second-highest behind Carson Palmer’s injury-shortened season. On third downs, Romo averaged 6.82 Net Yards per Attempt and 7.74 Adjusted Net Yards per Attempt.

RkPasserTm3DAvg To GoExp Rt.Exp ConvAct 1D1D Rate1D Ov Exp1D Rt Ov ExpNY/AANY/A
1Tony RomoDAL141739.8%56.17049.6%13.99.8%6.827.74
2Aaron RodgersGNB1487.139.4%58.47148%12.68.5%7.899.44
3Drew BreesNOR1806.939.9%71.88446.7%12.26.8%7.085.44
5Ben RoethlisbergerPIT1677.338.9%657544.9%106%7.668.44
7Colin KaepernickSFO1597.737.8%606943.4%95.6%6.896.36
8Carson PalmerARI807.737.8%30.23948.8%8.810.9%7.738.6
9Eli ManningNYG1997.338.8%77.28442.2%6.83.4%7.056.9
10Matt RyanATL1826.940%72.87842.9%5.22.8%6.86.66
11Alex SmithKAN1517.937.4%56.46140.4%4.63%5.745.93
12Mark SanchezPHI1027.239.1%39.84443.1%4.24.1%6.995.96
13Peyton ManningDEN1706.940%687242.4%42.3%7.158.24
14Andy DaltonCIN1577.837.6%59.16138.9%1.91.2%6.825.61
15Josh McCownTAM1098.436%39.24137.6%1.81.6%4.433.1
16Andrew LuckIND1696.939.9%67.46940.8%1.60.9%7.538.74
17Nick FolesPHI857.438.6%32.83440%1.21.4%5.945.35
18Teddy BridgewaterMIN1247.638%47.14838.7%0.90.7%7.086.64
19Austin DavisSTL897.438.6%34.43539.3%0.60.7%5.386.45
20Shaun HillSTL717.438.7%27.52738%-0.5-0.6%5.465.11
21Cam NewtonCAR1417.638.1%53.75337.6%-0.7-0.5%5.234.17
22Joe FlaccoBAL1667.239.1%64.86438.6%-0.8-0.5%6.486.54
23Ryan TannehillMIA1657.438.7%63.86338.2%-0.8-0.5%4.944.45
24Drew StantonARI757.438.6%28.92837.3%-0.9-1.2%6.797.59
25EJ ManuelBUF456.840.3%18.11737.8%-1.1-2.5%5.784.22
26Matthew StaffordDET1937.638%73.37237.3%-1.3-0.7%76.77
27Kyle OrtonBUF1457.338.8%56.35537.9%-1.3-0.9%6.676.15
28Ryan FitzpatrickHOU1047.638.2%39.73735.6%-2.7-2.6%6.225.93
29Mike GlennonTAM539.134%181528.3%-3-5.7%4.915.57
30Jay CutlerCHI1617.438.6%62.15936.6%-3.1-1.9%6.145.42
31Kirk CousinsWAS586.840.2%23.32034.5%-3.3-5.7%4.031.71
32Michael VickNYJ447.239.2%17.31431.8%-3.3-7.4%2.81.66
33Russell WilsonSEA1518.136.7%55.55234.4%-3.5-2.3%6.135.33
34Colt McCoyWAS42836.9%15.51228.6%-3.5-8.3%3.832.76
35Geno SmithNYJ1287.139.5%50.64736.7%-3.6-2.8%6.516.51
36Jake LockerTEN477.438.5%18.11327.7%-5.1-10.9%4.532.94
37Charlie WhitehurstTEN687.737.8%25.72029.4%-5.7-8.4%5.075
38Zach MettenbergerTEN577.737.7%21.51424.6%-7.5-13.2%4.254.51
39Robert GriffinWAS677.538.4%25.71826.9%-7.7-11.5%4.31.91
40Blake BortlesJAX1508.635.5%53.34429.3%-9.3-6.2%4.43.17
41Derek CarrOAK2086.940%83.17335.1%-10.1-4.9%5.384.81
42Brian HoyerCLE1547.937.4%57.74529.2%-12.7-8.2%5.124.63
• The players with the lowest expectation third down rates are those quarterbacks whose third down pass attempts were disproportionately coming on third and longs. So it’s not surprising that the three lowest expected rates came from the quarterbacks on Tampa Bay and Jacksonville. The 4th lowest expected rate, however, came from Russell Wilson. That’s more of a reflection of Seattle’s tendency to run (or Wilson to scramble, perhaps) on third and short. Wilson had just 20 throws on 3rd and less than 4 yards (he converted 11); as a point of comparison, Matt Ryan had 43.
• At the risk of igniting a Manning/Brady debate, the numbers here are interesting. Manning averaged 7.15 NY/A and 8.24 ANY/A on third downs, both excellent numbers. Brady was a little worse, at 6.33 NY/A and 6.74 ANY/A. But Brady converted 1 more first down despite having 11 fewer opportunities. Brady’s third down rate was excellent, while Manning’s was merely very good. Manning had 11 conversions go for at least 30 yards, compared to just 3 for Brady. Neither ANY/A nor 3rd down conversion rates tell the story — getting the first down is key, but it’s also better to gain more yards than less, so a 30-yard pass on 3rd-and-5 is better than a 6-yard one. As usual, pick your flavor here.
• Colin Kaepernick had a bad year, right? Well, not so much on third downs, where he was one of the most productive players in the league.

With three years of data, let’s look at the 22 quarterbacks who have had at least 70 third down attempts in each year since 2012. The table below shows each quarterback’s 1st Down Rate Over Expectation for the three years.2 You won’t be too surprised to see which quarterbacks top the list:

QB201220132014Avg
Peyton Manning9.5%10.7%2.3%7.5%
Drew Brees6.2%7.4%6.8%6.8%
Aaron Rodgers2.4%8.3%8.5%6.4%
Tony Romo7.2%0%9.8%5.7%
Philip Rivers-0.2%9.9%6.7%5.5%
Matt Ryan7.7%4.4%2.8%5%
Ben Roethlisberger5.9%1.5%6%4.5%
Colin Kaepernick1%5.1%5.6%3.9%
Matthew Stafford6.8%2.8%-0.7%3%
Carson Palmer-2.4%-0.4%10.9%2.7%
Nick Foles-1.6%7.7%1.4%2.5%
Ryan Fitzpatrick3.5%5.9%-2.6%2.3%
Russell Wilson3.6%2%-2.3%1.1%
Jay Cutler-0.5%4.6%-1.9%0.7%
Cam Newton-1.8%3.3%-0.5%0.3%
Eli Manning-3.2%-0.2%3.4%0%
Andrew Luck3.6%-5.5%0.9%-0.3%
Alex Smith-2.8%-2.1%3%-0.6%
Andy Dalton-8.2%5%1.2%-0.7%
Joe Flacco-2.4%-1.6%-0.5%-1.5%
Ryan Tannehill-3.1%-1.2%-0.5%-1.6%
Avg1.8%3.1%3%2.6%
• I’m not nearly the Ryan Tannehill lover that some out there seem to be, and these numbers cast the Dolphins passer in a poor light. Tannehill’s good enough to be a regular starting quarterback, but has yet to show much evidence that he’s an above-average starter.
• The correlation coefficient between these rates in 2013 and 2014 was just 0.08, indicating that there’s quite a bit of luck involved here (as is often the case with small sample sizes). It was only slightly higher, at 0.17, from 2012 to 2013. Andy Dalton went from -8.2% to 5% from 2012 to 2013, before settling in at a closer-to-average 1.2% last year. Tony Romo was at 7.2%, then 0%, and then 9.8%. In other words, there’s a lot of variation here (unless you’re Drew Brees or Joe Flacco, another player who has been really unproductive on third downs with some notable exceptions).
1. Note that I have discarded all third down attempts where the distance to go was greater than 20 yards. []
2. Note that in 2013, I was lazy and used the 2012 rates, which may explain why quarterbacks looked slightly better, on average, in 2013 than they did in 2012 or 2014. In reality, third down rates were up league-wide, albeit slightly, in 2013 compared to 2012. []
• Anders

Advancedfootballanalysis’ tools is not available right now, but I remember that in terms of EPA or WPA, its better to try to get 6 yards than 30 yards on a 3rd down attempts because the odds of getting 6 yards is easier than 30 yards and most of the EPA/WPA comes from the conversion and not the yards.

Of course those models do not take into account that the QB is Peyton Manning throwing to D. Thomas

• Kibbles

I suspect there’s an element of game theory to it, as well. Getting 6 yards on 3rd and 5 is a lot easier if you occasionally attempt to get 30 yards, too, than it is if you simply try to get 6 yards every time. It’s similar to how rushing is more efficient than passing on 3rd-and-short, but you occasionally have to pass on 3rd-and-short anyway to break tendencies. It’s a good example of Braess’ Paradox in that the goal isn’t to mix the two until they achieve some sort of efficiency equilibrium, but rather to deliberately sprinkle in less efficient plays to avoid draining the effectiveness of the more efficient play too much, thereby making the entire offense more efficient. (Yes, calling less efficient plays can make an offense more efficient; there’s a reason it’s a paradox.)

Not necessarily saying this as a specific defense of Peyton Manning vis-a-vis Tom Brady. If those 30-yard attempts were making everything easier, you’d expect that additional ease to already be reflected in the conversion percentage.

• Yes, this is also correct, although (over time) that would also just show up in the numbers here.

• Sure, but you still get more EPA/WPA from a 30-yard pass than a 6-yard pass. The marginal value of each yard goes down, but there’s still more to third downs than just getting first downs. That’s all I was saying.

• sacramento gold miners

35 year old Tony Romo has had an impressive NFL career, and leads this category. But the postseason has been his weakness, as Don Meredith and Danny White have been more successful QBs in this area. Two playoff wins is beyond disappointing, and his window for a strong HOF resume is closing fast.

This stat does help provide part of the reason for Romo’s struggles in the playoffs, individual numbers don’t always tell the full story, timing is so important in these usually tight playoff games. It’s not just the total numbers during a postseason game, but when those plays are made. Three of four of those Dallas losses came by a TD or less, and just a few plays could have made a difference. Romo’s third down/fourth down pass play conversion rate was a paltry 12-34 in those losses, and that’s part of the story behind those losses. Other more successful postseason QBs may not have a good conversion rate either, but those QBs either made more plays on first and second down, or avoided mistakes Romo could not. Last season’s playoff loss to Green Bay was a good example, Romo completed a glittering 79% of his passes, but was only 1-7 in this specific category in that game.

• sacramento gold miners

Yes, I took the research a little further, and it was surprising to see Romo didn’t perform better in this specific category in the playoffs. The postseason is a different animal, and we may never know why certain QBs don’t play as well in these crucial games as others.

If Ben Roethlisberger played for the Cowboys, the Dez Bryant catch would have counted. #7 knows how to get it done when it matters.

• Just to be clear, I was the first person to call Romo a choker today.

• Jonathan Aicardi

You do know Romo’s Passer Rating in the playoffs is now higher than Tom Brady’s and both Mannings, yeah? And had that play to Bryant been called a catch and the Cowboys prevail, Romo’s playoff record would’ve gone to 3-3 while Rodgers’ would have dropped to 5-5.

But that’s what people mean when they say beware of small sample sizes. It only took TWO games to bring Romo’s playoff numbers up.

• Tom

As much as I love these discussions, the bottom line is, the only way to really evaluate these guys is to do the impossible (or impractical): watch every single play of every game the QB has ever played and mark down those plays where a completed or incomplete pass is the QB’s fault, the WR’s fault, or the offensive line’s fault, etc. That Romo play is a good example…that goes down in the books as an incomplete pass, a “failure” on Romo’s part. I watched an old Indy playoff game recently on YouTube where Manning threw a perfect pass to Harrison and he drops it…flat out drops it. However you want to slice it, Manning performed perfectly on that play and the other guy didn’t. Or Welker missing that beautiful pass by Brady in the 2011 Super Bowl, or Kearse not holding on to Wilson’s pass in the 3rd quarter in this past Super Bowl, etc., etc. Apart from the obvious – Manning’s pick-sixes in the Super Bowls – we just can’t say these ridiculous things like “this guy didn’t get it done”, etc., without all the data. Maybe he DID get it done, and the other guy didn’t.

• sacramento gold miners

Unfortunately, we just don’t have those kind of stats yet, and fault can definitely be passed around. At the end of the day, the QB is still the most important position on the field, and his performance impacts both sides of the football.

No one is suggesting the highest rated QBs should all have great postseason records, but when a short list guy like Peyton Manning has a losing record in over 20 tries, that’s a problem. Ditto for a Tony Romo, who has just a couple playoff wins. A winning playoff QB is usually getting it done during those key times in these tight playoff games, and that’s all that matters. The glittering numbers are fine as long as they lead to points or a lack of mistakes. In Romo’s case, we can speculate what would have happened on the Bryant pass, but that’s not reality. During a given postseason game, there are dropped passes by receivers, but great catches as well. While the QB doesn’t deserve all the blame, there still must be accountability.

• Tom

Agreed, especially about the fact that along with the dropped passes, there are great catches (the Swann/Stallworth effect, etc.), and on top of that, pass interference calls (the Flacco effect). I guess was just mostly commenting on how if we TRULY wanted to judge/rank/evaluate these guys, we’d need to see every play. That being said, it seems to me that, as far as winning goes, the best stat we have right now is Win Probability. Brady threw two picks in the last Super Bowl, but because he didn’t throw a pick on the Pats’ last drive, when it mattered most, he’s a champion (also thanks to Malcom Butler, etc.)

• Richie

According to the PFR filter, I countRomo at 13-45 on 3rd/4th down in the playoffs in his career (worse than your stat).

But you’re right, Romo sucks on 3rd/4th down. Except when he doesn’t.
In playoff games he won, Romo was 14/26 (54%) conversion rate. In losses he dropped to 13/45 (29%).

Brady was 97/211 (46%) in wins and 33/92 (36%) in losses. Brady chokes too. Except when he doesn’t.

• JoeS

Romo’s worst enemy isn’t himself, it’s those in the media and online who took a small sample from years ago and “decided” that Romo is a “choker”. That’s it. End of discussion forever. Romo can make 99 plays that are worthy, but, they’ll hunt for the 100th that isn’t and say – See! I told ya! He’s a choker!
I would never say that Romo is one of the all-time greats. But, he is a solid, if not much better, QB. You want to know what modern QB made the most “choke” plays? Brett Favre (and by a wide margin). But, early on, he was deemed “clutch”, so no matter how many bonehead plays (like tossing a football far up in the air downfield in OT of a PLAYOFF game) – Favre got a pass.

• sacramento gold miners

I have Brett Favre as a top ten QB, so he would be vastly superior to someone who currently, wouldn’t make Canton in my opinion. Romo’s had a very good career, but so did Danny White, and when you play for a franchise like the Cowboys, postseason success is magnified. Only having two playoff wins right now is a major problem for Romo’s legacy, despite the good stats, he just hasn’t made the timely plays other QBs have made in those crucial situations. At 35, Romo is running out of time to even lead the Cowboys to an NFC title game, something White did for three straight seasons.

It’s true, Favre went 4-7 after a 9-4 start to his postseason career, and his gunslinger’s approach did result in some untimely picks. But at least he was still winning games, and had the Vikings in an NFC TG, at an advanced age. Romo’s back issues may prevent him from doing the same, and the only way he can help himself is to win, regardless of what the stat line is. A QB can have a lower passer rating, but if he hits on those key plays, it can make all the difference in the playoffs.

• JoeS

I agree that Favre was the superior QB, but, he also had superior teams around him for the most part. Once Jerry Jones took over the Cowboys from Jimmy Johnson, the team has never been fully complete. You can’t pin that on Romo.
Further, my main point is that Romo is still though of as the guy that fumbled the snap in Seattle (conveniently forgetting that he drove the team downfield on that critical drive late in the game). He has never lived it down. Favre did win the one Super Bowl, although it should be noted that was only after Jimmy Johnson’s carryover effect on the Cowboys had waned (remember Favre’s 0-7 record vs. the great Dallas teams?). Because of that one win, he was forever proclaimed as clutch despite a myriad of dumb plays. There’s a good reason why Favre has the most interceptions of any QB ever – and, no, not all them are related to his longevity.
Favre is the better QB (Not Top 10), but Romo is underrated.

It seems as though third down conversion rates share a similar property with turnovers – highly volitale from year to year, but disproportionately important in determining wins and losses.

• Jonathan Aicardi

Every time we end up with a list like this, it seems we’re not surprised as to who performs best among QB’s. Lists of conversion rates, total QBR’s, ANY/A+, etc … The actual order within talent tiers may vary from year to year but the tiers themselves are pretty consistent regardless of situation or leverage of play, which is to say that success is virtually always dictated by true talent. “Clutch plays” exist as discrete events, but QB’s accumulate them as a function of true talent as opposed to an actual skill of “clutch.”

And this isn’t a new idea. Fangraphs has documented the same thing in baseball, where the idea of a “clutch player” is more a myth of personal narrative than a product of data. Different players in different seasons will be “most clutch” but over time the average level of clutch begins to reflect certain players simply having more skill than others.

That should make sense, given that we can measure basic QB skills with things like WPA added and QBR and ANY/A+, and the like. But to say anyone is a “clutch player” is conferring a complex and abstract psychological skill of performance under pressure. Was Joe Montana great in critical situations or was he just great ALL THE TIME?

Exactly. With large sample sizes, the best QB’s always rise to the top. I feel very confident that if any QB played in 500 playoff games, his performance level would almost exactly match what he’s done in the regular season. “Clutch” is the equivalent of randomly picking one hour from your day and having it represent the other 23.

• Richie

Please pick 10am-11am for me, because I am AWESOME at sleeping in.

• Tom

I’m pretty good at around 2am-3am, when I’m dead asleep and not screwing anything up.

• sacramento gold miners

The postseason just doesn’t work that way, if Tony Romo wants the opportunity to improve his playoff record, he’ll have to earn another postseason appearance. Jim Plunkett was something like 9-2 in the playoffs, but we didn’t see that level in the larger sample size of the regular season. We really don’t know if a larger sample size would reflect the regular season in Romo’s case, it hasn’t worked out that way with Peyton Manning, who has had many more chances.

It may not be fair, but it’s simply the nature of the postseason, and that’s part of a player’s resume. Players aren’t fixed mathematical equations. Joe Montana was usually great in clutch situations, occasionally not. His rating in this topic’s category would likely be higher than a less successful QB like Romo. There have definitely been great players who have been more “clutch” than other great QBs. It’s just not a lottery out there, the QB position does have some control over the outcome.

• Richie

No, we haven’t seen it in Peyton Manning. Manning has played in 256 regular season games and 24 playoff games. That’s not nearly enough playoff games to be certain that his results are representative of his actual ability.

• sacramento gold miners

And that’s unfortunate, and we can’t help him. As I’ve said before, the postseason doesn’t overshadow the entire career, but it’s part of the legacy. No one knows why Manning has made some bad decisions in the postseason, and not had a record more representative of his career, but that’s life.

I think we have to arrive at the inevitable conclusion we don’t have all the answers. Manning has been tremendous under pressure in the regular season, but why it hasn’t translated in the playoffs is a mystery.

• Richie

I don’t think it’s mysterious. I think it is due to a couple things:
– the randomness of small sample size
– the higher level of competition
– the fact that if you lose a playoff game, you don’t get to play any more that year.

I feel that all QB statistics will regress towards their career averages, and probably slightly below their career averages (due to playing only stronger opponents). But, we’ll never know for sure.

• sacramento gold miners

Agreed, we’ll never know for sure, and that’s what makes sports enjoyable. I don’t want to see a best of three playoff format, and the Super Bowl played in March. We have no idea, even with a larger sample size, when the QBs would play more like the regular season. Everyone plays under the same rules, and has the opportunity to add to the quality of their career.

I guess we don’t have data on the older QBs, such a table would be great for wide receivers as well.

• Tom

Well said.

• Richie

Russell Wilson is surprisingly low. But he’s also a guy who seems to run for a few first downs. It’s possible that he might see the biggest improvement if rushing were included.

• Yeah, I think that’s right. He also may not take many chances on 3rd downs when Seattle is winning, or maybe it’s just a product of a small sample size. I would generally think of him as a pretty good 3rd down player, although remember, Wilson here is just a proxy for the Seattle passing offense, which can be pretty underwhelming.

• Tim Truemper

Put Tony Romo into the discussion and look what happens.

• I know, right?

• JoeS

Get over it, Tim. Romo is underrated. Period.

• Anonymous

I read somewhere that about 25% of completions on third and long situations fall short of the first down marker. Can someone explain to me why this happens? Is the qb under so much pressure in some of these situations that he thinks it’s best to dump it to someone 5 yards short of the first down and hope that the guy can run for the first down yardage? I’m sure in some of these cases the receiver might lose track of the first down marker and catch it half a yard short of the line, but that wouldn’t explain the 5 yard passes on third and 10. Do some coaches just give up on third and long and decide that we might as well put the opponent five yards back in field position? Considering how conservative coaches are on fourth down, I think we know that they’re almost never trying to set up an easier to convert fourth down.

• JoeS

Contrary to what one might think, I am not a Romo “apologist”. As a Cowboys fan, I see his strengths and weaknesses (do I!). But, also, it means I have seen virtually every play Romo has participated in (including his pre-starter exhibition games). Romo gets a permanent bad rap because of a few admittedly bonehead plays. He deserves that criticism, but, also deserves MUCH more praise than he is presently given.

• Tim Truemper

To Joe S below. “Get over it.” Not sure why you say that. So to make my point clear, I have followed the Cowboys since 1966. I see a lot of Meredith and Romo comparisons insofar as the early part of Romo’s career. I think he is great and feel bad that he is often in the center of so much controversy regarding his capabilities. I tell everyone who criticizes Romo in an unfair way that Dallas is lucky to have him. Looking at all the other teams and their investment into QB’s with high draft picks, or in trading for an established veteran, and getting often mixed results, it is a miracle that Dallas got Romo as a free agent. Thank you Sean Payton for noticing him when you were the QB coach for Dallas.