Previous post:

Next post:

Guest Post: Projecting Pass/Run Ratios in Division Round

by Chase Stuart on January 9, 2014

in Playoffs, Statgeekery, Statistics

Steve Buzzard has agreed to write another guest post for us. And I think it’s a very good one. Steve is a lifelong Colts fan and long time fantasy football aficionado. He spends most of his free time applying advanced statistical techniques to football to better understand the game he loves and improve his prediction models.


Last month, I wrote about how to project pass/run ratios using offensive Pass Identities and the point spread. However, this methodology only considers one side of the ball. Can we actually improve our projections model using both offensive and defensive Pass Identities? As it turns out the answer is yes.

First, I started off by creating defensive Pass Identities using the same methodology found here. The first thing I noticed was the standard deviation of pass ratios for defenses was only 3.0% compared to 5.1% for offenses. This led me to believe that offenses control how much passing goes on in a game more than defenses. I was glad to see this as it confirmed most of my previous research as well. Given this, it wasn’t appropriate to use a standard deviation of 3.0% for defenses in my projection while using a standard deviation of 5.1% for offenses. Instead, I used the combined standard deviation of all 64 offensive and defensive pass ratios, which turned out to be 4.17%. This doesn’t change the order of passer identities much but obviously does increase the deviation from the mean for the offensive side of the ball and decrease it for the defensive side. [Chase note: Determining the best way to handle the differing spreads between offensive and defensive pass ratios is a good off-season project; in the interest of time, I advised Steve to split the difference and move ahead with the analysis.]

Now that we have a Pass Identity grades for both sides of the ball, we can add a strength of schedule adjustment, too. To make the SOS adjustment, I simply took the average of the defensive Pass Identities played by each offensive unit and the average of the offensive Pass Identities played by each defensive unit. As expected the SOS adjustments had a much larger impact on the defensive Pass Identities than the offensive Pass Identities.

Here is the full list of adjusted offensive Pass Identities using the 4.17% standard deviation mentioned above and the strength of schedule adjustment. A positive SOS adjustment means the team faced teams that, on average, other teams passed against more often than average. If a team has a positive SOS adjustment, that means we need to reduce the team’s pass identity.

Rk
Team
Game Script
StDev GS
Pass Ratio
StDev PR
SOS Adjustment
Pass Identity
1DEN6.392.0361%0.620.02139.4
2NOR1.990.6363.7%1.27-0.02128.9
3ATL-1.23-0.3968.2%2.350.04128.8
4DAL1.60.5164.6%1.490.08128.7
5CLE-1.84-0.5867.6%2.210.1122.9
6MIA-0.96-0.366%1.820.1121.3
7KAN4.371.3957.1%-0.320116.1
8PIT-0.07-0.0261.5%0.740.11109.2
9NWE1.960.6258.3%-0.030.04108.3
10DET0.610.1960.1%0.410.09107.5
11IND-0.5-0.1660.6%0.53-0.12107.2
12ARI0.210.0759.1%0.17-0.12105.3
13BAL-0.83-0.2661.1%0.650.06104.9
14CIN3.090.9856%-0.580.1104.5
15CHI-1.79-0.5760.3%0.450.0897.1
16NYG-3.12-0.9961.8%0.810.0596.5
17GNB0.190.0657.2%-0.290.0496
18HOU-4.31-1.3761.9%0.84-0.193.5
19SEA5.611.7848.6%-2.36-0.1293.1
20SDG1.320.4254.5%-0.94-0.0693.1
21CAR3.781.251.3%-1.71-0.0392.9
22PHI2.830.952.7%-1.370.0592.2
23SFO5.861.8647.7%-2.57-0.0990.6
24TAM-1.14-0.3657.1%-0.32-0.0590.6
25STL-0.63-0.255.1%-0.8-0.1587.4
26JAX-6.56-2.0863.1%1.13-0.187.2
27MIN-2.7-0.8658.5%0.020.0786.3
28TEN-0.93-0.355.3%-0.75-0.185.7
29OAK-2.94-0.9356.5%-0.46-0.0379.6
30WAS-6-1.959.1%0.17-0.0174.2
31BUF-0.79-0.2551.5%-1.660.0271
32NYJ-3.46-1.152.1%-1.520.0460.2

For example, the Steelers had an almost perfectly average Game Script (-0.02) and had a Pass Ratio that was 0.74 standard deviations above average. However, Pittsburgh’s schedule was loaded with teams that other teams passed against, too — in other words, the Steelers faced a schedule that was pass-friendly. As a result, the SOS adjustment of .11 reduces their Pass Identity to 109.2 (100 + 15*.61).

Here is the full list of defensive Pass Identities.

Rk
Team
Game Script
StDev GS
Pass Ratio
StDev PR
SOS Adjustment
Pass Identity
1ARI0.21-0.0764.5%1.48-0.25124.8
2MIN-2.70.8660.9%0.610.13120
3NYJ-3.461.159.8%0.360.24118.3
4JAX-6.562.0853.4%-1.17-0.18116.3
5NYG-3.120.9958.3%-0.01-0.02115
6WAS-61.955.4%-0.710.32113
7DET0.61-0.1961.6%0.790109
8CLE-1.840.5858.3%-0.020.01108.3
9OAK-2.940.9357%-0.310.12107.5
10HOU-4.311.3753.2%-1.23-0.16104.4
11TEN-0.930.356.6%-0.42-0.27102.3
12CAR3.78-1.263.9%1.330.06101.2
13DAL1.6-0.5160.1%0.41-0.17101
14PIT-0.070.0258.2%-0.03-0.04100.4
15CIN3.09-0.9863.1%1.130.14100.1
16MIA-0.960.356.2%-0.5-0.1499
17SDG1.32-0.4260.8%0.590.2598.8
18TAM-1.140.3657.1%-0.310.1598.6
19IND-0.50.1656.3%-0.48-0.2198.4
20ATL-1.230.3954.6%-0.89-0.3197.2
21PHI2.83-0.961.5%0.750.0696.9
22STL-0.630.256.6%-0.420.0296.5
23BAL-0.830.2657.4%-0.230.2996.1
24NOR1.99-0.6359%0.15-0.0693.8
25BUF-0.790.2556.7%-0.380.3193.4
26GNB0.19-0.0657.4%-0.220.292.8
27KAN4.37-1.3959.9%0.37-0.0986
28SFO5.86-1.8661.1%0.66-0.2685.9
29CHI-1.790.5752.7%-1.350.1785.7
30NWE1.96-0.6257%-0.330.1284
31DEN6.39-2.0360.9%0.61-0.281.8
32SEA5.61-1.7857.4%-0.23-0.2273.1

Since the defensive Pass Identities are new to the site, let’s take a closer look at some of the teams. Let’s start at the bottom of the list which contains the defenses that are least likely to be passed on. Seeing Seattle as the team that most teams are afraid to pass against makes a lot of sense: the Seahawks had a historically great pass defense, finishing with an ANY/A average that was 2.93 standard deviations better than average. It is also not surprising to see Chicago near the bottom. That’s not because of how great the Bears pass defense was, but instead due to the fact that Chicago allowed half a yard more per rushing attempt than the 31st worst run defense. Finally, seeing New England and Denver in the bottom four isn’t completely surprising either as teams try to keep the ball away from Tom Brady and Peyton Manning by shortening the game.

On the other hand, a couple of the teams that are most likely to be passed on are a little surprising, especially the number one team. The Cardinals were very good against the run (ranking 3rd and allowing only 3.7 yards per attempt) but were also very good against the pass (ranking 6th and allowing only 5.1 ANY/A). The Giants are another team that stands out as a little unusual. Like Arizona, New York had a pretty good run defense (ranking 4th giving up 3.8 yards per attempt) but had a passing defense that was nearly as good (ranking 6th at 5.1 ANY/A). In both of these cases I would have expected the pass defense to rule coaches’ decisions more than the rushing defense but it seems to be the other way around. Minnesota and their 30th ranked pass defense and the Jets and their #1 ranked rushing defense aren’t nearly as surprising to be at the top of this list.

So now let’s combine the offensive Pass Identities with the Defensive Pass identities and Vegas point spreads to get the projected pass ratios for this weekends playoff games. We will use the same methodology as last month but the following is a quick summary.

We start by obtaining the Vegas point spread for each of the games this week and taking one half of it to get the approximate Game Script for the game. Next, we determine how many standard deviations from the mean this Game Script will be. The standard deviation for Game Scripts this year was 5.86. Then, we convert the offensive and defensive Pass Identities listed above back to their standard deviation form. Now we can predict the pass ratio for each team by using the following formula: League average pass ratio + (A+B+C)*Standard Deviation of pass ratio where

(A) = number of standard deviations above/below average the projected Game Script
(B) = number of standard deviations above/below average the offensive side of the ball is in Pass Identity
(C) = number of standard deviations above/below average the defensive side of the ball is in Pass Identity

Where the league average pass ratio is 58.3% and the standard deviation of all offensive and defensive pass ratios is the one derived above of 4.17%.

Here is a full list of the projections for this week. Here’s how to read the table below. The team projected to be the most pass-happy this week is New Orleans. The Saints are 7.5 point underdogs this week against the Seahawks, which translates to a projected Game Script of -3.8. That means the Saints Game Script is projected to be 1.19 standard deviations below average against the Seahawks. During the regular season, the Saints had a Pass Identity of 129, which was 1.92 standard deviations above average, while the Seahawks defense had a defensive Pass Identity of 73.1, which was 1.79 standard deviations below average (meaning less pass-friendly against). This means we would project the Saints to be 1.32 standard deviations above average in terms of pass ratio this week. That gives New Orleans a projected pass ratio of 63.8%, which is right at their year to date average. In other words, the two competing factors — likely to be trailing and facing a dominant pass defense — are projected to essentially cancel each other out.

Team
Opp
Point Spread
Game Script Value
Stdev GS
Off Pass Identity
Off Pass Identity (St Dev)
Def Pass Identity
Def Pass Identity (St Dev)
Total St Dev
Proj Pass ratio
2013 Pass Ratio
% Change
NORSEA7.5-3.81.19128.861.9273.1-1.791.3263.8%63.7%0%
DENSDG-9.54.8-1.51139.442.6398.8-0.081.0462.6%61%3%
INDNWE9-4.51.43107.210.4884-1.070.8461.8%60.6%2%
SDGDEN9.5-4.81.5193.11-0.4681.8-1.21-0.1657.6%54.5%6%
SFOCAR-10.5-0.1690.61-0.63101.20.08-0.7155.4%47.7%16%
NWEIND-94.5-1.43108.260.5598.4-0.11-0.9954.2%58.3%-7%
CARSFO1-0.50.1692.88-0.4785.9-0.94-1.2653.1%51.3%3%
SEANOR-7.53.8-1.1993.11-0.4693.8-0.42-2.0749.7%48.6%2%

What can we gather from these projections? One thing to note: we should see a general uptick in passing this weekend, since the combined year to date pass ratio of these eight teams is 55.7% and their projections this week are about 57.3%. That said, all four games are outdoors in January, so the weather could result in depressing some of the pass ratios, which means we might not see much of an uptick after all.

Kaepernick may have to carry his team to the conference championship game.

Kaepernick may have to carry his team to the conference championship game.

Secondly, the most key offensive unit this week may be Colin Kaepernick and the 49ers passing game. San Francisco has typically only passed on about 47.7% of their plays, but this study projects them to pass closer to 55% of the time. That’s an increase of about 16% from their year to date average. That’s because San Francisco has played with an average Game Script of 5.86, but (1) this game is expected to be even, and (2) teams tend to pass against Carolina, as a result of Luke Kuechly and the Panthers’ great front seven combined with the team’s less-than-stellar secondary. And while the 49ers were not very successful when trying to pass against Carolina during the regular season (Kaepernick gained just 46 net yards on 28 pass plays), Michael Crabtree was not active for that game.

Some of these extra dropbacks may simply result in more scrambles for Kaepernick, but either way, I suspect that more will be placed on the quarterback’s shoulders than in all but two other games from 2013. The first of those games was when Kaepernick lost to the Saints 23-20 in week 11. Against New Orleans, Kaepernick was only 17-31 for 127 yards, and PFR graded the San Francisco passing offense as being -0.26 expected points added that day. The second game was in week 17, when the 49ers beat the Cardinals by that same score of 23-20. Against Arizona, Kaepernick played much better (with Crabtree around) going 21-34 for 310 yards, producing 13.37 expected points.1 If the 49ers hope to win this week at Carolina, Kaepernick will have to play significantly better. My guess is that he will, continuing to build on his successful playoff narrative.

Another interesting game that sticks out is the Patriots only being expected to pass on 54.2% of their plays. That would be their 6th lowest pass ratio for the year. Can we really expect the Patriots to go into a playoff game with a healthy Tom Brady and pass so little? There are two main reasons to believe so. First, the Patriots aren’t really that pass happy this year due to so many changes in the passing game, as New England ranked only 9th in Pass Identity. Second, the Patriots have played a lot of close games this year and their projected game script of 4.5 would be the team’s 6th largest game script of the year, depressing the team’s pass ratio. In the other 5 games with large Game Scripts, New England’s average pass ratio was only 50%. This includes the last two weeks against the Ravens and the Bills, where New England only passed on 41% of all plays.

Finally, the matchup that might be the most intriguing is the Sean Payton’s high flying pass offense vs. the all time great Seahawks pass defense, where this model expects the Saints to pass on 63.8% of their attempts which is right at their year to date numbers. However, what’s interesting is this is 11% higher than the 57.4% that teams have passed on the Seahawks so far this year. During their first meeting, a 34-7 blowout loss, Drew Brees attempted a pass on a whopping 70% of the team’s snaps. As you probably remember, very few of those pass attempts were positive as New Orleans’ pass offense produced -19.5 expected points on 38 passing attempts for 147 yards and a fumble. Of course, the Saints didn’t do much better running the ball, either, gaining only 44 yards on 17 attempts (worth -4 expected points). It doesn’t take advanced stats to know the Saints will have to do better than that this time around if they want to win. But what is amazing is the miserable 3.8 ANY/A the Saints posted was already better than the 3.2 ANY/A the Seahawks gave up this year (one big reason: it was one of just 5 games where Seattle did not force an interception). In other words, it’s going to take a close to perfect game from New Orleans to beat Seattle by passing on 64% of all plays, given the dominant Seattle pass defense. That task may be even more difficult, if we assume that the lack of penalties in the playoffs will help the Seahawks ferocious pass defense.

  1. In week 1 against the Packers, Kaepernick of course had an excellent game. But against Green Bay, the 49ers only passed on 55% of their plays. []

{ 8 comments… read them below or add one }

Ajit January 9, 2014 at 2:02 am

I have to say, I am really surprised by these findings for this reason – I thought the great offenses encouraged teams to pass and be more aggressive than otherwise.

I noticed this pattern a few years back in 2010, when big ben was suspended for the first four games. The Pittsburgh defense played really well from a defensive scoring standpoint in those 4 games. When Ben came back, the defense got worse. I noticed this pattern also held up between the 07 pats and the 08 pats.

This led me to believe that teams with great offenses will probably have understated defenses and conversely, teams with great defenses but lousy offenses(teams like the ravens of the past and most of this decade’s bears) will have overstated defenses. I’d like to examine this more in the summer.

Reply

buzz January 9, 2014 at 9:27 pm

I was also fairly surprised by the way some of the defensive pass ratios turned out. You always hear about teams wanting to shorten the game against Brady and Manning and clearly that is what the teams try to do. But I always thought a strategy of trying a higher variance play such as passing more often would be at least as good if not better. Most teams it seems rather just wait to start passing until the game script tell them to – And then some. The interesting aspect about this is passing is generally a better strategy than running in general so teams are playing a nonoptimal strategy against the best teams. You have to wonder if that helps the great teams be even better.

Now the Cardinals and the Giants are another story. Somehow they have convinced teams to pass more heavily on them than I would have expected given their stats. Since it is generally a losing strategy to have teams pass on you more than what the game script dictates I have to wonder if this deflated their winning % somewhat.

Reply

Nate January 9, 2014 at 9:50 pm

Shortening the game is a high variance strategy. (It’s probably not a good one, but c’est la vie.) It may be selective memory, but it does seem like underdog teams have been better about taking risks this year. Jacksonville’s 2-point attempt against Denver was an excellent example.

Reply

Ajit January 9, 2014 at 2:05 am

Of course, this would also be true for the NYJ

Reply

Eric January 9, 2014 at 11:31 am

I would love to see the projected ratios compared with the actual ratios over the entire season to see the margin of error for this exercise.

Also might weighting recent games give a more accurate prediction to control for player health and other major changes in offensive style (personnel, player development)?

Reply

Danny Tuccitto January 9, 2014 at 3:22 pm

That was my first thought as well: Potentially great application of pass identity; would love to see the prediction residuals. It’s obviously only one week’s worth of evaluation, so nothing anywhere near definitive, but figured I’d look at what happened back in Week 14. Turns out there was a mean absolute error of 9.7%. If you were to just make the projection based on the offense’s actual P/R Ratio at the time, the MAE would have been 9.8%, with half of the projections being worse, and half being better. In both cases, though more so with Steve’s method, there was a tendency for higher P/R Ratio projections to miss on the high side and lower projections to miss on the low side. Like I said, only looking at one week isn’t enough, so I’m by no means trying to rain on anyone’s parade here. Just throwing it out there to continue the discussion because I think Steve’s idea has a ton of potential.

Here’s the list (sorry I couldn’t format it any better):

Team Proj PR Obs PR Error
HOU 53.4% 76.3% 22.9%
WAS 50.0% 72.3% 22.3%
NWE 53.1% 72.7% 19.6%
IND 60.0% 79.3% 19.3%
BAL 53.7% 65.8% 12.1%
BUF 54.4% 65.6% 11.2%
PIT 57.6% 67.7% 10.1%
NOR 66.3% 72.1% 5.8%
DEN 59.9% 64.4% 4.5%
OAK 56.5% 60.9% 4.4%
NYG 59.8% 63.0% 3.2%
NYJ 43.3% 44.8% 1.5%
MIN 59.0% 60.3% 1.3%
CAR 61.8% 62.9% 1.1%
STL 61.4% 61.2% -0.2%
SEA 56.8% 55.1% -1.7%
ATL 62.8% 61.0% -1.8%
JAX 53.1% 50.9% -2.2%
SFO 54.0% 49.2% -4.8%
GNB 58.1% 52.9% -5.2%
CHI 58.9% 53.6% -5.3%
ARI 57.5% 51.6% -5.9%
CIN 56.8% 50.0% -6.8%
MIA 67.5% 60.0% -7.5%
CLE 73.5% 64.3% -9.2%
TEN 64.1% 54.2% -9.9%
TAM 53.5% 42.6% -10.9%
SDG 56.1% 42.9% -13.2%
DET 65.3% 46.3% -19.0%
PHI 53.8% 32.8% -21.0%
DAL 70.5% 49.1% -21.4%
KAN 61.5% 37.7% -23.8%

Reply

buzz January 9, 2014 at 9:18 pm

Eric and Danny,

Really great points and questions. First of all maybe I should have prefaced this in the article but this model is certainly in its infancy. I back tested on both week 13 and the same week 14 that Danny used. Week 13 had results similar to what Danny posted for week 14 namely that this model did about the same as using the year to date pass ratio. In both weeks it projected if a team passed above or below their average slightly better than 50%. Obviously none of this shows any special predictive value for the model but again it is a small sample.

What I do think the model gives is an interesting starting point for predicting pass ratios that is at least as accurate as the year to date pass ratios that most people use but is often quite different from these ratios so you can use a starting point that is different than the masses and in my opinion is a more reasonable if you want to start adjusting for injury etc. Most importantly, I think it gives a good starting point for improving the model in the offseason. A couple of things I already know 1) I am pretty sure the defensive pass identity is weighted too high. My model without it seemed to actually do better (again a small sample size) but theoretically I think including it is better so I kept it there for now. 2) There is an inherent limitation of the accuracy of the point spread. For example, if it was possible to know the actual game scripts I am pretty certain the model would improve 10-20% based on my findings. That itself is useful to know and I can narrow that number with more testing or test other factors.

Reply

Danny Tuccitto January 10, 2014 at 1:43 am

100% agree that, regardless of whatever we both found in a small validation sample, there’s still likely a benefit in terms of anti-conventional wisdom arbitrage for the model as is. And yeah, was assuming this was just a first or second step; hope I didn’t come across as meaning otherwise.

In terms of going forward, I definitely concur with moving away from the point spread as your game script predictor, though it’s generally the most logical (and easiest) place to start. I’d imagine the solution would be coming up with your own win probability model that fills in the blanks of where the point spread goes way wrong. Or, before doing that, you could use Burke’s game probabilities (archived on his site via the NYT), since they’ve done pretty well against the spread over time. The other, more important, thing to address (to me, at least) is the last part of my previous comment. Don’t think you want the model to be biased at the extremes, with the residuals being dependent on a team’s P/R Ratio standing.

Good luck with the offseason tweaks. Eager to see what comes next.

Reply

Leave a Comment

You can use these HTML tags and attributes: <strong> <em> <pre> <code> <a href="" title="">

{ 1 trackback }