≡ Menu

Two years ago, I set a baseline for what a pre-season projection system should hope to accomplish. The simplest baseline of all would be to project each team to go 8-8. That would require no thought at all: a person could wake up 49 years and eight months from now and project each then-existing team in the NFL (or its successor league) to go .500 in the 2067 season. So any projection system has to beat that, at a minimum. As I wrote two years ago:

If you did that in every season from 1989 to 2014, your model would have been off by, on average, 2.48 wins per team. This is calculated by taking the absolute value of the difference between 0.500 and each team’s actual winning percentage, and multiplying that result by 16. So that should be the absolute floor for any projection model: you have to come closer than that.

Now, let’s flash back to July of this year. The USA Today published its preseason predictions in a rather provocative fashion, particularly with respect to two teams. I see a ton of preseason projections every year and forget them minutes later, so please forgive me if you feel like I am picking on the USA Today here. That is not the intent, and other publications have made more egregious errors but are not at my fingertips. But this publication picked the Patriots to go 16-0 and the Jets to go 1-15, which was rather extreme.

Upon further view of their predictions, many of them are pretty ugly. So I decided to compare those predictions to the “every team in the same” test, or the “8-8” system.

There were six teams the USA Today got really wrong, where an 8-8 projection would be at least 3.0 games closer to being accurate:

The Buffalo Bills are 8-7 (off of 0.500 by 0.5 wins); the USA Today predicted them to go 4-12 (off by 4.5 wins).
The Green Bay Packers are 7-8 (0.500 projection is off by 0.5); the USA Today predicted them to go 12-4 (off by 4.5).
The Los Angeles Rams are 11-4; an 8-8 projection would be off by 3.5 games, but the USA Today had the Rams are 4-12, off by 7.5 games.
The Oakland Raiders are 6-9 (off by 1.5); the USA Today had them at 11-5 (off by 4.5).
The Detroit Lions are 8-7 (off by 0.5); the USA Today had them at 5-11 (off by 3.5).
The Tennessee Titans are 8-7 (off by 0.5); the USA Today had them at 12-4 (off by 3.5).

You might say the loss of Aaron Rodgers shouldn’t be held against them, and that the Rams success caught everyone off guard. Both of those things are true, but we also know that superstars get hurt and surprise teams happen every year. Both of those facts should urge predictors to be more conservative in the aggregate.

The USA Today thought the Titans would be great and the Bills terrible; both are 8-7. And the USA Today had the Raiders at 11-5 and the Lions at 5-11; instead, Detroit has two more wins than Oakland. These, again, are signs that we shouldn’t be too overconfident in our preseason projections (a look at the Giants and the Rams projected wins totals would also work to that effect).

Now, there were also three teams where the USA Today beat the 8-8 system by at least 3 games.

The Indianapolis Colts were projected to go 5-11 by the USA Today; they are 3-12 (USAT off by 1.5, .500 prediction off by 4.5).
The Cleveland Browns were projected to go 4-12; they are 0-15 (USAT off by 3.5, .500 prediction off by 7.5).
The Pittsburgh Steelers were projected to go 12-4; they are 12-3 (USAT off by 0.5, .500 prediction off by 4.5).

In the case of the Colts, Browns, and Bears, predicting them to be bad — but not terrible — was the wise move. (That would have worked with the Jets, too: the 1-15 projection is now off by 4 or 5 wins, giving another win to the 8-8 system). The Steelers were projected to be very good, and that prediction was nailed. The USA Today even wins the Patriots bet as we stand right now (12-3 is closer to 16-0 than 8-8), but a more conservative approach would have been better.

Overall, the USA Today fared worse than the blind 8-8 system. The USA Today projections were closer than the 8-8 system for 13 teams. Meanwhile, the 8-8 system was closer on 15 teams, with four teams (Houston, Kansas City, Jacksonville, New Orleans) currently graded as ties. Take a look:

TeamWLWin%USAT W%DiffAbs Val Diff (USAT)Abs Val Diff (8-8)Closer?
New York Giants2130.1330.6250.4920.4920.3678-8
Tampa Bay Buccaneers4110.2670.6250.3580.3580.2338-8
Oakland Raiders690.4000.6880.2880.2880.1008-8
Green Bay Packers780.4670.7500.2830.2830.0338-8
Cleveland Browns0150.0000.2500.2500.2500.500USAT
Houston Texans4110.2670.5000.2330.2330.233tie
Tennessee Titans870.5330.7500.2170.2170.0338-8
New England Patriots1230.8001.0000.2000.2000.300USAT
Arizona Cardinals780.4670.6250.1580.1580.0338-8
Dallas Cowboys870.5330.6880.1540.1540.0338-8
Atlanta Falcons960.6000.7500.1500.1500.1008-8
Indianapolis Colts3120.2000.3130.1130.1130.300USAT
Denver Broncos5100.3330.4380.1040.1040.167USAT
Seattle Seahawks960.6000.6880.0880.0880.100USAT
Miami Dolphins690.4000.4380.0380.0380.100USAT
Los Angeles Chargers870.5330.5630.0290.0290.033USAT
Chicago Bears5100.3330.313-0.0210.0210.167USAT
Baltimore Ravens960.6000.563-0.0380.0380.100USAT
Pittsburgh Steelers1230.8000.750-0.0500.0500.300USAT
Cincinnati Bengals690.4000.313-0.0880.0880.100USAT
Kansas City Chiefs960.6000.500-0.1000.1000.100tie
Washington Redskins780.4670.313-0.1540.1540.0338-8
Jacksonville Jaguars1050.6670.500-0.1670.1670.167tie
San Francisco 49ers5100.3330.125-0.2080.2080.1678-8
Detroit Lions870.5330.313-0.2210.2210.0338-8
New Orleans Saints1140.7330.500-0.2330.2330.233tie
Minnesota Vikings1230.8000.563-0.2380.2380.300USAT
New York Jets5100.3330.063-0.2710.2710.1678-8
Buffalo Bills870.5330.250-0.2830.2830.0338-8
Carolina Panthers1140.7330.438-0.2960.2960.2338-8
Philadelphia Eagles1320.8670.563-0.3040.3040.367USAT
Los Angeles Rams1140.7330.250-0.4830.4830.2338-8

So how bad were the USA Today’s predictions? Well, they have been off by an average winning percentage of 0.197 per team. How bad is that? The 8-8 system is currently off by an average of 0.169 per team. On a per-16 game basis, the 8-8 system is off by 2.70 wins per team, while the USA Today’s projections are off by 3.16 wins per team. That is… a lot.

Now recall, in an average year, the 8-8 system will be off by 2.48 wins per team, so this is not a year where teams are huddled around .500 more than usual. If you want to throw the USA Today a bone, you might argue that the standings are more unpredictable than usual (although some data to support that claim would be wise), but they are not more tightly bunched.

But again, the intent here is not to criticize one publication, whose intent may not have been accuracy, and certainly wasn’t accuracy vs. a robot system to be judged by a crazy blogger five months later. The true lesson is to understand that there is a lot of uncertainty in the world, and more concretely, in the NFL. And if your projections are less accurate than a system that says every team will finish with the same record, you should probably rethink your projection system.

{ 5 comments }
  • Ryan

    Nice alliterative title. I’d be interested to see where the 8-8 and USAT systems end the year vs. the other projection system approaches from your previous article, as well as others like Vegas and Football Outsiders.

    • Thanks, Ryan. The data is all publicly available, so if you or anyone wants to put together a guest post on that, I’d be happy to publish.

  • Anders

    The way any system score unless you give very high points for been more aggressive its always better to predict most teams 3-4 wins around 8-8 rather than predict some very good and very bad teams.
    We know there will be some 2 win teams or 13 win teams the problem is predicting who

    • Well, it’s better to be accurate than aggressive, no?

      • Jason Lisk

        USA Today doesn’t get page views by predicting every team to go 8-8. So of course better in this case is which one gets more attention, unless readers actually punish them for being outlandish and wrong in the past, and give them less attention (see, Boy Who Cried Wolf).