If various opinion polls conducted over the years are any indication, the public is ready to move on from the BCS to next season’s “plus-one”-style playoff system. But before it bids farewell forever, how does the BCS grade out relative to other playoff systems in terms of selecting the best team as a champion?

Back in 2008, I concluded that it didn’t really do much worse of a job than a plus-one system would have. But that was more of an unscientific survey of the 1992-2007 seasons than a truly rigorous study. Today, I plan to take a page from Doug’s book and use the power of Monte Carlo simulation to determine which playoff system sees the true best team win the national title most often.

(Note: If you just want the results and don’t want to get bogged down in the details, feel free to skip the next section.)

The Details

The basic framework of this post is essentially a college football version of Doug’s fabled Ten Thousand Seasons NFL post. For each of the 125 FBS schools, I generated a random Simple Rating System value with a mean of the school’s average SRS since 1998 and a standard deviation of 6.44 (the typical FBS school’s yearly variance about its overall average in the BCS era). For non-FBS schools, I assigned a constant SRS of -23.4, which is the implied SRS of all non-major schools since ’98, based on the scores of their games and their opponents’ ratings.

I then plugged all of those ratings into the 2013 season’s schedule, simulating wins/losses for each team based on Wayne Winston’s normal distribution win probability method (if you’re curious, the standard deviation of scoring margin around pregame predictions in college football is 13.89, which is remarkably close to what Hal Stern found for NFL games in the early 1980s).

After simulating an outcome for every regular-season game, I tallied up conference records (with SRS “talent” as the tie-breaker) to determine conference-championship game participants in the ACC, Big Ten, MAC, and SEC. Following those games, I computed a makeshift BCS computer ranking using the same structure as college basketball’s RPI, but with the following weights instead of 25%-50%-25%:

RPI = 75% * Winning Percentage + 5% * Opponent’s Winning Percentage + 20% * Opponents’ Opponents’ Winning Percentage

(These weights were chosen in order to most closely correlate with Jeff Sagarin’s Pure Elo Chess rating, which typifies the kind of computer rankings the BCS uses.)

With that, it was time to program various playoff systems for the universe I laid out above. The following formats were considered:

• Standard BCS – The current BCS system in place for a final time in the 2013 season… #1 plays #2 for all of the marbles.
• 4-Team Playoff – The system that will be in place next season. #1 plays #4 and #2 plays #3 in the semis, then the winners face off in the championship game.
• 6-Team Playoff – In this setup, #1 & #2 have 1st-round byes. #3 plays #6 and #4 plays #5 in the opening round; the worst-seeded remaining team after round 1 plays #1 in the semis, with the other 1st-round winner playing #2.
• 8-Team Playoff – A straightforward bracket that looks like this.
• 10-Team Playoff – In the first round, #7 plays #10 and #8 goes against #9. The worst seed among those winners gets to face #1 with #4 facing #5 in the same half of the bracket; the other winner from round 1 faces #2 in the same half of the bracket with the #3-vs-#6 matchup.
• 12-Team Playoff – A bracket that looks like this.
• 16-Team Playoff – Your standard NCAA Basketball Tournament regional bracket; looks like this.

The Results

For each of the playoff systems detailed above, I ran 5,000 simulations apiece and tracked:

• The average BCS ranking of the “true” most talented team — aka the team with the highest SRS in the simulated universe (note that this is the same in every playoff variant)
• The average talent ranking of the BCS/playoff champion
• How often the BCS/playoff champion is the nation’s “true” most talented team
• How often the BCS/playoff field contains the nation’s “true” most talented team

Here were the results:

The purpose of this exercise is to estimate the sweet spot where two factors intersect for a playoff system: the probability of the field actually including the true best team, and the ease of the true best team in winning once they are in the bracket.

The current BCS fares worst among the proposed systems above because it fails in the former category; the best team has a very good chance of winning if they do rank among the top 2 (needing to win just a single game), but the likelihood of them being selected as one of the top 2 teams is relatively low. Even the new 4-team bracket includes the true best team less than 60% of the time.

Meanwhile, the more teams that are included in the bracket, the more likely it is that the true best team will be included — but also the more likely it is that they’ll be upset before winning the championship. (This is the problem faced by the 16-team bracket, which offers the best team as champion at a rate barely above the current BCS despite including the best team in the field more than twice as frequently.)

Balance seems to be achieved somewhere in between. In terms of most frequently seeing the true best team win, the 6-, 10-, and 12-team brackets came out ahead of the rest of the pack, with the 10-team playoff’s champ ranking the highest in true talent on average (coincidentally, Chase’s preferred college football playoff is the 10-team structure). Why? One major reason is that all 3 setups give superior teams a boost via 1st-round byes, something not offered in the 8- and 16-team bracket varieties. As a favored team, it’s always better to not have to play a game than to introduce the chance of an upset, however slim.

The best team won’t always win (or even usually win) no matter which setup is used, but this simulation suggests that moving to a Final Four next season won’t mean college football finally has the most optimal system possible. The powers that be always speak of “bracket creep” as though adding more teams is a bad thing, but here’s hoping they eventually give in to the temptation of a bigger tournament and settle at 6, 10, or 12 teams.

The basic conundrum for Andrecheck revolved around the very existence of a postseason tournament, since — logically speaking — such a thing should really only be invoked to resolve confusion over who the best team was during the regular season. To use a baseball example, if the Yankees win 114 games and no other AL team wins more than 92, we can say with near 100% certainty that the Yankees were the AL’s best team. There were 162 games’ worth of evidence; why make them then play the Rangers and Indians on top of that in order to confirm them as the AL’s representative in the World Series?

Andrecheck’s solution to this issue was to set each team’s pre-series odds equal to the difference in implied true talent between the teams from their regular-season records. If the Yankees have, say, a 98.6% probability of being better than the Indians from their respective regular-season records, then the ALCS should be structured such that New York has a 98.6% probability of winning the series — or at least close to it (spot the Yankees a 3-0 series lead and every home game from that point onward, and they have a 98.2% probability of winning, which is close enough).

This style of setup may seem strange (and, admittedly, the 1998 Yankees are an extreme example), but it preserves the integrity of the regular season by tying the odds of postseason success quite directly to performance during the 6 months leading up to the playoffs. And despite the long odds, there’s still an opportunity for the underdog to turn the tables and advance. It would take an incredibly improbable sequence of events, but that’s what a team should have to accomplish in order to undo 162 games worth of evidence in the opposite direction.

As for football, the NFL obviously doesn’t play series, but the same concept can still be applied; instead of spotting games in a series, we can spot a team the lead in points before kickoff. In the NFL, Chase & I once found that a team’s “true” talent can be estimated by adding eleven games of .500 ball to its regular-season record. Using that, we can calculate the probability of either team’s true talent level being higher in a given matchup, and add points until the pregame win expectancy matches said probability.

Take tomorrow’s Bengals-Chargers tilt. Cincinnati went 11-5 (true talent: .611), while San Diego went 9-7 (.537); both of those talent estimates come with a standard deviation of .096. Based on their records, the probability of the Bengals’ true talent being higher than San Diego’s is 70.7% (this is derived from the means/standard deviations listed above and the mathematical proofs laid out here). In order for the pregame win probability to be 70.7%, we must spot the Bengals about 7.3 points to begin the game — however, the game is also in Cincinnati, and we know this typically means they will start with a built-in 2.5-point advantage, so we’d only need to add about 5 points (spotting them a 5-0 lead to begin the game) in order to bump their win probability up to the level deserved by their regular-season record relative to San Diego’s.

Here are the number of points we’d have to add, rounded to the nearest integer, for all of this weekend’s games:

Note that this also addresses the seeming inequity of having 8-7-1 Green Bay host the 12-4 49ers; the Packers can be at home, but we’ll spot San Francisco a 15-0 lead to start the game — 12.8 for the pure difference in regular-season records and 2.5 more because they’re having to play on the road. Likewise, the Saints (11-5) would get an automatic 6-0 lead to start their game with the 10-6 Eagles since the game is in Philly, and the Chiefs would start out leading 3-0 against Indy because they’re having to play on the road despite both teams posting identical 11-5 records during the season.

Another benefit of this setup is that every regular-season game matters. No longer would teams have nothing to play for and rest their starters in week 17, when an extra win could very easily make the difference between winning and losing a playoff game.

Finally, if we dislike that the NFL playoffs seem to be getting more random in recent seasons, this process will nip that trend right in the bud. For instance, good luck to the 10-6 Giants going into the Super Bowl against the 16-0 Patriots, facing an instant 22-0 deficit — which is what this system would produce by dint of the biggest disparity in records of any playoff game since 2002. (The difference between the 2011 Giants and Packers’ records was equally large, but Green Bay would only start that game with a 19-0 lead under this system because they were at home).

Of course, maybe such unpredictability isn’t a bad thing — the NFL’s popularity has never been greater than during this period of wacky playoff outcomes — but if the goal is purely to make the playoffs fairer and give regular-season games more meaning, a handicapping system like this would reduce the role of randomness and ensure that the best team is rewarded more often with postseason success.