Q: Aaron, can you go into specifics on the type of work you do for NFL teams? Do you envision ultimately working for a team?

As far as consulting with teams, I’ve done two different sorts of things. First, I’ve done some in-game decision analysis, some fourth down stuff as well as some analysis on when to accept or decline penalties. Second, I’ve done reports for teams in February that gave analysis of the season with our stats, looking at what issues were likely to statistically regress and what issues really needed to be addressed, along with suggestions for possible free agent signings. Actually, it’s more accurate to say “we’ve done” rather than “I’ve done.” Some consulting I’ve done alone, and sometimes two or three guys on the FO staff work together.

Consulting for teams is great, but as advanced analysis people gradually move into front offices I don’t think I will be one of them. I don’t know about the various other folks who have followed in FO’s footsteps, but my heart has always been with the media, going back to my days running my high school paper, through my time as a radio disc jockey, doing the Lycos 50, and now Football Outsiders. I set out to revolutionize the way people analyzed the NFL, not the way they managed teams. If I end up improving the way people manage teams a little bit too, that’s just extra coolness.

Q: You publish your DVOA rankings every week, one of the most popular football articles on the web. Have you ever gotten flak from a team for them (i.e., how come we’re hiring you, we have a winning record, and you have us 24th!)?

No flak, no. A couple times I’ve had teams that I’ve worked with or that I’m otherwise in contact with ask me why their rating is particularly low in one area. However, unlike a lot of fans, people who work for teams understand that our stats are objective based on a general formula and don’t get tweaked to favor one team over another depending on how we feel each week. I think when people ask me why their team is low in one area, they often ask so that they can improve that area. And when a team hits rock bottom, I mean, they know it. The Jacksonville people don’t need to ask me why the Jaguars are ranked 30th in DVOA, or whatever it is this week. They don’t care as much about their DVOA right now as they do about their DVOA (and record) next year or two years from now. Jim Schwartz has told me he would rather have his defense ranked highly in DVOA than in yards per game. Of course, he’d rather have more wins than either. (In case it’s not clear otherwise, I should point out there are more teams where I’ve got contacts among various coaches and front office people than there are teams that I have actually worked for and received a check from.)

Q: Fumble luck is a drum FO has beaten for awhile. Is this something that you’ve discussed with NFL teams?

A little bit. I think they basically buy the concept, at least the people I talk to. People I don’t talk to? I mean, I would guess Lovie Smith does believe that his coaching leads to a higher fumble recovery rate, not just a higher fumble forced rate, but I don’t know, I’ve never talked to him.

A lot of it gets back to something I’ve said over and over again: when it comes to dinosaur thinking in the NFL, the media is a much bigger problem than front offices are. There certainly are things I would like to see teams be smarter about — fourth downs, understanding aging curves when they sign free agents, understanding where their scheme requires them to spend money and where their scheme lets them be more frugal — but there certainly are teams that aren’t living in the old world of “establish the run” anymore. More coaches are old school than front office people. But a good number of front office people, and a few coaches, do understand concepts of risk and randomness.

The media, on the other hand… ugh. Beat reporters are better now than they were 10 years ago — there are some really smart beat reporters now, who read FO and PFF and Advanced NFL Stats and even your site and they understand where we are all coming from. There are people like Greg Bedard who will do their own tape analysis articles, and writers like Mike Sando who will do their own stat analysis. However, you still have some guys trapped in 1975. Most color commentators still stand behind coaches when they make ridiculously conservative game decisions. I don’t know if Mike Mayock does, he’s so smart about everything else that maybe he’s smart about this too, I can’t remember a situation where I’ve noted it one way or another. And all the former players and coaches on the various studio shows know a lot about motivation and what it’s like to prepare for games, and sometimes they know about X’s and O’s, but they don’t understand advanced stats or the concept of probability at all, and they constantly grab onto completely absurd storylines just so they can fill airtime.

Mike Smith's team either just converted a 4th-and-1 or won another one-point game.

Yeah, it has been frustrating. You see a couple of famous decisions like some of Mike Smith’s get all this attention, and it makes you think maybe coaches are being more aggressive, and then you look at overall numbers and actually they seem to be getting less aggressive over the last couple years, even guys like Belichick seem to be less aggressive than they were a couple years ago. However, I do think (and hope) this is a situation where fans of advanced analysis need to just wait things out. As people move into front offices, eventually they will get the ears of the coaches, and I do think eventually you will have some coaches who are more aggressive. Maybe not as aggressive as Brian Burke’s win probability calculator thinks they should be, but certainly more than now.

Chip Kelly coming to the NFL might help also, especially if he has success quickly. I’ve said this before, but what we really need is for an owner to come out in public and say that he wants his coach to be more aggressive on fourth down. You need an owner who gives public backing for smarter decisions to try to get younger coaches over that hump of being worried about their job security if results don’t work out in the short term.

Q: Do you think 20 years from now, advanced analytics, especially of the type FO provides, will become widespread? On a scale of 1 to Baseball, if the NFL is at a 4 right now, how long do you think until we get to an 8 or 9?

Hmmmm… well, I guess there are two different issues here. Advanced analytics spread in two ways. The first is the ideas — color commentators understanding that fielding percentage is pretty useless, general managers understanding the value of walks or Ultimate Zone Rating, managers using more bullpen flexibility. Then you have the stats — OPS showing up everywhere, or WAR, or in the case of basketball, Houston putting Dean Oliver’s Four Factors up on their scoreboard.

In football, I would expect that the ideas will spread much easier than the stats. The fact is that many of the stats are proprietary. I explain what’s basically in DVOA, but I don’t give out the specific baselines. ESPN doesn’t give the specific formula for Total QBR. We both have business reasons behind these decisions. The “charting” type stats are also proprietary. You have all these groups doing charting and for the most part we don’t share data so the numbers aren’t out there for everyone to use and they don’t even agree with each other, ESPN may have one count of drops and PFF another and Stats Inc. a third.

I would hope that the “ideas” part of things can get to an 8 or 9 in 20 years. I still think the “stats” part will be something that’s there for the teams to use, and for a particularly hardcore subset of fans. They aren’t going to show up much on broadcasts.

Q: Thanks for taking the time, Aaron. Last question: We know that records in close games has little predictive value, and we’re seeing the Falcons again ride a string of close wins to a great record. Is that something you’ve talked about with teams? I picture this being difficult to bring up with a head coach, who only sees his team’s grit and a never-say-die attitude.

I’ve never talked about this with a team specifically, no. But I’ll say this. We know that in the long run, records in close games have little predictive value. In the short run, however, obviously teams often do specific things to win specific close games. They make the right coaching calls and they make plays when it counts. Not every close win is based entirely on one lucky bounce. So I don’t want players thinking that records in close games are random. They better think that it matters that they do things right because in each specific game, considered on its own, it often does. And I don’t think it’s a problem for a coach to think his team has grit and a never-say-die attitude. What is important is that when you go into the offseason, the personnel people don’t misread how good the team truly is because that team happened to have a 6-2 record in close games thanks to its grit and never-say-die attitude, because that same team is likely to show a sometimes-say-die attitude the following year.

Like a lot of issues with regression towards the mean, the problem isn’t patting yourself on the back for something you have accomplished, it is being unprepared for the future because you were deluded about the long-term meaning of those accomplishments.

My bigger question about this stuff is not close games, but things like red-zone performance. We know that red-zone performance regresses towards the mean significantly from season to season, where your red-zone performance tends to move back towards your overall performance if it is really high or really low. What does that say about the importance of designing specific plays for the red zone? I honestly don’t know.

Pete Gogolak, not Brian Burke.

But Gogolak’s impact wasn’t limited to identifying the optimal technique for kicking a football: he also helped usher in an era of specialists. In the early days of the NFL, there was no room for a specialist as rosters were tiny and players played on offense, defense and special teams. Unlimited free substitution wasn’t permanently instituted until 1950, and as recently as 1963, teams were limited to just 37-man rosters.

Once teams were allowed to roster more players, and a certain unique brand of kicking was proven to be superior, a more specialized NFL emerged. In 1949, nobody would have signed a soccer-style kicker, or any person who could only kick a football. We joke now that kickers aren’t real football players, because back in 1949, a kicker would also need to play tight end or free safety. The idea that 5’11, 182-pound, 42-year-old Jason Hanson could be a contributing member of an NFL team is as noncontroversial in 2012 as it would have been laughable in 1952. It’s not going to take 60 years before an advanced statistical analyst — perhaps the front office version of a kicker — becomes a contributing member of an NFL organization.

This weekend, I sat down with Brian Burke, the founder of Advanced NFL Stats, a fantastic website on football, statistics and game theory. Burke’s win probability calculator has been one of the most exciting innovations in our industry. In Part II of this series, I’ll be interviewing Football Outsiders’ Aaron Schatz. Neither person is a threat to Ron Rivera’s job security anymore than Jason Hanson is a threat to steal Calvin Johnson’s job. Specialization is the way of the world, and hiring someone trained in the art of decision-making isn’t any different than choosing to hire a lawyer or doctor. We can’t expert anyone to be an expert in everything.

There are some things NFL coaches do wonderfully well. They know how to train and develop players, how to teach proper technique, and how to line up against different coverages. Coaches know how to design a play and how to craft a defense to stop such a play. They’re experts at designing an offense and sprinkling in the right constraint plays. Coaches aren’t better at this than the average person because coaches are geniuses; they’re better at it because that’s what they do. They’ve been trained to do these jobs for decades, and they’re very good at them.

But what does knowing how far to shade a defensive tackle have to do with knowing whether to go for it on 4th and 1 from the 50? In my view, it is about as similar as the skills involved in sprinting 30 yards down the field and catching a pass and the skills needed to kick a 45-yard field goal. Why should we expect there to be overlap in these areas? Why do we expect Marvin Lewis to know the inner workings of a win probability model? We shouldn’t, at least not anymore than we’d expect Calvin Johnson to be able to kick a field goal. That’s where guys like Schatz and Burke come in. Both have consulted with NFL teams. Below is my interview with Brian.

Q: Brian, how have teams reached out to you? Has your work with the New York Times and Washington Post been helpful in increasing your exposure?

Teams initially contacted me through email. In most cases, they set up a phone call or asked to meet with me in person before we went any further. In other cases, it’s strictly been an online relationship.

The New York Times and Washington Post exposure definitely helps. It lends credibility to the work, and it spreads the word. I feel a great debt of gratitude to the Times in particular. It was my analysis there of the 2009 Patriots-Colts 4th down play that catapulted Advanced NFL Stats into its 15 minutes of fame. But I’m learning now that my exposure to the league has also come thanks to early readers of my website. Young front office guys and analysts who buy into the analytics movement were readers of ANS in its early days when they were in grad school or even younger. They grew up speaking the language and using the tools ANS developed, so even many of the analysts for teams I’ve had no direct contact with ‘speak ANS.’ The big breakthrough was probably in 2008/09 when I was able to start doing analysis that could truly alter the way teams approached the game. Until then, advanced football stats had primarily been team ranking and player ranking stuff, which is great fodder for fans but is of little real value to teams.

Q: NFL teams are notoriously secretive: do teams ask you if you’re working with other teams? Have any teams asked you to sign an exclusivity agreement with them?

Whenever I do formal work for a team I’ll sign a non-disclosure agreement. That way the teams can be assured I’ll compartmentalize their own ideas and data. It works both ways; the stuff I do stays within their organization. No one has asked for a total exclusivity agreement, just exclusivity within the their division. Most of my portfolio is available for free online to everyone, but some of the projects I’ve worked on recently are proprietary to one team or another. That’s one reason I haven’t been posting as often as I had in the past.

Q: What type of work do you do for NFL teams? Are they mostly looking for work related to your Win Probability Model?

I can’t comment specifically on what I do for them. Teams are secretive about these things for obvious reasons. But it’s no secret that the win probability model has a lot of direct applications that teams would be interested in. Different teams are interested in different things. Some are interested in decision analysis (4th downs, onside kicks, and clock management, for example). Others are interested in player analysis or salary analysis. I’ve become aware of a couple teams that have been using my Expected Points model for their own purposes even though they had never contacted me. (The full model is freely available.)

Q: As someone who has been an outsider to the NFL, what has been the most surprising part of working with NFL teams?

The first thing that surprised me was how smart coaches really are. They didn’t get to where they are by being dull. They’re thinking about considerations that never dawned on guys like us. We’ll criticize them for sub-optimum decisions, but they have a hundred different things to juggle on any different play–clock, play, personnel package, opponent personnel, injuries, etc. It’s unfair for us to expect them to be game theorists and microeconomists on top of everything else they do.

The second surprising thing is that even teams that have analytic staff don’t often listen to them. The analytic staff is usually separate from the coaching staff on the org chart, and they don’t always have the coaches’ attention. It may be another half-generation before analytic-friendly coaches are in positions to take full advantage of what we’ve learned.

But the biggest surprise, in all honesty, was that some teams were taking advanced stats seriously. I first noticed hits at the site from various team facilities. (My old hit counter service made that very easy.) I thought they must be stray clicks simply because my site comes up on the first page when you Google ‘nfl stats’. My thinking was that we were doing all this analysis in a vacuum just for consumption by like-minded statheads, but it turns out ANS had a few secret admirers.

Q: Why do you think coaches aren’t listening to their analytic teams?

Right now the league is in a sub-optimum equilibrium. Nobody is playing the ‘optimum’ way. The thing is, if every coach plays outdated strategies, there’s little inducement to change. Coaches see the Super Bowl teams playing the same risk averse way that they do, so they mistakenly think that’s the optimal strategy.

Q: Are there any particular stats teams like from ANS, such as Air Yards?

Expected Points is very useful, but I would say Air Yards is one of those things for fans rather than teams. QB performance metrics like that aren’t that useful to teams because there is tape on every quarterback in the league. They can just watch the tape on a guy and learn a lot more than any stat can tell you. Stats are great for when you don’t have time to watch dozens of hours of film. They’re also useful for calibrating your expectations and checking your biases, but with only 16 games in a year, film can is more valuable to an expert than any statistic.

Q: You’ve written a lot about how the running game is overvalued and paying running backs in particular is a losing proposition. How would you sell this to an NFL team?

It should be an easy sell. All you have to do is ask, “Remember all those top running backs who led their teams to Super Bowl wins? Yeah, neither do I.” But I have a feeling that the view of running backs will be tough to overcome because there is so much emotion wrapped up around ‘franchise’ running backs. They are typically very popular players in their home city, so letting a top free agent walk would be very tough for an owner, even if the front office bought into what the stats say.

Q: Thanks again for your time, Brian. Last question: Do you think 20 years from now, every team will use a Win Probability Model to help them make decisions?

I think win probability models will have an impact on in-game decisions, but not the way you might think. NFL rules prohibit the use of computers during games, so the WP model will serve to recalibrate coaches’ intuitive judgments. Maybe we’ll be able to create some rules of thumb. Certainly the EPA-based rules are easy enough to use during games though. But I have no doubt that things like WP models will be used for many things. I wouldn’t say I’m only scratching the surface with its applications, but there is much more to be done.

I think there will be a tipping point in a few years. There will be a head coach that adopts the analytics mindset, and when other teams see his success they’ll quickly follow. The advantage is there for the first adopters. Just like with ‘Moneyball,’ once a critical mass of teams are all using the same analytical tools, the advantage is neutralized. Now is the time for teams to capitalize, before that tipping point occurs.