Tuesday, November 17, 2009

Econometricians are all over themselves congratulating a losing move in this week's Indianapolis Colts versus New England Patriots football game. Econometrician Steven Levitt sets the scene:
I respect Bill Belichick more today than I ever have.

Last night he made a decision in the final minutes that led his team the New England Patriots to defeat. It will likely go down as one of the most criticized decisions any coach has ever made. With his team leading by six points and just over two minutes left in the game, he elected to go for it on fourth down on his own side of the field. His offense failed to get the first down, and the Indianapolis Colts promptly drove for a touchdown.
Here's his defense of Belichick's move:

The data suggest that he actually probably did the right thing if his objective was to win the game. Economist David Romer studied years worth of data and found that, contrary to conventional wisdom, teams seem to punt way too much. Going for a first down on fourth and short yardage in your end zone is likely to increase the chance your team wins (albeit slightly).
Here are the problems with Levitt using Romer's paper as justification for Belichik's move.

1. Romer only studied first quarters of football games not fourth quarters.

2. Second Romer used third down data not fourth down data!

Romer writes:
Decisions to go for it on fourth down (i.e., not to kick) are sufficiently rare, however, that they cannot be used to estimate the value of trying for a first down or touchdown. I therefore use the outcomes of third-down plays instead. I then compare the values of kicking and going for it to determine which decision is better on average as a function of where the team is on the field and the number of yards it needs for a first down or touchdown.
This demonstrates one of the problems with econometricians. They are often not even working with the data that will supposedly yield answers to their questions. I have to wonder if Romer has ever watched a professional football game from the view of anything but a college cheerleader. Does he really think third down, first quarter data is going to tell him anything about fourth down, fourth quarter success rates? Does he realize that any football team in the first quarter is playing with a much wider field that the defense must guard against and a much greater number of possible plays? Is he aware that an offense may try plays, or a defense might try a coverage, in the first quarter just to test an opponent? Is he aware that a significant increase in knowledge may be assembled about the play of the game by either the defense or offense or both, by the fourth quarter?

Romer claims to make some adjustments from some of these deficiencies, which means throwing more mathematical assumptions on top of his current assumptions.

Further, Romer makes no adjustment for the quality of the offensive team that will take the field if the first down attempt is not made. My guess is that you test giving Indianapolis Colts' Peyton Manning the ball with only 28 yards to go for a touchdown, even the stats are going to show that it is idiotic to give him even a small chance of that happening.

Bottom line: Romer's study is pretty much useless for fourth down plays (Especially against Peyton Manning). He basically has taken the old joke about economists making assumptions and taken it to a new level of Dali-like absurdity. Yes dear, assume a fourth down and assume a fourth quarter, while your at it. If Belichick was using this study as a reference in making his decision, well then, he got what he deserved, a big fat loss. And Levitt in his post about this continues to do what he does best, jump up and down about half baked data that make for interesting stories, but just like in his book Freakonomics are dangerous if implemented.

UPDATE: Not surprisingly, Greg Mankiw is a big fan of the third down, first quarter, is fourth down, fourth quarter club. He writes:
Chapter 2 of my favorite textbook [his own] has a box on David Romer's work on 4th down strategies in football. One fan of this work is Patriots' coach Bill Belichick, who recently applied Romer's analysis...It did not work out well in this particular case, and Belichick is coming under some heat for his call. This does not mean Romer and Belichick are wrong. Some strategies that fail ex post might be optimal ex ante. Randomness is a fact of life, even if Patriots' fans do not fully appreciate it.
I am beginning to understand how these guys can be Keynesians.

1 comment:

1. Wenzel,

You can learn a lot from another man. Unless you're Mankiw. Then, you can learn a lot from yourself.