It seems like every year, we see an MLB team make an epic comeback or collapse in the playoff race. In 2007, the Mets shot themselves in the foot, resulting in their postseason chances going from 99.8% to 0% in 18 days (and 96% to 0% in the final 5 days); the Phillies were roughly 200-1 dogs on September 13. Additionally, Arizona won the NL West as a 70-1 dog on July 21.
Last year, the Twins were at one point a 500-1 shot to win the AL Central. In 2005, the Astros made it as a 240-1 dog, and the Indians blew a 96.5% chance in the final week.
How do we account for all these longshots hitting in a span of only three years? There are three possibilities:
A) We've witnessed several anomalies
B) The BP Postseason Odds aren't perfect
C) A little of column A, a little of column B
Well, B is certainly true--I'll bet even Clay Davenport would agree with this--but I think we're looking at C. Even if we have a perfect playoff odds model, it's not going to project the Mets to dump seven games in a row to the Phillies, plus some more to the Nationals and Marlins. It won't expect the Diamondbacks to have a 17-game stretch starting July 19 where they go 13-4 with a -2 run differential.
These were examples of what Football Outsiders would call Non-Predictive Events: certainly the Phillies and D-Backs played well to reach the playoffs, but if they had to do it again, they'd probably come up short.
For now, though, I'd like to focus on B. There are several things standing between between the BP Odds and a perfect model, but how many of them can be practically implemented?
I think this is doable. Coolstandings already includes tiebreakers in their simulations. In the first half of the season, it's usually difficult to forecast how the tiebreakers are going to pan out, but by August you can usually tell.
If the report doesn't break ties, it could at least display the probability of a tie so that readers can make manual adjustments.
- Differences in Starting Pitcher Quality
This is certainly impractical for use in April, but it could be added for late September, or certainly for the playoffs themselves. Facing Brandon Webb instead of Livan Hernandez is a pretty big deal, and the September 27 (or mid-October) odds report should account for that.
- Shifting Team Composition
This is tricky. Certainly you want to adjust the odds report when Chris Carpenter goes down for the season or Mark Teixeira is traded, but where do you draw the line between significant and insignificant changes? I'm sure if Clay gambled on his reports, he would account for these things, but he can't be expected to account for every little injury or trade.
The ELO report is designed to account for these changes, but it's not very effective in that regard. How long does it take for the Teixeira trade to show up in the ELO numbers? A month? If Tex doesn't hit in the first few weeks, you may never notice the difference.
- Accurate Regression Numbers
Now we're talking. There's a good discussion on THE BOOK blog, with the authors concluding that the current Davenport formula does not regress each team's stats far enough toward their preseason projections, especially early in the season.
Take another look at the list of longshot winners. The 2007 Diamondbacks, 2006 Twins, and 2005 Astros were all expected to be serious contenders, and all were much better teams than their early-season records would have you believe. Did the BP Odds read too much into their slow starts? MGL and DFL seem to think so in posts 15 and 21 from the thread linked above.
Could it be that the '06 Twins, despite being 12 games out in the division race, were actually 50-1 dogs rather than 500-1? Between their disappointing start and the first-half over-achievement of the Tigers and White Sox that year, I think it's entirely possible.
Tom Tango and others have done good work determining the proper amount of regression to use in projections. I think this is one area the Odds Report can easily improve upon.
I don't mean to deride the BP Odds Report, a useful tool that sparked my interest in betting sports futures. But as with all things baseball, when I see something that's working at 80% efficiency, I want to see that number move closer to 100.
Can I just build a better model myself? Maybe. I'd need some help with computer programming, if anyone's willing to volunteer, but it could be an interesting project to keep me busy in the offseason.