As a follow-up to VegasWatch’s look at expert team predicitons, I want to use 2007 Pythagorean wins instead of actual wins to judge the predictors. Why? Because Pythagorean record is a better indicator of team talent than actual record. It doesn’t explain things like properly leveraging a great bullpen (which is a team skill), but at least it ignores teams that have been lucky to give up large chunks of runs at opportune times (largely not a skill). Now, I’m sure some of the experts, especially the computer simulations, built things like bullpen leverage into their predictions, which means that using Pythagorean wins has its own problems. (It also doesn’t account for any type of luck/statistical variation in runs scored and allowed.) But I’d argue that it’s at least as good of a test as actual record. You can, as always, judge for yourself.
Here are the average number of wins each system differed from 2007 Pythagorean wins:
4.4 Diamond-Mind 4.5 Neyer 4.6 Sports Interactive (Vegas) 4.9 PECOTA 5.0 MGL 5.2 BP Hit List 5.4 Gammons 5.5 Karabel 5.7 Kurkjian 6.0 Crasnick 6.0 2006 Pythag wins 6.1 Phillips 6.1 Law 6.3 Stark 6.7 Caple 7.1 Olney 7.1 2006 actual wins 7.3 81 wins
There are a couple important baselines in this list. One, it’s nice to see everybody beat the most basic prediction of 81 wins for every team. 2006 actual wins only beat that one by a small margin.
Next is 2006 Pythagorean wins, which you would guess is a pretty decent measure of 2007 Pythagorean wins, given that team talent doesn’t change all that much year to year. Phillips, Law, Stark, Caple, and Olney were unable to beat this baseline, but Olney and Caple were by far the worst.
Only two sets of predictions were able to beat the Vegas line — Rob Neyer and Diamond-Mind. Why were more systems able to beat Vegas in actual wins? Because there’s more luck involved in actual wins. A percentage of all reasonable predictors are bound to be luckier than Vegas.
I wouldn’t be consistent if I didn’t point out that Steve Phillips’ predictions look a lot better when compared with 2007 Pythagorean wins. That likely means he was the victim of some bad luck.
For fun, here’s one last table showing the average errors based on actual wins and Pythagorean wins, sorted by the average of the two.
Predictor AVG Actual Pythag Neyer 4.8 5.0 4.5 Diamond-Mind 4.9 5.3 4.4 PECOTA 5.0 5.1 4.9 SpInt (Vegas) 5.1 5.5 4.6 MGL 5.2 5.4 5.0 BP Hit List 5.3 5.3 5.2 Karabel 5.6 5.7 5.5 Gammons 5.7 6.0 5.4 Crasnick 5.8 5.5 6.0 Kurkjian 6.1 6.4 5.7 Law 6.1 6.1 6.1 Caple 6.3 5.9 6.7 Stark 6.3 6.3 6.3 '06 Pythag 6.5 6.9 6.0 Phillips 6.5 6.8 6.1 Olney 6.8 6.4 7.1 '06 wins 7.6 8.1 7.1 81 wins 7.9 8.4 7.3
I’m not sure what it says about me, but I really like seeing the computer systems at the top of the list (which include MGL), along with two of the writers I enjoy the most: Gammons and Neyer.
Update I decided to let the non-Neyer ESPN writers compete as a team, averaging their predictions. Together, these “experts” do much better. Their average error from actual wins is 5.4 — same as MGL. Their average error from Pythagorean wins is also 5.4. Overall, they’re at 5.4 (really!) — behind Vegas, MGL, and the computer systems — but better than any one of them did alone. Who doesn’t prefer a group test?
Popularity: 2% [?]
Share This
Sky is a baseball fan and racket sport afficianado living in upstate NY. His favorite color is orange and is just about ready to give up on his life-long dream to become the next Magnus ver Magnuson (World's Strongest Man). His favorite baseball teams are the Yankees and Red Sox, proving that there's hope in the Middle East.
September 29th, 2007 at 1:09 am
Neat!
I’d love to see a multi-year study. PECOTA predictions, at least, should still be available, no? -j