Updated Forecaster's Challenge results
So scratch the end of my last post. I'm not 3rd anymore in the Forecaster's Challenge. According to the updated results here I am now 6th. Slipped a little, but still right up there. Last year my Achilles heel was picking a bunch of pitchers that were out for the season. So in the offseason I redoubled my efforts to figure out who was projected to actually play, and my performance has certainly improved as a result, and I was right up there at the All-Star break. I'll have to dig a little deeper now (whenever I get the chance - i.e. in the offseason) to figure out what is distinguishing the winners from the losers this year, and why I've slipped in the second half. There is certainly a lot of variance in these things, although it is impressive how last year's winner has done very well again this year. I imagine he is a very experienced Fantasy Leaguer.What’s with the 2 sets of preseason predictions?
For those of you that have viewed the preseason predictions before, you may have noticed some things like (why are no pitchers projected to have more than 12 wins?) and (why is Albert Pujols projected to 480 Abs?) There are a couple of reasons why these numbers are lower than you probably would have expected (especially in retrospect). For one, we expect performance to regress to the mean a bit (google "regression to the mean in baseball" if you don't know what this is all about). And for another, these are average expected performances, not most likely expected performances. Where that comes into play most notably is in accounting for injuries. For instance, there may be a 10% chance a player will get injured, and when injured he has 0 at bats and when not injured he has 600. So the expected value is 540 AB (600*90%), but the most likely value is 600.
This explains a lot of why these playing time projections are a little off of your expectations. The one other reason that these projections are off is that some of my methods for projecting these things are just not that well-developed. Those of you that are familiar with my research know that the main models I have built are intended to estimate the probabilities of specific outcomes of specific at bats, and then based on that I have built out systems to predict outcomes for games and seasons. But I have not developed sophisticated models to predict playing time and injuries and such over the course of a season, and as a result some of the projections that are based on overall playing time and player usage, like at bats and wins and saves may be a little off or just not that great.
This is where the second set of projections come in. Knowing the weaknesses of my models, I have leveraged public sources of playing time and usage projections to leverage (what I believe to be) better information about these variables (most notably the community projections at fangraphs). Incorporating this information (by generally blending some of the usage based estimates with mine) I have augmented my projections. You will notice though that although the aggregate numbers change here, the ability based estimates like batting average and OPS have remained the same.
The last big question then is, how do these projections compare? Well on true player ability metrics that I would still maintain are the most important for actual estimation of player ability and impact (excepting propensity to injury), they are the same. For the aggregate metrics, which are no doubt a bit more important to applications like fantasy sports, they differ a bit. To evaluate one versus the other, one piece of anecdotal information would be to look at my rankings in the Forecasters Challenge (at www.insidethebook.com) last year – when I used only my algorithms – and this year when I augmented my results a la version 2 above. Last year I was in the middle of the pack. This year I am ranking third overall in the main challenge, and 1st in one of the alternate challenges.
Bug fix
Keeping in the spirit of my last post, I just fixed a bug that was arbitrarily throwing off the Win/Loss projections of several teams by up to a game. This doesn’t sound like much, but in close division races where it pushes teams in the opposite direction it can have a big impact. For example, the White Sox were again on the wrong side of this one, which is why you may have seen them projected to finish 13 games back two days ago (doing slightly better now, but not much).Caveats / Data needs
People have been asking me about these projections, particularly the Game and Season projections, and in particular asking “How good are they?” But what I think they really mean is, “Can I make money off of this?” So I figured I should clarify a few things. First of all, I would NOT recommend taking these projections as truth and trying to get rich off of them. They are generally good and informative, but this effort is pretty large scale and since I am not monitoring all of them, there are a few key developments that could throw things out of line. For example: 1) I use announced probable pitchers, and these sometimes change. The good thing about this is that for specific games, you can see the pitcher I am using. 2) I project lineups based on recent lineups used by the team. If there is reason to expect the lineup to change (such as a trade or player coming off or going on the DL), I won’t catch it. 3) There could be some other random weird glitch in the data that has gone undetected. These issues obviously effect game predictions, but could also potentially significantly impact rest of season projections for a team, if for instance a team has been without its star batter for the last few games, I will likely project him to be out for the remainder of the season, and the team’s projections will suffer. (The White Sox have recently suffered in my projections at the hands of such a circumstance having fielded particularly weak offensive lineups of late). In any case, I would greatly appreciate any and all feedback you have about the site, in particular regarding numbers that look somewhat unbelievable (they just may be). I also like compliments. Also towards these issues above, I would also always welcome with open arms any offers of useful data. In particular: 1) data sources with when players have gone on and are expected off of the DL 2) data on recent trades 3) data on projected trades or teams looking to trade or acquire certain players or sorts of playersJSM talk
JSMtalk2 For those of you looking for the slides for my talk at JSM 2010 Vancouver, here they are: JSMtalk2 On second thought, for some reason I am having trouble getting these slides uploaded, so if you are interested, just drop me a line (brad at this domain).Forecaster's Challenge
Midseason results for Tom Tango's forecaster's challenge are in, and our model is sitting in 3rd. www.insidethebook.com/ee/index.php/site/article/forecasters_challenge_2010_mid_season_results/Coming soon - Daily player ranking updates
For those that have been asking, the daily player evaluation and ranking updates that I have been using in the Strasburg watch should be posted daily on the site within the next week or so, so that you will all be able to scan all the way through the rankings to your heart's delight. So keep a lookout!Strasburg Watch IV
Just a quick note to commemorate the first day that Stephen Strasburg's expected ability has swung in the wrong direction. After yesterday's 6 inning, 9K, 0 walk, 1 run, 9 hit performance, Strasburg now projects to a .672 OPSA (from .658 after his last start) dropping him fro 22nd to 29th in our rankings.Strasburg Watch III
So after yet another stellar performance from Stephen Strasburg, he has climbed to 22nd on our pitcher rankings. Here's the top 25:| E OPSA | ||
| Jonathan | Broxton | 0.566 |
| Carlos | Marmol | 0.602 |
| Luke | Gregerson | 0.609 |
| Mike | Adams | 0.619 |
| Joe | Nathan | 0.623 |
| Mariano | Rivera | 0.624 |
| Matt | Thornton | 0.628 |
| Tim | Lincecum | 0.629 |
| Billy | Wagner | 0.631 |
| Joakim | Soria | 0.632 |
| Brian | Wilson | 0.641 |
| Rafael | Soriano | 0.645 |
| Jose | Valverde | 0.647 |
| Andrew | Bailey | 0.649 |
| Adam | Wainwright | 0.650 |
| Josh | Johnson | 0.651 |
| Takashi | Saito | 0.654 |
| Francisco | Rodriguez | 0.654 |
| Ubaldo | Jimenez | 0.654 |
| Hong-Chih | Kuo | 0.655 |
| Neftali | Feliz | 0.656 |
| Stephen | Strasburg | 0.658 |
| Huston | Street | 0.663 |
| Roy | Halladay | 0.664 |
| Arthur | Rhodes | 0.665 |