Tuesday, March 15, 2011

Are There Biases In Top Prospects Lists?

Two of the most common criticisms of baseball top prospects lists are that they are biased both towards high draft picks and recent signings who haven't played much professionally. These complaints seem to apply to all of the major prospecting sites: BA, BP, John Sickels, etc. (And even our lists.) Let's see if the biases exist, and if so, why.

To address the first question I took BA's 2011 Top 10 Prospects for each team (30 x 10 = 300 prospects) and divided them up by where (draft round and international signs) and when (by signing year) each player was acquired. I used the BA rankings for two reasons, they were readily available and BA lists overlap with BP and others a great deal, over 80% most years.

Here are how the 300 players were acquired, broken down by draft round and international signings:

How Acquired               #         %
1st round                 75      25.0
1st - supplemental        33      11.0
2nd round                 35      11.7
3rd round                 23       7.7
4th round                 11       3.7
5th round                 10       3.3
6th-10th round            20       6.7
11th - 20th round         12       4.0
21st - 50th round         11       3.7
Non-drafted free agent     1      0.03
International sign        69        23
TOTALS                   300       100

Nearly half (47.7%) of the 300 top prospects were picked in the first two rounds of the draft, and nearly two-thirds (62.4%) in the first five rounds. If you remove international signings who aren't eligible for the draft, those percentages rise to 61.9% of top prospects coming from the first two rounds and 81.0% from the first five rounds. There is no set definition of 'high draft picks', but I think we can agree that players chosen in rounds one through five certainly qualify. When over 62% of all prospects and 81% of draft-eligible prospects are acquired via 'high draft picks', it seems fair to say there is a significant bias towards high draft picks in top prospects rankings.

Next up is 'recent signings', here are the 300 prospects by the year they were signed:

Year          #         %
2010         62      20.7
2009         69        23
2008         54        18
2007         51        17
2006         37      12.3
2005         15         5
2004         12         4
TOTALS      300       100

As with 'high draft picks', there is no set definition of 'recent signings'. Again, I think we can agree that players signed in the past two years should qualify, which includes 43.7% of the top prospects. If we extend 'recent signings' to the past three years it increases to 61.7%. While the percentages aren't as high as with 'high draft picks', there is certainly a tendency towards 'recent signings' when nearly two-thirds of the prospects were acquired in the past 3 years. One other thing to remember, this data includes not only drafted players but also international signs, many of whom were only 16 or 17 at the time and are still teenagers even two or three years later, or about the age they would be drafted out of high school in the U.S.

As an aside, since I have the data available, here are the international signings broken down by country of origin and year signed:

Country               #        %
Dominican Republic   37     53.6
Venezuela            19     27.5
Cuba                  3      4.3
Curacao               3      4.3
Mexico                3      4.3
Korea                 1      1.4
Columbia              1      1.4
Panama                1      1.4
Australia             1      1.4
TOTALS               69      100

Year          #         %
2010          4       5.8
2009          6       8.7
2008         10      14.5
2007         17      24.6
2006         17      24.6
2005          7      10.1
2004          8      11.6
TOTALS       69       100

One last note from the BA lists that applies to both 'high draft picks' and 'recent signings', 24 of the 300 top prospects, or 8%, had yet to play a single professional game. The ultimate in potential over performance in prospect ranking.

I originally asked: if there is a bias, why? The data does show biases towards high draft picks and recently acquired players, but much of that can be explained by real world considerations in putting together top prospects lists. Looking at high draft picks, it just makes sense that if a team drafts Player A in the first round and pays them a huge signing bonus, their scouts believe that they are a better prospect than Player B who they drafted in the 15th round and paid a small bonus. The folks making the prospect lists don't have nearly the information available about each player that the team does, so it makes sense to rely on the team's evaluation to some degree.

Turning to the rankings for recent picks who have played little or no pro ball, there are several reasons for this to occur. First, there is some overlap with the first bias, top picks, since top picks tend to sign late and not play much that season. Second, the nature of creating prospect lists is to identify players earlier than the competition. If for example BA waited until a player reached AA before listing them as a prospect, they would lose their readership to list makers who identify prospects years earlier. And third, players signed in 2004 or 2005  generally are no longer prospects. Either they have reached the majors or flamed out after five or six minor league seasons (high school picks who miss a year or two with injuries and young international signs are the exception). If we wait several seasons to see a consistent record of professional performance, top prospects lists would be very short.

Bottom line: The biases do exist, but the complaints about them aren't as valid when you take into consideration the real-world factors affecting top prospects rankings. Agree, disagree, anything I've overlooked?


  1. Interesting analysis Doug.

    I think the money and draft slot have lots to do with how players are evaluated. It seems it's human nature for scouts and front-office types to justify why they paid what they did and selected where they did.

    What I see when it comes to top picks is lots of excuses made (for the player) when they don't pan out. Conversely, when a low-rounder produces, people look for reasons to denigrate them. In other words, labels tend to follow a player.

    Looking at recent examples, Tim Beckham is a prospect but he is not a first pick in the draft prospect imho and no matter what the Tampa brass says, they cannot justify that pick. Not saying he won't develop into a fine MLB player someday but with that pick you are looking for Justin Upton, Mike Trout, or Buster Posey type value. A player Beckham might turn out to be like is Brandon Phillips who wandered in the minor leagues for several years before figuring things out.

    My son is obviously a low rounder and deals with the stereo-type everyday. All I can do is compare his numbers with other players in the leagues he has played in. For two years now, his slugging has been at .450 or above in the NYPL and the Midwest League. He has been in the top 10 in both leagues in OPS, base stealing, batting average, on base percentage and so forth in both leagues and has out-performed almost every higher round pick in the respective leagues. I see people bringing up his age a lot for example and his size and his draft slot as a means to "explain" his numbers. I see higher round picks who produce far less and they are praised by Basebal America types and ancellary issues such as age are never brought up. It's frustrating but a player cannot control those things. Attitude and effort are all they can control and I pray that one day, at least one MLB team will give him a chance. I hope he can be the next Ian Kinsler or like the closer for the Giants with the blackbeard who were both low-rounders and who have now made good.

    I expect some criticism in advance for this post by someone named anonymous.

  2. I agree with almost everything you say except for the comment on Beckham-at least as I understand that comment.

    It is absolutely true that he has not performed as one expects a 1/1 to do, and that his stock has fallen. And perhaps for that reason, the fact that he remains on many top 100 lists is not legitimate.

    But I don't think you can fault the Rays for that pick. At the time of the draft, there was no clearcut choice. As a matter of fact, there were supporters of Posey, Alvarez and even Smoak and Gordon Beckham at the time. The results may prove that the Rays blundered, but as far as the decision at the time, it was well within the bounds of a justifiable pick. I don't think you can criticize an organization for a prospect's failure if what was known at the time pointed to success.

    As for your son, I think he has an excellent chance to succeed in the Rays' organization. They have promoted him regularly-in fact, it seems to me even quicker than is their wont with prospects-and have rewarded his outstanding performance with regular playing time. Is there something I am missing in the way he is being treated by the Rays?

    In the end, it really doesn't matter whether a player is highly regarded or not so long as he produces and succeeds. As far as I can tell, the Rays are far more impressed by what a player does than with how various publications and media outlets rank him, and that bodes well for Tyler.

  3. I think you are absolutely correct about team employees trying to justify picks, human nature as you say. As in any business, if you authorize or recommend a purchase, you are going to stick up for that product even when it's obvious (or beginning to look) to others that it wasn't a good choice. No one likes to admit mistakes, especially when your job could be on the line. I was hoping that by using BA's picks that would be lessened, but since they get a lot of their info from the org's, the bias probably does bleed through, not sure to what extent though. And to be clear, I'm not saying any of this is prevalent regarding the Rays, but in general across the 30 org's.

    You also brought up another good point - age. I'll go back and get the data by year of birth so we can see that, and I might as well get each player's highest level played in 2010 while I'm at it. Let me know if anyone thinks of anything else besides age and highest level.

    As far as the Beckham vs. someone else pick, usually Posey, I don't think it's fair to just go back and re-draft the bad picks. Should have to re-do the good ones too. For example, instead of Price what if we would have taken Wieters or Vitters (the other 2 players the Rays considered taking with the top pick). I'd rather have Beckham & Price than Posey & Vitters. Or look at 2006, what if it was anyone other than Longoria? The draft involves skill, but also a lot of luck. And I hate to see Beckham get too much grief, he didn't draft himself. (I'm not saying this so much in response to you Daniel as to all the people on various sites I've heard lament the Beckham instead of Posey pick.)

    And I don't think you have to stick up for Tyler around these parts, he's pretty popular here. I'll leave any dissenting comments to the anons too.

  4. My comment was in response to Daniel, Robert is just a much faster typist than I. And apparently Robert and I agree on Beckham, for different reasons, but agree overall.

  5. I agree with what Robert Rittner has posted as well. Thanks for those insightful comments.

    I agree that luck is involved as well Doug. Some teams seem luckier than others however. Longoria and Price were obviously great picks along with Crawford, Hellickson, Moore, etc.

  6. This takes "double-talk" to a new level.

    First, according to your analysis, yes, bias towards high draft exists, but it's justified because we should be biased towards high draft picks. Huh? Second, you state, "The folks making the prospect lists don't have nearly the information available about each player that the team does, so it makes sense to rely on the team's evaluation to some degree." Again, huh? Let me get this right - you relied on incomplete information from a biased source, and expect us to applaud your objectivity and analytical acumen?

    At the end of the day, these lists are interesting thought experiments, but don't result in anything positive, except to foment anger (from those who disagree) and disappointment (about those that don't live up to the hype). Why bother. Let's play ball!!

  7. Anon, you are drastically selling these lists short. Their main job is to give a list of prospects to the average fan so he can know who is his team's top prospects. There is some bias, but the lists still provide basic information on where a player is ranked. There is a reason us Ray's fans were willing to not be mad at Price after 2009. If it was a lower prospect, thn we might have thought poorly of him. But, instead, we knew he was a top prospect, and not giving judgement turned out to be a wise decision.