Scouting and Development: Budgets and Ratings

[MikeS21]

There have been MANY threads about adjusting scouting budgets and ratings so that scouting information can be hidden or made unreliable. I fully understand why many who play OOTP follow that strategy. I know that NO scout is perfect and there is no way they can be 100% accurate. However, the fact is, there are some scouts who are better than other scouts. And handful of scouts are pretty daggone good at finding that needle in the haystack who develops into a superstar. They're not perfect, but they have a knack that others don't have.

For my franchise, I want to find a "gifted" scout who has extremely high scouting ratings, and build my organization through the draft, minor league development, and by trading veteran players for other teams' young prospects. My goal is for the scout to be accurate about 35-40% of the time. I am not so much worried about a scout reporting on upcoming opponents.

What scouting ratings do you think I should look for? Should the scout favor tools or ability or neutral? If I set my scouting budget to the maximum amount, what percentage allocation for each of the four scouting areas do you recommend? How high would you go on player development budget?

[RonCo]

I don't have a good answer for you, but I'm interested in what you mean by "accurate 35-40% of the time." Feel free to ignore the ultimate question if you feel I'm derailing your thread.

Let's take, for example, a scout who looks at a pool of 20 players (of whom two will eventually become serviceable players). Let's call the two good players #1 and #2.

Now let's say the scout does suggest two players will become serviceable, but he identifies these two as #19 and #20.

In this case, he is correct in 16 out of 20 cases, making him 80% accurate. I don' think this is what the average person means by an accurate scout. There are several ways of looking at the problem. How are you looking at it?

[sreem]

I'm not trying to sound condescending, so please don't take it that way. I am genuinely interested in knowing who these scouts are that you are referencing that have a "knack" for consistently finding players, and not just getting lucky, or choosing the obvious studs.


It is my opinion, and only my opinion, that there aren't any out there that consistently get it right based on their "knack" and not just straight up luck.


I am of course not including those who utilize statistical analysis. I don't see that as "knack", but an ability that anyone can learn.

[NoOne]

Originally Posted by MikeS21

There have been MANY threads about adjusting scouting budgets and ratings so that scouting information can be hidden or made unreliable. I fully understand why many who play OOTP follow that strategy. I know that NO scout is perfect and there is no way they can be 100% accurate. However, the fact is, there are some scouts who are better than other scouts. And handful of scouts are pretty daggone good at finding that needle in the haystack who develops into a superstar. They're not perfect, but they have a knack that others don't have.

For my franchise, I want to find a "gifted" scout who has extremely high scouting ratings, and build my organization through the draft, minor league development, and by trading veteran players for other teams' young prospects. My goal is for the scout to be accurate about 35-40% of the time. I am not so much worried about a scout reporting on upcoming opponents.

What scouting ratings do you think I should look for? Should the scout favor tools or ability or neutral? If I set my scouting budget to the maximum amount, what percentage allocation for each of the four scouting areas do you recommend? How high would you go on player development budget?

invest as much as you can afford in development and scouting for greater consistency in developing and scouting talent. sometimes retaining a player can be a good reason to cut into the budget in certain situations, but not all.

scouting ratings are important too. the budget and ratings will just give you greater accuracy relative to your settings. a great scout can still make mistakes (even bad suggestions at 100% accuracy). also, you can never avoid bad luck, but you can minimize it. refer to manual about focusing on talent or ability.

the accuracy setting will just add some error to what the scout reports for each individual rating for a particular player. he might be fairly accurate on contact, but off by alot on power. the accuracy will be per rating, not per player. so, not sure how that relates to your idea on wanting 35-40% accuracy... i wouldn't blindly choose that figure. test it out on a league when you find it, and see how it influences decisions over the course of the next 50 years.. you need a baseline before doing any test.

you'll have to play with those settings to get them where you want. i'd suggest changing the setting, re-run league scouting, and then compare scout results to what you see in player profile for many many players to understand the accuracy within that setting. i'd also suggest using an average scout and average budget (relative to league competition). that way you have a better idea of the range of results from bad-to-good.

other settings, such as ratings scale, can have an affect on what you are trying to accomplish. changing one factor at a time to see its influence is best, then move on to next setting.

so, you can make the information given to be built in with error so that decisions are not a cake walk. the more you lower accuracy, the more you want to use statistics (with proper sample size, or understand the amount of error in the one you have) to verify what the scout is saying.

[MikeS21]

Originally Posted by RonCo

I don't have a good answer for you, but I'm interested in what you mean by "accurate 35-40% of the time." Feel free to ignore the ultimate question if you feel I'm derailing your thread.

Let's take, for example, a scout who looks at a pool of 20 players (of whom two will eventually become serviceable players). Let's call the two good players #1 and #2.

Now let's say the scout does suggest two players will become serviceable, but he identifies these two as #19 and #20.

In this case, he is correct in 16 out of 20 cases, making him 80% accurate. I don' think this is what the average person means by an accurate scout. There are several ways of looking at the problem. How are you looking at it?

I suppose I was a little vague. I guess I am thinking that if a scout brings in a pool of 25 players thru a draft or international signing, there may be 6-7 players out of that group who has a real potential to advance and not turn out to be busts. I'm also thinking about a scout who can scour the late rounds of the draft and find a serviceable player that a lesser scout might miss.

I guess my ultimate goal is to find/hire the scout who will help me create the best farm system (system with best overall minor league talent) in my league game in the least amount of time. And to know how to adjust all the financial and other settings for scouting and development (and coaching), to take the greatest advantage of that scout's abilities.

Originally Posted by sreem

I'm not trying to sound condescending, so please don't take it that way. I am genuinely interested in knowing who these scouts are that you are referencing that have a "knack" for consistently finding players, and not just getting lucky, or choosing the obvious studs.

It is my opinion, and only my opinion, that there aren't any out there that consistently get it right based on their "knack" and not just straight up luck.

I am of course not including those who utilize statistical analysis. I don't see that as "knack", but an ability that anyone can learn.

Maybe "knack" is the wrong word. But you honestly don't think that there are real life scouting directors who are slightly better than other scouting directors for finding talent?

Of course, there is no scout who is anywhere near 100% accurate. But if the average scout will scout a pool of 100 players and find 5 players with higher-ceiling talent, I'm looking for that one scout who will consistently scout a pool of 100 players and find 6 or 7 players with higher-ceiling talent. Call it "luck" if you want. Is it possible that there scouts who are more "lucky" than other scouts on a reasonably consistent basis? And I am not at all opposed to scouts who use statistical analysis. I welcome it as long as they help me to develop the most loaded farm system in the league.

I'm not necessarily talking about finding "studs." I just want to know what ratings to look for if I want to hire the most effective scout in the game. How how can I adjust my league settings, and my organization's scouting and development settings to maximize that scout's effectiveness?

As I said, I completely understand the reasoning why so many posters want the scouting to be unreliable. I get it. They consider the uncertainty of scouting to be more realistic. But I happen to want the exact opposite. My thinking is that every GM in baseball hires the one scouting director they think is the most effective and they depend on him to stock the farm system with the best talent (both quality and quantity) they can find. It just seems to me that they would want to eliminate as much uncertainty in scouting as they can. After all, isn't that why statistical analysis is used?

Within the OOTP game, I want to find the best (most accurate) scout I can and trust him to him to advise me as to the best young talent to draft or trade for. I want my scouting to be as reliable as the AI will allow. I know it will never be 100% or even 75% accurate. I just want the most reliable scouting information I can get.

[sreem]

Originally Posted by MikeS21

I suppose I was a little vague. I guess I am thinking that if a scout brings in a pool of 25 players thru a draft or international signing, there may be 6-7 players out of that group who has a real potential to advance and not turn out to be busts. I'm also thinking about a scout who can scour the late rounds of the draft and find a serviceable player that a lesser scout might miss.

I guess my ultimate goal is to find/hire the scout who will help me create the best farm system (system with best overall minor league talent) in my league game in the least amount of time. And to know how to adjust all the financial and other settings for scouting and development (and coaching), to take the greatest advantage of that scout's abilities.


Maybe "knack" is the wrong word. But you honestly don't think that there are real life scouting directors who are slightly better than other scouting directors for finding talent?

Of course, there is no scout who is anywhere near 100% accurate. But if the average scout will scout a pool of 100 players and find 5 players with higher-ceiling talent, I'm looking for that one scout who will consistently scout a pool of 100 players and find 6 or 7 players with higher-ceiling talent. Call it "luck" if you want. Is it possible that there scouts who are more "lucky" than other scouts on a reasonably consistent basis? And I am not at all opposed to scouts who use statistical analysis. I welcome it as long as they help me to develop the most loaded farm system in the league.

I'm not necessarily talking about finding "studs." I just want to know what ratings to look for if I want to hire the most effective scout in the game. How how can I adjust my league settings, and my organization's scouting and development settings to maximize that scout's effectiveness?

As I said, I completely understand the reasoning why so many posters want the scouting to be unreliable. I get it. They consider the uncertainty of scouting to be more realistic. But I happen to want the exact opposite. My thinking is that every GM in baseball hires the one scouting director they think is the most effective and they depend on him to stock the farm system with the best talent (both quality and quantity) they can find. It just seems to me that they would want to eliminate as much uncertainty in scouting as they can. After all, isn't that why statistical analysis is used?

Within the OOTP game, I want to find the best (most accurate) scout I can and trust him to him to advise me as to the best young talent to draft or trade for. I want my scouting to be as reliable as the AI will allow. I know it will never be 100% or even 75% accurate. I just want the most reliable scouting information I can get.

I guess for me I'm more analytical, so with there being a distinct lack of viable and consistent measurement that can quantify that a scouts picks are based off of some measureable "ability" (like stolen bases/speed, strikeouts/movement, or defense/accuracy) it just seems to me that "luck" plays a larger factor into identifying talent than any trait or ability inherent in the scout.
Now, I agree that if you take Joe Shmow of the street and Mr. Scout Man with 20 years baseball experience, Mr. Scout Man will be more consistently able to project success compared to Joe who doesn't know a baseball from a football.
But in an apples to apples comparison I don't know how you could verify that a scouts good choices on players can be attributed to anything but chance or personal opinion/taste, which is NOT an ability.
I'm completely open to correction, so if you do know of such a measurement that certifiably measures a scouts ability and eliminates chance, I would love to hear about it. I'm also interested to know who these scouts you referenced in your original post are that are consistently better than others are.
Again, these are just my opinions and I am not trying to sound condescending. You are entitled to your opinion and I do respect it, I just don't agree with it based on currently available data.

[RonCo]

Originally Posted by MikeS21

I suppose I was a little vague. I guess I am thinking that if a scout brings in a pool of 25 players thru a draft or international signing, there may be 6-7 players out of that group who has a real potential to advance and not turn out to be busts. I'm also thinking about a scout who can scour the late rounds of the draft and find a serviceable player that a lesser scout might miss.

I guess my ultimate goal is to find/hire the scout who will help me create the best farm system (system with best overall minor league talent) in my league game in the least amount of time. And to know how to adjust all the financial and other settings for scouting and development (and coaching), to take the greatest advantage of that scout's abilities.

So it seems your definition of accuracy seems to be:

A = Guys he said would be good, and panned out
B = All the guys he said would be good

Accuracy = A/B

I therefore suggest that the measure of a scout's lack of accuracy might be made of two parts--guys he said would be good, who don't turn out to be + guys he did not identify as potentially good who turned out to be good, then divide this by all the players he looked at:

C = Guys he said would be good, but did not pan out
D = Guys he looked at but did not identify would be good, but who eventually became good
E = All the guys he looked at

Error Rate = (C+D)/E

This is interesting to think about to me, because the fact that the denominator is different for these two things helps frame the problem.

At the end of the day, the only thing a team really cares about above is A (the number of good players they get from their scouting department). They also want to limit "C" guys because they represent investment that has no return. Folks don't tend to talk too much of "D" players except in the abstract.

In the old days, I think scouting staffs distiguished themselves through sweat equity, meaning a scout who drove through po-dunk cities for days and days to see 1,000 players was a better scout than one who saw only 100--merely because that increased the number of guys he said would be good, and assuming all scouts are have generically the same hit-rate, would yield more "A" players.

Today, scouting is ubiquitous. Everyone sees everyone. So the impact of sweat equity is considerably less (except in some foreign scouting areas...though that is changing, too). Given that, I suspect that the actual value of one professional scout (guys who have spent their lives doing this) over another is very, very slim.

In the above, I suggest the best measure of a scout might actually be:

Value = (A-C)/(A+B+C+D) (or some weighted factor of these)

And that the differentiation between experienced scouts is which guys fit into A and D (though the numbers of As and Ds are probably about the same).

Just my complicated .02

[RonCo]

And, all that said...I don't think that's really what OOTP scouts are doing.

[Josquin]

Originally Posted by RonCo

And, all that said...I don't think that's really what OOTP scouts are doing.

Agreed. They are probably estimating the true underlying ratings for a player, and for a scout in OOTP to be more accurate means that their estimate will be close to the player's true rating more often.

[Josquin]

Originally Posted by RonCo

So it seems your definition of accuracy seems to be:

A = Guys he said would be good, and panned out
B = All the guys he said would be good

Accuracy = A/B

I therefore suggest that the measure of a scout's lack of accuracy might be made of two parts--guys he said would be good, who don't turn out to be + guys he did not identify as potentially good who turned out to be good, then divide this by all the players he looked at:

C = Guys he said would be good, but did not pan out
D = Guys he looked at but did not identify would be good, but who eventually became good
E = All the guys he looked at

Error Rate = (C+D)/E

This is interesting to think about to me, because the fact that the denominator is different for these two things helps frame the problem.

At the end of the day, the only thing a team really cares about above is A (the number of good players they get from their scouting department). They also want to limit "C" guys because they represent investment that has no return. Folks don't tend to talk too much of "D" players except in the abstract.

In the old days, I think scouting staffs distiguished themselves through sweat equity, meaning a scout who drove through po-dunk cities for days and days to see 1,000 players was a better scout than one who saw only 100--merely because that increased the number of guys he said would be good, and assuming all scouts are have generically the same hit-rate, would yield more "A" players.

Today, scouting is ubiquitous. Everyone sees everyone. So the impact of sweat equity is considerably less (except in some foreign scouting areas...though that is changing, too). Given that, I suspect that the actual value of one professional scout (guys who have spent their lives doing this) over another is very, very slim.

In the above, I suggest the best measure of a scout might actually be:

Value = (A-C)/(A+B+C+D) (or some weighted factor of these)

And that the differentiation between experienced scouts is which guys fit into A and D (though the numbers of As and Ds are probably about the same).

Just my complicated .02

If you look at scouting as a pure binary classification problem (i.e. looking at all the prospects and categorizing them as "good" or "bad") then there are established measures like F1-score to measure accuracy of a classifier. As you discuss, you want to minimize both the false positives (players who are classified as "good" but don't pan out) and the false negatives (players who are classified as "bad" but turn out to be good).

As you point out in another post, this is probably not the right way to look at scouting in OOTP. The right way to look at it is how often the scout's estimate of a prospect's potential ratings is close to his "true" ratings (which are visible in the Edit Player screen in Commissioner Mode). There is no binary threshold between "good" and "bad" prospects -- it is more of a continuum, and the scout's estimated ratings may or may not be close to reality. The scout's accuracy would thus be measured as the average deviation between estimated and true ratings.