I've played since v4 in 2002. The game changed with a rewrite in 2006 to it's current base code.
I see you say you have turned off the 40 man but to me that and the other roster rules are what have made OOTP challenging. The only thing close to a house rule for me is not trying to fleece the AI by offering multiple players in trades. So on to your questions and the point of the OP.
I use the 40 man with lowest level of minors having a 50 man roster. With this setting IMHO the AI has done fine handling their 40 man roster and whole organization. Some suggest no limit on low minors but I use ghost players and have found 50 to be more than enough.
Add to this all current roster rules (rule 5, days of service for FA, arbitration etc.) the game offers to stop one from stockpiling talent and forcing decisions on both me and the AI. These rules are what have made me work to field a winner where before they were introduced winning was easy.
Because of these rules I have won one world series in the last 10 years and have missed the playoffs several times. Something that never happened in pre 2006 versions but I think v9 or v10 is when the game really started to get harder.
I too use low trading and very hard\favor prospects. I think this helps the AI hold on to it's young talent and build winning teams. I play a MLB game that has went from real players to fictional by attrition and am now in the 2027 season (2002 was my 1st season). In that time the Rays appeared in a WS with the league's lowest payroll after all real players were long gone so a team built by the AI. I play as the Cubs and faced a Brewers team that was my biggest rival for several years due to the AI building an amazing pitching staff through the draft. The Cardinals in my game are just starting an era like that with young P's they drafted.
I also use ghost players. I find they make dealing with minors much easier and I think it also helps the AI to an extent as it also is not forced into moves due to injuries. IOW it lets the AI leave players where they should be instead of it shuffling players to fill holes.
I play at default on almost everything. I figure Markus and the beta team have done more testing than any of us and so I trust default will perform just fine.
Then I just play the game. I don't get too excited about little things that might look funny and to be honest I don't see many of those things anyway.
I have found, at least it works for me, that using roster rules, making realistic trades, and simply using default settings has made OOTP a challenging and fun game. I think some players over think things because of all of the optional and adjustable things that are offered in the game. That's ok too, more power too them. I've just never found the need to change much of anything.
YMMV