Data leakage. Naive random splits let information from the future bleed into training and inflate the score. The fix was strict time-ordering and walk-forward backtesting across seasons — so every prediction is made only from the past.
An honest metric beats an impressive one. Top-3 hit-rate under walk-forward validation tells the truth; a shuffled split just tells you what you want to hear.