Predictions, uncertainty, overfitting and backtesting
- Read a forecast with its uncertainty
- Recognise overfitting and look-ahead bias
- Evaluate a backtest critically
Every forecast has a range
A useful forecast says how uncertain it is. ‘Expected excess return +2%, with a wide range from −15% to +18%’ is more honest than ‘+2%’. If the range is wider than the estimate, treat the estimate as a weak tilt, not a prediction.
Backtests
A backtest applies a strategy’s rules to historical data to see how it would have performed. It is a useful filter: a strategy that failed in the past deserves suspicion. It is not proof that it will work in the future.
Overfitting
Overfitting happens when a strategy is tuned so closely to past data that it captures noise rather than a real pattern. Testing hundreds of rule variations and keeping the best is a classic way to overfit: something will look great by chance.
Warning signs: many parameters, very precise thresholds, results that collapse when a parameter changes slightly, and stellar returns with few trades.
Look-ahead and survivorship bias
Look-ahead bias uses information that was not available at the time, such as trading on a day’s close before it happened. Survivorship bias tests only companies that still exist today, ignoring those that went bankrupt, which flatters results.
VEXA’s backtests trade at the next day’s close to avoid look-ahead, charge costs, and say clearly where today’s index membership introduces survivorship bias.
Out-of-sample and walk-forward
Better tests reserve data the strategy never saw while it was designed, called out-of-sample data, or use walk-forward testing: design on one period, test on the next, then roll forward.
A backtest shows 45% a year with only 12 trades over ten years, using an RSI threshold of 27.3 and a moving average of 37 days. Changing 27.3 to 28 cuts returns to 6%.
That sensitivity and the odd precision are classic signs of overfitting.
Key terms
- Backtest
- A simulation of a strategy’s rules on historical data.
- Overfitting
- Tuning a strategy so closely to past data that it captures noise.
- Look-ahead bias
- Using information in a test that was not available at the time.
- Survivorship bias
- Distortion from testing only investments that survived to today.
- Out-of-sample
- Data not used when designing a strategy, reserved for testing it.
- Walk-forward testing
- Repeatedly designing on one period and testing on the next.
- Uncertainty range
- The spread of plausible outcomes around a forecast.
Run a backtest in the Research Lab, then change one setting slightly (for example the RSI threshold). Note how much the result changes and what that tells you.
Key takeaways
- Read forecasts together with their uncertainty.
- Backtests filter ideas; they do not prove them.
- Watch for overfitting, look-ahead and survivorship bias.
- Prefer out-of-sample and walk-forward evidence.