Almost any trading model can be made to look brilliant on past data. Keep adjusting it and eventually it will fit the history almost perfectly. Then real money shows up and the magic disappears. Walk-forward testing is one of the best defenses against fooling yourself this way.
The problem with an ordinary backtest
A normal backtest runs a strategy over old data and reports the result. The trouble is that the person building it has usually seen that same data many times. Each tweak quietly leaks knowledge of the past into the strategy. This is called overfitting: the model has learned that particular stretch of history and not anything lasting.
Another trap is look-ahead bias, when a test accidentally uses information that was not available at the time. Using a day's closing price to make a decision at the open is a classic example. Even a tiny leak can turn a losing idea into a beautiful chart.
How walk-forward testing works
Walk-forward testing copies how a model would really have lived through time:
- Train the model on an older block of days.
- Test it on the days right after, which it has never seen.
- Slide the window forward and repeat.
- Judge the model only on those combined test days.
The result is much closer to what you could have experienced in real time. It is usually far less exciting than the backtest, and that is the point.
The bar our models have to clear
At My Own Stock Guru, a new model does not replace the current one because it looks clever. It has to earn the spot:
- It is graded on unseen days only. Results from its training data do not count.
- It has to beat what is already running. Matching it is not enough.
- It is judged on real outcomes. Results come from what the market did after each alert.
- It starts quietly. A new model first runs in the background, making calls that are recorded but not acted on.
Plenty of our experiments fail this test. We count that as the system working. A rejected model is a loss that never reached a member.
Questions to ask about any trading AI
You can use this on any service, including ours:
- Were the results measured on data the model was not trained on?
- Are the results from live, timestamped signals or only from a backtest?
- Does the record include the losers?
- How does it compare with simply holding the market?
- Has anyone ever seen a model get rejected?
If the answers are vague, treat the performance claims the same way.
Where to go from here
Our graded alert history is open on the Alert Results page, and the FAQ covers how alerts work. Past results never guarantee future ones, but you should always be able to look at them.
Sources and further reading
- Bailey, Borwein, López de Prado and Zhu, The Probability of Backtest Overfitting (SSRN)
- Gu, Kelly and Xiu, Empirical Asset Pricing via Machine Learning (NBER working paper)
This article is educational and is not investment advice.
