Technical track
Testing a strategy without fooling yourself
Backtest overfitting is not an exotic mistake made by careless people. It is the default outcome of searching hard enough, and the reason is arithmetic rather than character.
The number that is missing from almost every backtest
Bailey, Borwein, Lopez de Prado and Zhu put the problem in a sentence worth reading twice: "a backtest where the researcher has not controlled for the extent of the search involved in his or her finding is worthless, regardless of how excellent the reported performance might be."17
What they are pointing at is this. A backtest result is not a measurement of a strategy in isolation. It is the best result from however many strategies you tried. Ten attempts and one thousand attempts can produce the same headline number, and the second number means far less. So the honest reporting unit is not the winner. It is the winner plus the count of everything you threw away getting there.
Why searching harder makes results worse, not better
Suppose you test 200 variations of a rule set: different moving average lengths, different stop multiples, different sessions. If you keep anything that clears a 95% bar, then roughly five percent of those variations will clear it by luck alone. That is about ten variations that look like genuine findings and are pure noise.1618
Nothing about your skill or your software caused that. It is what sampling does. The larger the search, the more certain it becomes that a good looking result exists, and the less that result tells you about the next hundred trades.
A rule set that needs precise parameters to work is usually a fitted curve. If the results fall apart when a 20 period average becomes 21, you have measured noise rather than a market behaviour. Robust findings tend to keep working, less well, across a range of neighbouring settings.
How many trades before a win rate means anything
A 55% win rate over 100 trades feels convincing and is not. The uncertainty around a win rate shrinks with the square root of the sample, which means it shrinks slowly.
| Trades | Standard error | 95% interval | What it means |
|---|---|---|---|
| 100 | 5.0% | 45% to 65% | Cannot rule out a coin flip |
| 400 | 2.5% | 50% to 60% | Barely clears a coin flip |
| 1,000 | 1.6% | 52% to 58% | Starting to mean something |
Read the first row literally. A hundred trade sample producing 55% is fully compatible with a strategy that wins half the time, and you would not be able to tell the difference from the results. This is why trade count matters more than the number of years covered. Ten years of a slow system can still be a hundred trades.
A test protocol that reduces the damage
- Write the rules before you look at the data. If the rule set changes after you have seen the results, the results are void. Write it down, date it, then test it.
- Log every variation you try. Keep a running count in the same file as the results. This is the missing number from the quote above, and it is the only way to know how much to discount your own best result.17
- Set aside a holdout you never touch, and prefer walk forward testing. Standard statistical guards like a simple holdout are known to be unreliable in the context of investment backtests, which is why the probability of backtest overfitting exists as its own measured quantity.18
- Discount the headline number for the search. Bailey and Lopez de Prado's point is that a Sharpe ratio has to be adjusted for how many trials produced it, because selection bias and backtest overfitting both inflate the number you are looking at.17
- Include costs and slippage, then make them worse. Commission, exchange and clearing fees and a realistic adverse fill per trade. Then run it again with worse assumptions and see whether the edge survives.
- Look at the shape, not just the total. The worst peak to trough drawdown in the test, the longest losing run, and how long the system spent recovering. Compare those against the drawdown rules you will be trading under.
- Check the distribution of results, not only the average. A system that makes its money from three outsized trades in a sample of two hundred has a different risk profile from one that grinds, even with the same total.
- Forward test small, with real money, before you buy an evaluation. A simulated account that is scored on someone else's fills is not the place to discover that you cannot execute the plan.
What a backtest can and cannot tell you
| It can tell you | It cannot tell you |
|---|---|
| Whether the rules are internally consistent and can be followed mechanically | Whether the behaviour you found will persist in a different market regime |
| Whether costs and fees consume the edge at your trade frequency | Whether the result is a real effect or the best of a large search |
| Roughly what drawdown and losing run length to expect | Whether you personally can sit through that drawdown without changing the rules |
| How often the setup appears, which sets your realistic trade count | How the strategy behaves when you are down and behind on rent |
The single most valuable output of a good backtest is not a profit figure. It is a number for how bad a losing run you should expect to sit through without concluding that the strategy is broken. That number is what lets you hold the rules when they are being tested, and it is the number most people never compute.
Related
Losing streaks and how to tell a cold run from a broken strategy and Position sizing and risk per trade.
Sources
The number next to each claim above is a stable source id. The same number means the same source on every page of this site.
- 16Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceBailey, Borwein, Lopez de Prado and Zhu, Notices of the American Mathematical Society 61(5)
- 17The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-NormalityBailey and Lopez de Prado, The Journal of Portfolio Management 40(5)
- 18The Probability of Backtest OverfittingBailey, Borwein, Lopez de Prado and Zhu, The Journal of Computational Finance 20(4)