PropReady

Technical track

11 min read · Reviewed 2026-11-16

Testing a strategy without fooling yourself

Backtest overfitting is not an exotic mistake made by careless people. It is the default outcome of searching hard enough, and the reason is arithmetic rather than character.

The number that is missing from almost every backtest

Bailey, Borwein, Lopez de Prado and Zhu put the problem in a sentence worth reading twice: "a backtest where the researcher has not controlled for the extent of the search involved in his or her finding is worthless, regardless of how excellent the reported performance might be."17

What they are pointing at is this. A backtest result is not a measurement of a strategy in isolation. It is the best result from however many strategies you tried. Ten attempts and one thousand attempts can produce the same headline number, and the second number means far less. So the honest reporting unit is not the winner. It is the winner plus the count of everything you threw away getting there.

Why searching harder makes results worse, not better

Suppose you test 200 variations of a rule set: different moving average lengths, different stop multiples, different sessions. If you keep anything that clears a 95% bar, then roughly five percent of those variations will clear it by luck alone. That is about ten variations that look like genuine findings and are pure noise.1618

Nothing about your skill or your software caused that. It is what sampling does. The larger the search, the more certain it becomes that a good looking result exists, and the less that result tells you about the next hundred trades.

The tell

A rule set that needs precise parameters to work is usually a fitted curve. If the results fall apart when a 20 period average becomes 21, you have measured noise rather than a market behaviour. Robust findings tend to keep working, less well, across a range of neighbouring settings.

How many trades before a win rate means anything

A 55% win rate over 100 trades feels convincing and is not. The uncertainty around a win rate shrinks with the square root of the sample, which means it shrinks slowly.

95% confidence interval around an observed 55% win rate, computed from the standard error of a proportion.
TradesStandard error95% intervalWhat it means
1005.0%45% to 65%Cannot rule out a coin flip
4002.5%50% to 60%Barely clears a coin flip
1,0001.6%52% to 58%Starting to mean something

Read the first row literally. A hundred trade sample producing 55% is fully compatible with a strategy that wins half the time, and you would not be able to tell the difference from the results. This is why trade count matters more than the number of years covered. Ten years of a slow system can still be a hundred trades.

A test protocol that reduces the damage

  1. Write the rules before you look at the data. If the rule set changes after you have seen the results, the results are void. Write it down, date it, then test it.
  2. Log every variation you try. Keep a running count in the same file as the results. This is the missing number from the quote above, and it is the only way to know how much to discount your own best result.17
  3. Set aside a holdout you never touch, and prefer walk forward testing. Standard statistical guards like a simple holdout are known to be unreliable in the context of investment backtests, which is why the probability of backtest overfitting exists as its own measured quantity.18
  4. Discount the headline number for the search. Bailey and Lopez de Prado's point is that a Sharpe ratio has to be adjusted for how many trials produced it, because selection bias and backtest overfitting both inflate the number you are looking at.17
  5. Include costs and slippage, then make them worse. Commission, exchange and clearing fees and a realistic adverse fill per trade. Then run it again with worse assumptions and see whether the edge survives.
  6. Look at the shape, not just the total. The worst peak to trough drawdown in the test, the longest losing run, and how long the system spent recovering. Compare those against the drawdown rules you will be trading under.
  7. Check the distribution of results, not only the average. A system that makes its money from three outsized trades in a sample of two hundred has a different risk profile from one that grinds, even with the same total.
  8. Forward test small, with real money, before you buy an evaluation. A simulated account that is scored on someone else's fills is not the place to discover that you cannot execute the plan.

What a backtest can and cannot tell you

Being clear about which questions a historical test answers.
It can tell youIt cannot tell you
Whether the rules are internally consistent and can be followed mechanicallyWhether the behaviour you found will persist in a different market regime
Whether costs and fees consume the edge at your trade frequencyWhether the result is a real effect or the best of a large search
Roughly what drawdown and losing run length to expectWhether you personally can sit through that drawdown without changing the rules
How often the setup appears, which sets your realistic trade countHow the strategy behaves when you are down and behind on rent
The uncomfortable part

The single most valuable output of a good backtest is not a profit figure. It is a number for how bad a losing run you should expect to sit through without concluding that the strategy is broken. That number is what lets you hold the rules when they are being tested, and it is the number most people never compute.

Related

Losing streaks and how to tell a cold run from a broken strategy and Position sizing and risk per trade.

Sources

The number next to each claim above is a stable source id. The same number means the same source on every page of this site.

  1. 16Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceBailey, Borwein, Lopez de Prado and Zhu, Notices of the American Mathematical Society 61(5)
  2. 17The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-NormalityBailey and Lopez de Prado, The Journal of Portfolio Management 40(5)
  3. 18The Probability of Backtest OverfittingBailey, Borwein, Lopez de Prado and Zhu, The Journal of Computational Finance 20(4)