Guide / Trading research
How to tell if a backtest is lying to you
Every backtest looks good, because bad ones get deleted before anyone sees them. The academic literature on this is blunt: most published trading results are probably false positives. Here are the specific signs that a backtest is overfit, and the checks that separate a real edge from a curve fit to history.
Why the default assumption is "overfit"
Bailey, Borwein, Lopez de Prado, and Zhu showed the core problem mathematically in a 2014 paper in the Notices of the American Mathematical Society: if you try enough strategy configurations on the same data, you are essentially guaranteed to find one with an impressive in-sample Sharpe ratio, even when no configuration has any real edge. With five years of data, testing as few as 45 independent configurations is enough to expect a Sharpe of 1.0 from pure noise (Pseudo-Mathematics and Financial Charlatanism). Since most researchers iterate through hundreds of variants, a strong backtest is the expected outcome of the process, not evidence of skill.
The same problem infects academic finance. Harvey, Liu, and Zhu catalogued the hundreds of published return factors and concluded that, once you correct for how many things have been tried, a new result should clear a t-statistic of 3.0 rather than the usual 2.0, and that most claimed findings in the field are likely false (...and the Cross-Section of Expected Returns). If professionals with referees fool themselves at that rate, a solo researcher with a parameter slider should assume the worst by default.
Sign 1: no true out-of-sample period
The first question for any backtest: which data did the strategy never touch during development? If every year of history was available while rules and parameters were being chosen, there is no out-of-sample evidence at all, only in-sample fit. The honest structure reserves a slice of data (or better, live paper trading going forward) that gets used exactly once, after development is frozen. QuantConnect's research guide recommends exactly this discipline: hypothesis first, walk-forward testing, and a held-out period the researcher cannot peek at (QuantConnect research guide). Beware the quiet cheat: "out-of-sample" data that was tested, found wanting, and then used to revise the strategy is in-sample data with extra steps.
Sign 2: performance collapses when parameters wiggle
A real edge is a region, not a point. If the strategy earns 40 percent annually with a 21-day lookback and loses money at 19 or 23, the backtest found a coincidence. The check is cheap: sweep each parameter across a sensible range and plot the results. Smooth, gradual degradation is what economic effects look like. Sharp cliffs around one magic value are what noise looks like. The same logic applies to start dates: a strategy whose results depend heavily on whether the test starts in January 2016 or June 2016 is telling you the equity curve is a few lucky trades wearing a trend line.
Sign 3: the data knows the future
Look-ahead bias means the simulation used information that was not available at the moment of each decision: trading on a close price before the close, using earnings figures on the announcement date when they were filed weeks later, or normalizing indicators with statistics computed over the full dataset. It produces spectacular and completely fictional results. Survivorship bias is its quieter sibling: testing on today's index members means every company that went bankrupt or got delisted was silently excluded, which flatters almost any long strategy. The fix is point-in-time data that includes dead tickers, and an audit of every input for when it actually became knowable.
Sign 4: costless trading
Commissions, spread, slippage, and borrow costs decide whether high-turnover strategies survive contact with a real account. A backtest that assumes fills at mid-price with zero fees can turn a money-losing system into a beautiful equity curve, and the more the strategy trades, the bigger the lie. Reasonable practice is to model realistic fees and slippage, then re-run the test at double the assumed costs. An edge that dies when costs double was never robust enough to trade.
The probabilistic Sharpe ratio
A raw Sharpe ratio ignores how much data produced it and how ugly the return distribution is. The probabilistic Sharpe ratio (PSR), from Bailey and Lopez de Prado's Sharpe Ratio Efficient Frontier paper, computes the probability that the true Sharpe exceeds a benchmark, penalizing short track records, negative skew, and fat tails (The Sharpe Ratio Efficient Frontier). A Sharpe of 1.5 over eighteen months of smooth returns and a Sharpe of 1.5 over three months of lumpy ones are very different claims, and PSR says by how much. The companion deflated Sharpe ratio goes further and discounts for the number of trials you ran to find the strategy (The Deflated Sharpe Ratio). QuantConnect reports PSR on every backtest; a PSR below roughly 95 percent is a flag, not a footnote.
The checklist
- A held-out data period or forward paper-trading window the strategy never saw during development.
- Parameter sweeps showing a plateau of profitability, not a single spike.
- Every input audited for look-ahead; test universe includes delisted assets.
- Realistic costs modeled, and the strategy re-tested at double those costs.
- PSR reported alongside Sharpe, with the number of configurations tried disclosed.
Running this gauntlet is exactly what VerifiedDesk's validation suites and backtest audits do: an adversarial pass over your strategy's data handling, sensitivity, and statistics, delivered as a cited report. Whether you hire that out or do it yourself, the standard is the same: a backtest is a hypothesis test, and the burden of proof sits on the equity curve.
Sources: Bailey, Borwein, Lopez de Prado, Zhu, Pseudo-Mathematics and Financial Charlatanism, Notices of the AMS (ssrn.com); Bailey, Borwein, Lopez de Prado, Zhu, The Probability of Backtest Overfitting, Journal of Computational Finance (ssrn.com); Harvey, Liu, Zhu, ...and the Cross-Section of Expected Returns, Review of Financial Studies (ssrn.com); Bailey, Lopez de Prado, The Sharpe Ratio Efficient Frontier (ssrn.com); Bailey, Lopez de Prado, The Deflated Sharpe Ratio, Journal of Portfolio Management (ssrn.com); QuantConnect research guide (quantconnect.com). This article is research education only. Backtested performance is hypothetical, and nothing here is investment advice or a recommendation to trade any strategy.