Backtesting without lying to yourself

A backtest with a Sharpe of 3.0 is almost always a lie

Show a hedge fund quant a strategy with a 14-year backtest, Sharpe ratio of 3.0, max drawdown of 8%, and a 73% win rate. Their first question will not be "how do I deploy this." Their first question will be "where is the bug."

That is the right instinct. In live markets, a Sharpe ratio above 2.0 is exceptional; above 3.0 is almost certainly an artifact of methodology. Renaissance Technologies' Medallion Fund — by reputation the most successful quantitative hedge fund of all time — has a multi-decade Sharpe ratio that sits roughly in this neighborhood and charges 5-and-44 to access it.

If your homebrew backtest of a momentum strategy on Yahoo Finance data shows Sharpe 3.0, something is wrong with the backtest. This article catalogs the seven most common somethings and the walk-forward methodology that controls for all of them.

Bias 1: Survivorship bias

This is the most famous backtesting failure and still the most common. If your historical universe contains only stocks that exist today, your backtest is automatically excluding every company that went bankrupt, got delisted, was acquired below the offer, or otherwise failed.

A "small-cap value" screen run on today's Russell 2000 looks profitable because it inherits the survivors. The ones that did the worst version of small-cap value (highly levered, marginal business models) are not in your dataset because they no longer exist.

The fix: use a survivorship-bias-free dataset. CRSP, Compustat, Refinitiv, and a handful of other vendors maintain point-in-time universes that include delisted tickers with their last-known trading data. If you can't afford a paid feed, use IPO-date-anchored universes (every stock that IPO'd before date X, regardless of whether it still exists) and accept the data-collection cost.

Bias 2: Look-ahead bias

Look-ahead bias is using information in your backtest that wouldn't have been knowable at the time. The classic mistake: testing a strategy that uses annual earnings figures and assuming the figures were known on January 1 of the year they were reported. In reality, fiscal-year earnings are usually reported in February or March, and you only had access to them after the 10-K filed.

Less obvious examples:

• Using closing prices to make decisions that would have required

intraday data

• Using revised macro data (final GDP revisions) instead of the

vintage data that was available at the time

• Using analyst estimates from the "consensus as of today" instead

of the consensus that existed on the trade date

• Using index membership as it exists today instead of the

point-in-time membership

The fix: every input to the backtest must be lagged to its actual publication time. For company fundamentals, use the SEC filing date not the period-end date. For macro data, use ALFRED (the St. Louis Fed's vintage database) which gives you each series as it was first published.

Bias 3: Overfitting

Overfitting is the disease at the heart of backtesting. You run a strategy with 8 parameters across 20 years of data, optimize each parameter to maximize Sharpe, and discover a magical combination that backtests to a Sharpe of 4.2. You deploy live and the strategy returns -2% in three months.

What happened: with enough parameters and enough data, you can overfit any random sequence. The 4.2 Sharpe wasn't measuring strategy edge. It was measuring how well your parameter search located the specific noise of your specific data.

The fix: out-of-sample testing. Reserve at least 30% of your data for a holdout period that the parameter optimization never touches. If the strategy's Sharpe collapses in the out-of-sample period — which it usually does — you have an overfit strategy, not a real one.

Bias 4: Selection bias in strategies themselves

This is the bias that catches everyone, including professionals. You tested 50 strategies. 3 of them have a backtest Sharpe above 2.0. You publish those 3 and ignore the other 47. The 3 you publish are the survivors of a hidden search process.

Even if each strategy was tested honestly, the act of selecting from 50 strategies produces 3 false positives by chance alone. The mathematical relationship is sometimes called the "deflated Sharpe ratio" — the true expected Sharpe of a strategy must be discounted for the number of strategies you tested to find it.

The fix: report every strategy you tested, including the failures. If you tested 50 and 3 survived, your effective null hypothesis is not "this strategy works" but "this strategy works better than the 47 that didn't." The deflated Sharpe will typically be 60-80% of the naive Sharpe.

Bias 5: Transaction-cost ignorance

A strategy that backtests to Sharpe 2.5 with zero transaction costs will often backtest to Sharpe 0.4 with realistic costs. Many backtests quietly omit:

• Commissions (smaller today but still nonzero)
• Bid-ask spreads (the single largest cost for most retail strategies)
• Slippage on size (you can't actually trade your size at the touch)
• Borrow costs for shorts
• Market impact for liquid trades and illiquid trades alike

The fix: model transaction costs as a fixed haircut per round-turn. For US large caps, use 3-5 basis points per side. For small caps under $1B market cap, 15-25 basis points. For shorts, add the historical borrow rate from a securities-lending data feed. If a strategy survives costs at the rate you'd actually pay, it's plausibly real.

Bias 6: Regime-dependent edge

A strategy that works perfectly in 2010-2020 may have worked because that decade was a single low-volatility, low-rate, equity-bull regime. Test it across regimes — 1973-1975 bear, 1987 crash, 1998-1999 dot-com bubble, 2007-2009 crisis, 2020 pandemic, 2022 rate shock — and the picture often changes.

A strategy that doesn't survive regime change is not a strategy. It's a coincidence of a specific market environment.

The fix: never publish a backtest that covers fewer than two distinct market regimes. For US equities, that's roughly a 15-year minimum covering at least one recession and one extended bull market. For options strategies, include both 2008 (high vol) and 2017 (low vol) periods.

Bias 7: P-hacking via "robustness checks"

The newest professional sin. After your backtest succeeds, you run "robustness checks" — testing the strategy on slightly different universes, slightly different parameter values, slightly different time windows. The strategy holds up across all of them. You declare the strategy robust and publish.

But your "robustness checks" were themselves a form of optimization. If you ran 20 robustness checks and your strategy was robust on 18 of them, you have not proven robustness. You have proven that 18 of 20 selected variations confirmed the original finding — which is the expected statistical outcome of a strategy with no edge but moderate randomness in any individual test.

The fix: define your robustness check protocol *before* you see the original result. Write down "I will test these specific 5 variations" and report all 5 outcomes regardless of which ones pass.

The walk-forward method

The methodology that controls for most of the above is called walk-forward optimization. The procedure:

1. Split your data into time blocks. For 20 years of daily data, use rolling 5-year training windows and 1-year out-of-sample testing windows.

2. Train on the first window. Use the first 5 years to optimize your strategy's parameters.

3. Test on the next year. Run the optimized strategy on year 6 without changing parameters. This is genuinely out-of-sample because year 6 wasn't seen by the optimizer.

4. Slide forward. Now train on years 2-6, test on year 7.

5. Repeat. You end up with 15 years of true out-of-sample results from a strategy that was periodically re-optimized.

The walk-forward result is what your strategy would have actually returned if you'd been trading it live and periodically re-tuning it. If the walk-forward Sharpe is positive and stable, you may have a real edge. If it collapses, you have an overfit.

The strategies that actually survive

After a decade of working through backtests at hedge funds, here's the empirical observation: strategies that survive honest testing usually have these three properties.

• They have an economic logic, not just a statistical one. "When

small-caps are cheap, they outperform" has a logic. "Buy stocks with tickers that contain the letter Z on Tuesdays" does not. The former survives. The latter is overfit to noise.

• They have a Sharpe in the 0.5-1.5 range, not 2.0+. Real edge

is small. The frictionless 3.0 backtest is almost always wrong.

• They make money slowly and lose money slowly. Strategies with

long flat periods followed by sudden P&L spikes are usually capturing rare events that aren't likely to repeat. Strategies with smooth, modest equity curves usually capture something real.

What we publish at Veridion's Methodology Lab

The Veridion Score's information coefficient (IC) — the rank correlation between our score and forward returns — is published live at /static/lab.html with a 30/60/90-day rolling window.

If our methodology were overfit, the IC would be positive in our internal test data (the period we built the model on) and zero or negative in the live, out-of-sample period that's published in the lab. The fact that we publish it daily, in public, with the explicit caveat "if unverifiable, not displayed" is the honest version of a walk-forward result.

You can run the same test on us. That's the whole point.