Backtest overfitting means the rules were tuned so closely to past data that they describe noise, not a repeatable pattern. Warning signs include many tweaked settings, few trades and results that collapse on data the rules have never seen. Check nearby settings, then test on a stretch held back until the end.
Quick answer: Backtest overfitting means the rules were tuned so closely to past data that they describe noise, not a repeatable pattern. Warning signs include many tweaked settings, few trades and results that collapse on data the rules have never seen. Check nearby settings, then test on a stretch held back until the end.
So, what is backtest overfitting? It is what happens when you keep adjusting a strategy until it fits the past beautifully, and most of that fit is noise. The backtest looks excellent, then the next stretch of candles disagrees. This post covers how it happens, how to spot it and how to check your own plan.
What is backtest overfitting?
A backtest replays your rules over historical candles to show what they would have done. Overfitting is when the rules have been shaped to that one stretch of history, so they describe its quirks instead of a pattern that repeats. In the words of a well-known 2014 paper, the model "targets particular observations rather than a general structure".
Think of memorising last year's exam paper. You score full marks on that paper and learn nothing about the subject.
Two terms help here. In-sample data is the history you used to design and tune the rules. Out-of-sample data, or unseen data, is history the rules never touched. An honest backtest performs about the same on both, and an overfitted one does not.
Why do crypto traders run into it so often?
Three things push crypto traders towards curve fitting, which is simply another name for overfitting:
Short history. Many coins have only a few years of candles, so there are fewer different market conditions to test against.
Plenty of settings. Indicator length, entry threshold, stop, target and timeframe all multiply the number of versions you can try.
Shifting regimes. Markets move between trending, ranging and quiet stretches, so rules tuned to one stretch can fall flat in the next.
How many tries does it take to fool a backtest?
Hypothetical example, not a real strategy. Say a rule buys when RSI (a momentum indicator) drops below a threshold and sells when it recovers, with a stop and a take-profit. Four settings can move: RSI length, entry threshold, stop distance and take-profit distance. Give each five values and that is 5 × 5 × 5 × 5 = 625 versions of one idea. Illustrative arithmetic only.
Test all 625 on the same year of candles and rank them. Something will land on top, even if the idea has no edge at all, because 625 tries means 625 chances for a lucky fit. Afterwards the winning settings are easy to explain, which makes them feel earned.
Researchers have measured how quickly this builds. In Pseudo-Mathematics and Financial Charlatanism (Notices of the American Mathematical Society, 2014), Bailey, Borwein, López de Prado and Zhu calculate that if none of the versions has any real skill, trying just seven independent configurations on a two-year backtest is enough for the best one to be expected to show an in-sample Sharpe ratio of 1. Its expected out-of-sample Sharpe ratio is zero. (The Sharpe ratio is a common measure of return for the risk taken.) The seven-version figure is theoretical, and your own versions are rarely fully independent, so treat it as a sense of scale, not a rule of thumb.
The authors also point to what is missing from most backtests. They write that "a backtest which does not report the number of trials N used to identify the selected configuration makes it impossible to assess the risk of overfitting."
A strategy someone else shows you hides that number. Your own does not, so keep count.
What are the warning signs of an overfitted backtest?
Many tweaked settings. Each extra setting, filter or exception is another way to fit the past.
Few trades. A handful of trades can look convincing and still be luck. Evidence Lab withholds its grade when a strategy has fewer than 30 completed trades.
A result that collapses on unseen data. The strong backtest weakens the moment the rules meet candles they were not tuned on.
Settings that only work at one value. If a 14-period version looks great and 12 or 16 falls apart, you found a spike, not a plateau.
A near-perfect curve. Smooth results with tiny pullbacks usually say more about the tuning than the market.
How do you check a backtest for overfitting?
Four checks, in the order most traders find easiest to run:
Nudge the settings. Move each one a notch either way and rerun. Evidence Lab's fragility check does this by varying nearby settings and increasing execution costs, to see whether the result depends on one narrow configuration.
Hold back the final stretch and look once. Choose your rules on earlier history and keep the latest part aside. Evidence Lab keeps the final 30% of the tested history hidden until you choose Reveal. The reveal is recorded and cannot be undone, so you cannot peek, adjust and peek again.
Count your trials. Write down how many versions you tested. Evidence Lab's overfitting meter records your strategy trials and adjusts for repeated testing, so the count does not rest on memory.
Add realistic costs. Fees and slippage eat into short-timeframe results. Strategy Lab records the market, data range, fees and slippage used for each test, so read those before you read the result.
If the result only survives on the cheapest assumptions or the luckiest settings, treat it as a lead to investigate, not a finding. Our post on the 5 checks before you trust a strategy you saw on YouTube covers the wider set, including random entries.
What are the common mistakes?
Peeking more than once. Once you have looked at the held-back stretch and changed the rules, it has joined your design data.
Forgetting the count. Thirty quiet reruns count as thirty tries.
Fixing a weak result by adding rules. Every new filter is one more setting to fit.
Treating a strong backtest as proof. A backtest is a historical simulation under stated assumptions, and it does not predict future results.
Leaving gaps in the rules. If you build with Pulse, its Assumed labels mark rules you never chose. Each one is a setting worth testing before you trust the result.
FAQ
How do you avoid overfitting a backtest?
Keep the rules simple, with as few settings as the idea needs. Check nearby settings, hold back a final stretch of history and look at it once, and keep a count of every version you try.
Is my backtest overfitted?
One number cannot tell you, but a pattern can. If the result weakens on unseen data, falls apart when you nudge a setting or rests on only a few trades, overfitting is a likely cause.
What is curve fitting in a crypto backtest?
Curve fitting is another name for overfitting. The rules get drawn around the curve of past prices, so they match history closely and describe little else.
Can a backtest prove a strategy works?
No. A backtest shows how rules would have behaved in the past under stated assumptions. Passing a check is a reason to keep testing, not permission to trade.
Ready to test yours?
Take a saved backtest and see how it holds up. Challenge your backtest with Evidence Lab's checks.
Educational only. Not advice. Crypto is high risk.
