Research workspace
Overfitting Explainer
A strategy search can reward the luckiest historical variant. Explore a simple selection-bias example, then inspect how Evidence Lab scores robustness checks.
How many variants did you try?
More attempts create more chances to find a result that looks good by chance, even when none has a real edge.
Chance of at least one false positive
64.2%
At 5% per test across 20 independent tests.
Strategy variants are usually correlated, so this calculation is a teaching example—not a PBO estimate or a result for any EdgeTheory strategy.
Selection risk as the search grows
At the selected per-test rate, the formula is 1 − (1 − α)n, where n is the number of independent tests.
The curve does not adjust for correlation between variants. When many variants share rules or data, their tests are not independent; this chart does not estimate an effective trial count.
Evidence Lab score: the full 100 points
The weights, grade bands and critical-check cap below are read from the same scoring constants used by Evidence Lab.
| Check | What it covers | Published rule | Maximum |
|---|---|---|---|
| Luck comparison | Compare the saved result with random-entry alternatives. | Pass at or above 95% random-entry percentile; caution from 80%; fail below. | 20 points |
| Unseen data | Test on the held-out part of the historical window. | Pass only with complete coverage, positive held-out net P&L and Sharpe, and held-out Sharpe at least 50% of in-sample Sharpe. | 15 points |
| Overfitting | Account for repeated strategy trials and selection. | Fail below 80% Deflated Sharpe or above 50% PBO. If there is no measured failure, a missing Deflated Sharpe is unavailable. Pass at or above 95% Deflated Sharpe with PBO below 20% or unavailable; other complete combinations are caution. PBO needs at least 8 aligned trial series. | 20 points |
| Bad-luck stress | Measure how alternate trade sequences affect drawdown. | Pass when the 95th-percentile bootstrap drawdown is within the report's stated tolerance (default 25%); otherwise caution. | 10 points |
| Fragility | Check nearby settings and higher execution costs. | Pass when at least 70% of the valid cost, noise and nearby-setting stress runs remain profitable; otherwise fail. | 15 points |
| Other markets | Award breadth only where other markets were tested. | Requires at least three evaluated other markets. Pass when at least 70% are profitable, caution if some but fewer pass, and fail if none pass. Points equal floor(10 × profitable share). | 10 points |
| Enough data | Withhold confidence when the completed-trade sample is thin. | Pass when held-out trades meet the minimum-track-record calculation and the 100-trade target; caution at 30 or more but below that requirement; fail below 30. A grade is withheld when total completed trades are below 30. | 10 points |
| Total | Available score before grade bands and caps. | 100 points | |
A pass earns full points; a caution earns 60% of the check weight, rounded down; fail or unavailable earns zero. Market-breadth points use the profitable-share formula shown above. A failed luck or overfitting check caps the grade at D. The grade is also withheld for fewer than 30 completed trades, incomplete fixed-calendar history or an insufficient one-year candle sample. The walk-forward and regime views are reported separately and carry zero points in this grade.
Published check thresholds
These thresholds come from the current methodology constants; each report also records its methodology version and actual data coverage.
- Random-entry comparisonPass ≥ 95% · caution ≥ 80%
- Deflated SharpePass ≥ 95% · caution ≥ 80%
- Probability of backtest overfittingPass < 20% · caution ≤ 50%
A pass is one historical check under its recorded assumptions. It does not prove that a strategy has a durable edge or predict future performance.
Read the complete versioned Evidence Lab methodologyThese tools explain calculations and historical evidence under the assumptions shown. They are not forecasts, trading advice or proof of future performance.