Overfitting in backtests

Overfitting happens when a rule is adjusted until it matches quirks in the data used to design it. A strategy can look convincing on those same prices because its settings have absorbed both useful structure and random noise. The risk grows when many alternatives are tried and only the most attractive result is shown.

A backtest is a measurement, not a time machine. Historical prices contain patterns, but they also contain chance movements. If you keep changing entry thresholds, exit rules, markets and date ranges after seeing each result, eventually one version will line up with an unusual run. That result may describe the sample rather than a repeatable feature.

Overfitting is not limited to complex machine learning. A person who tests a handful of moving-average lengths and reports only the best has made a selection from alternatives. A spreadsheet with a few parameters can be tuned just as easily as a large model. The number of attempts matters because every try creates another opportunity to find a lucky-looking outcome.

A worked example

Suppose a researcher tests five fictional settings on an earlier training period. The measured scores are 1, 2, 1, 8 and 2 points. The setting scoring 8 is selected. On a separate later period, the same fixed setting scores 1 point. These round scores are made up and have no financial meaning. They show why the best training result can shrink when checked on data that did not guide the choice.

The selection itself matters. If the researcher had picked the setting before seeing any prices, the later result would answer a different question. Once the settings were compared on the training sample, that sample became part of the selection process. Reporting the winner as though it had been specified in advance understates how much searching took place.

How a held-out test helps

A held-out period is reserved from the selection process. First write down the candidate rules and choose settings using the training data. Then freeze the decisions. Finally, evaluate the unchanged rule on the held-out data. This asks whether the choice made earlier also describes a later segment that was not used to select it.

Time order is important for trading research. A sensible split puts training data before the test data, because a real strategy cannot use future prices to make past decisions. The split should be planned before the result is inspected. If you repeatedly look at the holdout and adjust the rule, it stops being untouched and becomes another training set.

A held-out result is not proof. It may reflect an unusual market regime, a short sample, data errors or chance. One split can be fortunate or unfortunate. The rule may also be tested on many assets, variants or holdouts, creating a larger search that needs to be disclosed. A clean separation reduces one source of optimism; it does not remove uncertainty.

What else should be controlled?

Record the full search, including failed settings, and avoid changing the evaluation window because the first result is disappointing. Compare against a clear baseline over the same dates. Check whether trade timing uses only information available at the time. Include delisted names where data allows, and state coverage limits. These checks address different biases and should not be collapsed into one score.

Costs are another separate issue. StrategyLens backtests use the zero-cost basis set out in ADR 0033: no commission or slippage is charged, and the buy-and-hold comparator is treated the same. Real trading costs money, so real results can be lower. A held-out test under zero costs can still be useful for evaluating the process, but it does not establish a cost-adjusted real-world outcome.

A further check is stability. If a rule only appears favourable at one exact setting and nearby settings are very different, that sharp peak can be a warning that the result depends on a narrow fit. Stability does not prove a real effect, and a broad plateau does not guarantee future performance. It simply adds context to the held-out comparison.

The number of experiments should be part of the story. If a researcher screened dozens of strategies and publishes one, readers need to know the selection was made. Statistical methods can adjust for multiple comparisons, but their assumptions also need to be clear. A single holdout cannot erase extensive search if the holdout itself was repeatedly consulted.

How StrategyLens describes its method

A backtest runs explicit rules on historical prices. Its result depends on the rule, data, period and evaluation choices. We do not claim that passing a held-out test proves a strategy works. We report the test as evidence about that defined sample, with limitations. Past results, real or simulated, do not tell you what will happen next.

The most useful interpretation is modest: a rule that fails on data it did not use has not travelled well in that test. A rule that holds up has survived one check, not every possible market or implementation. Further independent periods, assets and carefully disclosed experiments can add information, but each must be treated as evidence with boundaries.

This is research, not financial advice. Past results, real or simulated, do not tell you what will happen next.

The same discipline applies to choosing the split point. If a researcher tries several dates and retains the split with the strongest result, that choice has also been tuned. A predeclared date or a documented chronological procedure makes the evaluation easier to interpret. There is no single split that suits every dataset; the important point is to explain the choice and keep the evaluation observations from steering it.

What is overfitting in a backtest?

Overfitting is choosing a rule or settings that match quirks in the data used to develop them, rather than a repeatable pattern.

How does a held-out test help?

It reserves later or otherwise unseen data for evaluation, so choices made on the training data can be checked without using the same observations twice.

Does a holdout prove a strategy works?

No. A holdout reduces one source of bias, but can still be noisy, reused, or unrepresentative. It is evidence about that test, not a promise.