Overfitting and out-of-sample testing, in plain English

Overfitting is tuning a strategy so closely to past data that it captures noise rather than anything that will happen again. An overfitted strategy looks excellent on the data it was built on and disappoints on anything else.

Out-of-sample testing is the main defence: judging a strategy on data that played no part in choosing it.

How overfitting happens

Every strategy has settings: how far a price must fall, how long to look back, when to sell. Try enough combinations and some will fit the past well purely by chance, the way a few people in a large room will win several coin tosses in a row. The more combinations you try, the better the best one looks, and the less that tells you.

It also happens by small steps. A tester sees a bad patch, adds a rule to avoid it, sees another, adds another. Each change is reasonable, and together they describe the past rather than a lasting pattern.

In-sample and out-of-sample

In-sample data is what you used to choose the settings. Out-of-sample data is what you did not touch. A simple version: choose settings using prices up to a cut-off year, then see how those exact settings did on the years after it, without changing anything.

If the result holds up on the unseen years, that is some evidence. If it falls apart, the in-sample result was probably fitting noise. Our pages use the labels "held up", "did not hold up" and "inconclusive" for this reason.

Other ways to test for it

Walk-forward testing repeats the split many times, choosing settings on a window and testing on the next one, then moving forward. Testing on other tickers or markets asks whether the pattern is general. Small changes to the settings ask whether the result sits on a narrow peak or a broad plateau; a result that vanishes when a setting moves slightly is fragile.

Allowing for how many things were tried

Statistical corrections exist for this. The false discovery rate correction adjusts for testing many ideas at once. The deflated Sharpe ratio adjusts a strategy's Sharpe ratio for how many variations were tried. Both make it harder for a lucky result to pass, and both are used in our research.

What our own tests show

We run the same rules in 553 backtests on every published ticker, which is the kind of repeated testing in which overfitting thrives. That is why we publish how often the strategies beat holding across all of them, not just the best one. Across the 38 published tickers, 2,565 of 21,014 backtests beat holding the same ticker over the same dates (12.2%). On 37 of the 38 tickers, fewer than half of the backtests beat holding.

Figures are from the 38 StrategyLens published ticker cards, updated 29 October 2026. They are hypothetical backtests with no trading costs, before tax, and they will change as the cards are refreshed.

Reporting only the best setting on each ticker would have produced a far more flattering list, and it would have been mostly luck.

Signs a result may be overfitted

Many rules or exceptions. Very precise settings, such as a 13.7% fall over 41 days. A perfect or near-perfect record. No out-of-sample test. No mention of how many variations were tried. Results that only appear on one ticker or one period.

A coin-toss way to see it

Imagine 1,000 people each tossing a fair coin ten times. On average, about one of them will get ten heads in a row. That person did nothing skilful, and if they toss again, they are no more likely to get heads than anyone else. Testing 1,000 strategy settings is similar: a few will look exceptional by chance, and picking the best one after the fact selects the lucky toss.

This is why the number of attempts matters as much as the best result. One setting that beat holding out of ten tried is more interesting than one that beat holding out of a thousand.

Simplicity as a defence

Fewer rules and fewer settings give overfitting less room. A rule that can be written in one sentence and tested with a handful of round-number settings is harder to fit to noise than one with many conditions and finely tuned values. Simple rules can still fail, but when they work across many tickers and periods, the result is easier to trust.

A good test plan is written down before the results are seen: which settings, which tickers, which periods, and what would count as success. Changing the plan after looking is where many overfitted results begin.

A last check costs nothing: before trusting a result, ask what would have been concluded if the test had come out the other way. If a failure would have been explained away with a new rule, and a success taken at face value, the test was not really able to fail. A result that could have failed, and did not, is the only kind worth acting on, and even then only after it has been checked on data it never saw.

This is research, not financial advice. Past results, real or simulated, do not tell you what will happen next.

What is overfitting in trading?

Tuning a strategy so closely to past data that it captures noise. It looks good on the data it was built on and disappoints on new data.

What is out-of-sample testing?

Judging a strategy on data that was not used to choose its settings, such as later years held back from the start.

How many settings is too many?

There is no fixed number, but the more you try, the stronger the evidence you need. Corrections such as the deflated Sharpe ratio allow for the number of attempts.