Survivorship bias in backtests

Survivorship bias occurs when an analysis looks only at things that survived until the end of the period. In investing, that can mean testing a rule on companies that still trade today while leaving out companies that failed, merged, were acquired or were delisted. The missing cases can make the historical picture look healthier than the full set of choices was.

The problem is easy to miss because a current list feels complete. A list of today’s large companies contains familiar names and clean price histories. But if you take that list backwards in time, you are quietly asking: how would this rule have performed on companies that we now know survived? A researcher at an earlier date would not have known which firms would make it to the present.

This creates a selection problem. The companies that disappear may have disappeared for many reasons, and not all delistings mean failure. Some are bought at a premium; others stop trading after distress or bankruptcy. The direction and size of bias depend on which cases are missing and how they are treated. The key point is not that every omitted company lost money, but that the test no longer represents the historical universe it claims to represent.

Why it matters

Suppose a strategy selects the strongest shares from a list of current constituents. Shares that fell out of an index or stopped trading are absent. The test can then measure performance among eventual survivors, not among all the shares that could have been selected at the time. If failures are missing, an apparent record may partly reflect knowledge that was unavailable to the rule in real time.

Survivorship bias can combine with other errors. Look-ahead bias uses information that was not available on the decision date. Data snooping tries many ideas and reports the best. A backtest can have all three: a present-day universe, future information in its membership list, and settings selected after trying many variations. Each issue asks a different question, so fixing one does not fix the others.

A worked example

Imagine a fictional universe that began with ten companies. Five still trade today and five were delisted. A simple rule would have earned £10 on each survivor and lost £10 on each delisted company in a made-up test. Looking only at today’s survivors shows five gains and no losses. Including all ten shows gains and losses offset. The round figures are invented to explain the selection effect; they are not market evidence.

The example does not imply that every delisted company would have lost, or that every survivor prospered. It shows why the membership rule matters. If the question is how a historical investor could have applied a rule, the data should represent the companies available at each date as far as possible. If the data cannot do that, the limitation should be stated clearly.

How StrategyLens treats delisted prices

Historical price coverage is imperfect. StrategyLens tests on delisted prices where it can, because omitting former companies can create a survivor-only view. That does not mean every past security is present or that every delisting outcome is fully captured. Data coverage varies, and a missing terminal price can prevent a fair test. We describe the method and its limits rather than treating an incomplete universe as complete.

A delisted security may stop having prices before the event that matters. It may also have a final payment, merger consideration, or other corporate action that ordinary price files do not represent. A price series alone may not capture the full outcome. Researchers need to know whether returns include distributions and how corporate actions were handled before interpreting a result.

Our published performance figures come from simulated backtests on past data. Under ADR 0033, the runs charge no trading costs and the buy-and-hold comparison is charged the same. Real trading costs money, so real results can be lower, particularly for rules that trade often. This cost basis is separate from survivorship bias: a test can treat costs consistently and still omit historical companies.

Ways to check a backtest

Ask what the universe contained at each date, not only what symbols it contains now. Ask how delisted names, mergers, bankruptcies and missing prices were treated. Ask whether the strategy had access only to information available at the time. If the source cannot answer, the result may still be useful as a limited exercise, but it should not be read as a complete historical record.

The period matters too. A database may cover recent years well and earlier years poorly. Its symbol list can change over time, and a backtest may use a fixed list for convenience. A transparent note should say what was tested and what was unavailable. Where data is absent, inventing a smooth final price or silently dropping the company can create a false sense of precision.

Survivorship bias is not a reason to dismiss all backtests. It is a reason to match the data to the question and to make the remaining gaps visible. A carefully bounded test can explain how a written rule behaved on the available cases. It cannot establish how the rule would have behaved on an unknown complete universe unless the missing history is addressed.

This is research, not financial advice. Past results, real or simulated, do not tell you what will happen next.

A useful audit is to compare the dataset's historical membership with an independent record, then list symbols with missing final observations. That audit does not automatically recover missing prices, but it makes omissions countable and reviewable. It also prevents a present-day list from quietly standing in for a historical list. Readers can then distinguish a limitation in the source from a conclusion about the trading rule itself.

What is survivorship bias?

It is the distortion that occurs when an analysis includes only entities that survived to the end, leaving failed or delisted ones out.

Why does survivorship bias matter in a backtest?

A universe made only of current stocks can omit companies that disappeared, making historical results look more favourable than the full historical opportunity set.

Does StrategyLens include delisted prices?

Where suitable delisted price data is available, StrategyLens tests on it. Coverage is incomplete, so the notes describe that limitation rather than claiming complete delisting history.