Lessons › Advanced track › Backtest traps: bias, overfitting and costs

World 10 · Advanced track · lesson 11 · level 5

Backtest traps: bias, overfitting and costs

A backtest is easy to fool, so check it for peeking, missing losers, too many tweaks and missing costs.

In one line

A backtest shows a smooth, soaring equity line. Which of four traps inflated it?

Explained simply

A backtest can fool you like a magic trick: the result looks amazing until you see what was hidden. It might peek at the future, leave out the losers, be tweaked until it fits the past perfectly, or ignore fees. Checking for these four tricks shows you the real result.

The lesson

Look-ahead bias uses information that was not available at the time, and survivorship bias tests only on assets that still exist, leaving out the ones that failed or were delisted. Overfitting happens when rules are tuned so many times that they fit past noise, which is also called data snooping when many ideas are tried on the same data. Ignoring costs and slippage inflates results, especially for frequent trading. Each trap makes a strategy look better on paper than it will be in reality.

A worked example

Illustrative example: a backtest shows 500 trades averaging +0.15R before costs, a total of 500 x 0.15 = 75R. Costs and slippage average 0.12R per trade, so the real edge is 0.15 - 0.12 = 0.03R and the total shrinks to 500 x 0.03 = 15R. The test also used only the 80 stocks in today's index, but 100 stocks were in it when the test began, so 100 - 80 = 20 names, a fifth of the list, were left out, many of them because they fell and were removed.

The same idea at four levels

1. Beginner. A backtest can look better than reality if it hides problems.
2. Foundation. The four traps are peeking at the future, testing only survivors, tweaking too much and ignoring costs.
3. Intermediate. Test on the stocks that were in the index at each past date, not just today's members, and include every cost.
4. Advanced. Count your tweaks, because every extra setting tuned on the same data raises the chance that the best version is just fitting noise, so keep rules few and simple.
5. Expert. Report expectancy before and after costs with realistic slippage, and trust only strategies whose edge survives costs and stays positive on data that was never used for tuning.

Mistakes to avoid

• Testing only on the stocks in today's index.
• Adding another rule every time the backtest disappoints.
• Reporting results before costs as if they were real.

Goal of this lesson: Identify the main biases that make backtests look better than real trading.