Lessons › Advanced track › Out-of-sample, walk-forward and Monte Carlo
World 10 · Advanced track · lesson 12 · level 5
Out-of-sample, walk-forward and Monte Carlo
Test your idea on data it has never seen, and shuffle its trades to see how bad the bad luck could get.
In one line
Your backtest showed one path. What would a thousand other orderings of the same trades look like?
Explained simply
Studying with last year's exam paper is fine, but the real test is a paper you have never seen. Out-of-sample testing keeps some data hidden as that unseen paper. Shuffling your trades is like replaying the season in a different order to see how rough the losing runs could get.
The lesson
Out-of-sample testing keeps part of the data untouched while the rules are built, then tests on it once at the end, for example building on about 70% of the data and keeping 30% for the final test. Walk-forward testing repeats this in steps: tune on one window, test on the next, then roll forward. A Monte Carlo simulation reshuffles or resamples the trade results thousands of times to show the range of possible drawdowns and streaks, not just the one path that happened. If results collapse out of sample, the in-sample edge was probably fitting noise.
A worked example
Illustrative example: ten years of data are split so that 10 x 70 / 100 = 7 years are used to build the rules and the last 3 stay hidden. In-sample, the rule averaged +0.30R per trade. On the hidden years it made 60 trades, 24 wins at +1.8R and 36 losses at -1R: 24 x 1.8 - 36 x 1 = 7.2R, or 7.2 / 60 = 0.12R per trade. The edge fell by 0.30 - 0.12 = 0.18R, keeping 0.12 x 100 / 0.30 = 40% of it. Reshuffling those 60 trades 1,000 times gave the same 7.2R total every time, but in the worst 5% of orderings the deepest drawdown reached 13.2R or more, against 9R in the order that actually happened.
The same idea at four levels
- Beginner. Keep some data hidden while you build a strategy, then test on it once at the end.
- Foundation. In-sample data is for building, and out-of-sample data is the unseen final exam.
- Intermediate. Walk-forward testing tunes on one window, tests on the next, then moves forward and repeats.
- Advanced. Compare expectancy in and out of sample, since a big drop means the rules were tuned to noise, and every peek at the out-of-sample data before the final test quietly turns it into in-sample data.
- Expert. Use Monte Carlo reshuffles to size for the bad-luck tail, such as the worst 5% of drawdowns, rather than the single backtest path, because the future will not repeat the exact order of the past.
Mistakes to avoid
- Peeking at the out-of-sample data while building the rules.
- Expecting the future to follow the one backtest path you saw.
- Re-running the final test after every tweak until it looks good.
Check yourself
What is out-of-sample data?
Data kept hidden while building, for a final test. It is the unseen exam paper.
What does walk-forward testing do?
Tunes on one window, tests on the next, then rolls forward. It repeats build-then-test through time.
What does a Monte Carlo reshuffle show?
The range of drawdowns and streaks the same trades could produce. Same trades, many possible orders.
With a 70/30 split of 10 years, how many years are out-of-sample?
3. 30% of 10 years is 3.
In-sample +0.30R and out-of-sample +0.12R per trade. How much of the edge survived?
40%. 0.12 is 40% of 0.30.
Goal of this lesson: Validate a strategy on unseen data and estimate the range of outcomes it could produce.