Overfitting in Manual Backtesting

The Manual Forms of Overfitting

No code required:

• Variation shopping. Test six versions of a strategy on the same three months, keep the winner. The winner won partly by fitting that window’s randomness — and won’t repeat.
• Peek-and-tweak. Checking stats every 20 trades and nudging the rules toward what’s working so far. This is curve-fitting in slow motion, and it’s seductive because each tweak “improves” results.
• Over-precise rules. “Enter when RSI is between 31 and 34” — suspiciously exact thresholds are usually fitted to specific past trades, not robust behavior.
• Condition stacking. Adding filters until the losers disappear. Five conditions that perfectly separate winners from losers in-sample almost always describe noise.

How to Recognize It

The signature of an overfit strategy is fragility: the edge collapses when you change anything small. Practical probes —

• Nudge each parameter ±20%. A robust edge degrades gracefully; an overfit one falls off a cliff.
• Count conditions. More than ~3–4 stacked filters on a discretionary strategy is a yellow flag.
• Check trade count per condition. If a filter only acted on 4 trades, it’s fitted to 4 trades.
• Look at the equity curve: overfit strategies often show a beautiful in-sample curve that falls apart out-of-sample.

The Defenses

1. Out-of-sample reservation. Split your history: develop on 70–80%, freeze the rules, then run once on the untouched 20–30%. Matching expectancy → real edge. Collapse → you fitted noise, and the reserved slice saved you the live version of that lesson.

2. Robustness over precision. Prefer rules that work across ranges, not points: “RSI below 35” beats “RSI 31–34”; structure zones beat exact prints (broker feeds differ anyway). An edge that needs precision isn’t an edge, it’s a fit.

3. Analyze once. Set the sample-size target, reach it, then look. The discipline of not peeking is the single biggest manual-overfitting defense — every mid-stream glance is a temptation to tweak.

4. Light walk-forward. For parameter-sensitive strategies, develop on block 1 → validate on block 2 → roll forward. If re-tuning each block doesn’t transfer to the next, the strategy is fitting noise on a schedule.

5. Out-of-instrument sanity check. A robust concept often shows some edge on a related instrument. Total failure everywhere but one symbol suggests symbol-specific fitting.

The Underlying Tension

Every added rule improves the backtest and risks fitting noise — improvement and overfitting look identical in-sample. That’s why in-sample results can’t distinguish them and out-of-sample testing is non-negotiable. The mindset that protects you is counterintuitive: be suspicious of a strategy that looks too good, because in backtesting, “too good” is usually “too fitted.”

Frequently Asked Questions

How many rules is too many?

There's no hard limit, but each added condition should earn its place by acting on many trades and surviving out-of-sample. For discretionary strategies, more than three or four stacked filters is a yellow flag — and any filter that only affected a handful of trades is almost certainly fitted to those specific trades.

Is optimizing parameters always overfitting?

No — choosing sensible parameters is normal; fitting them to a single window's noise is overfitting. The tell is robustness: if the strategy works across a range of parameter values and on out-of-sample data, you optimized legitimately. If it only works at exact values on the development window, you curve-fitted.

How big should my out-of-sample slice be?

Typically 20-30% of your history, and crucially the most recent portion, since recency carries the most predictive weight. It must contain enough trades to be meaningful (aim for 30+) and stay genuinely untouched until the rules are frozen — peeking at it during development turns it into more in-sample data.

Can manual backtesting overfit as badly as automated?

It can overfit, though usually less severely — humans test fewer variations than a parameter sweep that tries thousands of combinations. The manual danger is subtler: peek-and-tweak and unconscious rule drift accumulate fit gradually across a test, which is exactly why 'analyze once' and out-of-sample reservation matter.