I Retired a Strategy That Passed Its Backtest: How Multiple Comparisons Manufacture Fake Winners

After years of mediocre results from a crypto trading bot's signal strategies, I ran a redesign project: 3 strategy families × 4 symbols = a 12-cell backtest grid with a strict pass/fail gate. Two cells passed. And instead of deploying the winners, I retired signal trading entirely. This is a write-up of how a "passing backtest" can be a statistical illusion — and what a failed project can still leave behind.

1. The Smell: Performance Indistinguishable from Chance

The bot ran faithfully — entering on technical signals, honoring stop-losses and take-profits. The problem: its results were indistinguishable from a bot with no rules at all. Net of fees, it was a coin flip. So I took the honest route: redesign the strategies from scratch, build a backtest harness, and keep only what passes.

2. The Design: Twelve Cells, One Gate

The redesign was a grid — three strategy families of different character (trend-following, mean-reversion, volatility-breakout) applied to four major symbols, twelve combinations in all. Every cell faced the same gate: minimum trade count, win-rate floor, profit-factor floor, max-drawdown ceiling. Parameters were fit on the in-sample window (the earlier part of history), then gated.

Result: 2 of 12 cells passed. A specific strategy on specific symbols cleared the win-rate and profit-factor bars. Had I stopped there, the conclusion would have been the pleasant kind: "redesign successful, two combinations go live."

3. The Twist: Total Collapse Out-of-Sample

I re-ran the two winners on the held-out validation window (out-of-sample). The result was brutally clear: both collapsed. Win rates fell below the bar; profit factors dropped under 1. The in-sample pass did not reproduce in the future it had never seen.

There's a way to know this wasn't bad luck — run the arithmetic backwards. If a single cell can clear the gate by pure chance with, conservatively, 15–20% probability, then across 12 attempts the odds of producing about two "lucky winners" exceed a coin flip. The in-sample result was never evidence that the strategies were good; it was a re-derivation of the fact that if you roll twelve dice, some roll high. Statistics calls this the multiple comparisons problem.

4. The Decision: Retirement Is Also a Result

Three options: ① lower the bar until something passes, ② add more combinations and roll again, ③ shut down signal trading. Option ① is self-deception. Option ② makes multiple comparisons strictly worse — more attempts breed more lucky winners. So ③: signal-based entries went behind a permanent kill switch.

What survived the retirement:

• The backtest harness. The plugin strategy structure, gate constants, and in-sample/out-of-sample split are reusable assets. The next idea gets onto the test bench in hours, not days.
• A switch from prediction to discipline. The system moved to rule-based laddered buying — price-level tranches, fixed stop-losses, hard budget caps. Once you admit you have no predictive edge, discipline is the only edge left.
• A reinterpretation of past performance. A separate backtest of historical picks showed no selection alpha at all — the real edge had been coming from exit and rotation discipline the whole time. Consistent evidence for the retirement.

5. Lessons

• A backtest without out-of-sample data is a wish, not a validation. In-sample passes always appear. Only held-out data imitates the future.
• Discount by the number of attempts. If 2 of 12 pass, the first question is not "are these good?" but "what are the odds of 2 passing by chance?" The more combinations you try, the stricter the bar must get.
• Retirement is a conclusion, not a failure. Proving "no edge" and stopping outperforms believing in an edge and burning fees.
• Projects die; harnesses remain. Even when the verdict is retirement, the validation infrastructure and codified discipline raise the starting line of the next attempt.

FAQ

Q. Wouldn't longer data change the outcome?

More data stabilizes estimates, but the moment you roll twelve cells again, the multiple-comparisons problem returns intact. It's not a data-length problem; it's a validation-design problem relative to the number of attempts. Without out-of-sample separation and attempt-count correction, longer data reproduces the same illusion.

Q. What if the parameters had been tuned more carefully?

Finer tuning reliably improves in-sample results — and overfits by exactly that much, making the out-of-sample collapse more dramatic. Degrees of tuning freedom are multiple comparisons by another name.

Q. So is there nothing individuals can do with trading automation?

Separate prediction from execution. Proving an edge in prediction is extraordinarily hard. But execution — buying in tranches by rule without emotion, honoring stops, enforcing budget caps — is where automation beats humans decisively. What got retired was the prediction, not the automation.