Methods · Nº 003 · 16 October 2026 · 9 min read
The best of 348 settings showed 17% a year in backtests, on prices we made up
We generated 1,000 random markets in which no strategy can have an edge, and searched each one for its best moving-average setting. The winners showed a median of 17.2% a year in their backtests, and 43% of them had a backtested Sharpe ratio (risk-adjusted return) above 1, a level few funds sustain for long. On the next two years, the typical winner lost 2.4% a year, the same as a setting picked at random. This is how a perfect-looking backtest is born, and how to catch one.
Every trader who has used a strategy tester knows the feeling. You start with a sensible idea, say a moving-average crossover. The first settings are disappointing. So you adjust: a faster average, a slower one, allow short trades, run the optimiser. Somewhere around the fortieth attempt, the equity curve suddenly climbs from the bottom left to the top right. It feels like discovery. It is usually something else.
To show exactly what, we ran the search on markets where we know there is nothing to discover.
"I tested hundreds of settings and kept the one that worked best. The backtest proves it."
The experiment: searching for gold in pure noise
We generated 1,000 artificial markets. Each one is a random walk: every day the price moves up or down by a random amount, with about 1.2% daily volatility (similar to a broad stock index), and on average the price change is zero. In these markets the past contains no information about the future. No indicator, pattern or setting can have a genuine edge, because there is nothing for it to find.
On each market we did what a diligent trader with a strategy tester would do:
- Defined a strategy family: a moving-average crossover. (A moving average is the average closing price of the last N days; a "fast" one uses few days, a "slow" one many.) Go long when the fast average is above the slow one; otherwise hold no position ("flat") or, in a second version, go short (in India, that means futures).
- Searched the settings: fast average from 5 to 50 days, slow average from 20 to 200 days, long-only or long/short. That is 348 combinations, a modest search by the standards of today's optimisers.
- Backtested every combination on two years of data, charging 0.1% each time the position changes (0.2% for flipping from long to short), and kept the best one.
- Traded the winner on the following two years, data it had never seen. This is the out-of-sample test: the closest a backtest gets to the future.
"Best" means the highest Sharpe ratio: return divided by volatility, scaled to a year. It rewards smooth gains, not just big ones. Above 1 is generally considered very good; many professional funds never sustain it.
The more you search, the better the backtest looks
Here is what the chosen strategy looked like, depending on how many settings were tried before picking the best. Each row summarises 1,000 markets.
| Settings tried | Backtest: return a year | Backtest: Sharpe | Next 2 years: return a year | Next 2 years: Sharpe |
|---|---|---|---|---|
| 1 (no search) | −2.6% | −0.18 | −2.4% | −0.18 |
| 10 | +8.0% | 0.46 | −2.6% | −0.17 |
| 50 | +12.7% | 0.71 | −2.6% | −0.15 |
| 100 | +14.4% | 0.79 | −2.3% | −0.15 |
| All 348 | +17.2% | 0.91 | −2.4% | −0.14 |
With no search at all, a random setting on a random market loses a little each year: the costs, plus "volatility drag" (a 10% fall followed by a 10% rise leaves you 1% down, so volatility alone drags down compounded returns even when the average change is zero). As the search widens, the winner's backtest improves steadily, from −2.6% a year to +17.2%. Its performance on new data barely moves: between −2.3% and −2.6% a year, whether we tried one setting or 348. The search is not finding a better strategy. It is finding a luckier fit to the past.
After the full search, 43% of the winning strategies had a backtested Sharpe ratio above 1, and 58% showed more than 15% a year, on markets that are pure noise. Any of them, presented as a clean equity curve, would look like a serious trading system.
The 1,000 winning strategies, in their backtest
Backtested return a year of the setting with the best backtested Sharpe, one per market. Each bar is a 5-point band starting at its label; the end bars hold everything below −25% and at or above +40%. 98.4% of the winners are profitable in their backtest.
The same 1,000 strategies, on the next two years
Out of sample, the "winners" behave like untuned settings: centred below zero, with 60% losing money. Nothing they learned from the past carried forward, because there was nothing to learn. Note, though, that 40% were still profitable and 151 of 1,000 made 10% a year or more, purely by luck.
A backtest chosen as the best of many is biased upwards, and the bias grows with every setting you try. On random data, searching 348 settings manufactured a typical backtest of +17.2% a year. Out of sample, it was worth exactly as much as no search at all.
Why a search manufactures winners
Each setting you test is like a lottery ticket drawn on the same history. (Neighbouring settings are closely related tickets, so 348 settings are fewer than 348 truly independent tries, which makes the result more striking, not less.) Some tickets always do better than others, purely by chance. When you keep the best ticket and throw the rest away, you have not found a better ticket; you have found the one that happened to match this particular past. The future is a new draw, and the luck does not carry over.
The danger is that nothing in the winning backtest warns you. The trades follow the rules. The costs are included. The equity curve is honestly computed. The only misleading thing is invisible: the 347 other curves that were thrown away. It is the same survivorship problem we saw with traders who got lucky, applied to strategies instead of people.
More history helps, but it doesn't cure
Two years is a short test, though a common one. So we reran the whole experiment with six years of data to search and four years to test.
| History used for the search | Backtest: return a year | Backtest: Sharpe above 1 | Out of sample: return a year |
|---|---|---|---|
| 2 years (then 2 years out of sample) | +17.2% | 43% of winners | −2.4% |
| 6 years (then 4 years out of sample) | +8.9% | 6% of winners | −2.6% |
A longer history roughly halved the backtested return and made a Sharpe above 1 seven times rarer, because it is harder for luck to stay consistent over six years than over two. But the winners still showed a comfortable +8.9% a year in their backtests, and still lost money afterwards. Longer data makes overfitting less extreme. It does not make it go away.
"But real markets aren't random"
True. Real markets contain some structure, and some strategies do have genuine edges. That is exactly why this experiment matters. A random walk is a control: a market where the right answer is known to be "no edge." If a research process reports a strong edge on a random walk, it will report strong edges on real data too, whether they exist or not. You can only trust a method that would have said "nothing here" in our 1,000 fake markets.
How to tell a real edge from a lucky fit
None of these checks can prove a strategy will work. Each of them makes it much harder for luck to pass as skill, and they are worth demanding from anyone who shows you a backtest, including yourself.
- Keep data you never touch. Before you start, set aside the most recent part of your history. Search and tune on the rest. Look at the held-back period once, at the end. If you peek and re-tune, it stops being out of sample. And two years of unseen data is a small sample: in our noise markets, 40% of the curve-fitted winners were still profitable out of sample, and 15% made 10% a year or more. Passing once is not proof.
- Count your attempts. Write down how many settings, filters and variations you tried, including the ones you abandoned. The more you tried, the higher the bar a result must clear. On two years of noise, the median best Sharpe was 0.46 after 10 tries, 0.71 after 50 and 0.91 after 348. A result that does not clearly beat the figure for your number of tries is not evidence.
- Walk it forward. Tune on one window, trade the next unseen window, roll forward and repeat, then judge the strategy only on the stitched-together unseen periods. We did not test this here, but it is the closest a backtest gets to how a strategy is actually run: always on data the settings have not seen.
- Prefer plateaus to peaks, but don't stop there. If 50/80 is brilliant but 45/80 and 55/80 are poor, you have probably found a spike of luck. The reverse is not reassuring on its own, though. Neighbouring settings trade almost the same signals, so luck spreads to them too: in our noise markets, for 30% of the winners the neighbouring settings averaged at least 70% of the winner's Sharpe, and those winners still lost 2.2% a year out of sample. A plateau is a weak filter, not a pass.
- Have a reason. Know why the edge should exist: who is on the other side of your trades and why they keep paying you. A rule found only by searching, with no explanation, deserves the most suspicion.
- Paper trade before you fund it. Run the finished system on live data without real money and compare it with the backtest. It is the cheapest truly out-of-sample test there is.
A backtest is the best tool a trader has. It is the only way to see a strategy's full range of outcomes before risking money, and we are firm believers in it. But it answers the question you ask it. Ask "which of these 348 settings fitted the past best?" and it will answer honestly every time, even when the true answer to the question you care about, "will this make money?", is no.
Methodology
- Markets simulated
- 1,000 random walks
- Daily volatility
- 1.2%, zero average price change
- Strategy family
- MA crossover, fast 5–50 (step 5), slow 20–200 (step 10)
- Modes
- long/flat and long/short
- Combinations
- 348
- Costs
- 0.1% per unit of position change
- Selection
- highest in-sample Sharpe
- Split
- 2y in / 2y out; repeated with 6y / 4y
- RNG
- mulberry32, seed 20261016
Moving averages are computed on log prices, through the close of each day, and the resulting position earns the next day's return, so there is no look-ahead (the signal and the fill share the same close, which is slightly optimistic). Short positions earn the inverse of the simple daily return, reset each day. The first 200 days of each market are warm-up only; each test starts flat and pays the cost of its first entry. "Average price change is zero" refers to simple returns; log prices drift down slightly. "Settings tried" rows use a random order of the 348 combinations, the order a researcher might happen to try them in. Returns are annualised. Table figures are medians across markets, percentages are shares of the 1,000 markets, and histograms are counts. The plateau check compares the average backtested Sharpe of the winner's neighbours (fast ±5, slow ±10, same mode) with the winner's. Every number comes from sim-overfit.js (plain Node.js: node sim-overfit.js 2 2 and node sim-overfit.js 6 4).