Tips

How to tell if a backtest is overfit

Ask how many variants were tried before this one was picked. With five years of data and a target Sharpe ratio of 1, no more than 45. Five checks you can run yourself, a worked example from the deflated Sharpe ratio paper, and the faults no statistic can see.

Published
Author
Ben
Filed under
Tips

A backtest is overfit when its result comes from the search for a good result. The market had little to do with it. The quickest test is one question: how many variants were tried before this one was picked? If nobody knows, the result can’t be judged. And if the answer is “a lot”, the Sharpe ratio has to clear a much higher bar than the one it was judged against.

That isn’t our opinion. It comes from two papers by David Bailey, Marcos López de Prado and colleagues. The rest of this post is how to use them.

The number nobody reports

Try enough variants of a strategy on the same data and one of them will look good by chance. Add a filter, move a threshold, try another market, and keep the best. Each variant is a trial, and the best of many trials is partly luck.

In 2014 Bailey, Borwein, López de Prado and Zhu worked out how much data a given number of trials needs. They published it in the Notices of the American Mathematical Society. Their example, for an annual Sharpe ratio of 1: with five years of data, “no more than forty-five independent model configurations should be tried”. Past that, you are almost certain to find a variant with that Sharpe ratio in the backtest. Its expected Sharpe ratio after the backtest is zero. With two years of data, seven variants are enough to get there.

Their conclusion is blunt. A backtest that doesn’t report the number of trials makes it “impossible to assess the risk of overfitting”.

A worked example

The second paper, in The Journal of Portfolio Management, defines the deflated Sharpe ratio, and it has an example. A strategist tests 100 variants of a trading idea on five years of daily data. The best one has an annual Sharpe ratio of 2.5, which sounds excellent. The returns lean to the left, with a skewness of -3, and have fat tails, with a kurtosis of 10.

Once the 100 trials and the shape of the returns are counted, the chance that the true Sharpe ratio is above zero is 90%. The paper’s investor turns it down, because 90% is below the 95% bar. Had the strategist stopped at 46 variants, the same result would have passed.

You can run that example, or your own backtest, in our deflated Sharpe ratio calculator. It is checked against the papers’ figures.

Five checks you can run yourself

1. Count every variant. Include the ones you threw away. Other parameters, other markets, other periods, other filters. If you used an optimiser, every point it evaluated is a trial. Variants that move together count as fewer independent trials, but the count is where the honesty starts.

2. Deflate the Sharpe ratio. Put the count, the spread of the variants’ results and the shape of the returns into the deflated Sharpe ratio. If it lands below 95%, the result is within reach of luck.

3. Keep data you never looked at. Set aside a period before any testing starts, and run the final variant on it once. The second time you look at it and adjust, it has become part of the search.

4. Move the parameters a little. Say a 20-day average works, and the 18 and 22-day versions lose money. Then the strategy found a spike in the past. Before we trust a setting, we want its neighbours to work too.

5. Check what the statistics can’t see. The deflated Sharpe ratio corrects for the search. It assumes the backtest is otherwise honest. Three faults break that assumption:

  • Look-ahead. The strategy uses a number before it existed, such as a closing price traded at the open of the same day.
  • Missing losers. The test only holds assets that still exist today. The ones that failed are gone from the data, and the past looks kinder than it was.
  • Free fills. Orders fill at prices nobody offered, with no fee, no spread and no slippage.

A backtest with one of these can pass every statistical test and still lose money from the first trade.

What an overfit backtest does next

It goes live and stops working. The first paper puts a number on it. When none of the variants had any skill, the pick’s expected Sharpe ratio after the backtest is zero. Live trading is the out-of-sample test, paid for with real money.

That is why we run every bot in stages. Paper trading first, then a small live size, each compared with what the backtest said would happen. A strategy that fails there costs little.

About to put money behind a backtest? Or running a bot whose live results drifted away from the test? We can check either. That is our backtest audit.

Start a conversation

Have a strategy you want to automate?

Book a strategy call