Auditing my own backtests

In July and August 2026 I withdrew four backtest results that I had earlier accepted. Each had a specific flaw:

• a market-regime filter could see the closing price of the day it was trading in;
• a coin-selection rule used data from the bar it was about to trade;
• a combined strategy made nearly all of its profit in three bars;
• an arbitrage model understated trading costs by about 11 times.

All four made a result look better than it was. They were found by going back over the work, not by any automated test.

Those findings, along with a few standard tests, became automated checks that my crypto strategies now run through. They include tests of:

• look-ahead;
• how concentrated the returns are;
• leverage;
• a Sharpe ratio deflated for the number of trials;
• costs;
• stability under small changes to parameters;
• capacity;
• bootstrap ruin.

The checks are themselves tested on deliberately broken strategies, for example one with a planted one-bar look-ahead, to confirm that they reject them.

In September a later review found two more problems, this time in the backtest engine. It had been silently dropping 35% of funding settlements, because the exchange’s timestamps carried millisecond offsets that did not line up with the hour. And it valued each portfolio as if it were rebalanced to its target weights every hour at no cost, when a real account holds fixed quantities between rebalances. After both fixes, the two paper-traded versions described in Near-high signals were re-run, and neither one’s verdict changed. A check on funding coverage was added.