The returns that were never there

A momentum backtest run on today's index list quietly reads the future. Putting the companies that fell out of the index back in took a fifth off the headline result, and it was the most valuable fix of the year.

The first backtest of the momentum strategy looked excellent, and nothing in it was strictly wrong. It used real exchange prices, filled every order at the next day's open, charged realistic costs, and kept a holdout period it never tuned on. It still overstated the result, because of one quiet assumption: which stocks were eligible on each day.

Today's list leaks the future twice

The easy way to define "the Nifty 500 in 2022" is to take the Nifty 500 as published today and run it backwards. For most strategies that is a small error. For a momentum strategy it is a large one, and it leaks in two separate ways:

• Promotion. Companies join an index because their shares have risen. A momentum screen that ranks by recent gains will find those names early, before they could actually have been bought as index members. The backtest trades the rally that later earned them their place.
• Deletion. Companies that collapsed, were demoted or were delisted are simply absent from today's list. Those are disproportionately the losers, and a backtest that never sees them never holds them.

Both errors push the same way, and neither shows up as a bug. The code is correct; the universe is not.

The first fix was free and only half a fix

Promotion is easy to block: freeze the universe at the start of the test and ignore any company that was not already large and liquid on that date. That alone cut the backtested return sharply, and every headline figure since has used it.

Deletion cannot be fixed that way. To keep the departed companies in the test, you need to know when each one was in the index, and that history is not published in a form a program can read. The index provider's site builds its announcements in the browser with no public interface, and the exchange's circulars feed carries only a rolling few days. The one thing that can be downloaded is today's constituent list, which is the very file that causes the problem.

The answer was in an archive

That constituent file has lived at the same address for years, and the Internet Archive kept dated copies of it. Each capture is a complete roster and the date it was true. Nothing has to be reconstructed or guessed: every row came from a file the exchange itself served on the day it was saved.

Stitching the captures together produced 724 membership spans covering 699 companies, 199 of which left the index during the test window (15 of them later rejoined). Before that history is allowed anywhere near a backtest, an installer checks it for the four ways a real file goes wrong:

1. overlapping membership spans for the same company;
2. a Nifty 50 file in the right format, which would pass every other check;
3. a history that starts after the test window opens, so it cannot correct the test it is meant for;
4. companies with no price data behind them.

What it cost

With the departed companies back in the test, about a fifth of the headline annual return disappeared, and the worst drawdown got deeper by more than a third. The win rate and the profit factor both fell as well. No setting changed. The strategy was simply measured against the stocks it could really have traded.

Even this is not exact, and the notes that go with it say so:

• The archive captures are irregular, with gaps from 19 days to more than a year. Membership is exact at each capture and assumed in between, so a company that left and rejoined between two captures is never seen leaving.
• 31 of the 199 departed companies have no price history at all, mostly mergers and renames, so their final months are still missing.

The right description is materially better controlled, not correct. To stop the gap growing, the system now saves the roster itself every week, at a resolution the archive never had.