Making configuration prove itself

A setting that reads like a control but cannot change anything is worse than no setting at all. An audit found four of them, and the cure has been spreading to every number the project publishes.

Configuration files are reassuring. They list the controls by name: a market filter, a risk limit, a ceiling on an indicator. Reading them, you feel you know what the system does. An audit on 1 August showed how far that feeling can drift from the truth.

Four controls that were not there

Each of these had been in place for months, documented and reviewed:

• A market filter with no tests. The regime filter was switched on, but both of the tests it could apply were switched off. With nothing to check, it declared every day a good day: zero bad-market days in 985 sessions, on the strategy carrying the largest share of the profit.
• A brake on a value that was already capped. The "half" setting halved the risk per trade in a falling market. But a stricter limit elsewhere already capped every position below the halved figure, so halving it changed no position size at all. It behaved exactly like the setting that does nothing.
• A limit that was silently clamped. One indicator ceiling was overridden in code for one strategy. Raising it in the configuration had no effect and produced no warning.
• A backtest without the circuit breaker. The backtest never called the live risk circuit, so its equity curves included paths the live system would have stopped.

One more lesson came out of the same review. The market filter's default had been to warn: detect a falling market, print a caution, and trade at full size anyway. A filter that detects and then does nothing is not a filter. The default became skip.

The cure: make the build fail

Fixing the four settings was easy. Stopping the class from coming back needed a check that runs on every change. A configuration audit now reads every strategy profile and reports any setting that is inert or contradicts another. Each finding names the setting, explains why it cannot fire, and says how to fix it. Errors fail the automated build, so a dead control cannot be merged without someone deciding, on the record, to keep it.

The same disease in published numbers

The pattern turned out to be wider than configuration. Anything copied from where it is computed to where it is read can go stale without anyone noticing.

Backtest results. Every published result carried a fingerprint of the strategy's settings, so a changed parameter would be caught. Then two genuine bug fixes landed in the backtest engine: one where a holding that could not be priced on the last day stayed on the books, and one where data-provider holiday placeholders were counted as trading days. The results changed. The settings did not, so the fingerprint still matched, and the published numbers silently stopped being reproducible. Results now carry a second fingerprint, of the engine code itself. If either one differs, the monitor reports a breach and the build fails.

Documentation. The number of automated tests was quoted in six places across the documentation, and it drifted apart twice in three days. A small test now checks that every document states the same number, and that the number is not below a floor counted from the code. It is deliberately not an exact match: a check that failed every time someone added a test would be deleted within a month. It even caught the change that added logout to a dashboard this week, which is exactly its job.

What the checks have in common

• They test whether a statement is still true, not whether the code runs.
• They fail loudly, in the build, where the change is being made.
• They are cheap enough to keep. A check that is too strict or too slow gets switched off, and then it protects nothing.