Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents
Eric Benhamou, Jean-Jacques Ohana, Beatrice Guez, Ethan Setrouk, Chamyl Saadi, Thomas Jacquot
Correct, carefully engineered anytime-valid promotion gate for adaptive strategy research, but the theory is a direct application of Ville plus a union bound and the empirical evidence is mostly synthetic calibration with no real promotion.
Abstract
Financial research agents can generate and monitor trading rules faster than a human can document the resulting selection process. Candidates appear at data-dependent times, weak rules are abandoned, and promising ones are watched until they cross a reporting threshold. We introduce PROMOTE, a protocol that assigns every candidate predictable testing wealth at birth and accumulates a cost-aware e-process only from subsequent returns. If lifetime allocations sum to at most one, a weighted Ville argument bounds the probability of ever promoting a rule whose declared bounded net score has always had nonpositive conditional mean. The result permits dependent candidates, adaptive births, and indefinite monitoring. We audit the implementation in two settings. A retrospective pseudo-out-of-sample replay contains 24 panel--rule hypotheses, 3,000 shared-sign null paths, injected alternatives, and public ETF and industry data. Repeated unadjusted tests produce a family crossing in 94.5% of null paths, versus 0.0% for PROMOTE; at a 15-bp aligned ETF-component perturbation, PROMOTE has 70.1% target power and median crossing session 2,327. A separate exact mean-zero experiment lets an agent select candidate identities and birth times from past scores; the funded gate has 1.77% false-family crossings across 3,000 programs. The market replay produces zero promotions after costs. The certificate controls historical false promotion under the declared null; it is not a certificate that an earlier edge remains profitable at deployment. AI-use disclosure. Generative AI tools, including OpenAI Codex, assisted manuscript preparation. For this preprint version, Codex assisted with LaTeX editing, author and venue metadata, compilation, and layout checks.
AI Impact Assessments
(1 model)Core Contribution
PROMOTE is a protocol for deciding when an adaptive research process, human or LLM-agent, may "promote" a trading rule. Each candidate receives predictable testing wealth π_h at birth, with lifetime allocations summing to at most 1. Its e-process is a product of (1+λY) factors built only from post-birth, cost-adjusted, clipped returns. The e-process is a fixed mixture over a 5-point bet grid. A candidate is promoted when its e-value reaches 1/(απ_h). Ville's inequality plus a union bound gives familywise control over ever promoting a null rule, even with dependent candidates, data-dependent births, and indefinite monitoring. Two extensions are added. One is a ledger design with hashed rule packages, idempotent updates, and a one-period execution delay. The other is "funded epoch" restarts (Prop. 2) that allow expiring authorizations without breaking validity.
The authors say plainly that the statistical ingredients are established. The math is a weighted online Bonferroni over e-processes, which is close to textbook. The contribution is a careful operationalization for financial research agents, framed around the "ever promoted" event instead of a terminal backtest.
Methodological Rigor
The theory is correct and simple. The proofs are short and valid. The null is a uniform conditional mean ≤ 0 on a clipped score. The paper says clearly that this is a stronger and narrower estimand than "no edge." Careful handling of predictability, cost timing, and the no-rolling-product and no-free-restart rules shows real understanding of where such tests leak.
The empirical audit has two parts:
Both are sound as calibration checks. The market replay promotes nothing, and the null is conservative (0/3000). The simulations mostly confirm what the theorem already guarantees, so they validate the implementation more than the idea. The naive-baseline comparison is a strawman, and the authors admit it is "intentionally common." No comparison is made with e-process alternatives such as alpha-investing, SAFFRON, or other online FWER methods, or with a stronger sequential baseline such as a mixture-martingale confidence sequence. The power result (70% at a 15 bp shift, median crossing around session 2,327 of 2,868) shows the practical price of Bonferroni-style wealth splitting. Real calendar anomalies are weak, so the method would rarely promote anything in practice. The experiments use no real adaptive agent, no LLM-generated candidates, and no profitable known-edge market data.
Potential Impact
The idea is useful as a governance and audit primitive for autonomous quant-research pipelines. Anytime-valid, wealth-budgeted promotion gates with append-only ledgers could be adopted by systematic funds and AI-research-agent platforms. The "evidence ledger versus economic ledger" separation and the funded-restart accounting are practical contributions. Statistically, it will not change how e-process researchers work, because the theory is a straightforward application. In finance it may add to the literature on backtest overfitting (Harvey–Liu–Zhu, Bailey–López de Prado) by moving from terminal-sample corrections to sequential, adaptive ones. The power cost of familywise control and the lack of profitable discoveries limit demonstrated practical value. Adoption would need evidence on real strategy libraries.
Timeliness
The timing is good. LLM-driven strategy-generation agents are an emerging concern, and "agentic p-hacking" is a natural framing for anytime-valid inference. ICAIF is an appropriate venue. The topic overlaps with a growing literature on safe AI-agent governance and auditability.
Strengths
Limitations
Overall
This is a solid, honest applied-statistics and finance-infrastructure paper with correct theory and well-designed audits. Its novelty and demonstrated empirical value are modest. Its likely impact is moderate and concentrated in quant-finance and AI-agent governance niches.
Generated Nov 3, 2026
Comparison History (0)
Correct, carefully engineered anytime-valid promotion gate for adaptive strategy research, but the theory is a direct application of Ville plus a union bound and the empirical evidence is mostly synthetic calibration with no real promotion.