Kurate.org
Rankings
Back to Rankings

Rethinking Agentic Model Orchestration for Time Series Forecasting

Tong Guan, Tianxiang Zhan, Shirui Pan, Ming Jin

Oct 21, 2026arXiv:2610.29504v1
cs.LGcs.AI
Share
Scorecard· 16/16
5.5/10 impact

Timely, well-framed controlled evaluation that reframes AMO gains, but workshop-scale, with small pools, no statistical testing, and partly adapted baselines.

Abstract

Time series foundation models (TSFMs) perform unevenly across forecasting tasks. Recent work uses agentic model orchestration (AMO) to select or combine TSFMs, often outperforming individual forecasters. Yet these gains may reflect stronger or more complementary candidates rather than better orchestration. To disentangle these effects, we introduce FairAMO, a controlled evaluation protocol with frozen candidate forecasts. Experiments on GIFT-Eval show that AMO improves accuracy, but its gains are bounded by the candidate TSFMs. The best orchestrators reduce MASE by less than 6% relative to the best single TSFM across all tasks. These gains depend on complementarity (different candidates excel on different tasks) rather than the number of candidates. Adding members yields diminishing returns, and a redundant member can erase these gains. The value of historical evidence also saturates quickly. A few recent windows capture all gains, and further history adds none. These findings separate system-level forecasting gains from the contribution of orchestration itself.

AI Impact Assessments

(1 model)

Paper type: Empirical evaluation/protocol paper (workshop-level, NeurIPS FMTS workshop), with a light taxonomy and reproducibility audit.

Core Contribution

The paper argues that leaderboard dominance of agentic model orchestration (AMO) systems on GIFT-Eval (all top 10 spots) conflates three factors: candidate strength, candidate complementarity, and orchestration ability. It proposes FairAMO, a controlled protocol built on a Frozen Forecast Bank. Candidate TSFM forecasts are cached once, with train, validation and test regions. Every orchestrator sees the same candidate forecasts and the same explicit evidence budget, and must commit to a selection or weights before test labels are revealed. The paper also introduces useful diagnostics: headroom (best single model minus window-level oracle), gain, and realization (the fraction of headroom converted).

Main findings on a 5-candidate pool:

  • The best deployable orchestrator (TSOrchestra SLSQP) improves MASE by 5.6% over the best single TSFM (0.658 vs 0.697).
  • Realization of the headroom is only about 30%.
  • Simple rules (Argmin Router, SLSQP weighting) match or beat LLM-mediated and learned routers. MoiraiAgent-Retrained and Small Patch Router realize about 4%.
  • Gains track complementarity, not pool size. The best single model stays flat while the oracle falls as members are added.
  • Redundant members can erase gains (Chronos2+Toto2-313m realizes 3.1%), while a weak but different member can help.
  • Evidence saturates after about 3–10 recent backtests, and more history can hurt.

    • Methodological Rigor

    Strengths:

  • The frozen-bank design is clean. It isolates the orchestrator and prevents test leakage by enforcing commit-before-reveal.
  • It reuses the official GIFT-Eval scorer, so numbers are comparable to the leaderboard.
  • The sweeps over pool size, evidence budget and exhaustive size-2/3 subsets are systematic.
  • Realization normalizes across pools, which is a thoughtful device.
  • The appendix tables give the full numbers.

    • Weaknesses:

  • Only a small pool (5 candidates, 7 including variants) is used. Subset analysis is heavily driven by TiRex2, so "complementarity" is partly a single-model effect, and complementarity is never quantified independently, for example via error correlation.
  • Single runs with no seeds, confidence intervals or significance tests. Several differences are 0.002–0.005 MASE and could be noise.
  • Several external methods are "style adaptations" (TimeRouter, Toto FnF), and the retrained MoiraiAgent is not the original LLM-agent system. Their poor performance may reflect adaptation or training-data limits, not the methods themselves. The claim that LLM-mediated orchestration is unclear is hedged appropriately, but the headline "rethinking" is partly built on weakly reproduced baselines.
  • Commitment is per config (dataset-term), which structurally caps realization against a per-window oracle. The oracle is a loose ceiling, so "two thirds uncovered" overstates what is attainable.
  • The best-single-TSFM baseline is chosen with test labels (acknowledged as optimistic), but this makes the "<6%" gains conservative in one direction.
  • Only MASE is reported in the main text, although CRPS is available. The benchmark is GIFT-Eval only.
  • Leaderboard systems that use different, stronger candidate pools are not recomputed, so the confounding is argued by construction, not shown directly on the leaderboard.

    • Potential Impact

    This is a methodological correction in a fast-moving niche. It gives the community a template for fair AMO comparison and a vocabulary (headroom, realization). The practical guidance is actionable: build pools by error-profile diversity, report pool composition with orchestration gains, use only a few recent windows, and prefer simple selection or weighting. If the frozen forecast bank is released and adopted, it could become a standard testbed. The conclusion that LLM agents add little over simple rules is potentially influential, though under-tested here. The work is narrow to TSFM ensembling and does not extend to other agentic or ensemble domains. The ensemble-learning literature has long known that diversity matters and that simple averaging or selection is hard to beat (the forecast combination puzzle), so the findings are confirmatory more than revelatory.

    Timeliness & Relevance

    Timing is good. AMO systems dominate GIFT-Eval, and agentic framing is popular, so a sober controlled audit is welcome. The result is a reminder that leaderboard gains from pools of strong models need careful attribution.

    Strengths & Limitations

    Strengths:

  • Clear problem framing and a controlled protocol.
  • Consistent narrative across the RQs.
  • Honest limitations, including the unclear contribution of LLMs.
  • A concise taxonomy and reproducibility audit.

    • Limitations:

  • Workshop-length scope.
  • Limited statistical treatment.
  • Reimplementation fidelity.
  • Small candidate pools.
  • Lack of a direct complementarity metric.
  • The evidence-saturation claim rests on a few methods and one pool.
  • No release of the bank is mentioned in the text. It is unclear whether code and data will be made available, which matters for the protocol's adoption.

    • Reproducibility: Metrics and protocols are well specified. Orchestrator training details, hyperparameters, and the candidate-model versions are thin in the main text. Some baselines are adapted, which hurts exact replication.

    Overall

    A useful, timely, but modest evaluation paper. Its value lies in reframing and disentangling confounds, not in a new method. It will likely be cited by subsequent AMO and TSFM-ensembling work as a caution and a protocol reference, but it is unlikely to be a field-changing contribution.

    Rating:5.5/ 10
    Significance 5.5Rigor 5.5Novelty 5Clarity 7

    Generated Oct 22, 2026

    Comparison History (0)

    No comparisons yet.