Harness In-Context Operator Learning with Chain of Operators
Minghui Yang, Ling Guo, Liu Yang
Abstract
Neural operators approximate mappings between function spaces, but often generalize poorly to other operators and usually require fine-tuning or retraining. In-Context Operator Networks (ICON) addresses this issue by prompting the model with numerical context so that the model learns specific operators from prompts and adapt to different operators without fine-tuning. However, ICON may still fail to generalize to out-of-distribution (OOD) operator tasks. Inpired by the success of harness engineering of Large Language models (LLMs), we introduce Chain of Operators (CHOP), a framework that harness a frozen ICON to OOD operator tasks without updating its parameters. Specifically, CHOP constructs a chain of operators consisting of explicit elementary transformations and the frozen ICON. Experiments on a scalar conservation law and a mean-field control problem show that CHOP reduces relative inference error over direct ICON evaluation, while each operator in the chain remains interpretable and in closed form. A chain constructed on one PDE family further generalizes to a different family, indicating shared mechanisms across harness systems.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "Harness In-Context Operator Learning with Chain of Operators"
1. Core Contribution
The paper introduces Chain of Operators (CHOP), a framework that wraps a frozen In-Context Operator Network (ICON) with explicit, closed-form prompt-side operators (F) and prediction-side operators (G), forming a composition F → ICON → G. The key idea is that instead of retraining or fine-tuning a neural operator for out-of-distribution (OOD) tasks, one can reformulate the prompt and post-process the prediction through interpretable mathematical transformations so that the induced operator falls within the model's in-distribution regime.
The analogy to Chain-of-Thought prompting in LLMs is conceptually appealing: decompose a hard problem into simpler, manageable sub-tasks. An agentic evolutionary search (EvE) discovers these operator chains automatically. The framework is tested on scalar conservation laws and mean-field control (MFC) problems, demonstrating 19–86% relative error reductions on OOD tasks, and cross-PDE transfer of discovered chains.
2. Methodological Rigor
Strengths in experimental design:
Weaknesses:
3. Potential Impact
Positive aspects:
Limitations on impact:
4. Timeliness & Relevance
The paper addresses a genuine bottleneck in scientific machine learning: the brittleness of neural operators under distribution shift. In-context learning for PDEs is an active and timely research direction, and adapting frozen models without retraining is practically important. The conceptual connection to prompt engineering in LLMs is timely, though the analogy is somewhat superficial—the "chain" here is a composition of mathematical operators rather than a reasoning decomposition.
The work is relevant to the growing interest in foundation models for scientific computing, where adaptation without retraining is a key desideratum.
5. Strengths & Limitations
Key strengths:
Notable weaknesses:
Overall Assessment
The paper presents a well-motivated and cleanly formulated framework for adapting frozen neural operators to OOD tasks through interpretable operator chains. The core idea is sound and the experimental results demonstrate consistent improvements. However, the experimental scope is limited, comparisons with alternative adaptation methods are absent, and the discovered chains are relatively simple. The work represents an incremental but meaningful contribution to the neural operator adaptation literature, with potential for broader impact if scaled to more complex settings.
Generated Jun 11, 2026
Comparison History (18)
Paper 2 introduces a more novel conceptual framework (Chain of Operators) that draws an innovative analogy between prompt engineering in LLMs and operator learning, enabling OOD generalization without retraining. This cross-pollination of ideas between foundation model prompting strategies and scientific computing/PDEs is highly innovative and has broader potential impact across computational science. Paper 1, while rigorous and practically useful, represents a relatively incremental adaptation (adding a survival head to existing tabular foundation models), combining known components rather than introducing a fundamentally new paradigm.
Paper 2 likely has higher impact due to strong real-world relevance (reducing land-surface uncertainty in climate projections), clear pathway to operational deployment (open-source emulators, digital twin coupling, HPC scalability), and broad cross-field reach (climate science, remote sensing, Earth system modeling, AI/HPC). Its timeliness aligns with major initiatives (Destination Earth) and could influence modeling workflows widely. Paper 1 is methodologically novel for operator learning and interpretability, but its demonstrated scope is narrower (few PDE tasks) and near-term applications are less immediate, limiting expected impact breadth.
Paper 2 has higher estimated impact: it tackles adversarial robustness—a broad, timely, high-stakes problem in ML security—with systematic multi-dataset, multi-architecture evaluation and detailed mechanism analysis, plus a practical hybrid (RL-adv) that improves robustness against diverse attack classes. Its implications span security, optimization, and training methodology. Paper 1 is novel and elegant for operator learning and interpretability, but its demonstrated scope (limited PDE families) suggests narrower near-term real-world uptake and field breadth compared to adversarial defense advances.
Paper 2 bridges the highly successful concepts of in-context learning and chain-of-thought from LLMs to neural operators for PDEs. This zero-shot generalization approach to out-of-distribution scientific machine learning tasks offers broader cross-disciplinary impact and higher novelty compared to Paper 1's algorithmic improvement of graph priors in neural relational inference.
Paper 2 introduces a highly novel approach by adapting LLM prompt engineering techniques (Chain of Operators) to neural operators. This enables zero-shot generalization to out-of-distribution tasks without retraining, addressing a major bottleneck in scientific ML. While Paper 1 provides a valuable active learning method for data-efficient SINDy, Paper 2's connection between foundation model methodologies and operator learning offers broader transformative potential across computational physics and AI.
Paper 1 identifies a fundamental limitation of LLMs in in-context learning ('categorical prior lock-in'), which has broad implications across numerous domains relying on LLMs for structured data generation. While Paper 2 offers an innovative prompting technique for neural operators, its impact is largely confined to the specialized field of solving partial differential equations (PDEs). Thus, Paper 1 has a higher potential for widespread scientific impact due to the ubiquity of LLMs and tabular data.
Paper 1 introduces Chain of Operators (CHOP), a novel framework that extends in-context operator learning to out-of-distribution tasks without retraining, drawing creative parallels between prompt engineering in LLMs and operator composition in scientific computing. This bridges two active research areas (neural operators and in-context learning) with broad applicability across PDEs and scientific domains. Paper 2 addresses an important but narrower engineering problem of monitoring deployed safety classifiers, with solid empirical work but limited novelty beyond combining existing techniques (conformal prediction, sequential statistics, importance weighting). CHOP's transferability across PDE families suggests deeper theoretical implications.
APPO addresses a critical bottleneck in developing autonomous LLM agents by improving credit assignment and branching during multi-step tool use. Given the explosive growth and broad applicability of general-purpose AI agents across numerous domains, this methodological advancement offers significantly wider real-world impact and timeliness compared to Paper 2, which focuses on the narrower, albeit important, subfield of neural operators and PDE solving.
Paper 2 introduces Chain of Operators (CHOP), a novel framework that extends in-context operator learning to out-of-distribution tasks without parameter updates, drawing an innovative parallel to prompt/harness engineering in LLMs. This bridges neural operators and foundation model reasoning paradigms, with broad implications for scientific computing and PDE solving. The cross-family generalization result suggests fundamental insights. Paper 1, while solid, is a more incremental combination of existing techniques (reservoir computing, granular-ball computing, graph optimization) for time-series clustering, with narrower conceptual novelty and impact scope.
Paper 1 is more novel and potentially higher-impact: it extends in-context operator learning with a harness-style “Chain of Operators” that composes interpretable, closed-form transformations with a frozen neural operator to improve OOD generalization without fine-tuning. This targets a key bottleneck in scientific ML for PDEs and control, with broad downstream applications (physics simulation, engineering design, inverse problems) and cross-field relevance (foundation models, operator learning, interpretable systems). Paper 2 is rigorous and useful, but mainly refines practical design choices for noisy SGD, likely yielding incremental impact.