From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis
Liwei Dong, Jiahao Zhao, Nan Xu
A well-executed, honest paper with a useful conceptual framing (abstraction-execution gap, teacher-mediated concretization), but impact is bounded by single-benchmark evidence, modest held-out gains, and a densely crowded competing literature.
Abstract
Large language models increasingly solve scientific-computing tasks, but executable feedback from one problem rarely becomes durable capability on subsequent problems. We study scientific-computing experience consolidation: converting verified runtime experience into transferable procedural knowledge and persistent model improvement. This setting presents two challenges: trajectory-derived artifacts may encode source-specific repairs rather than cross-task computational mechanisms; and a weaker target model may be unable to operationalize an otherwise valid abstract procedure - an abstraction-execution gap. We introduce SciConsolidate, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand the consolidation data without requiring pre-existing reference answers. Because the target model may not directly execute these abstractions, a stronger model concretizes them into executable code supervision for standard, procedure-free SFT; a matched no-procedure teacher branch isolates the value of procedural guidance. On SciCode, runtime procedure injection improves Qwen3.6-27B by +3.85/+6.26 sub-step/main-problem points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap. After procedure-guided concretization, the 9B student improves under procedure-free deployment by +3.89/+6.25 points over the no-procedure SFT control and by +5.62/+11.25 over the original 9B model. These results establish an experience-to-capability pathway for scientific computing and provide a practical starting point for scaling self-improving scientific assistance.
AI Impact Assessments
(1 models)Scientific Impact Assessment
1. Core Contribution
The paper addresses a genuine problem in LLM-based scientific computing: executable feedback (runtime successes/failures) from solving one problem rarely translates into durable improvement on subsequent problems. The authors frame this as "scientific-computing experience consolidation" and propose SciConsolidate, a data-centric pipeline that (a) contrasts verified successes and failures to induce cross-task procedural knowledge, (b) selects procedures via a development-validation utility gate, (c) synthesizes new answer-free scientific-computing queries balanced across failure families, and (d) uses a stronger model (27B) to "concretize" abstract procedures into executable code supervision for procedure-free SFT of a weaker target (9B).
The paper's most conceptually interesting contribution is the empirical characterization of an abstraction–execution gap: runtime procedure injection helps the 27B model (+3.85/+6.26 sub-step/main points) but yields essentially no aggregate main-problem gain for the 9B model. This motivates a teacher-mediated concretization step — the key design insight that a weaker model cannot operationalize abstract procedures but can learn from the concrete executable traces those procedures produce in a stronger model.
2. Methodological Rigor
The experimental design is thoughtful with several commendable controls. The three-way split discipline (development-train / development-valid / frozen held-out) is well-motivated, and the authors are unusually candid that only the held-out split constitutes a clean generalization estimate. The matched no-procedure SFT control (same source pool, generation budget, construction protocol, same 27B teacher) is the critical baseline that isolates the value of procedural guidance beyond generic distillation — this is the right comparison and its inclusion strengthens the causal story considerably.
However, the rigor has clear limits the authors themselves flag. The abstraction–execution gap is inferred from just two differently-pretrained model sizes (9B vs 27B), which conflates capacity with pretraining differences — no controlled scaling or procedure-complexity study isolates the mechanism. The guided (2,095) vs control (1,985) datasets are not size-matched, weakening the causal interpretation. Most importantly, the evaluation rests on a single benchmark (SciCode) with only 80 main problems and 338 sub-steps, and the frozen held-out gains are modest (+1.46/+3.33 sub-step/main over the control). Much of the headline improvement is concentrated on the development distribution. There are no error bars, seeds are fixed at 42 (single run), and no statistical significance testing is reported — concerning given the small problem counts and per-domain buckets of 11–27 steps.
3. Potential Impact
The concretization insight — moving abstract-procedure consumption to a stronger temporary teacher during data construction while keeping the deployed student procedure-free — is a genuinely useful and transferable idea for the fast-growing "self-improving/skill-internalization agent" literature. If the finding replicates broadly, it reframes how the field should think about skill-internalization pipelines: don't train weak models to parse skills; use skills to shape teacher supervision. This could influence how memory/skill systems are built.
That said, the impact is bounded by the narrowness of the evidence and the crowded competitive landscape. The related-work section itself cites dozens of contemporaneous 2026 papers on experience reuse, agent skills, and skill internalization (SKILL0, Skill0.5, Skill1, SkillRL, MemSkill, etc.). This work is one entry in a very dense space, and its distinguishing claim (starting from the target's own verified execution experience rather than pre-existing skills) is incremental relative to that body.
4. Timeliness & Relevance
Highly timely. Self-improving scientific-computing assistants, executable-feedback-driven training, and skill internalization are active frontiers. The problem framing — turning transient runtime feedback into persistent capability — addresses a real bottleneck in long-horizon agentic systems. Code is released (GitHub link provided), which aids adoption.
5. Strengths & Limitations
Strengths:
Limitations:
Additional Observations
The paper is well-written and logically organized, with a coherent pipeline diagram and clear formalization. Reproducibility is aided by released code, full hyperparameters, and prompt templates, though the specific model checkpoints ("Qwen3.5-9B", "Qwen3.6-27B", "GLM-5.1") reference apparently future/renamed models and the arXiv date (2026/2027) suggests this is a very recent preprint whose reproducibility depends on access to those models. The work is technically solid graduate-to-specialist level, moderate resource intensity (8-GPU full-parameter SFT). It does not refute or replicate prior claims in any substantive sense — it introduces a new framing rather than contesting existing results.
Overall, this is a competent, honest, timely contribution with a genuinely useful conceptual insight (the abstraction–execution gap and teacher-mediated concretization), but with impact constrained by narrow single-benchmark evidence, modest generalization gains, and a very crowded contemporaneous literature.
Generated Jul 28, 2026
Comparison History (14)
Paper 2 addresses a timely, broadly applicable problem—converting execution feedback into durable model capability for scientific computing—with clear relevance to the rapidly expanding LLM-for-science field. Its identification of the abstraction-execution gap and the experience-to-capability pathway offers transferable conceptual insight and measurable gains across model scales. While Paper 1 is methodologically rigorous and clinically valuable, its impact is concentrated within the specialized iBMI decoding domain. Paper 2's contributions to self-improving scientific assistants have wider cross-field reach and align with current momentum in AI-for-science research.
Paper 2 likely has higher impact: it tackles a timely, broadly relevant challenge—turning runtime feedback into persistent, transferable capability in LLMs for scientific computing. The proposed SciConsolidate pipeline (success/failure contrast, validation gating, answer-free query synthesis, and teacher concretization) is a novel end-to-end method with clear empirical gains and ablation structure, and highlights a general “abstraction–execution gap” with practical mitigation. Applications span scientific software, autonomous agents, and self-improving assistants. Paper 1 is rigorous and insightful for human/AI optimization behavior, but its scope and downstream applicability are narrower.
Paper 2 offers broader impact across multiple scientific disciplines by addressing LLM self-improvement and the abstraction-execution gap in scientific computing. While Paper 1 presents a valuable, practical approach for embodied AI (prompt optimization for black-box VLAs), Paper 2 tackles a more foundational AI challenge: converting runtime execution experience into durable procedural knowledge. Its methodology for synthesizing executable code supervision allows weaker models to improve persistently without reference answers, advancing the highly relevant and scalable field of AI-driven scientific discovery.
Paper 2 likely has higher impact due to a clearer methodological contribution with theoretical guarantees (hybrid advantage derivation, unified critic proof) and broad applicability to agentic VLM reinforcement learning across many interactive domains. Its framework is general (token- and turn-level credit assignment) and timely as multi-turn VLM agents rapidly expand. Paper 1 is valuable for scientific-code self-improvement, but is more domain-specific, relies on a complex teacher–student pipeline, and its gains appear model-capacity dependent (abstraction–execution gap), which may limit immediate generalization.
Paper 2 addresses a fundamental, broadly applicable challenge: converting execution feedback into durable model capability for scientific computing, with clear methodological innovation (SciConsolidate, abstraction-execution gap) and measurable gains on established benchmarks. Its self-improving paradigm has wide relevance across AI-for-science domains. Paper 1 offers a valuable niche contribution (transaction-cost-aware persona simulation for energy policy) but is more domain-specific, limited to a single survey dataset, and narrower in breadth of impact. Paper 2's insights on transferable procedural knowledge and self-improvement scale more broadly across fields.
Paper 2 addresses a more fundamental and broadly impactful problem: converting execution experience into durable, transferable capability for scientific computing. Its identification and operational demonstration of the abstraction-execution gap is a novel conceptual contribution with clear implications for self-improving AI assistants across scientific domains. The methodology is rigorous with controlled ablations. Paper 1, while methodologically sound with strong efficiency gains, tackles a more incremental problem (step-level supervision in multimodal reasoning) within a crowded research area, offering narrower breadth of impact and less conceptual novelty.
Paper 2 is more likely to have higher impact due to a clearer algorithmic contribution (SciConsolidate) addressing a timely, widely relevant bottleneck: converting verified execution traces into durable, transferable capability in scientific computing. It presents measurable improvements, ablations (teacher branches), and an identified failure mode (abstraction–execution gap), indicating methodological rigor and generalizability to many code/science-assistant settings. Paper 1 is valuable infrastructure for FAIR/semantic access to mathematical models, but is more domain- and system-specific and appears less rigorously evaluated for broad performance gains.
Paper 1 has higher potential impact: it targets a timely, high-leverage problem—turning verified execution feedback into persistent LLM scientific-computing capability—introducing a consolidation pipeline (contrastive success/failure, validation gate, answer-free query synthesis, teacher concretization) that generalizes beyond a single benchmark. It also provides mechanistic insight (abstraction-execution gap) and an actionable training recipe with sizable gains on a smaller model, suggesting broad applicability to self-improving scientific assistants. Paper 2 is a solid, incremental objective improvement for semantic hashing with modest mAP gains and narrower domain scope.
AgentOmnia addresses a broader, more timely problem (full-scenario agentic scaling) with substantial empirical scope: thousands of environments, hundreds of thousands of tools, and dramatic performance gains outperforming much larger models. Its taxonomy, benchmark, and self-evolution pipeline offer wide applicability across consumer, business, and enterprise domains. Paper 1 is methodologically rigorous and identifies an interesting abstraction-execution gap, but its scope is narrower (scientific computing on one benchmark) with more modest gains. AgentOmnia's breadth of impact, industrial relevance, and infrastructure contributions suggest higher potential scientific and practical influence.
Paper 2 addresses a more fundamental and open problem—converting execution feedback into durable, transferable capability for scientific computing—with broader implications for self-improving AI assistants across scientific domains. It identifies and empirically demonstrates a novel conceptual insight (the abstraction-execution gap) and offers a generalizable methodology. Paper 1 presents an elegant, rigorous, and practical token-reduction method, but it is an incremental efficiency improvement within a crowded VLM optimization niche with narrower long-term impact. Paper 2's contribution to scaling scientific reasoning and knowledge consolidation is more timely and impactful across fields.
A well-executed, honest paper with a useful conceptual framing (abstraction-execution gap, teacher-mediated concretization), but impact is bounded by single-benchmark evidence, modest held-out gains, and a densely crowded competing literature.