KISS - Knowledge Infrastructure for Scientific Simulation: A Scaffolding for Agentic Earth Science
Ziwei Li, Liujun Zhu, Yuchen Liu, Yichen Zhao, Birk Li, Ruiqi Wu, Junliang Jin, Jianyun Zhang
Abstract
Process-based simulation models encode decades of scientific understanding across the Earth sciences, yet the communities most exposed to climate risk and resource scarcity are the least able to use them. Here, we introduce knowledge infrastructure (KI), an agent-actionable scaffold that externalizes expertise into validated modelling operators, staged domain protocols, and diagnostic recovery mechanisms. Across a 3,000-trial coupled-hydrology benchmark, agents equipped with KI produced physically plausible, verifiable end-to-end simulations in up to 84% of trials, while agents without KI plateaued below 40%. KI generalizes across disciplines. We packaged its construction into a Knowledge Dissection Toolkit (KDT) that autonomously produced KI enabling end-to-end agent execution of 117 additional process-based models across 14 Earth-science domains. Across all 119 KIs, modelling decisions and failure remedies converged despite different underlying physics, showing that operational expertise is structured and extractable rather than ad hoc. Demonstrations show KI-equipped agents lowering both the access barrier between non-specialist users and process-based simulation, and the integration barrier between modelling communities. Through this scaffold, process-based science can then evolve as a living scientific commons, answerable to whoever needs to know and extendable by whoever can contribute.
AI Impact Assessments
(1 model)Scientific Impact Assessment: KISS – Knowledge Infrastructure for Scientific Simulation
1. Core Contribution
The paper introduces Knowledge Infrastructure (KI), a structured scaffold that externalizes tacit operational expertise required to run process-based simulation models into three machine-readable layers: validated modelling operators (procedural knowledge), staged domain protocols (evaluative knowledge), and diagnostic recovery mechanisms (diagnostic knowledge). The key insight is that the barrier to using scientific models is not informational but operational — knowing which sequence of decisions, checks, and debugging steps turns a model binary into a valid simulation.
The accompanying Knowledge Dissection Toolkit (KDT) automates the construction of KI packages from model source code, documentation, and examples. The paper demonstrates this at two scales: a deep 3,000-trial benchmark on coupled VIC-Lohmann hydrology, and a breadth test across 119 process-based models spanning 14 Earth-science domains. The central empirical finding — that operational expertise converges structurally across models with different underlying physics (unit-conversion and I/O format errors account for 55% of failures universally; three decision categories appear in every domain) — is arguably the paper's most important scientific contribution.
2. Methodological Rigor
The experimental design is commendably thorough for an AI-for-science paper:
Weaknesses in rigor: The success criterion (NSE ≥ 0.2) is relatively lenient — this threshold indicates the model explains only 20% of discharge variance. The paper acknowledges runs are uncalibrated, but this makes it harder to assess whether agents are producing *scientifically useful* rather than merely *physically plausible* outputs. The 33 models validated only for "runnability" (not against observations) represent a weaker evidence tier that could inflate the breadth claims. The demonstrations in Fig. 6 are explicitly proof-of-concept and lack systematic evaluation — the Vietnamese farmer and MRV officer scenarios are compelling narratively but unvalidated scientifically.
3. Potential Impact
Immediate impact: If KI packages become widely adopted, this could substantially democratize access to process-based models. The paper correctly identifies that communities most vulnerable to climate risk are least able to run these models. The practical demonstrations (carbon credit verification, multi-model ensembles assembled in single sessions vs. years of coordination) point toward genuine workflow acceleration.
Broader scientific impact: The finding that operational expertise is structurally convergent across 14 domains is a meta-scientific insight. It suggests that the reproducibility crisis in computational science may be addressable through systematic knowledge externalization rather than ad hoc documentation efforts. This has implications well beyond Earth science.
Infrastructure impact: HydroCraft (the execution platform) and KDT together constitute a significant community resource. The 119-model library with standardized interfaces could lower barriers to multi-model intercomparison, potentially accelerating projects like AgMIP and ISIMIP by orders of magnitude in setup time.
Limitations on impact: The paper relies entirely on commercial, closed-source LLM agents. The top performers (Claude Sonnet 4.5, Claude Opus 4.5) are from a single provider. This creates fragility — API changes, pricing shifts, or deprecation could undermine deployed workflows. The paper doesn't address this vendor-lock concern.
4. Timeliness & Relevance
This paper is exceptionally well-timed. The convergence of capable coding agents (2025-2026 vintage), growing climate urgency, and the persistent reproducibility crisis in computational geoscience creates a clear window for this contribution. The paper explicitly benchmarks against 2026-era agents (GPT-5.x Codex, Gemini 3, Claude Opus 4.5), placing it at the frontier. The reference to coding agents achieving only 54% on adjacent reproduction tasks [17] positions KI as addressing a documented capability gap.
5. Strengths & Limitations
Key strengths:
Key limitations:
Overall Assessment
This is a high-impact systems paper that makes a compelling case for knowledge externalization as the missing layer between AI agents and scientific simulation. The empirical evidence is extensive and the conceptual framework is clean. The convergence finding across 119 models is the paper's deepest contribution. However, the work would benefit from higher success thresholds, systematic comparison with simpler baselines, and more honest engagement with the sustainability challenges of maintaining a living knowledge commons.
Generated May 19, 2026
Comparison History (33)
While Paper 1 offers a transformative framework for Earth science simulations, Paper 2 addresses a fundamental, cross-disciplinary vulnerability in modern AI-assisted research: prior contamination. By introducing a generalizable protocol to audit LLM memorization versus data-driven inference, Paper 2 improves the methodological rigor of any scientific field utilizing language models. Its proven efficacy across distinct domains like biology and finance indicates a broader, more foundational impact on how AI-driven science is validated.
Paper 1 demonstrates broader scientific impact by addressing accessibility of process-based simulation models across 14 Earth-science domains with 119 knowledge infrastructures, validated over 3,000 trials. It tackles a systemic barrier—democratizing complex scientific modeling for climate-vulnerable communities—with a generalizable framework (KDT) that scales autonomously. Paper 2's QED system is impressive in generating novel mathematical proofs for 3 of 5 open problems, but its impact is narrower, targeting research-level mathematics. Paper 1's breadth across disciplines, practical applications for climate adaptation, and vision of a 'living scientific commons' suggest wider transformative potential.
Paper 1 likely has higher scientific impact due to stronger novelty and broader real-world applicability: it proposes a generalizable “knowledge infrastructure” that operationalizes domain expertise into agent-actionable scaffolds for process-based Earth science simulation, validated at scale (3,000 trials) and across 117 models/14 domains. This could materially lower barriers to using and integrating scientific simulators, affecting climate risk, water resources, and cross-disciplinary modeling workflows. Paper 2 is timely and methodologically solid as an AI evaluation benchmark, but its impact is narrower (agent benchmarking/training) and less directly tied to high-stakes scientific/engineering deployment than Paper 1’s infrastructure for scientific simulation.
Paper 1 introduces a comprehensive infrastructure (KI/KDT) that democratizes access to 119+ process-based simulation models across 14 Earth-science domains, with rigorous benchmarking (3,000 trials) showing dramatic performance improvements. It addresses critical real-world problems (climate risk, resource scarcity) with broad cross-disciplinary applicability. Paper 2 provides valuable mechanistic insights into LLM emotion representations with alignment implications, but is narrower in scope (single model analysis) and more incremental. Paper 1's potential to transform how Earth science models are accessed and integrated gives it substantially broader and more lasting impact.
Paper 1 has higher potential scientific impact due to its methodological and conceptual novelty (agent-actionable knowledge infrastructure + automated “knowledge dissection”), strong empirical validation across many process-based models/domains, and broad applicability to accelerating and democratizing scientific simulation in Earth systems and beyond. It targets a major real-world bottleneck—making complex, validated process-based models usable and interoperable—potentially affecting climate risk assessment, water resources, ecology, and policy. Paper 2 is timely and impactful for scientific workflows, but its core contribution is more deployment/operations-focused and likely narrower in cross-disciplinary scientific advancement.
Paper 2 has higher likely scientific impact due to immediate, large-scale real-world applicability in healthcare and regulation, strong methodological rigor (massive dataset, scaling studies, >1,000 tasks, prospective and external validation, clear baselines), and broad downstream utility (prediction, expenditure forecasting, reduced bias in target trial emulation). It is timely given the surge in foundation models and demand for real-world evidence. Paper 1 is novel in agentic scientific simulation infrastructure and could be transformative, but its impact depends more on adoption and generalization beyond demonstrated Earth-science settings.
Paper 1 demonstrates a fundamentally new capability: an AI agent autonomously discovering and experimentally validating a previously unknown physical mechanism (optical bilinear interaction) on real hardware. This represents a paradigm shift in how scientific discovery can be conducted. The discovery itself—a physical analog to Transformer attention in optics—has direct implications for optical computing hardware. Paper 2 makes important contributions to democratizing simulation access via knowledge infrastructure, but it primarily improves accessibility of existing models rather than generating new scientific knowledge. Paper 1's end-to-end autonomous discovery milestone has broader transformative potential across all experimental sciences.
HealthFormer presents a generative transformer model for human physiology that enables personalized intervention simulation—a transformative capability for precision medicine. Its validation across independent cohorts, 30 disease endpoints, and 41 randomized trial comparisons demonstrates strong methodological rigor and immediate clinical applicability. The concept of a 'health world model' enabling clinical digital twins has enormous breadth of impact across medicine, drug development, and public health. While Paper 1 innovatively democratizes Earth science simulation models via agentic AI scaffolding, Paper 2 addresses a more universally impactful domain (human health) with a paradigm-shifting approach to in silico clinical trials and personalized medicine.
Paper 1 likely has higher scientific impact due to a more novel, technically ambitious multimodal generative foundation model spanning many biomolecular modalities, plus strong demonstrated performance (state-of-the-art on multiple tasks) and direct translational applications (splicing correction, binding-site–conditioned protein design). Its unified conditional generation framework could broadly influence genomics, structural biology, and protein/RNA engineering. Paper 2 is timely and valuable for democratizing Earth-science simulations, but impact may be more infrastructural/engineering-oriented and depend on adoption; methodological novelty is comparatively incremental vs broader foundation-model advances.
Paper 1 introduces a highly generalizable infrastructure that enables AI agents to operate complex scientific simulations across 14 Earth science domains. Its breadth of impact, bridging AI and climate/resource modeling, offers far wider real-world applicability and systemic innovation than Paper 2, which, despite its methodological rigor, is confined to a specific medical subfield (pulmonary diagnosis).