Deterministic Event-Graph Substrates as World Models for Counterfactual Reasoning
Fabio Rovai
Abstract
We study event-graph substrates: a class of world models that represent agent state as an append-only log of typed RDF triples and answer counterfactual queries by forking the log under a structured intervention vocabulary. Substrates are inspectable at the triple level, support exact counterfactuals, and transfer across domains without learned components. We formalize the class, prove a duality between explanatory and counterfactual queries that reduces both to the same causal-ancestor traversal, and evaluate a 1,400-line CLEVRER-DSL interpreter atop a domain-agnostic substrate runtime at full CLEVRER validation scale (n=75,618). The substrate exceeds the NS-DR symbolic oracle on all four per-question categories (by 9.89, 20.26, 17.65, and 0.80 percentage points), and exceeds the parametric ALOE baseline on descriptive and explanatory while lagging on predictive and counterfactual. We also introduce twin-EventLog, a 500-specification Park-canonical Smallville counterfactual benchmark on which the substrate exceeds Llama-3.1-8B with full context by 18.80 points joint accuracy.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper formalizes "event-graph substrates" — world models that represent agent state as append-only logs of typed RDF triples and answer counterfactual queries by forking the log under structured interventions. The three claimed contributions are: (1) a formal definition with a duality theorem showing explanatory and counterfactual queries reduce to the same causal-ancestor traversal, (2) empirical evaluation on CLEVRER at full validation scale, and (3) a new benchmark (twin-EventLog) for agent memory consistency under intervention.
The core idea is conceptually clean: rather than learning latent dynamics, maintain a typed event log and answer "what if" questions by deterministic replay after graph surgery. This is essentially a finite-state instantiation of Pearl's twin-network formulation, as the authors acknowledge. The novelty is more in the engineering formalization and systematic evaluation than in the underlying conceptual machinery.
Methodological Rigor
Formal framework. The substrate definition is clearly presented as a tuple (T, A₀, L, ρ, I) with well-defined intervention semantics. The ancestor duality theorem (Proposition 1) is stated under three explicit conditions (closed events, exogeneity of non-ancestors, no emergent interactions). However, the proof is only sketched, with the full proof deferred to an appendix that does not appear to be included. The theorem itself, while useful, is relatively straightforward — it essentially says that if you remove an object, only events causally downstream of that object are affected, which is almost tautological given the closed-event assumption. The conditions under which it holds are quite restrictive, and the paper acknowledges C3 (no emergent interactions) fails regularly in CLEVRER.
CLEVRER evaluation. The substrate consumes ground-truth annotation files (object properties, trajectories, collisions) rather than video pixels. This is a critical methodological point: the comparison with NS-DR is somewhat fair since NS-DR also uses a symbolic oracle, but ALOE processes video. The substrate's advantage over NS-DR on descriptive and explanatory queries (which are essentially lookups and graph traversals over ground-truth annotations) is unsurprising. The more informative comparison is the substrate's underperformance against ALOE on predictive (-18.00 pp) and counterfactual (-15.75 pp) per-question metrics, which reveals the fundamental limitation of the approach.
Twin-EventLog benchmark. The substrate is "correct by construction" on this benchmark because it generates the ground truth. This is circular — the benchmark grades against the substrate's own deterministic replay. While demonstrating that LLMs struggle with counterfactual consistency is valuable, benchmarking a system against its own outputs inflates the apparent advantage. The 18.80 pp gap over Llama-3.1-8B and 65 pp over Concordia-style baselines must be interpreted in this light.
Controlled comparison. The n=300 subset comparison where Llama-3.1-8B receives the same event log as natural language is informative but uses a very small sample. The claim that "structured execution is the load-bearing pathway" is reasonable but the experimental design doesn't fully isolate this — the LLM also faces parsing and format-following challenges.
Potential Impact
The paper addresses a genuine need in agentic AI: auditable, deterministic counterfactual reasoning. The approach has clear applications in domains requiring formal guarantees — compliance, safety-critical systems, and debugging agent behavior. The cross-domain transfer demonstration (CLEVRER, ComPhy, GQA, bAbI) is useful for establishing generality, though TBox authorship remains manual.
However, the practical impact may be limited by several factors: (1) the approach requires structured, complete event logs as input, which is unrealistic in many real-world settings; (2) the TBox must be hand-authored per domain; (3) the method fundamentally cannot handle hidden variables, partial observability, or emergent dynamics without ad-hoc heuristics.
Timeliness & Relevance
The paper is timely given growing interest in reliable AI systems, interpretable world models, and the limitations of LLMs for structured reasoning. The contrast between deterministic symbolic approaches and learned parametric models is an active and important debate. The honest characterization of where each approach excels is valuable to the community.
Strengths
1. Clarity of formalization: The substrate definition is precise and the intervention semantics are well-specified.
2. Honest reporting: The paper clearly identifies where the approach lags (predictive, counterfactual with emergent interactions) and doesn't oversell.
3. Comprehensive evaluation: Full validation scale on CLEVRER (n=75,618), multiple benchmarks, and meaningful ablations.
4. Reproducibility: Fully deterministic with no learned parameters; JSONL artifacts provided.
5. Useful ablations: The per-event reference frame ablation (+7.58 pp) and emergent collision heuristic (+11.84 pp) demonstrate clear algorithmic contributions.
Limitations & Weaknesses
1. Circularity in twin-EventLog: Grading against the substrate's own output is methodologically problematic.
2. Input assumption: Consuming ground-truth annotations rather than raw observations limits practical applicability and makes comparisons with perception-based systems uneven.
3. Theorem triviality: The duality theorem, while clean, is relatively obvious under its strong assumptions, and the most interesting cases (where C3 fails) require heuristics outside the theorem.
4. Limited novelty: The approach is a careful engineering of well-known ideas (SCMs, RDF stores, deterministic replay). The paper acknowledges the class is "not new in spirit."
5. Heuristic nature of key components: The emergent-collision heuristic and kinematic projector are domain-specific patches that undermine the "domain-agnostic" framing.
6. Missing formal proof: The full proof of Proposition 1 is deferred but not included.
7. Scalability concerns: The O(b-a) replay cost and manual TBox authorship raise questions about scaling to complex, long-horizon domains.
Overall Assessment
This is a competent systems paper that formalizes and evaluates a principled approach to deterministic counterfactual reasoning. It provides a useful reference point contrasting symbolic and parametric world models. However, the conceptual novelty is modest, the benchmark design has methodological issues, and the approach's practical applicability is constrained by strong input assumptions. The paper's main value lies in its honest, systematic characterization of where deterministic replay succeeds and fails relative to learned alternatives.
Generated May 18, 2026
Comparison History (25)
Paper 2 has higher potential impact due to a more foundational, cross-domain contribution: a deterministic, inspectable world-model substrate with formal guarantees (duality proof) and exact counterfactual forking. Its approach is broadly applicable to causal reasoning, planning, interpretability, and verification beyond LLM multi-agent routing. The evaluation is large-scale and compares against strong symbolic and neural baselines, plus introduces a new counterfactual benchmark. Paper 1 is timely and useful for LLM agent systems, but its novelty is more incremental (confidence/routing/calibration) and its impact is likely narrower and faster to be subsumed by evolving agent frameworks.
Paper 1 offers a comprehensive roadmap and taxonomy for AI-automated research, a highly timely and universally relevant topic. By analyzing the entire research lifecycle and providing practical guidelines, it has the potential to influence how research is conducted across all scientific disciplines, likely resulting in widespread adoption and massive citation impact compared to the narrower, domain-specific technical contributions of Paper 2.
Paper 2 addresses a broader and more practically impactful problem—AI-assisted scientific discovery in nanomedicine—a field with enormous translational potential. It introduces a novel system (pArticleMap) combining literature mapping, frontier detection, and LLM-based hypothesis generation with rigorous evaluation including retrospective benchmarks and human assessment. While Paper 1 makes solid contributions to causal reasoning with event-graph substrates, its impact is more narrowly scoped to symbolic AI and specific benchmarks. Paper 2's cross-disciplinary relevance (AI + nanomedicine + scientific discovery) and real-world applicability give it higher potential impact.
Paper 2 addresses the highly relevant and rapidly growing field of multimodal agents, offering an adaptive memory framework that shifts away from fixed schemas. Its focus on learning how to structure multimodal experience has broad applicability across embodied AI, robotics, and complex reasoning tasks. While Paper 1 is methodologically rigorous and advances neuro-symbolic counterfactual reasoning, Paper 2's flexible, learning-based approach to multimodal memory aligns better with current AI trends and promises a wider impact across multiple domains.
Paper 1 offers a more novel, generalizable scientific contribution: a formalized world-model substrate with provable properties (duality of explanatory/counterfactual queries), exact counterfactual semantics, and evaluations on large-scale, established (CLEVRER) and newly introduced benchmarks. Its approach could influence causal reasoning, programmatic world models, and neuro-symbolic AI beyond a single domain. Paper 2 shows strong practical gains for SRE agents, but is more application/benchmark-engineering focused, with narrower scope and less methodological novelty/formal grounding, making its broader scientific impact likely lower.
Paper 1 addresses fundamental challenges in AI world modeling and causal reasoning. By proving theoretical dualities and demonstrating high performance on counterfactual tasks without learned components, it significantly advances explainable AI. While Paper 2 offers a valuable and timely benchmark for evaluating coding agents, Paper 1's theoretical and architectural contributions to causality and reasoning provide deeper, longer-term scientific impact across broader domains.
NeuroMAS introduces a broadly applicable framework reconceptualizing multi-agent LLM systems as trainable neural network architectures, offering a novel scaling paradigm with theoretical grounding and practical implications across many AI applications. Its insight about progressive growth and organizational scaling is timely given the current focus on LLM scaling. Paper 1, while rigorous and demonstrating strong results on specific benchmarks, addresses a narrower problem (deterministic counterfactual reasoning via event graphs) with more limited applicability. Paper 2's framework has greater potential to influence the rapidly growing multi-agent systems and LLM communities.
PopuLoRA introduces a broadly applicable population-based self-play framework for LLM post-training that addresses a fundamental limitation (self-calibration collapse) in RLVR. It demonstrates consistent improvements across 10 diverse benchmarks in both code and math reasoning, suggesting wide applicability. The weight-space evolution operators for LoRA are novel and computationally efficient. Paper 1, while technically sound with strong results on CLEVRER and a new benchmark, addresses a narrower problem (deterministic counterfactual reasoning via event graphs) with more limited community interest. Paper 2's contributions align with the high-impact, rapidly growing field of LLM reasoning improvement.
Paper 2 is likely higher impact: it tackles a harder, timely problem—online self-supervised dynamics discovery under prior/lexical misalignment—directly relevant to robust agent learning and world-model induction beyond curated vocabularies. The proposed closed-loop mechanism (using preservation conflicts to generate structured counterexamples and drive exploration) is a broadly applicable idea for program/world-model learning and could transfer across domains, simulators, and robotics. Paper 1 is innovative and rigorously evaluated, but is more engineering-/representation-centric and depends on hand-specified intervention vocabularies and DSL scaffolding, limiting breadth.
Paper 2 is likely higher impact: it proposes a deterministic, domain-agnostic world-model substrate with exact, inspectable counterfactual reasoning, includes formal results (duality proof), and demonstrates strong empirical performance at scale on CLEVRER plus a new benchmark (twin-EventLog) outperforming both symbolic and LLM baselines in key metrics. Its applications span AI reasoning, interpretability, and causal modeling, and it aligns with timely needs for reliable counterfactual reasoning beyond opaque learned models. Paper 1 is novel but more niche, with impact mainly in developmental/self-organization modeling.
Paper 1 demonstrates higher scientific impact through its rigorous empirical evaluation on established benchmarks (CLEVRER with n=75,618), introduction of a novel benchmark (twin-EventLog), and quantifiable improvements over both symbolic and parametric baselines. It addresses the fundamental problem of counterfactual reasoning in world models with a concrete, domain-agnostic computational framework. Paper 2, while intellectually interesting in formalizing moral psychology algebraically, is more theoretical with limited empirical validation and addresses a narrower interdisciplinary niche. Paper 1's broader applicability to AI reasoning systems and stronger experimental methodology give it greater potential impact.
Paper 2 addresses a critical bottleneck in long-term autonomous agents: maintaining memory consistency when underlying information changes. Its rigorous formalization of the cascade update problem and elegant reduction to a classic s-t min-cut algorithm provide a highly practical, domain-agnostic solution. While Paper 1 presents a strong neuro-symbolic approach to counterfactuals, Paper 2's focus on scalable, reliable agentic memory has broader immediate applicability and tackles a pressing challenge in deploying real-world LLM agents.
Paper 1 tackles foundational challenges in AI, specifically counterfactual reasoning, causality, and world models, offering a novel deterministic symbolic approach. Its theoretical contributions and domain-agnostic framework have broad implications across general AI research. In contrast, Paper 2 presents a highly applied method for e-commerce personalization, which, while practically valuable, has a narrower scope and less fundamental theoretical impact.
Paper 1 introduces a formally grounded framework for counterfactual reasoning with event-graph substrates, proving a novel duality theorem and demonstrating strong empirical results on established benchmarks (CLEVRER) plus a new benchmark. It addresses fundamental questions in causal/counterfactual AI with broad applicability across domains. Paper 2, while addressing a practical problem in GUI exploration, is more narrowly scoped to desktop agent exploration and presents primarily empirical/engineering contributions without comparably deep theoretical insights or demonstrated downstream task improvements.
Paper 2 is more likely to have higher scientific impact due to a broader, timely agenda (interpretable world models and counterfactual reasoning), clear real-world applicability (auditable reasoning substrates transferable across domains), and strong empirical validation at scale with new benchmarks. Its formal duality result plus competitive performance against symbolic and LLM baselines suggests cross-field relevance (causal inference, knowledge representation, neuro-symbolic AI, evaluation). Paper 1 is rigorous and novel within RL theory, but its impact is narrower (finite-time analysis of constant-step-size Q-learning) and less directly application-facing.
Paper 2 is more novel and broadly impactful: it proposes a deterministic, inspectable, domain-agnostic world-model substrate enabling exact counterfactuals via log forking, with formal duality results linking explanation and counterfactual querying. This offers clear real-world applicability for trustworthy reasoning, auditing, and safety, and its methodology includes formalization, proofs, and large-scale evaluations plus a new benchmark. Paper 1 is a strong systems/algorithm contribution for skill libraries in LLM agents, but is more incremental within RL-for-agents and tied to particular task environments, with narrower cross-field reach.
Paper 2 likely has higher impact: autonomous, agentic neural architecture discovery targets a central bottleneck in foundation-model progress, with broad applicability across ML systems and potential to reshape model design workflows. It reports scalable evaluations (up to 3B), improved downstream performance, and better scaling efficiency—highly timely given current focus on scaling laws, compute efficiency, and automated R&D. Paper 1 is novel and rigorous for interpretable counterfactual world models, but its approach is more domain-structured/symbolic and may generalize less broadly than an automated framework for discovering next-gen architectures.
Paper 2 introduces a novel formal framework (event-graph substrates) for counterfactual reasoning with theoretical contributions (duality proof), a new benchmark, and demonstrated domain-agnostic transfer—touching causal reasoning, knowledge representation, and LLM evaluation. Its breadth of impact spans multiple fields (causal inference, NeSy AI, world models). Paper 1, while solid, is a more incremental extension of PPO with symbolic guidance in a well-explored neuro-symbolic RL space. Paper 2's foundational contributions and cross-domain applicability suggest higher long-term scientific impact.
Paper 2 offers broad, foundational contributions to AI through its formalization of event-graph substrates for counterfactual reasoning and mathematical proofs of query duality. It demonstrates high methodological rigor with large-scale evaluations (n=75,618) and comparisons against strong baselines. In contrast, Paper 1 is a domain-specific applied study with limited empirical rigor, evaluating only a proof-of-concept subset of 51 cases and explicitly omitting baseline comparisons. Paper 2's potential impact across neuro-symbolic AI and agentic world models far exceeds Paper 1's niche application.
Paper 1 addresses fundamental challenges in AI by proposing a novel, formal class of world models for counterfactual reasoning. Its theoretical contributions (proving duality between query types) and domain-agnostic approach offer broad applicability across AI subfields. In contrast, Paper 2 presents a practical but niche engineering framework for root cause analysis in microservices. Paper 1's focus on foundational reasoning mechanisms gives it higher potential for widespread scientific impact.