Hypothesis Generation and Inductive Inference in Children and Language Models
Jeffrey Qin, Wasu Top Piriyakulki, Zhuangfei Gao, Mia Radovanovic, Jessica Sommerville, Kevin Ellis, Marta Kryven
Abstract
Real world decision-making requires constructing mental models under uncertainty over evidence, over the underlying causal rules, and over the state of the world itself. Which computational principles underpin human inference under such conditions, and do LLM-based agents exhibit similar behavior given matching constraints? We address these questions using an inductive inference Box Task in which participants, human children and LLM-based agents, infer a latent cause through sequential interaction with an uncertain environment. We formalize this task as program induction with Bayesian particle-based inference, admitting two complementary interpretations: (1) as a constraint satisfaction process over hypotheses, and (2) as a program synthesis problem in which hypotheses are executable programs evaluated against evidence. Using the constraint-based formulation, we show that children's behavior is best explained by a combination of subjective evidence reliability and online hypothesis generation, accounting for both their evidence-seeking patterns and their dissociation between task completion and rule generalization. Using the program synthesis formulation, we treat LLM-based agents as model organisms: controllable systems that allow systematic manipulation of task conditions. Across backends, LLM-based agents replicate children's responses to changes in evidence reliability and observability, including discounting unreliable evidence, seeking to resolve partial information, and dissociating between task completion and causal generalization. At the same time, LLM-based agents tend to over-observe and over-comply with instructions relative to children. These results suggest that while children and LLM-based agents adapt similarly to environmental structure, their information-seeking behavior exhibits distinct underlying costs and inductive biases.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper presents a dual computational framework for studying inductive inference under uncertainty, applied to a physical puzzle-solving task (the Box Task) where agents must discover a latent causal rule from noisy, partially observable evidence. The main novelty lies in the twin formulations: (1) a Sets of Constraints (SoC) cognitive model fitted to children's behavioral data, and (2) an LLM-based Program Synthesis (LLM-PS) system where hypotheses are executable Python programs, enabling LLMs to serve as "model organisms" whose environmental conditions can be systematically manipulated. The key insight is that children's behavior requires both subjective evidence reliability (ρ) and online hypothesis generation to explain the observed dissociation between task completion (66%) and rule generalization (22%), and that LLM-based agents exhibit qualitatively similar but quantitatively distinct patterns under matched conditions.
Methodological Rigor
The POMDP formalization is well-specified, and the SMC-S inference framework provides a principled probabilistic backbone for both implementations. The SoC model comparison is methodologically sound: four model variants (lesioned, reliability-only, generator-only, full) are compared via per-child MLE with AIC correction, and the full model wins decisively (paired t-tests, all p < .001, Cohen's d ranging 0.39–1.34). The parameter fitting approach is honest about its limitations — the authors explicitly acknowledge that exact inference is intractable and resort to coarse grid search, which is appropriate given the sparse individual-level data (single trajectories of 10–70 trials).
However, several methodological concerns arise. The SoC probability tables are constructed from only 100 forward simulations per parameter setting, which may introduce Monte Carlo noise into the likelihood estimates. The ε = 0.01 floor applied before taking logarithms could systematically bias model comparisons, though the authors note rankings are stable across alternatives. The LLM-PS experiments lack formal statistical testing — results are presented descriptively across backends without confidence intervals on generalization rates or formal hypothesis tests comparing LLM behavior to children's. The number of LLM runs (under 300 total including debugging) is relatively modest for drawing robust conclusions.
The treatment of LLMs as "model organisms" is conceptually appealing but somewhat imprecise. Unlike biological model organisms where the underlying mechanisms are partially understood, LLM internals remain opaque, making it unclear what exactly is being "controlled" when environmental conditions are varied. The authors acknowledge this limitation but it weakens the interpretive framework.
Potential Impact
The paper makes contributions across several domains:
Cognitive science: The extension of the Sampling Hypothesis to settings with unreliable evidence and unbounded hypothesis spaces is a meaningful theoretical advance. The demonstration that children's approximate hypotheses can aggregate disparate causes without achieving unification offers a concrete computational account of a well-observed developmental pattern.
AI/LLM evaluation: The framework provides a structured methodology for comparing LLM reasoning to human reasoning under controlled conditions, going beyond simple accuracy metrics to examine process-level behaviors (information-seeking, hypothesis revision, instruction compliance). The finding that LLMs over-observe and over-comply relative to children is a useful characterization of LLM inductive biases.
Program synthesis: The single-particle finding for GPT-5.2 (task completion via sequential hypothesis revision without population-based inference) is practically interesting, suggesting that sufficiently capable models can approximate Bayesian updating through autoregressive generation.
The real-world applicability is somewhat limited by the specificity of the Box Task, though the POMDP formalization is general enough to extend to other latent-rule discovery domains.
Timeliness & Relevance
The paper addresses a timely intersection of developmental cognitive science and LLM evaluation. As the field increasingly uses LLMs as cognitive models or comparison systems, principled frameworks for such comparisons are needed. The work responds directly to calls for structured evaluation paradigms (Allen et al., 2024; Ying et al., 2025) and builds on the growing "LLMs as model organisms" perspective (Frank, 2023; Summerfield, 2023).
Strengths
1. Elegant dual formulation: The same inference machinery (SMC-S) supports both a cognitive model fitted to human data and an LLM-based system, enabling meaningful comparison.
2. Principled lesion analysis: The systematic ablation of model components (reliability, generator) provides clear evidence for which computational principles are necessary.
3. Interpretable hypotheses: LLM-generated Python programs are directly inspectable, allowing qualitative analysis of hypothesis evolution trajectories.
4. Ecological validity: Using an existing developmental psychology paradigm grounds the computational analysis in real behavioral data.
5. Honest treatment of limitations: The authors are forthright about fitting challenges, data contamination risks, and interpretive constraints.
Limitations
1. Small and specific behavioral dataset: N=100 children from a single study, single age range (7-10), single task. Generalizability is uncertain.
2. Asymmetric comparison conditions: Children face physical constraints (time pressure, motor demands, social context) absent from LLM experiments. The paper acknowledges but doesn't resolve this confound.
3. Limited LLM backends: Only three backends tested, with DeepSeekV3.2 failing entirely under partial observability and thus excluded, narrowing the comparative analysis.
4. No direct human-matched LLM condition: The authors note that adults/children were not tested under the fully observable/reliable conditions given to LLMs, weakening cross-system comparisons.
5. Potential data contamination: The Box Task is from a published study, and while the authors argue against direct retrieval, this cannot be conclusively ruled out.
6. Observation model simplification: OBSERVE actions in LLM-PS-P are fully informative, unlike human observations which may be partial — this asymmetry could explain the over-observation finding.
Overall Assessment
This is a well-crafted interdisciplinary paper that makes a genuine contribution to understanding inductive inference under uncertainty in both humans and machines. The cognitive modeling results are the stronger component, providing clear evidence that subjective reliability and online generation jointly explain children's behavior. The LLM comparison, while conceptually interesting, is more exploratory and descriptive. The paper would benefit from more rigorous statistical treatment of LLM results and closer matching of experimental conditions across systems.
Generated May 26, 2026
Comparison History (15)
Paper 1 bridges cognitive science, developmental psychology, and AI to explore fundamental mechanisms of inductive reasoning. By comparing children and LLMs using Bayesian program synthesis, it offers profound theoretical insights into human cognition and AI behavior. Paper 2 presents a rigorous neuro-symbolic approach for Legal AI, but its focus is primarily domain-specific. Paper 1's foundational exploration of hypothesis generation grants it broader interdisciplinary appeal and a higher potential for widespread scientific impact across multiple fields.
Paper 2 likely has higher scientific impact due to broader cross-field relevance (cognitive development, computational cognitive science, causal/inductive inference, and AI evaluation), timely questions about LLMs as models of human-like inference, and a principled formalization (Bayesian particle inference; dual constraint-satisfaction and program-synthesis views) with interpretable behavioral comparisons across humans and models. Its findings can influence both theory (mechanisms of hypothesis generation) and practice (benchmarking/diagnosing agent inductive biases). Paper 1 is strong and applicable for LLM-agent engineering, but its impact is narrower and more systems-focused.
Paper 1 likely has higher scientific impact due to a concrete, scalable solution to a pressing, widely felt problem in agent evaluation: artifact drift in benchmark/task generation. Anchor’s joint generation of instructions, environments, certified optimal solutions, and verifiers is a methodological innovation with strong rigor and immediate real-world applicability (enterprise workflows), plus a sizable released benchmark (ERP-Bench) that can become a community standard. Its impact spans ML evaluation, agent reliability, and enterprise automation. Paper 2 is timely and interesting for cognitive science–LLM comparisons, but its broader downstream tooling/standardization and practical leverage are less direct.
Paper 1 likely has higher scientific impact due to its cross-disciplinary novelty (linking child cognition, Bayesian/program induction, and LLM agents as experimental model organisms) and broader relevance to psychology, cognitive science, AI alignment/interpretability, and education. It offers a principled task formalization with complementary computational interpretations and tests mechanistic hypotheses about evidence reliability and information seeking, supporting methodological rigor. Paper 2 is timely and practically useful for long-horizon agent scaling, but its impact may be narrower (systems/engineering) and more contingent on benchmark generalization and competitive baselines.
Paper 2 likely has higher impact due to broader, timely relevance: it links developmental cognition with LLM agent behavior under uncertainty using a shared formalism (Bayesian program induction / program synthesis). This bridges psychology, cognitive science, AI alignment/agent evaluation, and human–AI comparison, with clear experimental paradigms and actionable diagnostics (evidence reliability, observability, information-seeking biases). Paper 1 is novel and rigorous within runtime algorithmic fairness/control, but its impact is more specialized and depends on adoption in deployed decision systems. Paper 2’s cross-field reach and immediacy in the LLM era boosts expected impact.
Paper 2 has higher potential impact due to broader cross-field relevance (cognitive science, developmental psychology, AI/LLM evaluation, Bayesian inference), a principled task formalization (program induction with particle-based Bayesian inference) and strong timeliness in using LLM agents as manipulable “model organisms” to study human-like inference. Its findings generalize beyond a single medical modality and can influence both theories of human learning and methodologies for probing/aligning AI systems. Paper 1 is innovative and clinically useful, but its impact is more domain-specific to ECG interpretation.
Paper 2 likely has higher impact: it introduces a general, theoretically grounded evaluation framework (trajectory-level strictly proper scoring rules) that cleanly targets a well-defined object (prefix-conditioned success probability), with proofs and extensions to censored trajectories. This is broadly applicable across agent benchmarks and model classes, timely for agentic LM evaluation, and can become a standard tool. Paper 1 is novel in linking child cognition and LLM agents via program-induction formulations, but its impact is more specialized (cognitive modeling + LLM behavior) and depends on task-specific generalization and empirical validity.
Paper 1 addresses multi-step reasoning efficiency, a critical bottleneck in modern AI. Using hyperbolic geometric signals to guide LLM reasoning paths is a highly novel and mathematically grounded approach. This method has immediate real-world applicability in enhancing LLM performance. While Paper 2 provides valuable cognitive science insights by comparing children and LLMs, Paper 1 has a broader potential impact across the rapidly expanding field of artificial intelligence by directly improving core model capabilities.
Paper 1 addresses a critical bottleneck in LLM program synthesis by introducing a property-guided repair loop with counterexamples, significantly reducing computational costs and improving program quality. Its integration of formal verification with LLMs has broad, highly practical applications across AI planning, code generation, and software engineering. While Paper 2 offers valuable cognitive science insights, Paper 1's methodological innovation solves a pressing scalability problem in AI, likely leading to wider adoption and greater technological impact.
Paper 2 likely has higher scientific impact due to its cross-disciplinary novelty and breadth: it links developmental cognition, Bayesian/program induction, and LLM agent behavior in a shared experimental task, offering a framework to compare human and model inference mechanisms. Its real-world applications span cognitive science (theory of hypothesis generation), AI evaluation (agentic uncertainty handling), and interpretability/safety (information-seeking biases). The methodological framing (dual formalizations, controlled manipulations, human–model comparisons) is broadly reusable and timely amid growing interest in LLMs as cognitive models.
Paper 2 has broader interdisciplinary impact, bridging cognitive science, developmental psychology, and AI through a novel comparative framework between children's inductive reasoning and LLM behavior. Its dual formalization (constraint satisfaction and program synthesis) offers theoretical depth, and treating LLMs as 'model organisms' for cognitive science is a timely, innovative paradigm. Paper 1, while technically sound, addresses a narrower engineering problem (selective prediction via prover-verifier protocols) with primarily empirical contributions on specific benchmarks, limiting its broader scientific reach.
Paper 1 likely has higher scientific impact due to a scalable, verifiable RL training pipeline plus a large open dataset/environments for computer-use agents—an area currently bottlenecked by reward verification and environment availability. Its methodological contribution (generator–discriminator reward synthesis with execution-based filtering) and demonstrated performance gains/transfer suggest immediate applicability for agent training and benchmarking across industry and academia, with broad downstream influence on RL, agentic LLMs, and UI automation. Paper 2 is novel and interdisciplinary, but its impact is more domain-specific (cognitive modeling) and less directly enabling for large-scale systems development.
Paper 1 likely has higher impact: it introduces a concrete, automated benchmark plus trained evaluators and fine-grained metrics for a fast-moving area (audio-video generative models), with clear downstream uses (model comparison, data filtering, differentiable reward for RLHF). This combination of infrastructure + methodology can standardize evaluation and accelerate progress across many AV generation systems. Paper 2 is conceptually novel and interdisciplinary (cognitive science + LLM agents) but its immediate real-world applications and field-wide tooling effects are narrower, and impact depends more on follow-up adoption and replication.
Paper 1 offers broader scientific impact by bridging cognitive science and AI, providing novel insights into human inductive inference (children's hypothesis generation) and using LLMs as 'model organisms' — a compelling methodological innovation. It contributes to multiple fields (developmental psychology, computational cognitive science, AI alignment) with a rigorous Bayesian framework. Paper 2, while technically sound, addresses a narrower optimization problem (efficient LRM decoding) that is more incremental and engineering-focused, with impact largely confined to the NLP efficiency community.
Paper 1 bridges developmental psychology, cognitive modeling, and artificial intelligence, offering fundamental insights into how both humans and LLMs perform inductive reasoning under uncertainty. This cross-disciplinary approach addresses core questions in both cognitive science and AI alignment, leading to a broader potential scientific impact. In contrast, Paper 2 presents a valuable but highly specialized data model update for a specific repository (AOP-Wiki), which, while important for regulatory science and toxicology, has a narrower scope and less fundamental theoretical novelty than Paper 1.