OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind
Sharmin Sultana Srishty, Kazi Mahathir Rahman, Malaika Parizat Sakkhi, Samia Shahid Prianna, Shaikhul Islam Sinat
Abstract
Large Language Models (LLMs) perform well on many language tasks, but their Theory of Mind (ToM) reasoning is still uneven in complex social settings. Existing benchmarks, including ExploreToM, do not always test the recursive beliefs and information asymmetries that make these settings difficult. This paper presents OSCToM (Observer-Self Conflict Theory of Mind), an approach for modeling nested belief conflicts in LLM-based ToM tasks. The key case is one in which an observer's view of another agent conflicts with the observer's own belief state. Such cases go beyond simple perspective-taking and require recursive, multi-layered reasoning. OSCToM combines reinforcement learning (RL), an extended domain-specific language, and compositional surrogate models to generate observer-self conflicts. In our experiments, OSCToM-8B gives the best overall result among the systems tested. It improves on the reported ExploreToM results on FANToM and remains competitive on Hi-ToM and BigToM. On the information-asymmetric FANToM benchmark, OSCToM reaches 76% accuracy, compared with the 0.2% reported by ExploreToM. The data-synthesis procedure is also 6x more efficient, indicating that targeted training data can help smaller models handle advanced cognitive reasoning. The project code is available at https://github.com/sharminsrishty/osct.
AI Impact Assessments
(1 model)Scientific Impact Assessment: OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind
1. Core Contribution
OSCToM introduces a framework for generating adversarial Theory of Mind (ToM) training data centered on "Observer-Self Conflict" — scenarios where an observer's recursive model of another agent's belief contradicts the observer's own knowledge. The system combines four components: (1) an extended domain-specific language (DSL) supporting 4th-order belief tracking and deceptive operators, (2) a compositional surrogate evaluation pipeline using six distilled DistilBERT modules, (3) a DQN-based RL generator that navigates the DSL state space, and (4) a two-stage curriculum fine-tuning strategy for Llama-3.1-8B-Instruct.
The headline result is dramatic: 76% accuracy on FANToM compared to ExploreToM's reported 0.2%, achieved with an 8B parameter model that outperforms much larger systems on several benchmarks.
2. Methodological Rigor
Strengths in design: The compositional surrogate pipeline is well-motivated — reducing per-story evaluation from ~14 seconds to <50ms enables practical RL training. The six-module decomposition (false belief detection, ToM depth, deception scoring, social complexity, temporal complexity, OSCT detection) provides interpretable reward signals.
Significant concerns about evaluation fairness: The comparison with ExploreToM is potentially misleading. ExploreToM is primarily a *data generation framework*, and the authors compare against its reported numbers which use iterative A* search at inference time, not a fine-tuned model. OSCToM-8B is a fine-tuned model evaluated via direct inference. These are fundamentally different paradigms — comparing ExploreToM's 0.2% on FANToM against OSCToM's 76% conflates the quality of generated training data with inference methodology. A fairer comparison would fine-tune the same base model on ExploreToM-generated data and compare.
Missing ablations: The paper lacks critical ablation studies. What is the individual contribution of (a) the extended DSL vs. standard DSL, (b) the RL-guided generation vs. random sampling from the DSL, (c) the two-stage curriculum vs. single-stage training, and (d) the surrogate pipeline vs. direct LLM scoring? Without these, it's unclear which components drive the improvements.
Baseline selection issues: The baselines are mostly base/instruct models without any ToM-specific training. The only trained baseline is ExploreToM, and that comparison is problematic as discussed. The paper would benefit from comparing against models fine-tuned on ExploreToM data, or other ToM-enhanced models.
Surrogate validation: The paper claims surrogates are distilled from Llama-3.3-70B annotations but provides no validation metrics (accuracy, F1, calibration) for the six surrogate modules themselves. If surrogates are poorly calibrated, the entire RL reward structure is compromised — a risk the authors acknowledge but don't address empirically.
3. Potential Impact
The framework addresses a genuine need: generating targeted, adversarial training data for specific cognitive reasoning capabilities. The idea of decomposing ToM difficulty into independent dimensions and using lightweight surrogates for RL-guided generation is architecturally sound and could generalize beyond ToM to other structured reasoning tasks.
The 6x efficiency improvement in data synthesis and the ability to train an 8B model that competes with 27-32B models is practically valuable, especially for resource-constrained settings. The Observer-Self Conflict concept itself — formalizing the distinction between an agent's own beliefs and their model of others' beliefs — is a useful cognitive framing.
However, the narrow focus on belief-location scenarios limits immediate applicability. Real-world ToM involves emotions, intentions, social norms, and contextual cues far beyond what the current DSL captures.
4. Timeliness & Relevance
This work is timely. ToM evaluation for LLMs is an active area with recognized limitations in existing benchmarks. The shift from passive evaluation to adversarial data generation aligns with broader trends in the field. The paper correctly identifies that existing programmatic approaches (ExploreToM) have limitations in their search strategies and that RL-based exploration could be more flexible.
The focus on information asymmetry and recursive belief tracking addresses documented failure modes of current LLMs. However, the field is rapidly evolving — newer models (GPT-4o, Claude 3.5, Llama 3.3) may already perform better on these benchmarks, making some comparisons quickly outdated.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Additional Observations:
The DQN comparison table (Table 1) shows A2C and Recurrent PPO achieving the same mean reward of -0.060, which seems suspicious. The claim that DQN was chosen for "stability" is reasonable but the comparison is shallow — only training time, mean reward, mean hardness, and a subjective "stability" label are reported.
The paper would benefit substantially from human evaluation of generated stories to verify that high surrogate scores correspond to genuinely challenging ToM scenarios rather than artifacts of the surrogate reward structure.
Overall Assessment
OSCToM presents an interesting architectural framework for adversarial ToM data generation with some promising results, particularly on FANToM. However, the evaluation methodology has significant gaps — most critically the unfair comparison with ExploreToM and the lack of ablation studies. The core ideas (surrogate-guided RL for cognitive task generation, Observer-Self Conflict formalization) have merit but need stronger empirical validation to support the paper's claims.
Generated May 21, 2026
Comparison History (25)
Paper 1 likely has higher impact: it targets a central bottleneck for LLM agents (skill lifecycle management), proposes a minimal, general “hygiene” recipe with clear ablations, provides sizable gains on widely relevant coding/agent benchmarks (MBPP+, SWE-bench Verified), and includes a stability/non-divergence argument—supporting methodological rigor and transferability. Its contributions apply broadly to autonomous agents, continual improvement without finetuning, and tool/skill libraries across domains. Paper 2 is novel for ToM benchmarking/data generation, but its impact is narrower to social reasoning tasks and relies more on synthetic data/RL specifics.
Paper 1 offers higher potential scientific impact by addressing a critical bottleneck in foundation models: the immense computational cost of processing long videos. Its training-free, spatio-temporal token selection method (balancing static similarity and dynamic differences) provides an immediate, scalable solution for efficient Multimodal Large Language Models (MLLMs). This enhances real-world applicability across numerous domains like autonomous driving, robotics, and content analysis. While Paper 2 presents impressive benchmark gains in Theory of Mind reasoning, Paper 1's generalizable architectural optimization impacts a broader range of immediate, high-demand downstream applications.
ArborKV addresses a fundamental infrastructure bottleneck in tree-based LLM reasoning, which is increasingly important as search-augmented reasoning (e.g., Tree-of-Thoughts, Monte Carlo Tree Search) becomes central to scaling LLM capabilities. Its 4x memory reduction enables broader adoption of these methods under fixed hardware constraints, with broad applicability across reasoning paradigms. Paper 2 advances Theory of Mind benchmarking, a narrower subfield, and while the FANToM improvement is notable, benchmark-specific gains on cognitive reasoning tasks have more limited downstream impact compared to enabling scalable inference infrastructure.
Paper 2 likely has higher impact: it targets a central, timely problem in agentic LLMs—maintaining self-evolving skill libraries—and shows large, reproducible gains with ablations, multi-seed evaluation, and a stability proposition. The “minimal hygiene recipe” is broadly applicable across domains (coding, software engineering agents) and can be adopted without model finetuning, increasing real-world utility. Paper 1 is novel in ToM benchmark/data generation and shows striking benchmark gains, but its impact is narrower (ToM-specific) and more tied to synthetic benchmark design than to general agent performance.
Paper 2 (IdleSpec) likely has higher impact due to broad, immediately deployable applicability across LLM agent systems where tool/IO latency is common (code execution, web, APIs). Its speculative planning framework is generic, improves accuracy while controlling latency, and is timely for agentic workflows and efficiency. The methodological contribution (idle-time utilization with adaptive drafting and posterior feedback) can transfer across domains and models. Paper 1 is novel for ToM benchmarking/data synthesis, but its impact is narrower (social reasoning benchmarks) and may be more sensitive to dataset/design choices.
Paper 2 addresses a fundamental cognitive capability in AI (Theory of Mind), which has broad, cross-disciplinary implications for human-AI interaction, cognitive science, and complex reasoning. Its novel combination of RL and domain-specific languages to solve recursive belief conflicts demonstrates significant methodological innovation. In contrast, while Paper 1 provides a highly practical benchmark for enterprise finance, its scientific scope is narrower and focuses on evaluating existing models rather than advancing fundamental AI reasoning architectures.
Paper 2 addresses a highly central problem in LLM development—simplifying the complex post-training pipelines for search-augmented reasoning. By demonstrating that self-evolution via vanilla GRPO and self-distillation can outperform complex, heavily engineered baselines, it offers a broadly applicable methodology that impacts general LLM training. Paper 1 offers impressive gains in a specialized cognitive domain (Theory of Mind), but Paper 2's methodological simplification has wider potential adoption and broader impact across various reasoning and agentic tasks.
Paper 2 addresses a fundamental research question in AI—Theory of Mind reasoning in LLMs—with novel methodology combining RL-guided adversarial generation and compositional surrogate models. It demonstrates dramatic improvements (0.2% to 76% on FANToM) and contributes to understanding cognitive reasoning in AI systems, with broad implications for AI safety, multi-agent systems, and cognitive science. Paper 1 is a useful engineering contribution (reducing boilerplate for API/MCP tool deployment) but is primarily a software framework with limited scientific novelty, addressing a narrow developer productivity concern rather than advancing fundamental understanding.
Paper 2 has higher potential impact due to stronger novelty and timeliness in LLM evaluation/training: RL-guided adversarial data generation for higher-order Theory of Mind with a new conflict formulation (observer-self) and measurable, large benchmark gains (e.g., 76% vs 0.2% on FANToM) plus open-source code. Its applications span AI safety, social reasoning, HCI, and cognitive science, giving broader cross-field reach. Paper 1 is valuable and rigorous for safety assurance, but is more domain-specific and incremental relative to established assurance-case frameworks.
Paper 2 has higher potential impact due to a more novel, technically innovative method (RL-guided adversarial data generation with a DSL and surrogate models) and strong quantitative gains on established ToM benchmarks, including a dramatic improvement on information-asymmetric FANToM plus improved data efficiency. It is timely for LLM reasoning research, broadly relevant across ML, NLP, cognitive modeling, and AI safety, and offers reusable code. Paper 1 provides valuable qualitative insights for organizational practice, but its narrower scope and less generalizable methodology suggest a more limited cross-field scientific impact.