CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Zhaoyue Sun, Hainiu Xu, Andero Uusberg, James J. Gross, Petr Slovak, Yulan He
Abstract
Emotion understanding is a core capability for LLMs to interact effectively with humans, yet existing evaluation paradigms rely on discrete emotion label prediction and fail to capture the cognitive processes underlying emotion generation. Grounded in appraisal theory, we introduce CAREBench, the first benchmark with complete inferential chain annotations from both first- and third-person perspectives on real-world narratives, spanning appraisal reasoning, appraisal ratings, and multi-label emotion annotation. We propose a process-level evaluation framework and conduct systematic experiments across six LLMs organized around four research questions. We find that stronger models match or surpass human observers on certain tasks, yet fall short on appraisal reasoning and positive emotion recognition; performance across chain steps and sensitivity to appraisal interventions exhibit dissociations across models; and current models have not internalized the mechanisms needed to capture human subjective heterogeneity. These findings suggest that downstream emotion prediction metrics may overestimate LLMs' true emotion understanding, and CAREBench provides a foundation for more diagnostically informative evaluation of LLMs' affective cognitive capabilities.
AI Impact Assessments
(1 model)Scientific Impact Assessment: CAREBench
1. Core Contribution
CAREBench addresses a genuine gap in how we evaluate LLMs' emotion understanding. Rather than treating emotion recognition as a classification task over discrete labels, the paper grounds evaluation in appraisal theory — a well-established psychological framework positing that emotions arise from cognitive evaluations of situations along dimensions like goal relevance, controllability, and accountability. The core novelty is threefold: (1) explicit annotation of the *reasoning process* underlying appraisals, not just appraisal ratings; (2) multi-label emotion annotations with intensity; and (3) dual-perspective (first-person experiencer + third-person observer) annotations across the full inferential chain (situation → reasoning → appraisal → emotion). This creates a process-level evaluation framework that can localize where LLMs fail in emotion understanding, rather than providing only a binary success/failure signal at the output level.
2. Methodological Rigor
Data collection is carefully designed with psychology expert involvement, using the micro-narratives framework to elicit real emotional experiences from 1,000 participants, with 500 narratives additionally annotated by 5 third-person observers each (2,500 third-person records). The annotation pipeline is well-motivated by appraisal process models, and the sequential collection of reasoning before ratings is a thoughtful design choice that scaffolds annotation quality. Quality controls include paste detection, typing speed anomaly checks, attention checks, and bot filtering.
Experimental design is structured around four progressive research questions, each probing a different facet of LLMs' emotion understanding. RQ1 evaluates step-by-step performance; RQ2 tests whether appraisal reasoning improves downstream predictions; RQ3 examines sensitivity to controlled appraisal interventions (a quasi-causal design); RQ4 probes distributional heterogeneity modeling. This progression from descriptive to mechanistic questions is well-organized.
Limitations in rigor: The appraisal reasoning evaluation relies heavily on surface-level automatic metrics (BLEU, ROUGE, BERTScore), which are poorly suited for evaluating open-ended reasoning about subjective mental states. The human evaluation (100 scenarios, 2 evaluators, κ = 0.25 for consistency) provides some complementary signal but with low inter-annotator agreement, underscoring the difficulty of the task. The RQ3 intervention design, while creative, substitutes naturalistic reasoning texts rather than performing controlled minimal-pair manipulations, making it difficult to isolate the causal effect of specific appraisal dimensions. The authors appropriately note this limitation but could have designed more controlled interventions.
The sample is restricted to English-speaking US/UK participants, limiting cultural generalizability. The negative emotion skew (~1.6:1 over positive) is acknowledged but not addressed through sampling strategies.
3. Potential Impact
For NLP/AI: The benchmark fills a clear need. Most emotion recognition benchmarks treat emotion as a flat classification problem, and CAREBench provides the community with a diagnostic tool that can reveal *why* models fail, not just *that* they fail. The finding that downstream emotion prediction metrics overestimate LLMs' true understanding is important for the field's self-assessment. The dataset and code are publicly released.
For affective computing and HCI: The dual-perspective annotations and multi-label structure better reflect real-world emotion complexity. The framework could inform the development of more emotionally intelligent AI systems, particularly in mental health applications where understanding the cognitive basis of emotions matters.
For psychology: The dataset provides a computational testbed for appraisal theory predictions, and the finding that LLMs mirror developmental acquisition hierarchies (relevance/congruence before control/accountability) is an intriguing cross-disciplinary insight that warrants further investigation.
Broader applications: Cognitive reappraisal systems, empathetic dialogue, and therapeutic AI applications could all benefit from the inferential chain framework.
4. Timeliness & Relevance
This work is highly timely. LLMs are increasingly deployed in emotion-sensitive applications (mental health chatbots, companion AI, customer service), yet evaluation of their emotional capabilities remains superficial. The paper arrives as the community is grappling with questions about what LLMs truly "understand" versus what they can pattern-match. The grounding in established psychological theory (appraisal theory) gives the benchmark theoretical legitimacy that many AI benchmarks lack.
The inclusion of very recent models (GPT-5.2, Claude-Sonnet-4.6, Qwen 3.5) ensures immediate relevance, though this also means the findings may quickly become dated as models evolve.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations
The paper is well-written and well-organized, with extensive supplementary materials. The connection to developmental psychology findings (easier appraisal dimensions being those acquired earlier in child development) is a compelling interpretive lens, though it remains speculative. The cost transparency (~£9,000 total) is appreciated. The dataset's CC BY-NC-ND 4.0 license is somewhat restrictive for a benchmark, potentially limiting derivative work and adaptation.
Generated May 19, 2026
Comparison History (23)
CAREBench addresses a fundamental gap in LLM evaluation methodology for emotion understanding, a topic with broad relevance across NLP, AI safety, human-computer interaction, and cognitive science. Its grounding in appraisal theory provides novel theoretical depth, and its benchmark format enables widespread adoption and reproducibility. Paper 1, while technically solid, addresses a narrower application domain (UAV detection via ISAC). The breadth of impact, timeliness given the LLM evaluation boom, and cross-disciplinary relevance give Paper 2 higher potential scientific impact.
Paper 1 offers a profound algorithmic breakthrough, improving runtime by 5 orders of magnitude and instance size scaling by 3 orders of magnitude for POMDP-based planning. This fundamental advancement in tractability will have a lasting, rigorous impact on autonomous agent design and robotics, surpassing the likely transient impact of an LLM benchmark.
TO-Agents presents a novel multi-agent AI framework bridging natural language design intent with topology optimization, offering clear real-world engineering applications and a paradigm shift in how designers interact with optimization tools. While CAREBench makes a solid contribution to LLM emotion evaluation through appraisal theory, it is primarily a benchmark within an already-populated evaluation space. TO-Agents demonstrates broader cross-disciplinary impact (AI, engineering design, manufacturing), introduces a more actionable and novel methodology, and has stronger potential for practical adoption in product design and additive manufacturing workflows.
Paper 2 addresses a novel and practical problem—multi-task unlearning—that has broad applicability given the prevalence of multi-task learning in modern AI systems and growing regulatory demands (e.g., GDPR right to be forgotten). It introduces a new problem formulation (full-task vs. partial-task unlearning), identifies a concrete technical challenge (interference through shared parameters), and proposes a principled solution with strong empirical results. Paper 1, while valuable, is more niche—creating an emotion evaluation benchmark grounded in appraisal theory—and its impact is primarily diagnostic rather than enabling new capabilities. Paper 2's methodological contributions are more broadly transferable across domains.
Paper 1 likely has higher scientific impact due to broader cross-field applicability and stronger real-world integration: it benchmarks and operationalizes multi-agent LLM workflows spanning simulation/optimization, RAG, HPC orchestration, and manufacturing (3D printing). This ties evaluation directly to reproducible, end-to-end engineering tasks with clear deployment relevance and timely importance as agentic systems move into production. Paper 2 is novel and rigorous for affective reasoning evaluation, but its applications are narrower and less immediately tied to measurable external-world task execution.
GeoX introduces an innovative self-play framework with verifiable rewards for spatial reasoning, addressing the major bottleneck of human data annotation in VLMs. This methodology aligns with cutting-edge AI advancements (like RL for reasoning) and has profound applications in remote sensing, urban planning, and autonomous navigation. While CAREBench provides a valuable evaluation tool for affective computing, GeoX's novel training paradigm offers a more foundational methodological leap with wider cross-domain applicability.
Paper 1 likely has higher impact: it proposes an integrated, end-to-end autonomous research system with concrete mechanisms (self-healing execution, verifiable reporting, cross-run learning, and calibrated human-in-the-loop modes) and demonstrates sizable benchmark gains, making it directly actionable for accelerating scientific workflows across domains. Its contributions are timely given rapid growth of agentic AI and have broad applicability beyond a single task area. Paper 2 is novel and valuable for affective evaluation, but its impact is narrower (primarily NLP/affective computing benchmarking) and less immediately transformative for real-world scientific productivity.
Paper 1 introduces a novel cross-disciplinary framework bridging robotics and AI safety for foundation models in socially sensitive domains. Its reframing of guardrails as runtime behavioral control over interaction trajectories (rather than individual outputs) is a significant conceptual contribution with broad applicability. The framework is validated across three real-world deployments, demonstrating practical impact. It addresses a critical and timely gap in AI safety with formal guarantees, potentially influencing both the robotics and LLM safety communities. Paper 2, while valuable as a benchmark, is more incremental—extending emotion evaluation with appraisal theory—and has narrower scope.
Paper 1 demonstrates significantly higher impact due to its immediate relevance to the critical, rapidly growing field of AI agent security. Unlike Paper 2, which focuses on the theoretical aspects of LLM emotion understanding, Paper 1 presents a battle-tested, production-scale system deployed at a major enterprise (Uber). It addresses urgent real-world vulnerabilities, introduces a comprehensive framework (sensor, explorer, detector), and provides a valuable benchmark (ADR-Bench). Its combination of methodological rigor, proven real-world application, and timeliness in securing LLM agents guarantees broad utility and high impact in both industry and academia.
CAREBench introduces a theoretically grounded benchmark (appraisal theory) with novel process-level evaluation of emotion understanding, addressing a fundamental gap in how we assess LLMs' affective capabilities. Its finding that downstream metrics overestimate true understanding has broad implications across NLP, affective computing, and human-AI interaction. While PersonaArena offers a useful simulation framework for persona role-playing, CAREBench's contribution is more foundational—it redefines how emotion understanding should be evaluated, has stronger methodological rigor through cognitive science grounding, and its insights generalize more broadly across LLM applications.