VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
Yuxin Chen, Yi Zhang, Zhengzhou Cai, Yaorui Shi, Zhiyuan Yao, Chenhang Cui, Jingnan Zheng, Yaqi Huo
Abstract
Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.
AI Impact Assessments
(1 models)Scientific Impact Assessment: VitaBench 2.0
1. Core Contribution
VitaBench 2.0 addresses a genuine and underexplored gap in LLM agent evaluation: the ability to infer, maintain, and leverage evolving user preferences from fragmented, long-term interactions. While existing agent benchmarks (τ-bench, WebArena, SWE-bench) focus on reasoning and tool orchestration with fully specified task contexts, VitaBench 2.0 introduces personalization as a first-class evaluation dimension. The benchmark evaluates three specific capabilities: preference extraction from noisy interaction histories, preference utilization during decision-making, and preference updating as user behavior evolves. Additionally, it introduces proactive tasks requiring agents to recognize missing contextual information and actively seek clarification.
The benchmark is organized around 56 users with 2,000+ preferences across 819 tasks spanning delivery, in-store consumption, and online travel agency domains, with 66 tools. The POMDP formulation is well-motivated, and the temporal task sequence design captures realistic long-horizon dynamics that static benchmarks miss.
2. Methodological Rigor
Strengths in design: The benchmark's construction methodology is thorough. User profiles are data-driven and manually curated, preferences are grounded in realistic scenarios with explicit drift mechanisms (addition, deletion, modification), and interaction histories blend signal with deliberate noise. The environment synthesis pipeline with multi-stage validation (LLM verification + human review) addresses the scalability challenge of creating thousands of executable environments.
The extensible memory interface enabling controlled comparison across agentic memory, RAG memory, and full-context settings is a well-designed experimental control. The evaluation uses rubric-based decomposition with both trajectory-level and outcome-level assessment, and each task is run 4 times for statistical stability (Avg@4, Pass@4, Pass^4 metrics).
Concerns: The use of GPT-4.1 as both user simulator and evaluator introduces potential bias, though the paper partially addresses simulator leakage by restricting the simulator's access to preferences. The programmatic construction of preferences, while enabling control, may not capture the full messiness of real-world preference expression. The benchmark covers only three domains (delivery, in-store, OTA), all within a life-services vertical, limiting generalizability claims. The paper also lacks inter-annotator agreement statistics for the manual curation process and does not report confidence intervals for the main results.
3. Potential Impact
Immediate impact: The benchmark fills a clear evaluation gap. As LLM agents increasingly serve as personal assistants (e.g., Apple Intelligence, Google Gemini, OpenAI's operator), the ability to evaluate personalization systematically is valuable. The finding that even state-of-the-art models (Claude Opus 4.6, GPT-5, DeepSeek-V4-Pro) achieve only ~0.5 Avg@4 and ~0.3 Pass^4 establishes a concrete capability gap that can drive focused research.
Research directions enabled: The benchmark can catalyze work on (a) memory architectures for long-term user modeling, (b) preference extraction from noisy signals, (c) proactive interaction strategies, and (d) personalization-aware training objectives. The finding that thinking/reasoning modes don't consistently help personalization is particularly insightful, suggesting this requires fundamentally different capabilities than chain-of-thought reasoning.
Broader influence: The memory interface design could become a standard for comparing memory mechanisms across different agent systems. The failure pattern analysis (Figure 5) showing that personalization errors dominate over tool-use errors in stronger models provides actionable guidance for model developers.
4. Timeliness & Relevance
The paper is highly timely. The LLM agent ecosystem is rapidly moving toward persistent, personalized assistants (evidenced by the models evaluated including GPT-5, Claude Opus 4.6, and other frontier 2025-2026 models). Existing benchmarks have not kept pace with this shift. The paper correctly identifies that as tool-use and reasoning improve, personalization becomes the next bottleneck—a claim substantiated by the failure analysis showing preference-related errors now dominate in stronger models.
The memory evaluation aspect is also timely given the explosion of memory systems (Mem0, A-MEM, MemAgent, MemGPT) that currently lack standardized evaluation.
5. Strengths & Limitations
Key Strengths:
Limitations:
Overall Assessment
VitaBench 2.0 makes a solid contribution by formalizing and operationalizing the evaluation of personalized agent behavior—a capability that is increasingly critical but poorly measured. The benchmark design is thoughtful, the experimental coverage is broad, and the findings are actionable. The main limitations are typical of benchmark papers (synthetic data, limited domains, evaluator validity). The paper's impact will depend on adoption, which is supported by the code release and extensible design.
Generated May 27, 2026
Comparison History (23)
Paper 2 likely has higher impact due to broader, timely relevance: long-term personalization and proactivity are central bottlenecks for real-world LLM agents across products and research areas (memory, HCI, evaluation, alignment). A well-designed benchmark plus extensible memory interface can become community infrastructure, enabling standardized comparison and accelerating progress across many methods and models. Paper 1 is novel and practically motivated, but AI-text detection is narrower, more adversarial/fragile over time, and its impact may be limited by shifting generation models and policy constraints, despite the valuable explainability angle.
Paper 2 likely has higher impact: it introduces a timely, broadly useful benchmark targeting a major real-world bottleneck (personalized, proactive long-term agents), with an extensible memory interface enabling systematic, reproducible comparisons across architectures. Benchmarks often catalyze community progress across many subfields (agent design, memory, HCI, evaluation). Paper 1 is innovative methodologically for spatial reasoning, but its impact is narrower (spatial planning tasks) and may depend on robustness/generalization of a specific training/RL scheme. Paper 2’s applicability and cross-field relevance are wider.
Paper 1 addresses a fundamental bottleneck in AI agent development—long-term personalization and proactivity—by providing a comprehensive benchmark. Benchmarks in nascent areas like agentic memory typically drive broad follow-up research across the AI community. While Paper 2 offers a highly practical architecture for speech translation, Paper 1 has broader applicability across the entire LLM agent ecosystem and will likely shape evaluation standards for future human-AI interaction models.
Paper 1 pairs a new, general framework (verification-based filtering of explanation claims against faithful tools) with an open-world RL benchmark explicitly targeting model-specific faithfulness—addressing a timely, high-stakes failure mode of LLM-mediated XAI (plausible but unfaithful explanations). The methodological contribution is actionable and could influence both XAI system design and evaluation standards across domains where faithful explanations matter. Paper 2 provides a valuable benchmark for personalization/proactiveness, but is primarily evaluative; its impact hinges more on adoption. Overall, Paper 1 offers broader cross-field methodological leverage and a clearer path to changing practice.
Paper 2 addresses a fundamental and heavily debated question in AI and cognitive science: whether LLMs build internal world models. Its highly rigorous methodology—featuring a six-level hierarchy, multilingual evaluation, and human baselines—provides profound theoretical insights into LLM working memory and spatial reasoning. While Paper 1 offers a valuable benchmark for practical agent development, Paper 2's focus on foundational model capabilities and its broader implications across linguistics and cognitive science give it a higher potential for deep scientific impact.
Paper 2 provides fundamental scientific insights into how human-like perceptual representations emerge in text-only models, bridging AI interpretability and cognitive science. While Paper 1 offers a valuable benchmark for agent development, Paper 2's findings on the transient nature of perceptual geometry in neural representations offer deeper theoretical implications for understanding LLM cognition and cross-modal learning, giving it broader interdisciplinary impact.
Paper 2 identifies a critical and counterintuitive flaw in Chain-of-Thought distillation—that final answer accuracy can improve while reasoning step factuality degrades. This challenges fundamental assumptions in LLM evaluation and distillation methodologies, especially in high-stakes domains like medicine. While Paper 1 introduces a valuable benchmark for agent personalization, Paper 2's findings have broader theoretical and safety implications for how the field assesses and trusts LLM reasoning traces, giving it higher potential scientific impact.
Paper 1 likely has higher scientific impact due to a novel, broadly applicable detection signal (entropy distribution shape/tails) plus strong methodological rigor (hypothesis-testing framing, finite-sample calibration via a new random-length DKW inequality, exponential-consistency guarantees). It offers a practical single-pass, black-box method with cross-model/task comparability and clear deployment value for hallucination mitigation across many LLM applications. Paper 2 is timely and useful as a benchmark for personalization/proactivity, but benchmarks typically have narrower cross-field impact and less foundational contribution than a theoretically grounded, general-purpose reliability method.
Paper 2 likely has higher scientific impact because it introduces a timely, broadly applicable benchmark targeting a major real-world bottleneck: long-term personalization and proactive behavior in LLM agents. Benchmarks often become community standards, shaping evaluation practices across academia and industry, and its extensible memory interface can catalyze method development across agent, memory, and HCI research. Paper 1 is innovative and methodologically interesting (mechanistic + difficulty-aware RLVR), but its impact is narrower (RLVR training dynamics) and may depend on adoption within a smaller subcommunity.
Paper 1 tackles the complex, real-world challenge of long-term, proactive, and personalized human-agent interaction, a critical bottleneck for deploying AI assistants. While Paper 2 provides rigorous diagnostic testing for memory systems, Paper 1's focus on realistic temporal tasks, heterogeneous interactions, and proactive behavior offers a more comprehensive framework with higher potential to drive next-generation agent architectures and broad real-world applications.
Paper 1 has higher likely scientific impact due to stronger timeliness and broader relevance: evaluating long-term personalization and proactive behavior is a central bottleneck for deploying LLM agents across many domains (assistants, healthcare, education, productivity). Its benchmark design (temporally ordered interactions, fragmented preference signals, proactiveness tests) and extensible memory interface can standardize evaluation and drive model/system research. Paper 2 is methodologically solid and highly applicable to e-commerce/spec QA, but its impact is more domain-specific and aligns with an active, already-crowded RAG/structured QA direction.
Paper 2 likely has higher scientific impact: it introduces a novel, generalizable pipeline that converts clinical guidelines into executable logic to generate factual/counterfactual supervision, directly improving model reliability in a high-stakes domain. The approach has clear real-world applicability (clinical decision support), strong methodological signals (multi-benchmark gains plus physician evaluation), and timely relevance given deployment pressures for medical LLMs. Paper 1 is valuable as benchmarking infrastructure for personalization/proactiveness, but its impact is more indirect (evaluation-focused) and may be narrower unless it becomes a widely adopted standard.
Paper 2 likely has higher scientific impact due to broader cross-domain relevance and timeliness: benchmarking personalized, proactive long-term agents is a central bottleneck for real-world LLM deployment. VitaBench 2.0 provides a reusable evaluation framework, tasks, and an extensible memory interface enabling controlled comparisons across architectures—assets that can be adopted widely by the community and influence model development. Paper 1 is novel and valuable but more domain-specific (legal indicators; French marine environmental law corpus) and may have narrower immediate uptake outside legal NLP.
Paper 1 likely has higher scientific impact due to its broader, timely relevance to LLM agent evaluation and deployment. VitaBench 2.0 targets a major open problem—long-term personalization and proactivity—providing an extensible benchmark and memory interface that can standardize comparisons across architectures and influence many subfields (agent design, memory, HCI, evaluation). Its potential applications span consumer agents, enterprise assistants, and safety/alignment testing. Paper 2 is methodologically solid with clear real-world value for batteries, but its impact is more domain-specific and its architectural novelty is narrower than a widely adoptable benchmark for the fast-moving LLM ecosystem.
Paper 1 tackles a highly timely and broadly applicable challenge—evaluating LLM agents in long-term, personalized interactions. Its focus on proactive and personalized agent behavior aligns with the rapid shift towards real-world AI assistants, offering broader impact across both academia and industry compared to the more specialized knowledge graph focus of Paper 2.
Paper 2 addresses long-term memory, personalization, and proactive behavior in AI agents, which are fundamental and widely applicable challenges across all human-AI interaction domains. While Paper 1 is highly rigorous, its focus is more narrowly constrained to mobile GUI navigation in Chinese applications. Paper 2's focus on the next frontier of agent capabilities—proactivity and long-term user alignment—gives it higher potential for broad scientific impact and general real-world application.
Paper 1 addresses the rapidly expanding and highly relevant field of LLM agents, focusing on the critical challenges of personalization, proactiveness, and long-term memory. Benchmarks in this domain typically drive significant downstream research and have a broad cross-disciplinary impact. In contrast, while Paper 2 provides a strong methodological advance, its focus on vehicle routing problems is more specialized, resulting in a narrower potential impact compared to the widespread applicability of interactive AI agents.
Paper 1 (TC-WM) presents a novel theoretical and empirical framework addressing a fundamental challenge in world modeling for reinforcement learning — bridging foundation model representations with task-centric planning. It offers theoretical guarantees (identifiability of latent factors), a principled architectural design, and demonstrates improvements across multiple benchmarks. Its contributions span representation learning, planning, and control, with broad applicability. Paper 2 (VitaBench 2.0) is a valuable benchmark contribution for personalized agents, but benchmarks typically have narrower methodological impact compared to new frameworks with theoretical foundations and demonstrated empirical gains across diverse domains.
Paper 1 presents a novel interdisciplinary framework integrating virtual water accounting into power system dispatch optimization using differentiable optimization layers—a methodologically innovative approach addressing the critical water-energy nexus for data centers. It offers concrete, quantifiable real-world benefits (3-5% freshwater withdrawal reductions) and bridges multiple fields (power systems, computing, water resources). Paper 2, while timely, is primarily a benchmark contribution for LLM agent evaluation—important but incremental in nature, with impact largely confined to the NLP/AI community and dependent on the evolving LLM landscape.
Paper 2 is likely to have higher impact: it introduces a timely benchmark targeting a central, broadly relevant gap in LLM agents—personalization and proactiveness over long-term interactions—with an extensible memory interface enabling systematic comparisons across architectures. Benchmarks often shape research directions across NLP, HCI, agent systems, and evaluation methodology, and can be widely adopted by both academia and industry. Paper 1 is innovative and rigorous for geometry-conditioned buildable brick generation, but its applications and audience are narrower (3D/graphics/robotic assembly), likely limiting breadth of downstream influence.