Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Paper 2 identifies a critical, counter-intuitive inverse scaling phenomenon in high-stakes forecasting domains like epidemiology and finance. Discovering that increased model capabilities degrade performance on tail-risk predictions addresses urgent AI safety and reliability concerns. Its profound implications for real-world decision-making and LLM evaluation methodology give it a broader, more significant potential scientific impact than Paper 1's narrower focus on personality perception.

Paper 2 introduces a novel conceptual framework (Grounded Personality Reasoning) that addresses a fundamental question about whether MLLMs truly understand or merely pattern-match—a question with broad implications across AI safety, fairness, and deployment. Its contributions (new task formalization, large dataset, comprehensive evaluation with novel failure-mode metrics, and benchmarking 27 models) are methodologically rigorous and reveal a striking 'Prejudice Gap' finding. This has broader cross-field impact (social AI, psychology, fairness/bias) compared to Paper 1, which provides a valuable but more domain-specific benchmark for spreadsheet tasks in finance.

Paper 1 introduces a foundation model trained on an unprecedented scale (5 million participants) for wearable health data, addressing critical challenges in personalized medicine. Its broad evaluation across 35 health tasks and integration into a clinician-validated Personal Health Agent suggest immense real-world applications and transformative potential in healthcare. Paper 2, while valuable for MLLM evaluation and AI fairness, is a benchmarking study with a narrower scope and less direct societal impact compared to the population-scale health insights enabled by Paper 1.

Paper 2 introduces a novel task, a new dataset (MM-OCEAN), and comprehensive benchmarks evaluating a critical issue in contemporary AI: whether Multimodal LLMs genuinely reason or rely on superficial prejudice. Benchmark datasets and evaluations of AI reasoning/bias typically garner high citation rates and spur broad downstream research. In contrast, Paper 1 is a synthesis chapter reviewing AI applications in serious games, which, while valuable, lacks the empirical novelty, rigorous benchmarking, and broad applicability across the rapidly moving fields of AI safety and multimodal model development seen in Paper 2.

Paper 2 offers significantly higher scientific impact by introducing a novel dataset (MM-OCEAN), formalizing a new task, and comprehensively benchmarking 27 models. It addresses a critical flaw in AI evaluation—whether models truly reason or just rely on superficial prejudices. In contrast, Paper 1 presents an interesting approach to emotion recognition but relies on a highly limited case study of a single 245-second speech. Paper 2's robust methodology, new failure-mode metrics, and large-scale analysis give it much broader relevance and citation potential across AI alignment and computational social science.

While Paper 1 provides a valuable infrastructure benchmark for agentic workflows, Paper 2 tackles a fundamental scientific question regarding how multimodal models reason about human attributes. By exposing the 'Prejudice Gap' and shifting evaluation from superficial score prediction to grounded behavioral reasoning, it addresses critical issues in AI safety, fairness, and interpretability, promising broader interdisciplinary impact across AI, psychology, and HCI.

Paper 1 addresses a fundamental gap in MLLM evaluation—whether models truly understand personality or rely on superficial cues—introducing a novel task (GPR), a comprehensive dataset (MM-OCEAN), and revealing the striking 'Prejudice Gap' across 27 models. This exposes a deep issue (right answer, wrong reasoning) relevant to AI safety, trustworthiness, and social cognition broadly. Paper 2 contributes a useful benchmark for agentic delegation but addresses a narrower, more engineering-focused problem with less surprising findings. Paper 1's conceptual contribution and implications for responsible AI deployment give it broader and more lasting impact.

Paper 2 introduces a fundamentally new evaluation paradigm for MLLMs in social cognition, exposing a critical 'Prejudice Gap' where models appear to succeed without genuine understanding. It contributes a novel task formalization (GPR), a large-scale dataset (MM-OCEAN), and benchmarks 27 models with new failure-mode metrics. This has broader impact across AI safety, fairness, and human-AI interaction. Paper 1, while solid engineering work on reasoning efficiency, is more incremental—combining known techniques (DPO, content editing) for a well-studied problem of verbose reasoning in LLMs.

Paper 2 introduces a novel evaluation framework (Grounded Personality Reasoning) with a new dataset, benchmark, and failure-mode metrics that expose fundamental limitations in MLLMs' social reasoning. It evaluates 27 models and reveals a striking 'Prejudice Gap' with broad implications for AI safety and deployment in human-facing applications. While Paper 1 makes valuable contributions questioning chess LLM claims, its scope is narrower (chess domain) and its core finding (pattern-matching over understanding) is less surprising. Paper 2's contributions span AI evaluation methodology, social cognition, and responsible AI deployment, giving it broader cross-field impact.

Paper 2 likely has higher impact due to timeliness and broad applicability: privacy-preserving LLM agents are an immediate deployment concern across consumer, enterprise, and regulated domains. POLAR-Bench offers a clear, diagnostic evaluation framework (policy axes × attack strategies) with deterministic scoring and adversarial interaction, directly informing safety/privacy alignment and model selection—especially for widely used smaller open-weight models. Paper 1 is novel and valuable for grounded social cognition and bias diagnosis in MLLMs, but its application scope is narrower (personality inference) and less universally critical than privacy-utility trade-offs in agentic systems.

Paper 2 addresses a critical, universal bottleneck in the deployment of autonomous LLM agents: data privacy and adversarial robustness. While Paper 1 provides excellent insights into multimodal social cognition and bias, Paper 2's focus on privacy-utility trade-offs impacts virtually every domain where AI agents interact with third-party systems. Furthermore, its specific finding that smaller, on-device models are highly vulnerable to data leakage provides immediate, highly actionable value for the open-source and AI security communities, granting it broader and more urgent real-world impact.

Paper 1 has higher potential impact due to its broader, timelier framing of frontier AI evaluation: it challenges benchmark-centric paradigms and proposes open-world evaluations applicable across domains (agents, safety, governance, deployment readiness). The CRUX proposal and real deployment case study (shipping an iOS app) are likely to influence evaluation practice and policy beyond a single subfield. Paper 2 is methodologically strong with a valuable dataset/metrics for grounded personality perception in MLLMs, but its scope is narrower (social cognition/personality) and impacts fewer adjacent fields.

While Paper 1 provides a rigorous and valuable benchmark for a specific domain (personality perception), Paper 2 addresses a fundamental limitation in how frontier AI capabilities are currently measured across all domains. By advocating for and demonstrating 'open-world evaluations,' Paper 2 has a broader potential impact on AI safety, policy, and general capability tracking, shaping future methodologies for evaluating autonomous agents in real-world scenarios.

Paper 2 addresses a critical flaw in Multimodal Large Language Models—the 'Clever Hans' effect in personality perception—by introducing a novel task, dataset, and benchmark. Its exposure of the 'Prejudice Gap' has broad implications for AI safety, fairness, and human-computer interaction, likely driving significant future research. While Paper 1 provides a strong technical solution for a specific subfield (multi-task unlearning), Paper 2's focus on foundational MLLM evaluation offers wider immediate relevance and broader interdisciplinary impact.

Paper 2 has higher potential scientific impact due to its broader relevance across AI safety, fairness, and social cognition research. It introduces a novel conceptual framework (Grounded Personality Reasoning), a reusable benchmark dataset (MM-OCEAN), and reveals a fundamental limitation ('Prejudice Gap') in 27 MLLMs that has implications for any human-facing AI deployment. The findings challenge assumptions about MLLM reasoning capabilities broadly. Paper 1, while technically rigorous, addresses a narrower domain (AV scenario generation) with incremental improvements over existing methods.