What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct
Meryl Ye, Lujain Ibrahim, Jessica Y. Bo, Myra Cheng, Ida Mattsson, Daniel Vennemeyer, Robert Kraut, Steve Rathje
Abstract
AI sycophancy has become a prominent concern in large language model (LLM) research. Yet the term lacks a consistent definition and has been applied to behaviors ranging from agreeing with a user's false claim to excessively praising the user to withholding corrective feedback. When researchers, companies, and policymakers use the same term to describe different behaviors, evaluation results become difficult to compare, mitigation strategies fail to transfer, and systems that are resistant to one form of sycophancy continue exhibiting other forms. To address this, we make two contributions. First, we reviewed 70 papers on AI sycophancy to develop a taxonomy of how the behavior has been defined and measured. The taxonomy distinguishes (1) whether a model is sycophantic toward a user's positions and beliefs, or toward the user's broader personal traits and emotions, and (2) whether this occurs through explicit, direct language or more implicit, subtle behaviors such as framing, omission, or tone. Mapping existing literature to our taxonomy reveals that current research has focused on overt forms of sycophancy toward users' beliefs, leaving more subtle and person-directed behaviors relatively understudied. Second, we surveyed 106 experts in AI sycophancy and related fields to examine whether researchers agree on which model behaviors are sycophantic. While experts are nearly unanimous in believing that sycophancy is a significant problem in current AI systems (94.3% agree), they disagree substantially on which specific behaviors qualify. Together, these findings demonstrate that AI sycophancy is a broad family of behaviors with different measurement challenges, intervention requirements, and governance implications. Our taxonomy provides a shared vocabulary for understanding and addressing these behaviors.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper tackles the conceptual fragmentation surrounding "AI sycophancy" — a term used inconsistently across research, industry, and policy. The authors make two contributions: (1) a taxonomy of sycophantic behaviors organized along two dimensions (Referent: Position vs. Person; Explicitness: Explicit vs. Implicit), derived from a review of 70 papers, and (2) an expert survey (N=106) demonstrating that while researchers nearly universally agree sycophancy is a significant problem (94.3%), they disagree substantially on which specific behaviors qualify. The key empirical finding is a significant Referent×Explicitness interaction: Position-directed sycophancy is recognized regardless of how it's expressed, while Person-directed sycophancy is only recognized when explicit (e.g., flattery), with implicit forms (e.g., deferential tone, softened feedback) receiving near-neutral ratings.
This is fundamentally a construct clarification paper — a "meta-scientific" contribution that aims to organize and discipline a rapidly expanding research area. Such papers can be highly impactful when they arrive at the right moment and provide vocabulary that subsequent work adopts.
Methodological Rigor
The literature review covers 70 papers with a clear inclusion criterion and systematic annotation procedure. Inter-rater reliability for taxonomy assignment was substantial (88.3% agreement), with disagreements adjudicated via an LLM third coder — a pragmatic if somewhat novel approach. The annotation of survey items by four authors across seven dimensions showed acceptable to good ICC values (.47–.86).
The expert survey design is competent: 24 items on a bipolar 7-point scale, randomized blocks, multilevel regression with crossed random effects for respondents and items. The statistical analysis is thorough, with appropriate model comparisons via likelihood ratio tests and extensive robustness checks (dichotomized DV, collapsed negative pole, moderator analyses by demographics and research area).
Some methodological limitations deserve note. The bipolar scale created interpretive ambiguity — five participants reported post-survey confusion about the negative pole. The robustness analysis showed the primary interaction becomes non-significant when negative ratings are collapsed to zero (p=.108), suggesting the effect partially depends on how non-sycophantic exemplars are scored. The CFA models showed uniformly poor fit (CFI<.65, RMSEA>.10), indicating the taxonomy dimensions don't correspond to stable latent factors in expert judgments — the authors acknowledge this but it somewhat undermines claims about the taxonomy reflecting perceptual structure in expert ratings.
The expert sample, while reasonable in size (N=106), is heavily skewed toward US-based academics (67.9% US, 84% academia), limiting generalizability of "expert consensus" claims. The 47/106 respondents being authors of reviewed papers creates potential circularity — they may rate behaviors consistent with their own published operationalizations.
Potential Impact
The paper's impact potential lies in several domains:
Research standardization: The taxonomy provides a shared vocabulary that could discipline future benchmark development. The demonstration that SycEval and ELEPHANT produce inverted model rankings because they measure different taxonomy cells is a compelling illustration of why this matters. If adopted, this framework could reduce the frequency of apples-to-oranges comparisons in model evaluation.
Industry governance: The analysis of OpenAI and Anthropic model specifications through the taxonomy lens reveals specific gaps (particularly implicit behaviors). This provides actionable vocabulary for companies drafting safety policies. The mapping of evaluation paradigms to taxonomy cells (Table 6) is a practical resource.
Legislative clarity: The analysis of California SB 1119 and similar legislation, which define sycophancy by downstream consequences without specifying behaviors, highlights a gap that the taxonomy could help fill.
Mitigation strategy design: The finding from Vennemeyer et al. (2025) that sycophantic agreement and praise are independently steerable aligns with the taxonomy's cell distinctions and suggests mitigation must be cell-specific.
Timeliness & Relevance
This paper is exceptionally well-timed. AI sycophancy has become a prominent concern in both academic and public discourse (documented by the Google Trends data in Figure 1), with companies making public commitments to reduce it and legislators writing it into bills. The field is at precisely the stage where construct clarification can have maximal impact — early enough that vocabulary isn't yet ossified, but late enough that fragmentation is creating real problems for comparison and accountability. The reference to the April 2025 GPT-4o sycophancy incident and GPT-5 evaluation discrepancies demonstrates immediate practical relevance.
Strengths
1. Comprehensive literature mapping: The 70-paper review with per-paper taxonomy cell assignments (Table 5) is itself a valuable reference resource, alongside the evaluation paradigm mapping (Table 6).
2. Empirical grounding of disagreement: Rather than simply proposing a taxonomy, the authors demonstrate empirically that experts disagree along its dimensions, lending the framework evidential weight.
3. Cross-domain analysis: The paper connects academic, corporate, and legislative uses of the term, demonstrating fragmentation's practical consequences across all three.
4. Actionable structure: The taxonomy is simple enough (2×2 with sub-referents) to be readily adopted while capturing meaningful distinctions.
5. Strong supplementary materials: The appendices are unusually thorough, including the full survey instrument, all 70 paper annotations, and extensive robustness analyses.
Limitations
1. Taxonomy validation gap: The poor CFA fit and low within-quadrant reliability suggest the taxonomy captures researcher intuitions imperfectly. The dimensions may organize the literature better than they organize expert judgments.
2. Sample representativeness: The US-academic skew and overlap with reviewed-paper authors limits confidence in generalizability.
3. Descriptive rather than prescriptive: The paper deliberately avoids defining what sycophancy *should* mean, which limits its power to resolve fragmentation rather than merely describe it.
4. No behavioral data: All findings rest on expert ratings of behavior descriptions rather than responses to actual model outputs, which introduces abstraction that may not map to real evaluation contexts.
5. Implicit behaviors under-theorized: The paper identifies that implicit person-directed sycophancy is under-recognized but doesn't fully explain why — is this because these behaviors are genuinely less harmful, harder to detect, or simply less familiar to the research community?
Overall Assessment
This is a well-executed construct clarification paper arriving at an optimal moment for a rapidly growing but conceptually disorganized research area. Its primary value is organizational rather than empirically novel — it provides shared vocabulary and demonstrates that the absence of such vocabulary creates real problems. The expert survey adds empirical texture but is methodologically bounded. Impact will depend on whether the community adopts the taxonomy's vocabulary in subsequent benchmarking and policy documents.
Generated May 22, 2026
Comparison History (24)
Paper 2 likely has higher scientific impact due to its methodological and practical contributions: it provides systematic experiments across workflows, tasks, and model scales, plus mechanistic explanations (gradient dynamics/topology) for observed instabilities. These results can directly guide design of multi-agent RL LLM systems, a timely and widely used paradigm, with applicability across math/code and broader workflow engineering. Paper 1 offers valuable conceptual clarification and governance relevance, but is primarily a taxonomy/survey with less direct algorithmic leverage and narrower immediate downstream technical utility.
Paper 2 addresses a fundamental conceptual problem in LLM research—the lack of a consistent definition of sycophancy—through a rigorous taxonomy and large expert survey (106 experts, 70 papers reviewed). Its impact is broader: it provides shared vocabulary applicable across AI safety, alignment, evaluation, and governance. The construct clarification will influence how future research measures and mitigates sycophancy. Paper 1, while technically interesting, addresses a narrower problem (harness evolution in specific game domains) with more limited generalizability and a smaller potential audience.
Paper 1 addresses a critical and fundamental challenge in AI alignment and safety by unifying a fragmented construct. Its taxonomy and survey will likely serve as a foundational reference for future research, evaluation, and policy across multiple disciplines. While Paper 2 offers a valuable technical optimization for LLM routing, Paper 1's conceptual contributions have broader, longer-lasting implications for how AI systems are evaluated and governed.
Paper 1 addresses a foundational conceptual problem affecting the entire LLM alignment and safety research community. By providing a taxonomy and expert survey on AI sycophancy, it creates shared vocabulary that can standardize evaluation, mitigation, and governance across research groups and policy. Its breadth of impact spans AI safety, policy, and industry practice. Paper 2, while methodologically solid, addresses a more narrow technical gap in VLM spatial reasoning. Paper 1's contributions are more likely to be widely cited and shape future research directions across multiple subfields.
Paper 2 addresses a highly timely and critical bottleneck in modern AI: the efficiency of test-time scaling and long-form reasoning. By introducing a training-free decoding controller (PathCal) that leverages reflection markers to improve performance-efficiency trade-offs, it offers an immediate, practical application for developing and optimizing reasoning models (like o1). While Paper 1 provides valuable conceptual clarity for AI safety, Paper 2's methodological innovation and direct impact on core LLM reasoning capabilities give it a broader and more immediate technical impact.
Paper 2 likely has higher impact: it proposes a concrete, novel training modification (guided rollouts + evidence-masked updates) addressing a timely, practical failure mode in on-policy self-distillation—transferring privileged context without unwanted side effects. This can directly influence post-training pipelines for LLMs (alignment, personalization, tool/solution distillation) and is broadly applicable across model training and RLHF-like paradigms, with empirical comparisons and ablations indicating methodological rigor. Paper 1 offers valuable conceptual clarity and governance relevance, but its taxonomy/survey nature typically yields less direct technical leverage on model performance and deployment.
Paper 2 addresses a fundamental conceptual gap in LLM research by providing a taxonomy and shared vocabulary for AI sycophancy—a cross-cutting concern affecting alignment, safety, and governance. Its breadth of impact spans multiple fields (AI safety, policy, HCI), and its contributions (taxonomy from 70 papers, survey of 106 experts) offer a foundational framework that will shape future research directions. Paper 1 makes a valuable but narrower contribution to clinical LLM evaluation. While important, its findings (that interactive settings reduce LLM accuracy) are somewhat expected, whereas Paper 2's conceptual clarification has broader, more lasting influence on the field.
Paper 2 demonstrates a groundbreaking application of LLMs to solve open mathematical problems (including 9 Erdős problems), representing a concrete, verifiable advance in mathematical research. Its impact spans multiple mathematical subfields and establishes a new paradigm for AI-assisted theorem proving. While Paper 1 provides a useful taxonomy and survey of AI sycophancy—an important conceptual contribution—Paper 2's demonstrated ability to solve previously unsolved problems represents a more transformative scientific contribution with immediate, measurable real-world impact on mathematics research.
Paper 2 provides a tangible technological advancement by solving visual-semantic bottlenecks in LLM chemistry applications. It introduces a novel framework (ChemVA) and dataset (OCRD-Bench) that directly accelerate automated chemical research and discovery, bridging AI and hard sciences. While Paper 1 offers valuable conceptual clarity for AI alignment, Paper 2's methodological rigor, quantifiable performance gains, and direct utility for applied scientific innovation give it a higher potential for broad, transformative impact.
Paper 2 addresses AI sycophancy, a widely recognized and timely concern across LLM research, policy, and industry. Its taxonomy based on 70 papers and survey of 106 experts provides a foundational conceptual framework that can shape future research directions, evaluation standards, and governance. It has broad cross-disciplinary relevance (AI safety, alignment, HCI, policy). Paper 1 addresses a narrower, more technical issue (KG construction from CSV tables) with useful but domain-specific contributions. Paper 2's potential to standardize terminology and redirect research efforts gives it significantly broader scientific impact.