IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures
David Gringras
Abstract
Ask a frontier model how to taper six milligrams of alprazolam (psychiatrist retired, ten days of pills left, abrupt cessation causes seizures) and it tells her to call the psychiatrist she just explained does not exist. Change one word ("I'm a psychiatrist; a patient presents with...") and the same model, same weights, same inference pass produces a textbook Ashton Manual taper with diazepam equivalence, anticonvulsant coverage, and monitoring thresholds. The knowledge was there; the model withheld it. IatroBench measures this gap. Sixty pre-registered clinical scenarios, six frontier models, 3,600 responses, scored on two axes (commission harm, CH 0-3; omission harm, OH 0-4) through a structured-evaluation pipeline validated against physician scoring (kappa_w = 0.571, within-1 agreement 96%). The central finding is identity-contingent withholding: match the same clinical question in physician vs. layperson framing and all five testable models provide better guidance to the physician (decoupling gap +0.38, p = 0.003; binary hit rates on safety-colliding actions drop 13.1 percentage points in layperson framing, p < 0.0001, while non-colliding actions show no change). The gap is widest for the model with the heaviest safety investment (Opus, +0.65). Three failure modes separate cleanly: trained withholding (Opus), incompetence (Llama 4), and indiscriminate content filtering (GPT-5.2, whose post-generation filter strips physician responses at 9x the layperson rate because they contain denser pharmacological tokens). The standard LLM judge assigns OH = 0 to 73% of responses a physician scores OH >= 1 (kappa = 0.045); the evaluation apparatus has the same blind spot as the training apparatus. Every scenario targets someone who has already exhausted the standard referrals.
AI Impact Assessments
(3 models)Scientific Impact Assessment: IatroBench
Core Contribution
IatroBench introduces a benchmark that measures a previously uninstrumented failure mode of LLM safety training: omission harm — the clinical damage caused when models withhold knowledge they demonstrably possess. The paper's central construct is the "decoupling eval," a matched-framing design where identical clinical content is presented in layperson versus physician framing, isolating identity-contingent withholding from capability limitations. The benchmark comprises 60 pre-registered clinical scenarios scored on dual axes (commission harm 0–3, omission harm 0–4) with acuity weighting, tested across six frontier models with 3,600 total responses.
The conceptual framing is the paper's strongest intellectual contribution: it draws a structural analogy between defensive medicine (where asymmetric malpractice liability drives unnecessary testing) and defensive AI (where asymmetric training penalties drive unnecessary refusal). The Goodhart's Law framing — that safety benchmarks measuring only commission harm create optimization pressure that systematically increases omission harm — is well-articulated and empirically grounded.
Methodological Rigor
Strengths. The pre-registration on OSF before Phase 2 data collection is commendable and unusual for AI evaluation work. The paper is transparent about hypothesis dispositions: H3 (safety-training rank predicts decoupling) and H5 (aggregate collision-type differences) are explicitly reported as unsupported, and the authors resist post-hoc reinterpretation. The specification curve analysis across 18 analytic paths strengthens robustness. The binary critical-action hit rate analysis (Table 5) is the most convincing evidence: the 13.1 pp drop on safety-colliding actions in layperson framing versus 1.7 pp on non-colliding actions (p < 0.0001 vs. p = 0.54) provides scorer-independent evidence of selective withholding.
Weaknesses. The most significant methodological concern is that Claude Opus serves as both a tested model and the primary structured evaluator. The authors provide four mitigations (physician validation, multi-judge concordance, binary hit rates, rubric provenance), and the κ_w = 0.571 against physician scoring is reasonable, but the self-evaluation confound remains non-trivial — particularly for H2, where bias would inflate the gap in the hypothesized direction. The Opus-excluded sensitivity analysis (+0.27 with Gemini Flash scoring, still significant) partially addresses this.
The physician validation (N=100, two raters) is limited in scale. The raw κ = 0.375 is below conventional thresholds, though the authors reasonably argue that κ is compressed by distributional clustering and that within-1 agreement (96%) better reflects practical reliability. Gold-standard responses authored by a single physician, while validated against published guidelines, introduce potential bias in what counts as "complete."
The physician-vs-layperson framing confounds several variables: register, displayed competence, question specificity, and identity claim. The authors acknowledge this and conduct a post-hoc probe with lawyer and informed-layperson framings that partially deconfounds — finding that any professional or knowledge signal unlocks engagement — but this is exploratory (5 pairs, N=592).
Potential Impact
Immediate practical impact. The finding that standard LLM judges assign OH=0 to 73% of responses that structured evaluation scores OH≥1 (κ = 0.045) has direct implications for every lab running RLHF pipelines. If the evaluation apparatus shares the training apparatus's blind spot, the self-reinforcing cycle (Figure 3) means omission harm cannot be detected or corrected from within. This is actionable: labs could integrate omission-sensitive evaluation into training loops.
Benchmark contribution. The dual-axis scoring framework (commission + omission, with acuity weighting) fills a genuine gap. Existing benchmarks (TruthfulQA, XSTest, OR-Bench, HarmBench) all measure variants of commission harm or treat over-refusal as uniform-cost UX failures. IatroBench's acuity weighting means a refused taper and a refused joke carry different costs — an obvious improvement.
Policy relevance. The paper documents that safety policies are structurally regressive: users who signal limited access to professional care (the primary LLM medical advice demographic per cited surveys) receive the least clinical value. This has implications for AI regulation, health equity policy, and liability frameworks.
Alignment implications. The paper provides empirical content for a theoretical concern: safety optimization on a measurable proxy that misaligns on unmeasured objectives and goes undetected by evaluation. Medicine is a useful test domain precisely because ground truth is verifiable; the paper correctly notes that this dynamic likely exists in domains without independent verification.
Timeliness & Relevance
This paper addresses a current bottleneck at the intersection of AI safety and deployment. As LLMs are increasingly used for health information (particularly by underserved populations), the tension between safety-trained refusal and clinical necessity is urgent. The paper arrives as OpenAI shifts toward "safe completions" and Anthropic's constitution explicitly ranks helpfulness above harmlessness when withholding causes harm — making the gap between stated principles and deployed behavior a live issue.
Strengths & Limitations
Key strengths:
Notable limitations:
Overall assessment. This is a well-executed, conceptually sharp paper that identifies and measures a genuine blind spot in AI safety evaluation. The core finding — identity-contingent withholding of clinical knowledge — is robustly supported by multiple evidence lines. The judge-miscalibration finding has potentially broader methodological implications. The paper's main limitation is the self-evaluation design, partially but not fully mitigated. The work should catalyze dual-axis evaluation in safety benchmarks and force explicit consideration of omission costs in RLHF training.
Generated Apr 10, 2026
Comparison History (279)
Paper 1 introduces a field-defining foundation model trained on an unprecedented scale of wearable data (1 trillion minutes, 5M participants). Its broad applicability across 35 health prediction tasks and integration with LLM agents represent a massive leap in personalized digital health. While Paper 2 provides a crucial and timely critique of AI safety mechanisms, Paper 1's sheer scale, novelty in multimodal physiological representation, and potential to catalyze widespread downstream clinical applications give it a higher overall scientific and real-world impact.
Paper 2 likely has higher scientific impact due to its large-scale foundation model (trillion-minute pretraining, 5M participants), broad validation across 35 diverse tasks, and clear pathway to real-world deployment via a Personal Health Agent with clinician-rated evaluation. Its methodological scope (scaling laws, few-shot/label efficiency, generative metric estimation, automated head search) and applicability across many health domains suggest wide cross-field influence. Paper 1 is novel and timely in highlighting iatrogenic harm from safety behaviors, but its narrower domain, smaller scale, and primarily evaluative nature limit breadth and downstream adoption.
Paper 1 likely has higher scientific impact due to broader cross-disciplinary scope and foundational contribution: a temporally grounded, controlled-knowledge benchmark (CUSP) for forecasting scientific progress across thousands of events. It addresses a general, timely question about AI’s role in science and provides reusable evaluation infrastructure with implications for scientometrics, AI evaluation, and research policy. Paper 2 is novel and important for AI safety/clinical deployment, but its domain is narrower (clinical safety framing effects) and may have more constrained breadth despite strong pre-registration and methodology.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm—with a rigorous pre-registered methodology that reveals a striking, actionable finding: identity-contingent knowledge withholding. Its clear demonstration that safety guardrails systematically harm vulnerable users who have exhausted standard referrals has immediate policy implications for AI deployment in healthcare. The finding that evaluation tools share the same blind spots as training systems is also deeply consequential. While Paper 1 (CUSP) is methodologically sound and interesting, its conclusions are largely negative (AI can't forecast science well), offering less actionable insight. Paper 2's specific, reproducible failure modes will likely drive concrete changes in AI safety design and regulation.
Paper 2 addresses a critical, life-altering flaw in current AI safety paradigms: 'safety' measures causing omission harm in medical crises. Its methodology is highly rigorous, utilizing pre-registered clinical scenarios and physician-validated scoring. While Paper 1 provides a valuable benchmark for AI research agents, Paper 2 has profound implications for global AI safety policy, medical AI deployment, and human health, giving it a significantly higher potential for broad, interdisciplinary real-world impact.
Paper 1 offers a broadly applicable, technically novel safety architecture (evidence-carrying authorization with typed certificates and deterministic gating) addressing a growing multimodal agent risk: hallucination-to-action. It demonstrates strong methodological rigor (formalization, verifier red-teaming at scale, quantified bypass reduction, end-to-end unsafe-action rates with confidence bounds, replay sanity checks) and has wide impact across agentic AI, security, HCI, and systems. Paper 2 is timely and important for clinical safety evaluation, but its impact is narrower (healthcare framing/policy) and more observational than architectural; it also depends on scenario design and scoring subjectivity despite preregistration.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm—with a rigorous pre-registered methodology, clear policy implications, and broad societal relevance. It reveals a systematic flaw (identity-contingent withholding) affecting frontier models deployed to millions, with direct real-world health consequences. The finding that safety measures can paradoxically harm vulnerable users who have exhausted standard referrals challenges fundamental assumptions in AI alignment. TRACE is technically strong but addresses a narrower problem (hallucination correction) in a crowded space. IatroBench's findings are likely to influence AI safety policy, medical AI deployment, and regulatory frameworks across multiple fields.
Paper 2 has higher estimated scientific impact because it addresses an urgent, high-stakes real-world domain (clinical safety) with immediate policy and deployment implications. Its pre-registered design, sizable response set, validated scoring against physicians, and identification of distinct failure modes make the evidence actionable for model training, safety engineering, and regulation, with broad impact across AI safety, evaluation science, and healthcare. Paper 1 is methodologically strong and novel for LLM-driven discovery, but its impact is more specialized to automated research/program synthesis and likely slower to translate into societal outcomes.
IatroBench addresses a critical and timely problem—AI safety measures causing measurable harm through identity-contingent knowledge withholding—with rigorous pre-registered methodology across frontier models. Its findings have immediate, high-stakes real-world implications for AI deployment in healthcare, AI safety policy, and model evaluation practices. The discovery that safety filters systematically harm vulnerable users who have exhausted standard referrals challenges fundamental assumptions in AI alignment. While Formal Conjectures is a valuable benchmark for mathematical reasoning, IatroBench's cross-disciplinary impact (AI safety, healthcare, policy, evaluation methodology) and its challenge to prevailing safety paradigms give it broader and more urgent scientific significance.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm through identity-contingent withholding of medical knowledge. Its rigorous pre-registered methodology across 6 frontier models with 3,600 responses, validated evaluation pipeline, and identification of three distinct failure modes has broad implications for AI safety policy, healthcare AI deployment, and alignment research. It challenges fundamental assumptions about how safety training is implemented. VibeServe is a solid systems contribution but addresses a narrower engineering optimization problem with less paradigm-shifting potential.
IatroBench introduces a novel, pre-registered benchmark exposing a fundamental and previously unquantified problem: AI safety measures causing iatrogenic harm through identity-contingent withholding. This challenges core assumptions in AI safety alignment and has immediate policy implications for how frontier models are deployed in healthcare. The finding that safety filters systematically harm the most vulnerable users (those who have exhausted referrals) is paradigm-shifting. Paper 2, while impressive in scale and practical value for symptom assessment, represents incremental progress in a well-studied area (AI-assisted diagnosis). IatroBench's cross-cutting implications for AI safety, medical ethics, and evaluation methodology give it broader and deeper impact.
Paper 1 has higher impact potential: it introduces a concrete, pre-registered benchmark for a societally critical failure mode (identity-contingent omission harm) with immediate safety, policy, and deployment implications for frontier medical advice systems. Its methodology includes multi-model evaluation, clear harm taxonomies, and validation against physician ratings, and it surfaces a tooling blind spot (LLM judges miss omission harm) that could affect many evaluations. Paper 2 is novel and broadly relevant for interpretability, but its downstream real-world consequences are less immediate and its utility depends on adoption in research workflows.
Paper 2 introduces a novel, generalizable methodology for understanding LLM reasoning by extracting search trees from CoT traces, revealing a fundamental insight about LLM planning (myopic despite deep traces). This has broad implications across AI interpretability, cognitive science, and alignment research. While Paper 1 (IatroBench) addresses an important and timely safety concern about identity-contingent withholding, its impact is more narrowly focused on AI safety policy and medical applications. Paper 2's methodological contribution and its fundamental insight about the gap between LLM deliberation and action have wider scientific reach and are more likely to influence multiple research directions.
Paper 2 has higher potential impact due to stronger novelty and cross-field relevance: it introduces a pre-registered, clinically grounded benchmark for iatrogenic harm caused specifically by safety interventions (identity-contingent withholding), with statistically supported findings and validation against physician ratings. The results directly challenge current alignment/safety paradigms and evaluation practices (LLM-judge blind spots), with immediate implications for deployment in healthcare and beyond. Paper 1 is practically valuable engineering for tool-call safety, but is narrower (agent tooling/security) and more incremental relative to existing runtime policy/guard systems.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm by withholding life-saving medical information based on user identity rather than clinical need. Its pre-registered methodology across 6 frontier models with 3,600 responses, identification of three distinct failure modes, and exposure of systematic evaluation blind spots have broad implications for AI safety policy, healthcare AI deployment, and alignment research. Paper 2 makes a useful engineering contribution to LLM routing but addresses a narrower optimization problem with incremental improvements. Paper 1's findings challenge fundamental assumptions in AI safety practices and could reshape how safety guardrails are designed across the industry.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm through identity-contingent withholding of medical knowledge. Its pre-registered empirical methodology across 6 frontier models with 3,600 responses, identification of distinct failure modes, and demonstration that standard LLM evaluation shares the same blind spots as training, has immediate policy and engineering implications for AI safety alignment. It challenges prevailing AI safety paradigms with concrete evidence of real-world harm. Paper 2 offers theoretical insight into shortcut learning via evolutionary game theory, which is intellectually interesting but more incremental and narrower in immediate practical impact.
Paper 2 has higher potential impact due to strong novelty and timeliness: it introduces a pre-registered benchmark quantifying safety-measure iatrogenic harm via identity-contingent withholding, a critical and under-measured failure mode in deployed frontier models. Its methodology is comparatively rigorous (pre-registration, multi-model evaluation, physician-validated scoring, statistical testing) and the findings generalize broadly to AI safety, evaluation, alignment, and high-stakes deployment beyond medicine. Paper 1 is solid and applicable for on-device agents, but its contributions are more incremental within memory/compression and likely narrower in cross-field and societal impact.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm by withholding medical knowledge based on user identity rather than clinical need. It introduces a novel, pre-registered benchmark with rigorous methodology (60 scenarios, 6 models, 3,600 responses, validated scoring pipeline), reveals systematic failure modes in frontier models, and exposes a fundamental flaw in LLM-judge evaluation. Its implications span AI safety policy, medical AI regulation, and model alignment research, with direct real-world consequences for patient safety. Paper 2, while useful, addresses a narrower engineering optimization problem (routing confidence estimation) with incremental contributions over existing baselines.
IatroBench addresses a critical and timely problem—AI safety measures causing iatrogenic harm by withholding medical knowledge based on user identity rather than clinical need. Its pre-registered methodology across 6 frontier models with 3,600 responses provides rigorous evidence of systematic failure modes with direct implications for AI policy, healthcare, and safety alignment. The finding that evaluation tools share the same blind spots as training is particularly impactful. While Paper 2 offers valuable theoretical insight into shortcut learning via evolutionary game theory, its impact is more incremental within the ML theory community. Paper 1's breadth of impact across AI safety, medicine, and policy gives it higher potential.
Paper 2 has higher potential impact: it introduces a timely, policy-relevant benchmark with pre-registered methodology and clinically grounded harm metrics that expose identity-contingent safety failures in frontier models—an issue affecting real-world deployment and regulation across AI safety, evaluation, and healthcare. Its findings generalize beyond medicine to any domain where models may withhold actionable guidance, and it also critiques evaluator/judge blind spots. Paper 1 is innovative and practical for on-device agent memory, but its contribution is more specialized to systems/memory compression and likely narrower in cross-field and societal impact.