Understanding helpfulness and harmless tension in reward models
Eshaan Tanwar, Pepa Atanasova
Abstract
Reward models are a key component of reinforcement learning from human feedback (RLHF), aligning language models toward both helpful and harmless behaviour. However, the internal mechanisms underlying these objectives and their conflicts remain poorly understood. We study alignment tension in reward models trained under helpfulness-only, harmlessness-only, and mixed-objective settings. We find that mixed-objective models often underperform single-objective models, indicating interference between objectives. Using activation-based methods, we identify neurons associated with each objective and study their functional roles via targeted ablations. We find that these neurons causally support their corresponding objectives while often negatively affecting the opposing one. We find that a substantial proportion of neurons are shared between helpfulness and harmlessness, and that these shared neurons exert a disproportionate influence on model behaviour, contributing to alignment tension. Additionally, our results provide insights and mechanistic interpretation into how alignment objectives are represented in reward models and why multi-objective alignment remains challenging, motivating future work on disentangled and controllable alignment methods.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper investigates the internal mechanisms behind the tension between helpfulness and harmlessness objectives in reward models (RMs) used for RLHF. The authors train three RM variants—helpful-only, harmless-only, and mixed-objective—and conduct both behavioral and mechanistic analyses to understand why jointly optimizing for both objectives leads to degraded performance. The key finding is that approximately 50% of neurons identified as important for helpfulness and harmlessness are *shared*, and these shared neurons exert disproportionate influence on model behavior, providing a mechanistic explanation for why multi-objective alignment is challenging. The paper reframes alignment tension not merely as a behavioral trade-off but as a *representational interference* problem arising from overlapping internal circuitry.
Methodological Rigor
The experimental design is reasonably sound but has notable limitations:
Strengths in methodology:
Weaknesses:
Potential Impact
The paper addresses a practically important problem. Understanding *why* multi-objective alignment fails mechanistically could inform:
1. Architecture design: Motivating modular or mixture-of-experts approaches where objective-specific pathways are structurally separated.
2. Training strategies: Informing curriculum design, gradient surgery, or regularization approaches that minimize interference in shared neuron populations.
3. Controllable alignment: The identification of objective-specific vs. shared neurons could enable targeted interventions (e.g., selective fine-tuning, activation engineering) for steering RM behavior.
However, the practical applicability is limited by the small model scale and the simplicity of the neuron identification approach. The insights are more directional than prescriptive—the paper identifies the problem mechanistically but doesn't propose solutions.
Timeliness & Relevance
The paper addresses a timely gap. While there is substantial work on behavioral characterization of alignment trade-offs and on improving RM training, mechanistic interpretability of RMs specifically regarding competing objectives is understudied. The finding that shared representations drive alignment tension connects the alignment and mechanistic interpretability communities in a useful way. The recent emergence of benchmarks like RewardBench2 and RM-Bench makes the evaluation landscape more mature for this type of analysis.
Strengths
1. Clear problem formulation: The paper cleanly decomposes the problem into behavioral characterization (§3), neural analysis (§4), and conflict analysis (§5), making the narrative easy to follow.
2. The shared neuron finding is genuinely interesting: The ~50% overlap between helpfulness and harmlessness neurons, combined with the disproportionate impact of ablating these shared neurons (Table 4), provides a concrete mechanistic hypothesis for alignment tension.
3. Comprehensive evaluation: The use of diverse benchmarks reveals nuanced patterns—e.g., harmlessness training improving adversarial robustness, asymmetric behavioral effects of ablation.
4. Behavioral retention analysis: The formalization and measurement of how mixed training degrades specialized capabilities is a useful analytical contribution.
Limitations
1. Scale concerns: All experiments use models ≤1.7B parameters. Modern RLHF systems use much larger models where representational dynamics may differ substantially.
2. Single dataset for training: All RMs are trained on HH-RLHF, which has known limitations in quality and diversity. The specific tension patterns observed may be dataset-dependent.
3. No parameter-efficient fine-tuning comparison: While the authors justify full fine-tuning for interpretability, LoRA and similar methods are standard in practice and may exhibit different interference patterns.
4. Limited actionable insights: The paper identifies the problem but stops short of proposing or testing solutions (e.g., neuron-level regularization, modular training).
5. Neuron-level analysis limitations: Modern interpretability work increasingly uses features (via sparse autoencoders) rather than individual neurons, as neurons are often polysemantic. The authors acknowledge this but don't address it experimentally.
6. The causal claims from ablation studies could be stronger: Zeroing out neurons is a relatively coarse intervention. More fine-grained causal methods (activation patching, path patching) could strengthen the mechanistic claims.
Overall Assessment
This is a solid exploratory study that provides useful initial evidence for the representational basis of alignment tension in reward models. The shared neuron finding is the most compelling contribution, offering a mechanistic lens that complements existing behavioral analyses. However, the work is limited by model scale, methodological simplicity in neuron identification, and the absence of proposed solutions. It opens interesting research directions but represents an incremental rather than transformative contribution to the field.
Generated Jun 12, 2026
Comparison History (14)
Paper 1 addresses a critical and highly timely challenge in AI safety and LLM alignment: the tension between helpfulness and harmlessness in RLHF. By applying mechanistic interpretability to uncover how these objectives interfere at the neuron level, it provides fundamental insights that can directly impact how future foundational models are aligned. While Paper 2 offers robust theoretical advancements in discrete diffusion models, Paper 1's focus on understanding and solving core bottlenecks in widely deployed LLM alignment gives it a broader potential impact across both AI research and real-world deployment.
Paper 1 addresses a critical and highly timely issue in AI alignment (RLHF), specifically the tension between helpfulness and harmlessness in LLMs. By providing mechanistic interpretability into reward models, it offers foundational insights that could broadly influence how safe and reliable AI systems are developed. While Paper 2 offers an innovative approach to model quantization, Paper 1's focus on AI safety and alignment has a broader potential impact across the rapidly expanding field of large language models and their real-world deployment.
Paper 1 is more novel and broadly impactful: it offers mechanistic, causal analysis of objective interference in RLHF reward models (neuron identification/ablation), directly addressing a central, timely bottleneck in AI alignment. The methodological contribution and interpretability insights can influence reward modeling, multi-objective optimization, safety, and model editing across many LLM systems. Paper 2 is valuable infrastructure (benchmark+corpus) for a narrower subfield; its impact depends on community adoption and is primarily evaluative rather than providing new mechanistic understanding.
While Paper 2 presents impressive empirical advances in robotics, Paper 1 tackles a foundational bottleneck in modern AI: the tension between helpfulness and harmlessness in LLM alignment (RLHF). By applying mechanistic interpretability to uncover how these conflicting objectives are represented at the neuron level, Paper 1 provides critical insights into the 'alignment tax.' Given the current dominance of LLMs and the urgent need for safe, controllable AI systems, this foundational research is likely to have a broader, more profound impact across the entire artificial intelligence landscape.
Paper 1 addresses a fundamental and timely problem in AI alignment—understanding internal conflicts between helpfulness and harmlessness objectives in reward models used for RLHF. This mechanistic interpretability work has broad implications for the safety and trustworthiness of large language models, a topic of enormous current interest. The findings about shared neurons and interference between objectives provide novel insights that could influence how alignment is approached. Paper 2 offers a solid engineering contribution to spatio-temporal forecasting with modest incremental improvements (~5%), but operates in a more narrow domain with less transformative potential.
Paper 1 addresses a critical bottleneck in LLM alignment (RLHF) by providing mechanistic insights into the tension between helpfulness and harmlessness. Given the massive current focus on AI safety and LLM capabilities, this has profound practical and theoretical implications. While Paper 2 provides a solid theoretical foundation for ASGD optimization and distributed training, its impact is narrower compared to the fundamental and highly timely issue of aligning foundational AI models.
While Paper 1 offers a strong methodological advance for diffusion models, Paper 2 addresses a highly critical and timely bottleneck in AI safety: the tension between helpfulness and harmlessness in LLM alignment. By applying mechanistic interpretability to understand RLHF reward models, Paper 2 provides insights that could broadly impact the development of safer, more reliable AI systems, giving it a higher potential for immediate real-world application and widespread scientific impact across the rapidly growing field of AI alignment.
Paper 1 addresses the highly timely and critical challenge of understanding how reinforcement learning enhances reasoning in LLMs. By identifying 'strategy selection' and 'strategy improvement' mechanisms, it provides actionable insights for scaling advanced reasoning capabilities. Given the current AI landscape's intense focus on reasoning models (e.g., OpenAI o1), this work has immediate, high-impact applications. While Paper 2 offers valuable mechanistic insights into alignment tension, understanding and scaling reasoning capabilities currently represents a more transformative frontier in AI development.
Paper 2 is likely higher impact due to a more broadly applicable, theoretically grounded framework (projection caustics) for abrupt transitions in diffusion/flow generative dynamics, a timely topic with wide relevance across generative modeling. It offers a unifying geometric explanation plus a practical diagnostic (CBD) demonstrated on toy, diffusion, and latent text-to-image models, suggesting immediate utility for analysis and control. Paper 1 is valuable mechanistic interpretability for RLHF reward models, but its scope is narrower (reward-model-specific) and its findings may generalize less broadly across fields and model classes.
Paper 1 addresses a highly timely and critical challenge in AI alignment: the tension between helpfulness and harmlessness in RLHF. Given the widespread deployment of large language models, mechanistic insights into reward models offer immense real-world applicability and broad societal impact. While Paper 2 presents rigorous, mathematically novel contributions to geometric deep learning and physics, Paper 1's focus on foundational AI safety mechanisms positions it for a broader, more immediate scientific and technological impact across the rapidly expanding field of artificial intelligence.