Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang, Bo Han, Yike Guo
Abstract
Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper identifies a critical implicit assumption in DPO's derivation: the RLHF-optimal policy must prefer human-preferred responses. The authors prove that DPO-RLHF equivalence is *conditional* on this assumption, which depends on reference policy quality. When the reference policy is sufficiently misaligned (δ_πref ≤ -Δr*/β), the RLHF-optimal policy can prefer dispreferred responses due to KL regularization dominance, causing DPO to optimize "relative advantage over the reference policy" rather than "absolute alignment with human preferences." The authors characterize an "undesirable solution space" U where policies decrease DPO loss while violating human preferences, and propose two fixes: CPO (soft constraint) and E-CPOC (hard constraint with conservative bounds).
Methodological Rigor
The theoretical development is thorough and well-structured. The core insight—that substituting π* into the Bradley-Terry model requires π* to respect preference ordering—is clearly articulated and formalized (Assumption 3.1, Proposition 3.2). The characterization of the undesirable solution space (Definition 3.3) and the gradient vanishing analysis (Proposition 3.4) provide concrete mechanisms for how DPO fails.
The paper provides a comprehensive set of proofs across extensive appendices. The CPO derivation from constrained RLHF (Definition 4.7, Theorem 4.8) is clean, and the approximation of 1/π* with 1/π_ref to achieve a stationary objective is well-justified with explicit error bounds (Proposition H.2). The E-CPOC development using conservative bounds (Φ_cons) and KKT conditions is mathematically sophisticated.
However, several concerns arise regarding rigor:
1. The assumption set for E-CPOC equivalence is extensive (Assumptions 4.1-4.5, Condition 4.6), and the ℓ2-to-ℓ∞ conversion via Lemma L.1 introduces a √N factor that could be problematic for large datasets, potentially making the pointwise bound vacuous in practice.
2. The approximation 1/π* ≈ 1/π_ref in CPO's margin term is precisely weakest where it matters most—when π_ref is misaligned (low probability on preferred responses). The authors acknowledge this self-correcting property (Remark H.3), but the theoretical guarantee degrades in the regime where the method is most needed.
3. Experimental validation is limited. Only one model size (Llama-3-8B-Instruct) is tested, and the improvements over DPO are modest (0.55% WR on AlpacaEval 2, 3.7% on Arena-Hard). The paper acknowledges that E-CPOC lacks experimental validation entirely—a significant gap given the extensive theoretical development devoted to it.
Potential Impact
The theoretical insight about DPO's conditional equivalence addresses a genuine concern in the alignment community. As DPO has become widely adopted, understanding its failure modes is valuable. The 45.5% violation rate measured on a standard setup (Figure 1, Appendix A.1) demonstrates this is not a corner case.
The geometric interpretation through margin ranking loss (Section 5) is particularly elegant—showing DPO implements margin ranking with potentially negative targets provides immediate intuition for practitioners. The connection to the learning-to-rank literature could enable cross-pollination of techniques.
However, practical impact may be limited by several factors: (1) the margin correction in CPO requires precomputing 1/π_ref(y|x), which can be numerically unstable for low-probability responses; (2) the hyperparameter γ requires tuning (sensitivity analysis in Table 5 shows significant performance variation); (3) reference-free methods like SimPO already sidestep the identified issue entirely, and the paper's positioning relative to such methods is primarily theoretical rather than empirical.
Timeliness & Relevance
The paper addresses a timely question as DPO adoption accelerates in production systems. Understanding when and why DPO fails is crucial for deploying aligned language models safely. The concurrent work by Shi et al. (2025) and others examining DPO's theoretical properties indicates growing community interest in this direction.
Strengths
1. Clear identification of a precise failure condition (Equation 11) with constructive characterization of when it occurs.
2. The margin ranking perspective (Section 5) provides accessible geometric intuition that complements the formal theory.
3. The misaligned reference experiment (Appendix A.2) directly validates the theory: DPO degrades while CPO remains robust under increasing corruption ratios, and the fraction-in-U training dynamics (Figure 2) beautifully illustrate the theoretical mechanism.
4. The assumption dependency map (Table 1) provides unusual clarity about what each theoretical result requires.
5. CPO's implementation simplicity—identical computational cost to DPO with only precomputed margin subtraction.
Limitations
1. Scale limitations: Only 8B model tested; no evidence the findings hold at larger scales where reference policies may be better calibrated.
2. E-CPOC remains purely theoretical with no experimental validation despite being presented as the primary theoretical contribution.
3. The relationship to existing margin-based methods (IPO, SimPO) could be more carefully delineated; the paper notes distinctions but doesn't provide systematic ablations.
4. The γ* formula (Equation 27) requires knowledge of r*(y_w) - r*(y_l), which is unknown in practice, creating a gap between theory and implementation.
5. Missing training dynamics analysis acknowledged by authors—loss curves, preference accuracy evolution, and fraction-in-U tracking on the main benchmarks would strengthen the empirical story.
Overall Assessment
This paper makes a genuine theoretical contribution by precisely characterizing when DPO's RLHF equivalence breaks down. The theory is well-developed, the geometric interpretation is insightful, and the proposed fixes are principled. However, the experimental evidence is thin relative to the theoretical claims, E-CPOC lacks any empirical validation, and the practical improvements are modest. The work is primarily a theoretical contribution with preliminary experimental support.
Generated May 21, 2026
Comparison History (50)
Paper 1 addresses a critical flaw in Direct Preference Optimization (DPO), a dominant method for aligning LLMs. By proving the limitations of DPO-RLHF equivalence and introducing a theoretically sound fix (CPO), it has immense potential for rapid, widespread adoption in AI. While Paper 2 offers an elegant, rigorous solution for Scientific Machine Learning, Paper 1's immediate relevance to state-of-the-art LLM training pipelines gives it a broader and more urgent real-world technological impact, likely driving higher citation counts and immediate industry implementation.
Paper 2 addresses a fundamental theoretical question about DPO vs RLHF equivalence that affects the entire LLM alignment community. It identifies a critical implicit assumption, characterizes failure modes, and proposes a principled solution (CPO) with both theoretical guarantees and empirical validation. The breadth of impact is larger since DPO is widely used across the field. Paper 1, while practically useful for inference efficiency, is more incremental—a training-free routing heuristic for reasoning models. Paper 2's theoretical contributions will likely influence how the community understands and improves preference optimization methods going forward.
Paper 1 addresses fundamental LLM alignment methodologies. By identifying critical theoretical failure modes in DPO and proposing a mathematically rigorous, empirically validated alternative (CPO), it directly impacts how foundation models are trained. While Paper 2 provides an excellent methodological correction for an interdisciplinary subfield (LLM psychology), Paper 1 offers a foundational advance in core AI optimization and safety. Its direct applicability to state-of-the-art model training pipelines ensures a wider, more profound technological and scientific impact.
Paper 1 challenges the foundational assumptions of DPO, the dominant paradigm for LLM alignment, by exposing critical failure modes and providing a mathematically rigorous alternative (CPO). Its deep theoretical contributions and immediate, widespread applicability to foundation model training give it exceptional timeliness and broad impact across the entire AI landscape. While Paper 2 presents an innovative and practical robotics framework, Paper 1's corrections to fundamental alignment theory promise broader, more foundational scientific shifts and immediate adoption in the rapidly scaling LLM industry.
Paper 2 addresses fundamental theoretical limitations of Direct Preference Optimization (DPO), a widely used algorithm for LLM alignment. Because DPO and RLHF are foundational to modern AI development, theoretical insights and improvements (like the proposed CPO) have immense, immediate applicability across the entire AI industry. Paper 1 offers strong contributions to autoformalization, but its impact is relatively confined to the narrower subfield of AI for formal mathematics, giving Paper 2 broader relevance and higher potential for widespread scientific impact.
Paper 2 identifies a fundamental theoretical gap in DPO—one of the most widely adopted alignment methods—proving its equivalence to RLHF is conditional, not universal. This has immediate, broad impact across the entire LLM alignment community, as DPO is used extensively in practice. The clear theoretical contribution (characterizing failure modes, proving divergence of objectives) combined with a practical fix (CPO) and reproducible code makes it highly actionable. Paper 1, while ambitious in scope for physical AI world models, is more of a systems/engineering contribution with incremental advances across multiple axes, and its impact is narrower, limited to embodied AI.
Paper 2 addresses a fundamental theoretical gap in DPO vs RLHF equivalence, which is central to current LLM alignment practice. It provides rigorous mathematical characterization of failure modes, proposes a practical fix (CPO) with provable guarantees, and validates with experiments. Its impact spans the entire LLM training community. Paper 1 presents an interesting empirical phenomenon (reward-channel addiction) but is more narrow in scope, uses a synthetic sandbox, and offers less actionable solutions. Paper 2's findings directly affect widely-used training methodologies.
Paper 2 addresses a fundamental theoretical gap in DPO vs RLHF equivalence, which is central to the rapidly growing field of LLM alignment. Identifying when DPO's implicit assumptions fail and proposing CPO as a provably correct alternative has broad implications for all LLM training pipelines. The work combines rigorous theoretical analysis with practical solutions. Paper 1, while addressing an important clinical AI problem, represents a more incremental contribution combining existing LLM capabilities with rule-based systems for a narrower domain. The alignment community's size and momentum give Paper 2 significantly wider potential impact.
Paper 2 provides fundamental theoretical insights into the relationship between DPO and RLHF—two of the most important techniques in current AI alignment. It identifies a critical implicit assumption violation, characterizes failure modes, and proposes a principled solution (CPO) with both theoretical guarantees and empirical validation. This directly impacts the rapidly growing LLM alignment community. Paper 1, while useful as infrastructure for evaluation standardization, is more of an engineering/community contribution (a schema and repository) with less fundamental scientific novelty. Paper 2's theoretical depth and practical implications for AI alignment give it broader and deeper scientific impact.
Paper 1 has higher likely impact: it delivers formal results clarifying widely cited DPO–RLHF “equivalence,” identifies concrete failure modes, and proposes a principled, implementable fix (CPO) with provable guarantees plus benchmark evidence. This directly affects a core alignment/training pipeline used across many LLM deployments, making it timely and immediately actionable. Paper 2 is conceptually interesting and cross-disciplinary (cog sci + interpretability), but its claims about human reasoning mechanisms are harder to validate rigorously and may have less direct downstream engineering leverage than a training-method correction.