When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop
Yang Zhang, Xiukun Wei, Xueru Zhang
Abstract
Foundation models are increasingly trained on synthetic data generated by prior model iterations rather than exclusively on real data. This self-consuming training paradigm can lead to model collapse, divergence, or bias amplification. Recent work (Ferbach et al., 2024) shows that incorporating human curation into the loop can steer a self-consuming model toward human-aligned behavior, but these analyses focus on a single, isolated model that solely consumes its own outputs. In practice, however, models often interact and train on input-output pairs produced by other models. This paper studies self-consuming training in the multi-model regime. We first formalize a framework for interacting self-consuming models and characterize when the resulting dynamical system converges to a stable point. We then examine how human curation of one model affects its own alignment (self-influence) and how such effects propagate to other models (cross-influence). Unlike isolated settings where human curation always enhances model alignment, we show that cross-model interactions can dampen or even invert this effect, ultimately degrading long-term alignment.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper extends the analysis of self-consuming generative model training from the single-model setting to the multi-model regime, where multiple models interact by training on each other's outputs. The central finding is that human curation — which provably improves alignment in isolated single-model loops (Ferbach et al., 2024) — can backfire when models interact, potentially degrading long-term alignment. The paper formalizes this through a dynamical systems framework, derives convergence conditions, and decomposes the effect of curation into "self-influence" (direct effect on the curated model) and "cross-influence" (propagated effect on other models). The key insight that cross-model interactions can invert the sign of curation's effect is practically significant and theoretically novel.
Methodological Rigor
The theoretical framework is built on standard assumptions (strong convexity, smoothness, distribution sensitivity) inherited from the performative prediction and self-consuming model literature. The analysis is technically sound:
Limitations in rigor: The strong convexity assumption is difficult to verify for neural networks. The authors acknowledge this but argue empirical results suggest robustness beyond these assumptions. The finite-sample analysis is absent — the theory operates at the population level, and the gap between finite-sample behavior and theoretical predictions is only addressed empirically. Example 4.6 is constructive but somewhat contrived; it demonstrates the mechanism exists but doesn't characterize how common such inversions are in practice.
Experiments
Three experimental settings validate the theory:
1. Gaussian models provide exact verification of Theorem 4.5, demonstrating blockwise amplification/attenuation of curation effects.
2. CIFAR-10 diffusion models with hue-based reward functions show non-monotonic curation effects across six interaction configurations (A1-A6).
3. Qwen2.5-0.5B demonstrates preference domain mismatch (PDM), where coupling effects are masked when evaluation domains differ from training domains.
The experimental design is reasonable for validating theoretical mechanisms, though the reward functions (hue-based for vision, length/copying-based for text) are admittedly stress tests rather than realistic alignment scenarios. The CIFAR-10 experiments showing monochromatic images under extreme preferences acknowledge this limitation.
Potential Impact
Practical relevance: The paper addresses a genuine concern in the ML ecosystem. Models like Alpaca (trained on ChatGPT outputs), CapsFusion (LLM-refined captions for multimodal training), and RLHF pipelines with conflicting helpfulness/safety objectives all instantiate multi-model self-consuming loops. The finding that curation can backfire has direct implications for data pipeline design at scale.
Diagnostic framework: The matrices S_p, S_q, C_p, C_q provide a diagnostic toolkit. Practitioners could estimate these locally (via perturbation experiments) to predict whether increasing curation will help or hurt before committing to expensive retraining.
Limitations on impact: The two-model framework, while capturing the minimal interesting case, may not scale easily to realistic ecosystems with dozens of interacting models. The authors note N-model extension is "conceptually straightforward" but "technically challenging." The strong convexity requirement limits direct applicability to modern overparameterized models.
Timeliness & Relevance
This paper is highly timely. The proliferation of synthetic data in training pipelines (web scraping of AI-generated content, model distillation, RLHF) makes multi-model self-consuming dynamics an increasingly pressing concern. The gap between single-model analyses (which dominate the literature) and real-world multi-model ecosystems is precisely what this work addresses. The connection to performative prediction literature is natural and well-exploited.
Strengths & Limitations
Strengths:
Weaknesses:
Overall Assessment
This paper makes a solid theoretical contribution to an important and timely problem. The multi-model self-consuming framework fills a genuine gap in the literature, and the finding that curation can backfire is both surprising and practically relevant. The mathematical framework is rigorous within its assumptions, and the decomposition into self/cross-influence provides useful conceptual tools. The main limitations are the gap between theoretical assumptions and practical neural network training, and the restriction to two models.
Generated May 29, 2026
Comparison History (20)
Paper 1 addresses a critical, macroscopic problem in modern AI: model collapse and alignment in a multi-model ecosystem reliant on synthetic data. Its theoretical framework and counterintuitive findings regarding human curation offer broad implications for the future of foundation model training. Paper 2, while methodologically rigorous and practically useful for LLM fine-tuning, focuses on a much narrower data artifact (Long-CoT continuations), limiting its broader scientific impact compared to the systemic issues explored in Paper 1.
Paper 1 addresses a fundamental and highly relevant problem in AI—model collapse and alignment in self-consuming, multi-model training loops. Its findings have broad, critical implications for the future of foundation model training and AI safety. Paper 2, while offering a practical and useful framework for mobile GUI agents, is more applied and narrower in scope compared to the foundational theoretical impact of Paper 1.
Paper 1 addresses a critical, ecosystem-level problem of multi-model self-consuming loops and presents a highly counter-intuitive finding that human curation can backfire. This has profound implications for how future foundation models will be trained on web data, likely sparking widespread follow-up research across the AI community. Paper 2 offers a valuable but narrower technical optimization solution for safety fragility.
Paper 2 likely has higher scientific impact due to greater novelty and broad relevance: it extends self-consuming/synthetic-data training theory from single-model to multi-model interacting regimes, formalizes convergence dynamics, and reveals counterintuitive failure modes of human curation via cross-influence—highly timely for foundation model development and governance. Its methodological rigor (formal framework + stability characterization) and applicability across many AI domains (alignment, data pipelines, multi-agent/model ecosystems) suggest wider cross-field and real-world impact than Paper 1’s valuable but narrower ethnographic study focused on music production workflows.
Paper 1 identifies a novel and counterintuitive inverse scaling phenomenon (larger LLMs being less robust to distractor instructions), introduces a practical benchmark (DistractionIF), provides mechanistic explanations via perplexity analysis, and offers a concrete mitigation strategy (GRPO). This directly addresses a critical practical vulnerability in widely-deployed RAG and agentic systems. Paper 2 makes important theoretical contributions on multi-model self-consuming loops, but its impact is more niche and theoretical. Paper 1's combination of immediate practical relevance, actionable solutions, and surprising empirical findings gives it broader and more timely impact.
Paper 1 likely has higher scientific impact due to its novelty and broad relevance: it extends self-consuming/synthetic-data training theory from single-model to interacting multi-model ecosystems, with formal dynamical characterization and a counterintuitive finding (human curation can backfire via cross-influence). This is timely for LLM deployment/training pipelines and has implications across alignment, ML theory, and AI governance. Paper 2 is practically valuable for finance workflows and systems design, but is more domain-specific and appears more architectural/engineering than theoretically generalizable, limiting breadth and long-term cross-field impact.
Paper 2 addresses a more fundamental and broadly impactful problem—model collapse and alignment dynamics in multi-model self-consuming training loops—which is highly relevant to the rapidly evolving foundation model ecosystem. Its theoretical contribution (formalizing multi-model dynamics, convergence analysis, and the counterintuitive finding that human curation can backfire) has broad implications across AI safety, alignment research, and generative AI training practices. Paper 1, while methodologically solid, addresses a narrower application domain (cross-building energy forecasting) with incremental contributions (transfer learning + uncertainty quantification) that have more limited cross-field impact.
While Paper 1 offers strong empirical improvements in LLM tool use, Paper 2 addresses a fundamental, systemic issue in modern AI: model collapse from synthetic data in multi-agent environments. Its counter-intuitive finding that human curation can backfire in multi-model loops provides critical, timely insights for the long-term viability of AI alignment and foundation model training.
Paper 2 has higher estimated impact due to broad, timely relevance to foundation-model training practices (synthetic/self-consuming data) across many domains. It offers a formal multi-model dynamical framework with clear conceptual novelty beyond single-model analyses, and its conclusions about when human curation can backfire are widely applicable to AI safety, alignment, and deployment ecosystems. Paper 1 is innovative and application-rich for BCI, but its impact is likely narrower, constrained by data availability, experimental variability, and domain specificity compared to the cross-field implications of Paper 2.
Paper 1 addresses a critical, field-wide challenge in AI: model collapse and alignment degradation in multi-model synthetic data loops. Its theoretical insights challenging the assumption that human curation always helps will broadly impact how foundation models are trained across all domains. While Paper 2 offers a valuable real-world application of LLM alignment in medicine, its scientific impact is narrower and more incremental compared to the foundational implications of Paper 1.
Paper 2 addresses a fundamental and timely problem in foundation model training—self-consuming loops with multiple interacting models—providing formal theoretical analysis of when human curation can backfire. This has broad implications across all of AI/ML as synthetic data training becomes ubiquitous. The counterintuitive finding that curation can degrade alignment in multi-model settings is novel and impactful. Paper 1 proposes an educational chatbot architecture that, while practical, is more incremental and domain-specific, with less potential for broad cross-field impact.
Paper 1 addresses a fundamental and increasingly critical problem—model collapse and alignment in multi-model self-consuming training loops—with rigorous theoretical formalization. As the AI ecosystem increasingly involves models training on each other's outputs, understanding cross-model influence on alignment is highly novel and broadly impactful. Paper 2 introduces a useful benchmark for evaluating LLM scientific reasoning, but benchmarks have shorter shelf lives and narrower theoretical contributions. Paper 1's theoretical insights about when human curation backfires in multi-model settings have deeper implications for AI safety and training methodology at scale.
Paper 2 addresses a fundamental, highly timely issue affecting the entire foundation model ecosystem: model collapse and alignment in multi-model self-consuming loops. Its insights into how cross-model interactions can invert the benefits of human curation have broad, paradigm-shifting implications for how AI models will be trained on the increasingly synthetic internet. In contrast, Paper 1 offers a valuable but more narrow technical improvement for step-level credit assignment in Agentic Search, making Paper 2's potential breadth of impact significantly higher.
Paper 1 addresses a specific, technically rigorous problem (multi-model self-consuming training loops) with formal analysis and novel theoretical contributions showing counterintuitive results about human curation backfiring. This has direct implications for the rapidly growing practice of training on synthetic data. Paper 2 proposes a cognitive framework for measuring AGI progress, but is more of a position/framework paper that builds on existing cognitive science taxonomies without strong empirical validation. Paper 1's mathematical formalism and surprising findings about cross-model interactions are more likely to generate follow-up research and influence training practices.
Paper 1 targets a timely, high-stakes problem: synthetic-data self-consuming loops and alignment failures in interacting model ecosystems. Its multi-model dynamical-systems framing and the finding that human curation can backfire via cross-model interactions is a novel and broadly relevant insight for real-world foundation-model training pipelines, governance, and deployment. The impact spans ML theory, alignment, dataset curation, and AI safety, with direct implications for industry practice. Paper 2 is innovative within multi-agent RL, but is more niche to hidden-role games/opponent shaping and likely has narrower cross-domain and near-term deployment influence.
Paper 2 has higher potential impact due to its immediate real-world deliverable (largest integrated marine Pb database from 230k papers), strong methodological validation (multi-layer checks, 92% expert-verified accuracy), and broad downstream utility for oceanography, pollution studies, and climate-related circulation research. It is timely (LLM agents for scientific extraction) and provides open infrastructure (visualization platform) that can be reused across domains. Paper 1 is novel/theoretically important for multi-model self-consuming dynamics and alignment, but likely more specialized and with less direct near-term empirical or cross-disciplinary uptake.
Paper 2 has higher potential impact: it addresses a timely, broadly relevant failure mode in modern foundation-model training (synthetic data self-consuming loops) and extends prior single-model analyses to a realistic multi-model ecosystem. Its formal dynamical-systems framework and results about when human curation can backfire offer general insights that can influence alignment, data governance, and training practices across many domains and model families. Paper 1 is strong and application-ready, but is more domain-specific (speech translation deployment) and likely impacts a narrower slice of ML compared to the cross-cutting implications of Paper 2.
Paper 1 identifies a concrete, previously undocumented failure mode (unfaithful capitulation) in multi-turn adversarial settings, with strong empirical evidence across datasets and models plus multiple validation mechanisms (independent judge, token-level probe, ablations on think/no_think). The finding is timely for real-world LLM deployment and evaluation, likely influencing benchmarking, safety, and training practices broadly. Paper 2 is conceptually important but more theoretical/qualitative from the abstract and its impact depends on empirical validation in realistic multi-model training pipelines. Overall, Paper 1 appears more immediately actionable and methodologically substantiated.
Paper 1 addresses a fundamental and increasingly critical problem—multi-model self-consuming training loops—with formal theoretical contributions (dynamical systems analysis, convergence characterization). It extends prior work on model collapse to the realistic multi-model regime, revealing counterintuitive results about human curation backfiring. This has broad implications for the entire foundation model ecosystem. Paper 2 presents a useful engineering contribution for LLM aggregation but is more incremental, building on existing Mixture of Agents ideas. Paper 1's theoretical framework is likely to have more lasting and cross-disciplinary impact as synthetic data training becomes ubiquitous.
Paper 2 likely has higher impact due to timeliness and direct relevance to current foundation-model training practices (synthetic data, model collapse, alignment). Its multi-model self-consuming loop formalization targets a real-world setting (models training on other models’ outputs) with clear safety and deployment implications, and offers concrete theoretical conditions (stability, self-/cross-influence) that could inform data curation and governance. Paper 1 is innovative and broadly applicable in game/optimization, but its impact may be more specialized and dependent on empirical validation and adoption of the proposed solver-synthesis framework.