Solve the Loop: Attractor Models for Language and Reasoning
Jacob Fein-Ashley, Paria Rashidinejad
Abstract
Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.
AI Impact Assessments
(1 model)Scientific Impact Assessment: "Solve the Loop: Attractor Models for Language and Reasoning"
1. Core Contribution
The paper introduces Attractor Models, a two-module architecture where a standard Transformer backbone produces an initial output embedding, and a smaller weight-tied "attractor module" refines it by solving for a fixed point using Anderson acceleration. Gradients are computed via implicit differentiation (specifically, a one-step approximation), decoupling training memory from the number of refinement iterations. This directly addresses three well-known pain points of looped/recurrent language models: (i) training memory scaling linearly with loop depth, (ii) instability during training, and (iii) train-test mismatch when varying iteration count at inference.
The key architectural insight is to place the fixed point in the tied output-embedding space (not a hidden state space as in DEQs), warm-start from a meaningful backbone proposal, and persistently inject this proposal at every refinement step. This design choice is well-motivated and the ablations (Tables 3, 4, 6) demonstrate each component's importance.
2. Methodological Rigor
Strengths in experimental design:
Concerns:
3. Potential Impact
Immediate practical impact:
Broader influence:
4. Timeliness & Relevance
This paper is extremely timely. The field is actively exploring test-time compute scaling, latent reasoning, and looped architectures (the reference list contains numerous 2025-2026 papers). The training instability and memory costs of looped models are widely recognized bottlenecks. The paper directly addresses these with a clean formulation.
The connection to the mechanistic finding by Blayney et al. (2026) — that looped LMs converge to fixed points for most tokens — provides strong empirical motivation for the approach.
5. Strengths & Limitations
Key strengths:
Notable limitations:
Missing comparisons: No comparison with Coconut (Hao et al., 2025), which also performs latent reasoning, or with other recent approaches like ChainGPT or AdaPonderLM.
Summary
This is a well-executed paper that makes a meaningful contribution to the active area of recurrent/looped language models. The core idea of replacing finite unrolling with fixed-point solving is clean and well-motivated, and the equilibrium internalization phenomenon is genuinely novel. The experimental evidence is strong within its scope, though scale limitations and selective baselines temper the broader claims. The constant-memory property alone makes this practically relevant.
Generated May 13, 2026
Comparison History (69)
Paper 1 offers a highly novel architecture that directly addresses critical bottlenecks in LLM scaling and reasoning. By using implicit differentiation for constant-memory training and demonstrating 'equilibrium internalization' for efficient inference, it achieves striking empirical gains—such as a 770M model outperforming a 1.3B Transformer and a 27M model solving reasoning tasks where frontier models fail. While Paper 2 provides rigorous theoretical unification for discrete diffusion models, Paper 1's massive efficiency improvements, practical applicability in NLP, and disruptive potential against the dominant Transformer paradigm give it significantly higher potential for broad scientific impact.
Paper 2 proposes a foundational improvement to Transformer architectures, addressing critical reasoning and scaling limitations in LLMs. By introducing Attractor Models with implicit differentiation, it achieves constant-memory training and massive performance gains, even outperforming frontier models on complex reasoning tasks. Its broad applicability across AI and NLP gives it a much wider, disruptive scientific impact compared to Paper 1's domain-specific, albeit strong, application of generative AI to materials science.
Paper 2 proposes a foundational architectural shift addressing core limitations of standard Transformers. By introducing Attractor Models with implicit differentiation, it enables constant-memory training and allows tiny models to outperform massive frontier models on complex reasoning tasks. Its novel 'equilibrium internalization' also allows highly efficient inference. While Paper 1 tackles a critical ethical and legal issue in AI copyright, Paper 2's potential to revolutionize the underlying architecture, efficiency, and reasoning capabilities of foundation models gives it a much broader and deeper scientific impact across the entire AI ecosystem.
Paper 1 introduces a fundamentally novel architectural paradigm (Attractor Models via fixed-point solving with implicit differentiation) addressing core challenges in scalable iterative reasoning. Its impressive empirical results across both large-scale language modeling and reasoning, plus the novel 'equilibrium internalization' phenomenon, offer broad conceptual influence across ML. Paper 2 is a solid, practical quantization method with strong engineering merit, but its impact is narrower and incremental within efficient inference for a specific model class (diffusion transformers). Paper 1's foundational nature and cross-domain relevance give it higher potential impact.
Paper 2 is more novel and broadly impactful: it introduces a new recurrent/fixed-point training paradigm (Attractor Models) with implicit differentiation, adaptive iteration, constant-memory effective depth, and an inference-time “equilibrium internalization” effect. The claimed gains span both large-scale LM pretraining and small-model reasoning, suggesting cross-field relevance (optimization, architectures, efficiency, reasoning). Paper 1 is a strong, timely systems optimization for million-token speculative decoding, with clear deployment value, but it is more incremental and narrower in scope (draft attention windowing) and primarily impacts inference engineering rather than core modeling.
Paper 2 introduces a fundamental architectural paradigm shift with Attractor Models, addressing critical limitations in transformer reasoning. By utilizing implicit differentiation and fixed-point solving, it achieves constant-memory training and massive performance leaps, such as a 770M model beating a 1.3B transformer and a 27M model solving reasoning tasks where frontier models fail. While Paper 1 offers excellent systems-level optimizations for edge deployment, Paper 2's novel 'equilibrium internalization' and potential to broadly disrupt foundational LLM pretraining architectures give it a much higher potential scientific impact.
Paper 1 introduces a fundamental architectural innovation (Attractor Models) that addresses a core limitation in LLMs: scalable iterative reasoning. By stabilizing looped transformers and demonstrating massive efficiency gains—such as a 770M model outperforming a 1.3B model and tiny models solving reasoning tasks where frontier models like GPT-o3 fail—it has profound implications for general AI. In contrast, Paper 2 provides a highly valuable, but narrower, systems-level optimization for equivariant GNNs. Paper 1's broader applicability across NLP and foundational AI architectures gives it significantly higher potential scientific impact.
Paper 2 has higher potential impact due to its high timeliness and broad applicability in large language models. By introducing Attractor Models, it solves critical bottlenecks in Transformer architectures like memory constraints and reasoning limitations. The impressive empirical results—outperforming larger models and frontier LLMs on complex reasoning tasks with a fraction of the parameters—suggest immediate real-world utility. While Paper 1 offers rigorous theoretical contributions to statistical learning verification, Paper 2's scalable approach to iterative refinement and novel 'equilibrium internalization' promise widespread adoption and paradigm-shifting impact across applied artificial intelligence.
Paper 1 introduces Attractor Models with strong empirical results across language modeling and reasoning, showing Pareto improvements over standard Transformers and outperforming much larger models. The equilibrium internalization phenomenon is novel and practically significant. The breadth of impact (scaling, efficiency, reasoning) and immediate applicability to large-scale ML systems gives it higher near-term impact. Paper 2 is theoretically elegant and important for AI security, but addresses a narrower problem with primarily theoretical contributions and only preliminary empirical results, limiting its immediate broad impact.
Paper 1 introduces Attractor Models, a fundamentally new architecture paradigm combining fixed-point solving with implicit differentiation for iterative refinement in Transformers. It demonstrates broad impact across language modeling and reasoning, with striking results (770M model beating 1.3B Transformer, solving tasks where frontier models fail). The novel 'equilibrium internalization' phenomenon is scientifically significant. Paper 2 presents an elegant but more incremental contribution to controllable image generation via reference-guided flow matching. While useful, its scope is narrower (image generation control) compared to Paper 1's potential to reshape how recurrence and depth are handled in foundation models.