Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Ali Hatamizadeh, Yejin Choi, Jan Kautz
Abstract
Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Gated DeltaNet-2
1. Core Contribution
Gated DeltaNet-2 addresses a specific structural limitation in delta-rule linear attention models: the use of a single scalar gate β_t to simultaneously control both how much old content is erased (key-side) and how much new content is written (value-side). The paper introduces two independent channel-wise gates — an erase gate b_t ∈ [0,1]^{d_k} operating on the key axis and a write gate w_t ∈ [0,1]^{d_v} operating on the value axis. This is a clean generalization that subsumes both Gated DeltaNet (scalar decay, scalar delta gate) and KDA (channel-wise decay, scalar delta gate) as special cases through gate tying.
The conceptual insight is well-motivated: erasing and writing act on different axes of the state matrix (key vs. value dimensions), and tying them to one scalar unnecessarily constrains the model's ability to manage its compressed memory. The paper also provides a fast-weight interpretation (Eq. 13) showing the update as the solution to a local online optimization problem, which gives theoretical grounding to the design choice.
2. Methodological Rigor
The paper demonstrates strong mathematical rigor. The chunkwise parallel training algorithm is carefully derived, showing how channel-wise decay can be absorbed into asymmetric erase factors to preserve the WY representation structure. The decay-normalized recurrence (Eq. 19) is the key technical insight enabling this — by absorbing cumulative decay into the rank-one factors, the algorithm retains the same triangular-solve-plus-dense-matmul structure as KDA.
The full backward derivation is provided in Appendix B, with explicit identification of why scalar post-scaling (valid for KDA) breaks for the decoupled gates (Appendix B.5). Numerical verification details (Appendix D) demonstrate fp64 gradient agreement to machine precision.
The experimental setup is reasonably controlled: all models share 1.3B parameters, identical training recipes (100B FineWeb-Edu tokens, same optimizer, same batch size), and matched recurrent state sizes. The comparison set is comprehensive and current, including Mamba-2, Gated DeltaNet, KDA, Mamba-3 (SISO and MIMO), and Transformers.
However, some limitations in rigor should be noted: (1) only one model scale (1.3B) is evaluated, leaving scaling behavior unclear; (2) training on only 100B tokens is modest by current standards; (3) the improvements on some benchmarks are within noise margins (e.g., some commonsense tasks show mixed results across methods).
3. Potential Impact
Direct impact on efficient sequence models: The paper makes an incremental but practically useful contribution to the delta-rule family of linear attention models. The decoupled gate mechanism is conceptually simple, adds minimal overhead (~5% throughput decrease vs. KDA), and provides consistent improvements across multiple evaluation axes.
Long-context retrieval: The most compelling results are on RULER needle-in-a-haystack tasks (Table 3), where Gated DeltaNet-2 shows clear advantages in multi-key retrieval — the hardest setting for fixed-state recurrence. At 4K context length with recurrent-only models, it achieves 37.8% on MK-NIAH-1 vs. 28.0% for KDA and 27.8% for Gated DeltaNet. The S-NIAH-2 at 4K shows 93.0% vs. 89.0% (KDA) and 87.2% (GDN).
Code availability: Open-source Triton kernels are provided, which lowers the adoption barrier significantly.
Broader influence: The paper's contribution is primarily within the linear attention/efficient transformer subfield. It does not introduce a fundamentally new paradigm but rather refines the gating mechanism. The impact is likely to be absorbed into future iterations of delta-rule models rather than spawning independent research directions.
4. Timeliness & Relevance
The paper is highly timely. It enters a very active research area where Mamba-2, Gated DeltaNet, KDA, and Mamba-3 have been published in rapid succession (2024-2026). The competition among these approaches is fierce, and the improvements shown here, while modest in absolute terms, address a well-identified bottleneck. The need for efficient long-context models with strong retrieval capabilities is a pressing concern for LLM deployment.
5. Strengths & Limitations
Strengths:
Limitations:
Overall Assessment
Gated DeltaNet-2 is a well-executed incremental contribution to delta-rule linear attention. The insight of decoupling erase and write is clean and mathematically grounded, the engineering is thorough, and the results are consistently positive across evaluations. The paper's strongest case is on long-context retrieval, where the architectural change most directly addresses interference in compressed memory. However, the improvements are modest in many settings, and single-scale evaluation limits confidence in the generality of the findings.
Generated May 22, 2026
Comparison History (23)
While Paper 1 presents a solid algorithmic advancement in linear attention, Paper 2 demonstrates higher potential impact due to its critical real-world applications in healthcare. Paper 2 introduces a framework addressing diagnostic errors and AI safety, a major bottleneck in medical AI. Its rigorous interdisciplinary evaluation—using literature cases, real-world emergency department notes, and blinded physician reviews—highlights immediate clinical utility, improved patient safety, and significant broader societal impact compared to the specialized architectural focus of Paper 1.
Gated DeltaNet-2 presents a concrete architectural innovation in linear attention with clear empirical improvements across multiple benchmarks, including strong results on long-context retrieval. It addresses a fundamental limitation in existing linear attention mechanisms and provides a practical, scalable solution with open-source code. While Paper 1 offers interesting theoretical insights about compression and generalization in LLM research agents, Paper 2 is more likely to have direct, broad impact on the rapidly growing field of efficient sequence modeling, influencing future architecture designs and enabling practical deployment of long-context models.
While Paper 1 offers a highly novel formal approach to AI safety testing, Paper 2 proposes a fundamental architectural advancement in linear attention by decoupling erase and write gates. By outperforming state-of-the-art sub-quadratic models like Mamba-2 and Mamba-3 on long-context tasks, it addresses a critical bottleneck in foundation model efficiency and scaling, which will likely drive broader adoption and foundational impact across the deep learning community.
Paper 2 has higher estimated impact due to broader applicability and timeliness: a systematic, controllable “text-space optimizer” for agent skills can improve many existing and future agentic systems without changing base model weights, and transfers across models/harnesses. Its methodology (held-out validation gating, bounded edits, learning-rate budget, stability mechanisms) resembles reproducible optimization and is likely to influence practice across AI agents, software engineering, and RL-style evaluation. Paper 1 is a strong, rigorous architectural advance for linear attention/sequence models, but its impact is narrower to efficient long-context model design.
Paper 1 likely has higher scientific impact due to its step-change in automating formally verified system construction: jointly synthesizing code and machine-checked proofs for distributed systems, achieving full success where strong baselines fail, with large speed/cost gains and performance-optimized verified outputs. This targets a major bottleneck in trustworthy software and can influence programming languages, verification, systems, and AI agent design. Paper 2 is a strong, timely incremental advance in linear attention with solid applications, but it is closer to ongoing architectural refinement in a crowded area and likely has narrower cross-field impact than scalable verified-system synthesis.
Gated DeltaNet-2 introduces a fundamental architectural improvement to linear attention mechanisms by decoupling erase and write gates, with strong empirical results across multiple benchmarks. This addresses a core limitation in efficient sequence modeling, has broad applicability beyond any single domain, and advances the frontier of efficient transformer alternatives—a highly active research area. Paper 1, while useful, is primarily a benchmark contribution for a specific application domain (finance spreadsheets) with findings that current agents fall short, offering less methodological novelty and narrower impact.
Paper 2 likely has higher scientific impact: it proposes a broadly useful architectural improvement for efficient long-context modeling (decoupled erase/write gating in linear attention), provides algorithmic/optimization contributions (chunkwise WY, gate-aware backward pass), and reports large-scale training with consistent gains across multiple benchmarks and settings, enabling real-world deployment benefits (constant-memory decoding, long-context retrieval). Paper 1 is insightful and timely for safety research, but its primary contribution is an attack/analysis on refusal mechanisms with narrower positive application scope and potentially shorter-lived impact as defenses evolve.
Gated DeltaNet-2 introduces a fundamental architectural improvement to linear attention mechanisms by decoupling erase and write operations, with strong empirical results across multiple benchmarks at 1.3B scale. This addresses a core limitation in efficient sequence modeling—a rapidly growing field with broad impact on LLM efficiency, long-context modeling, and inference cost. Paper 2 contributes a valuable evaluation metric for VLM explainability, but its scope is narrower (benchmarking/evaluation rather than model architecture), and its impact is primarily within the XAI subcommunity rather than the broader deep learning ecosystem.
Paper 2 likely has higher scientific impact: it advances core sequence modeling architecture by decoupling erase/write in linear attention, provides new algorithms (chunkwise WY, gate-aware backward) enabling efficient training, and demonstrates strong scaling and broad benchmark gains, especially for long-context retrieval—highly timely and broadly applicable across NLP and systems. Paper 1 addresses an important safety issue in multi-agent KV-sharing, but its impact is narrower (specific to latent KV communication setups) and depends on adoption of KV-sharing agents; it also hinges on a particular leakage definition (reconstruction) that may not cover all threat models.
Gated DeltaNet-2 introduces a fundamental architectural innovation in linear attention mechanisms—decoupling erase and write gates—with strong empirical results across language modeling, reasoning, and retrieval benchmarks. It generalizes multiple existing architectures (Gated DeltaNet, KDA), provides theoretical grounding (fast-weight update view, chunkwise algorithm), and addresses a core challenge in efficient sequence modeling. Its impact spans architecture design, efficient inference, and long-context modeling. Paper 1 identifies an interesting LLM evaluation bias (AMEL) but is more narrowly scoped to evaluation practices with a straightforward mitigation (fresh context per item), limiting its broader scientific impact.