This assessment is based on version 1 of this paper. Version 3 is now available on arXiv — the authors may have revised their methods, results, or conclusions.
The Defense Trilemma: Why Prompt Injection Defense Wrappers Fail?
Manish Bhatt, Sarthak Munshi, Vineeth Sai Narajala, Idan Habler, Ammar Al-Kahfah, Ken Huang, Blake Gatto
Abstract
We prove that no continuous, utility-preserving wrapper defense-a function that preprocesses inputs before the model sees them-can make all outputs strictly safe for a language model with connected prompt space, and we characterize exactly where every such defense must fail. We establish three results under successively stronger hypotheses: boundary fixation-the defense must leave some threshold-level inputs unchanged; an -robust constraint-under Lipschitz regularity, a positive-measure band around fixed boundary points remains near-threshold; and a persistent unsafe region under a transversality condition, a positive-measure subset of inputs remains strictly unsafe. These constitute a defense trilemma: continuity, utility preservation, and completeness cannot coexist. We prove parallel discrete results requiring no topology, and extend to multi-turn interactions, stochastic defenses, and capacity-parity settings. The results do not preclude training-time alignment, architectural changes, or defenses that sacrifice utility. The full theory is mechanically verified in Lean 4 and validated empirically on three LLMs.
AI Impact Assessments
(3 models)Scientific Impact Assessment: "The Defense Trilemma: Why Prompt Injection Defense Wrappers Fail?"
1. Core Contribution
The paper establishes an impossibility result for prompt injection defense wrappers: no continuous, utility-preserving function D: X → X can guarantee complete safety for language models with connected prompt spaces. The central argument is topological — a continuous defense that fixes all safe inputs (utility preservation) must have a closed fixed-point set, but the safe region S_τ is open and non-closed in a connected space, forcing the defense to also fix boundary points where alignment deviation equals the threshold τ. This is formalized as a "defense trilemma" (continuity + utility preservation + completeness cannot coexist) and developed through three progressively stronger results: boundary fixation, ε-robust constraints under Lipschitz regularity, and persistent unsafe regions under a transversality condition.
2. Methodological Rigor
The mathematical framework is clean and the proofs are sound. The core boundary fixation result is essentially a standard topological observation: a continuous retraction onto a non-closed set is impossible in a connected Hausdorff space. The Lean 4 formalization (45 files, ~350 theorems, no `sorry` statements) significantly strengthens confidence in correctness and sets a high bar for formal verification in ML security research.
However, several methodological concerns arise:
Modeling gap. The central question is whether D: X → X on a connected topological space faithfully models real defense systems. Actual prompt spaces are discrete token sequences, and the paper's bridge via Tietze extension only shows that continuous extensions *exist*, not that actual model behavior is well-captured by continuity assumptions. The discrete results (Theorem 8.3) avoid this concern but deliver a weaker conclusion (non-injectivity rather than impossibility).
Empirical validation is thin. The experiments validate "qualitative predictions" on a 2D behavioral surface (indirection × authority framing) for three models, but this is a highly compressed projection. The estimated Lipschitz constants (G ≈ 5, ℓ ≈ 1) are rough grid-based estimates, not rigorous measurements. The transversality condition — essential for the strongest result — is checked only informally.
Transversality as a strong assumption. The persistent unsafe region (Theorem 6.3) requires G > ℓ(K+1), and the paper acknowledges this is vacuous for isotropic surfaces (ℓ = L). The practical prevalence of sufficient anisotropy in real alignment surfaces remains unestablished.
3. Potential Impact
The defense trilemma framing could influence how practitioners reason about defense architecture design. The paper's engineering prescriptions (shallow boundaries, reduced Lipschitz constants, dimensionality constraints, monitoring over elimination) are actionable. The explicit scope limitations — clearly stating that training-time alignment, architectural changes, discontinuous filters, and ensemble methods are not precluded — prevent misinterpretation.
The Lean formalization could catalyze a norm of formal verification in ML safety theoretical work. The discrete defense dilemma (completeness forces information loss) may prove more practically influential than the continuous results, as it applies directly to token-level operations without topological assumptions.
However, the practical impact is tempered by the paper's own acknowledgment that if threshold-level behavior is benign (as in GPT-5-Mini where peak AD = 0.50), the impossibility is "mathematically true but practically harmless." Many well-designed systems already ensure boundary behavior is benign refusal.
4. Timeliness & Relevance
The paper addresses a timely concern as wrapper-based defenses (input classifiers, constitutional rewriting pipelines, output filters) proliferate in production LLM systems. Establishing fundamental limits on this defense class is valuable for the field's maturation. The connection to the broader adversarial robustness literature (robustness-accuracy tradeoffs) situates the work well.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Additional Observations:
The paper would benefit from engagement with the question of whether realistic prompt embeddings actually form connected spaces with the relevant topological properties. The assumption that embedding spaces are connected and that alignment deviation varies continuously is plausible but not self-evident — tokenization introduces discrete jumps, and model behavior can change dramatically with small prompt changes (the basis for adversarial attacks).
Overall Assessment
This is a well-crafted theoretical paper that provides a clean impossibility result for a specific class of defenses. The Lean formalization is a genuine contribution. However, the core mathematical insight is relatively standard topology applied to a debatable model of real systems, and the practical implications are substantially weakened by the acknowledgment that benign thresholds render the impossibility harmless. The paper is valuable primarily for its conceptual framing and for establishing formal verification norms in ML security.
Generated Apr 9, 2026
Comparison History (228)
Paper 1 establishes a fundamental, mathematically proven limit (a 'no-go' theorem) for LLM wrapper defenses, mechanically verified in Lean 4. By proving that continuity, utility preservation, and completeness cannot coexist, it redirects the AI safety field away from flawed defense strategies toward architectural or training-time alignment. While Paper 2 offers a highly practical and timely empirical auditing tool, Paper 1's foundational theoretical contribution will likely have a more profound, long-lasting scientific impact by establishing absolute mathematical bounds on AI security approaches.
Paper 2 likely has higher scientific impact because it provides a general, theoretically grounded impossibility/trilemma result about an entire class of defenses (prompt-injection wrapper preprocessors), with clear conditions, extensions (discrete, multi-turn, stochastic), mechanized verification in Lean, and empirical validation. Such negative results can reshape research direction across LLM security, alignment, and HCI by ruling out broad design space. Paper 1 is innovative and practically valuable for evolving red-teaming and online guardrail training, but its impact may be more incremental/competitive and can age quickly as models and attacks change.
Paper 1 addresses a high-stakes, timely domain (AI cryptanalysis) with a concrete benchmark, verifiable results, and genuinely novel findings (previously unknown attacks on SpoC and an error in KINDI's proof). It offers a practical tool for the security community and tracks a fast-moving frontier. Paper 2 is theoretically elegant with formal Lean verification, but its impossibility result under restrictive assumptions (continuous, utility-preserving wrappers) has narrower practical reach, especially since it explicitly excludes training-time and architectural defenses that dominate real-world practice. Paper 1's breadth of impact and immediate applicability are stronger.
Paper 2 likely has higher impact due to a general impossibility-style result for a timely, broadly relevant problem (prompt injection). Its theoretical characterization applies across models and defense designs (wrappers), potentially redirecting an entire line of security research, with mechanized verification adding rigor and credibility. Paper 1 is innovative hardware–algorithm co-design with strong efficiency results and real-world edge relevance, but its impact is more domain-specific (RRAM CiM training) and contingent on hardware adoption. Paper 2’s conclusions generalize more widely across ML safety/security.
Paper 2 has higher potential impact because it provides a general, formally proved impossibility result (“defense trilemma”) with clear assumptions and extensions (discrete, multi-turn, stochastic, capacity-parity), plus mechanized verification in Lean—high methodological rigor and broad relevance across ML security, robustness, and theoretical CS. Its conclusion directly guides an active, timely area (prompt-injection defenses), likely influencing many subsequent designs. Paper 1 is innovative and practical, but is a specific architecture in a narrower slice of agent security and may be more contingent on tooling/benchmarks.
Paper 1 addresses a timely, high-stakes threat—the weaponization of AI-driven autonomous science through indirect data poisoning—with strong empirical rigor (450 runs, three frontier systems, five socially-salient topics) and actionable mitigations that reduce attack success to zero. Its implications span scientific integrity, AI safety, and the open data ecosystem, offering broad cross-field impact. Paper 2 offers elegant, formally verified theoretical results on prompt injection defenses, but its impact is narrower and more theoretical, focused on impossibility results for a specific defense class. Paper 1's practical urgency and breadth give it higher estimated impact.
GPUBreach demonstrates the first GPU-side privilege escalation attacks via Rowhammer, escalating to CPU-side root access by defeating IOMMU protections. This has enormous practical security implications for GPU cloud computing, AI infrastructure, and hardware security. It opens an entirely new attack surface with immediate real-world consequences. While Paper 1 provides elegant impossibility results for prompt injection defenses, its conclusions are somewhat expected (perfect defense is impossible) and narrower in scope. Paper 2's novel attack primitive affects hardware security, cloud security, and AI security simultaneously, with broader cross-disciplinary impact.
Paper 2 establishes a mathematically verified impossibility theorem, checked in Lean 4, for LLM prompt injection wrapper defenses. Fundamental 'no-go' theorems historically have massive scientific impact, as they shift entire research paradigms away from futile approaches toward foundational architectural changes. While Paper 1 provides a highly practical systems-level framework for AI agent safety, Paper 2's exceptional methodological rigor and fundamental theoretical contribution to the urgent problem of prompt injection give it broader and longer-lasting scientific significance.
Paper 2 addresses a critical, highly timely issue in LLM security by proving a fundamental impossibility result for prompt injection wrappers. Its mechanical verification in Lean 4 demonstrates exceptional methodological rigor. While Paper 1 offers strong advancements in blockchain scalability, Paper 2's findings have broader, immediate real-world implications, effectively redirecting the AI security community's approach away from flawed wrapper defenses toward foundational architectural and training solutions.
Paper 2 establishes a fundamental, mathematically rigorous impossibility theorem for LLM wrapper defenses, mechanically verified in Lean 4. While Paper 1 introduces a highly novel hardware-based attack, Paper 2's theoretical bounds (the defense trilemma) shift the entire paradigm of AI safety by proving a widespread defense strategy inherently flawed. This combination of deep methodological rigor and broad applicability to the theoretical limits of AI security promises a higher enduring scientific impact.