Toward a Principled Framework for Agent Safety Measurement
Shuyi Lin, Anshuman Suri, Alina Oprea, Cheng Tan
Abstract
LLM agents emit actions, not just text, and once taken, those actions often cannot be undone. Yet today's agent-safety evaluations run greedy or a few sampled rollouts and report a single safe/unsafe rate -- blind to the long-tail trajectories where unsafe behavior may arise from low-probability but non-negligible actions. We argue agent safety should be measured by search, not sampling. We apply BOA, a framework that, given a deployment configuration (model, decoder, prompt, environment, judger, likelihood budget), searches the in-budget trajectory space and reports a safety score: the probability the agent stays safe under the configuration. BOA searches both within a single LLM round and across the agent-environment interaction tree under a given likelihood budget, and makes search practical via batched decoding/judging, prefix caching, and chunked tree expansion. On agent-safety workloads, BOA discovers unsafe trajectories that greedy and sampled evaluations miss. BOA can additionally be used for ranking models, defenses, and attacks, all on the same scale, with manageable GPU costs.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "Toward a Principled Framework for Agent Safety Measurement"
1. Core Contribution
This paper reframes agent safety evaluation from a sampling problem to a search problem. The key insight is that standard evaluation practices—greedy decoding or a handful of sampled rollouts—systematically miss low-probability but reachable unsafe trajectories. The authors propose BOA, a framework that, given a deployment configuration (model, decoder, prompt, environment, judger, likelihood budget), systematically searches the trajectory space and outputs a continuous safety score: the probability-weighted fraction of in-budget trajectories judged safe.
The core novelty lies in extending prior single-turn token-level search (Jailbreak Oracle) to the multi-turn agent setting, where the search tree interleaves model-generated chunks with deterministic environment responses. The formalization of a "length-relative likelihood budget" (τ(n) = ε · L_n) is a thoughtful design choice that prevents trajectory probabilities from becoming meaninglessly small at greater depths.
2. Methodological Rigor
Strengths in formalization: The problem definition (Definition 1) is clean and well-motivated. The length-relative threshold addresses a real issue with naive probability cutoffs in long sequences. The hierarchical decomposition into chunk trees (within-turn) and block trees (across-turn) is architecturally sound.
Concerns about experimental rigor:
3. Potential Impact
The paper addresses a genuinely important gap. As LLM agents move to production (tool use, code execution, financial transactions), the distinction between "what the agent happened to do" versus "what it could do" becomes safety-critical. The framework has several compelling applications:
However, the practical impact depends on scalability to more complex environments. The current evaluation uses Agent-SafetyBench's relatively constrained tool-calling setup. Real-world agents interacting with databases, APIs, file systems, and web browsers present much larger branching factors and longer interaction horizons.
4. Timeliness & Relevance
The paper is exceptionally timely. Agent deployment is accelerating rapidly (the paper cites real-world incidents from 2024-2026), and the safety evaluation gap it identifies is widely acknowledged but under-addressed. The framing of safety-as-search rather than safety-as-sampling is a conceptual contribution that could influence how the community thinks about evaluation methodology, even independent of BOA's specific implementation.
The paper also arrives at a moment when stochastic decoding strategies (top-p, top-k, temperature) are standard in production, making the argument against greedy-only evaluation particularly resonant.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations
The paper builds directly on the authors' prior Jailbreak Oracle work, extending it to the agentic setting. This lineage is clearly acknowledged, though it somewhat reduces the novelty—the core search machinery existed; the contribution is the adaptation to multi-turn agent interactions with environment feedback.
The finding that removing explicit override markers makes prompt injection *more* effective (Table 3) is a genuinely interesting empirical result that could influence attack/defense research independently of the BOA framework.
Generated May 5, 2026
Comparison History (32)
Paper 1 addresses a highly urgent and broadly applicable problem: the safety evaluation of LLM agents. By proposing a principled, search-based methodology to replace inadequate sampling methods, it offers a foundational tool for the rapidly expanding field of AI safety. In contrast, Paper 2 presents an impressive but niche advancement in hardware security (RF retroreflector attacks). Furthermore, Paper 2 targets an older standard (100BASE-TX Fast Ethernet), which limits its immediate real-world relevance compared to the ubiquitous and escalating deployment of autonomous AI agents.
Paper 1 addresses a critical and timely problem in AI safety evaluation for LLM agents, proposing a principled search-based framework (BOA) that reveals unsafe behaviors missed by standard sampling. Given the rapid deployment of LLM agents and growing regulatory interest in AI safety, this work has broad impact across AI safety, policy, and deployment practices. Paper 2, while technically impressive in extending RF retroreflector attacks to Fast Ethernet, targets a narrower hardware security niche with more limited cross-disciplinary impact and a smaller affected community.
Paper 1 likely has higher impact: it introduces a principled, deployment-config-aware measurement framework for agent safety that targets rare but consequential long-tail failures via search rather than sampling, with practical engineering to make exhaustive-like evaluation feasible. This is timely as LLMs increasingly act in tools/environments, and the method can standardize evaluation across models/defenses/attacks, enabling broad adoption in benchmarking and governance. Paper 2 is novel mechanistically and yields a strong attack, but its impact may be narrower (model-specific interpretability/steering) and could age faster as architectures and alignment methods change.
Paper 2 is likely higher impact: it introduces a principled, general evaluation framework for agent safety that addresses a pressing, broadly relevant gap (long-tail unsafe trajectories) and is directly actionable for benchmarking, deployment decisions, and comparing models/defenses/attacks. Its “search not sampling” framing and likelihood-budgeted safety probability are methodologically rigorous and extensible across agents and environments. Paper 1 is novel mechanistically and impactful for interpretability and attack/defense insights, but it is narrower (LLM refusal circuitry) and its standout result (REA) is primarily adversarial, which may limit adoption versus a measurement standard.
Paper 2 is more novel and broadly impactful: it introduces “trace inversion” to reconstruct hidden reasoning and demonstrates large downstream gains from training on synthetic traces, directly challenging a major safety/IP assumption (that hiding CoT protects capabilities). Applications span model distillation, auditability, and policy/security, with immediate relevance to current deployment practices. The methodology appears empirically grounded with concrete black-box transfer results on standard benchmarks. Paper 1 is valuable for agent-safety evaluation rigor, but it is more incremental (search vs sampling) and narrower in impact compared to the capability-leakage implications of Paper 2.
Paper 1 introduces a highly novel and timely method that challenges the current industry practice of hiding reasoning traces to protect proprietary models. By demonstrating substantial performance gains in student models using 'trace inversion,' it has profound and disruptive implications for model distillation, AI security, and the open-source ecosystem. While Paper 2 offers solid methodological improvements for agent safety, Paper 1's immediate relevance and broader impact on AI development dynamics make it more scientifically impactful.
Paper 1 is likely higher impact: it proposes a principled, general evaluation framework for agent safety that reframes measurement as search over low-probability trajectories, addressing a core, widely recognized weakness of current agent evaluations. Its likelihood-budgeted, configuration-specific safety scoring can standardize comparisons across models/defenses/attacks and applies broadly to many agentic settings, making it timely and cross-cutting. Paper 2 is a strong, practical defense for instruction-injection in RAG, but is narrower in scope and may be more sensitive to shifting attack strategies and NLI-model assumptions.
Paper 2 has higher potential impact: it proposes a principled, configuration-aware safety metric for LLM agents and replaces sampling-based evals with likelihood-budgeted search, addressing a fundamental evaluation gap (rare but consequential unsafe trajectories). This framework is broadly applicable across models, prompts, environments, and defenses, enabling standardized comparison and ranking—likely influencing benchmarks, policy, and deployment practices. Paper 1 is a strong, timely defense technique for prompt-injection sanitization, but is narrower in scope and may be more susceptible to domain-specific assumptions and adaptive attackers than a general measurement framework.
Paper 1 addresses a fundamental and timely problem in AI safety measurement for LLM agents, proposing a principled framework (BOA) that shifts evaluation from sampling to search. Given the rapid deployment of LLM agents with real-world action capabilities, this work has broad impact across AI safety, alignment, and responsible deployment. Paper 2, while practically important for cybersecurity, addresses a narrower problem (timing-based evasion of visual phishing detectors) with more incremental contributions. Paper 1's framework-level contribution and relevance to the fast-growing agent safety field gives it higher potential for broad scientific impact.
Paper 1 tackles a critical, rapidly evolving problem in AI: LLM agent safety. Proposing a principled search-based framework to evaluate long-tail safety risks offers a fundamental methodological shift that could broadly impact AI alignment and agent deployment. Paper 2 presents a clever and practical evasion attack against visual phishing detectors, but its scope is narrower and mostly isolated to a specific subfield of cybersecurity. Therefore, Paper 1 has broader relevance and higher potential for widespread scientific impact.