ADR: An Agentic Detection System for Enterprise Agentic AI Security
Chenning Li, Pan Hu, Justin Xu, Baris Ozbas, Olivia Liu, Caroline Van, Manxue Li, Wei Zhou
Abstract
We present the Agentic AI Detection and Response (ADR) system, the first large-scale, production-proven enterprise framework for securing AI agents operating through the Model Context Protocol (MCP). We identify three persistent challenges in this domain: (1) limited observability -- existing Endpoint Detection and Response (EDR) tools see file writes but not the agent reasoning, prompts, or causal chains linking intent to execution; (2) insufficient robustness -- static defenses constrained by pre-defined rules fail to generalize across diverse attack techniques and enterprise contexts; and (3) high detection costs -- LLM-based inference is prohibitively expensive at scale. ADR addresses these challenges via three components: the ADR Sensor for high-fidelity agentic telemetry, the ADR Explorer for systematic pre-deployment red teaming and hard-example generation, and the ADR Detector for scalable, two-tier online detection combining fast triage with context-aware reasoning. Deployed at Uber for over ten months, ADR has sustained reliable detection in production with growing adoption reaching over 7,200 unique hosts and processing over 10,000 agent sessions daily, uncovering hundreds of credential exposures across 26 categories and enabling a shift-left prevention layer (97.2% precision, 206 detected credentials). To validate the approach and enable community adoption, we introduce ADR-Bench (302 tasks, 17 techniques, 133 MCP servers), where ADR achieves zero false positives while detecting 67% of attacks -- outperforming three state-of-the-art baselines (ALRPHFS, GuardAgent, LlamaFirewall) by 2--4x in F1-score. On AgentDojo (public prompt injection benchmark), ADR detects all attacks with only three false alarms out of 93 tasks.
AI Impact Assessments
(1 model)Scientific Impact Assessment: ADR – An Agentic Detection System for Enterprise Agentic AI Security
1. Core Contribution
ADR presents an end-to-end enterprise security framework for AI agents operating through the Model Context Protocol (MCP). The system addresses a genuinely new attack surface: autonomous AI agents that can execute tools, access data, and make decisions via standardized interfaces. The three core components — the ADR Sensor (telemetry collection capturing the full causal chain from prompts to tool execution), the ADR Detector (two-tier online detection combining fast triage with deep contextual reasoning), and the ADR Explorer (offline evolutionary red-teaming) — together form a coherent security architecture.
The paper also introduces ADR-Bench (302 tasks, 17 techniques, 133 MCP servers), which is meaningfully broader than existing benchmarks in both threat coverage and MCP integration. The accompanying threat taxonomy (5 tactics, 17 techniques) synthesized from public incidents, frameworks, and operational telemetry is a useful contribution for the community.
The most compelling novelty is the *observability argument*: traditional EDR tools capture system-level events but miss the semantic layer (prompts, reasoning chains, tool invocations) that gives meaning to agent actions. This is a real gap, and the sensor architecture that reconstructs causal chains from local caches of agentic tools is a practical and elegant solution.
2. Methodological Rigor
The evaluation has notable strengths and weaknesses:
Strengths: The paper evaluates on both a public benchmark (AgentDojo, 93 tasks) and their own benchmark (ADR-Bench, 302 tasks), comparing against three relevant baselines (LlamaFirewall, GuardAgent, ALRPHFS). The ablation studies are informative, demonstrating the value of individual MCP context providers and the triage layer. The production deployment at Uber (10 months, 7,200+ hosts, 10,000+ daily sessions) provides strong evidence of practical viability.
Weaknesses: The benchmark design raises concerns. ADR-Bench was constructed by the same team that built ADR, creating potential overfitting risk — the system may be inadvertently optimized for the types of attacks and benign patterns included. The 42 malicious tasks (out of 302) is relatively small for drawing robust conclusions. The "zero false positives" claim, while impressive, should be interpreted cautiously given that the benign task distribution was curated.
The production deployment metrics are somewhat vague. The 49% false positive rate in deployment (versus zero on the benchmark) reveals a significant gap between benchmark and real-world performance, though the paper acknowledges this honestly. The credential detection results (97.2% precision, 206 TPs) are solid but represent a relatively narrow use case (regex-based pattern matching for secrets) rather than the full ADR detection pipeline.
The comparison with baselines may not be entirely fair: ADR uses GPT-4o and Claude Sonnet 4 (state-of-the-art models) while LlamaFirewall uses Llama Guard 3-8B (a much smaller model). Cost comparisons partially address this, but the capability gap is substantial.
3. Potential Impact
The practical impact potential is high. MCP adoption is accelerating rapidly (16,800+ public servers by 2025), and enterprise security teams urgently need tools designed for this paradigm. ADR's architecture — sensor + hierarchical detector + offline red-teaming — provides a reasonable blueprint that others can adapt.
The benchmark contribution (ADR-Bench) fills a real gap. Prior benchmarks cover only 3-6 of the 17 techniques identified, and most lack native MCP support. The comprehensive threat taxonomy grounded in real incidents is independently valuable for the security community.
The open-source release of the sensor, detection framework, and benchmark significantly amplifies potential impact by enabling reproducibility and community extension.
However, the impact is somewhat bounded by the rapidly evolving nature of both MCP and the attack landscape. The sensor's approach of parsing local caches of specific tools (Cursor, Cline, Claude Code) requires continuous maintenance as these tools change. The threat taxonomy, while comprehensive today, will need frequent updates.
4. Timeliness & Relevance
This paper is extremely well-timed. MCP has become the de facto standard for AI agent-tool integration in under a year, and the security implications are just beginning to be understood. The paper arrives at a moment when enterprises are rapidly deploying MCP-based agents but lack mature security tooling.
The shift from "securing LLMs" to "securing AI agents" represents a genuine paradigm shift in AI security, and ADR is among the first systems to address this comprehensively at enterprise scale. The 10-month production deployment at Uber provides credibility that few academic or industry papers in this space can match.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations
The evolutionary red-teaming approach (Explorer) is theoretically interesting but under-evaluated — the paper provides no standalone assessment of how many novel attack variants it discovers or how much it improves detection over time. The fitness function and convergence guarantees are stated but not empirically validated.
The paper's framing as the "first" enterprise-proven framework is reasonable given the evidence, though the rapidly growing MCP security space means this distinction may be short-lived.
Generated May 19, 2026
Comparison History (25)
Paper 1 (ADR) demonstrates higher scientific impact through its production-validated system deployed at Uber for 10+ months at scale (7,200+ hosts, 10,000+ daily sessions), addressing a critical and timely enterprise AI security problem. It introduces a novel benchmark (ADR-Bench) and outperforms baselines by 2-4x in F1-score. The real-world deployment evidence, practical security implications, and growing importance of agentic AI security give it broader impact. Paper 2, while methodologically rigorous and admirably transparent, reports largely negative findings—memory substrates don't reliably improve accuracy—limiting its actionable impact despite valuable insights about the copyability threshold.
Paper 2 reveals a fundamental and broadly consequential flaw in chain-of-thought distillation: answer accuracy can improve while reasoning quality degrades. This finding challenges core assumptions in the rapidly growing field of knowledge distillation and LLM reasoning, with critical implications for safety-sensitive domains like medicine. Its methodological rigor (multiple evaluators, clinical expert validation, extensive controls) and generalizable insight make it likely to influence evaluation practices across the entire LLM community. Paper 1, while practically valuable as an engineering contribution at Uber, is more narrowly scoped to enterprise AI security and MCP-based agent monitoring.
Paper 1 introduces a novel conceptual framework that bridges robotics control theory with foundation model safety, offering formally grounded behavioral guarantees for socially sensitive domains. This cross-disciplinary reframing—treating guardrails as runtime trajectory control rather than per-output filtering—is a genuinely new paradigm with broad applicability across AI safety research. Paper 2, while impressive as an engineering contribution with real-world deployment at Uber, is more of a systems/security paper solving a narrower enterprise problem. Paper 1's theoretical contribution has greater potential to influence multiple research communities and spawn new research directions.
Paper 1 is likely to have higher scientific impact: it introduces a novel, general training objective refinement (policy-aware reweighting for rubric-based RLVR) that can transfer across models, datasets, and future RLHF/RLVR settings, with clear methodological framing and controlled comparisons showing efficiency gains. Its contribution targets a core bottleneck in post-training—multi-criterion optimization with informative reward signals—relevant to many research groups. Paper 2 is highly impactful operationally, but is more systems/engineering- and deployment-specific (MCP/enterprise security), potentially narrowing breadth and long-term generalizability.
Paper 1 makes fundamental theoretical contributions—formalizing interface-constrained SMDPs, providing the first finite-sample guarantee for neural Q-learning under decentralized partial observability, and lifting the AIS framework to multi-agent SMDPs. These are novel, rigorous results with broad applicability beyond LLM pipelines to any multi-agent sequential decision-making setting. Paper 2 is a strong engineering/systems contribution with real-world deployment at Uber, but it is more narrowly focused on enterprise AI security and introduces less foundational methodology. Paper 1's theoretical novelty and breadth of impact across reinforcement learning, multi-agent systems, and workflow optimization give it higher long-term scientific impact.
Paper 1 demonstrates significantly higher impact due to its immediate relevance to the critical, rapidly growing field of AI agent security. Unlike Paper 2, which focuses on the theoretical aspects of LLM emotion understanding, Paper 1 presents a battle-tested, production-scale system deployed at a major enterprise (Uber). It addresses urgent real-world vulnerabilities, introduces a comprehensive framework (sensor, explorer, detector), and provides a valuable benchmark (ADR-Bench). Its combination of methodological rigor, proven real-world application, and timeliness in securing LLM agents guarantees broad utility and high impact in both industry and academia.
Paper 1 likely has higher scientific impact due to its timeliness (enterprise agentic AI security), demonstrated large-scale real-world deployment, and rigorous empirical validation (production metrics plus two benchmarks). Its contributions (telemetry/sensing, red-teaming workflow, scalable two-tier detection) are broadly applicable across enterprises adopting agent protocols, potentially influencing security tooling and standards. Paper 2 is novel and valuable for embodied/household agents, but its impact may be narrower (domestic robotics/assistants) and currently more evaluation-driven than deployment-proven.
Paper 2 presents a novel, production-proven system addressing the critical and timely problem of securing AI agents in enterprise environments. It introduces a comprehensive framework (ADR) deployed at scale at Uber for 10+ months, demonstrates strong real-world results, and provides a new benchmark (ADR-Bench). The problem of agentic AI security is highly relevant given rapid AI agent adoption. Paper 1 is an incremental improvement to PPO for multi-UAV coverage—a well-studied area with limited novelty (shared backbone is a known technique). Paper 2's breadth of impact, timeliness, and practical validation far exceed Paper 1's contributions.
Paper 2 likely has higher scientific impact due to strong timeliness and broad real-world relevance (enterprise AI-agent security), plus unusually convincing empirical validation: long-running production deployment at scale with clear metrics, and a new benchmark (ADR-Bench) enabling reproducibility and follow-on work. Its contributions span systems, security, and applied ML, potentially influencing tooling and standards around agent observability and defense. Paper 1 is novel and methodologically solid in combinatorial optimization/representation learning, but its immediate cross-field and real-world impact is narrower than production-proven security infrastructure.
Paper 1 presents a novel, production-proven enterprise security framework (ADR) for AI agents, addressing a critical and timely gap in securing agentic AI systems. Its deployment at Uber for 10 months with concrete metrics demonstrates real-world impact at scale. It introduces a new benchmark (ADR-Bench) and significantly outperforms baselines. The breadth of impact spans AI security, enterprise systems, and the rapidly growing agentic AI ecosystem. While Paper 2 makes a solid methodological contribution to AI safety annotation, its scope and immediate practical impact are narrower compared to the pressing need for agentic AI security infrastructure.