Empirical Evaluation of Task-Based Permission Scoping Architecture for AI Agents

Scientific Impact Assessment

Core Contribution

This paper is the empirical companion to a prior architecture proposal (Noyan, 2026, same author) for task-based permission scoping of enterprise AI agents. The central problem — the "task-context mismatch," whereby agents inherit static, role-wide credential envelopes that a compromised or misaligned agent can later abuse — is framed as a *prevention* problem (removing credentials) rather than a *detection* problem (monitoring behavior). The paper's chief novel act is executing the evaluation end-to-end on a 600-prompt synthetic dataset: it (1) shows a fine-tuned 355M RoBERTa-large security gate roughly matches few-shot Claude Haiku 4.5 (macro-F1 0.881 vs 0.886, precision 0.897 vs 0.842, residual risk 0.63 vs 1.12); (2) introduces a "severity-weighted attack-surface elimination" metric showing the classifier closes 84.4% of the exposed surface vs 27.9% for the role ceiling alone; and (3) evaluates a policy-derived prohibition layer that contributes nothing on top of a precise classifier but recovers 11.5–13.6% surface in ceiling-fallback. A conceptually sharp framing runs through the paper: AI agents are the *first principal type* for which task-granular least privilege is enforceable, because their tasks arrive as machine-readable text, collapsing the per-task provisioning cost that made task granularity impractical for humans.

Methodological Rigor

The experimental design is above-average for this emerging area: a weak floor (TF-IDF), a frontier baseline (Haiku few-shot), a six-configuration threshold sweep, an isolated-layer ablation ladder (C0/C1/S2/C2/C3), and an honest fallback analysis. The paper commits to a pre-registered deployability standard (undershoot <10%, precision ≥0.90) and candidly reports that *no configuration meets it* — a refreshingly non-triumphalist stance. The methodological observation that defense-in-depth backstops are systematically undervalued by end-to-end marginal metrics, and must be evaluated by ablating preceding layers, is a genuinely useful insight. However, rigor is capped by real weaknesses: the test set is only 100 records (the author himself notes the 0.5-point F1 gap is negligible), all data is synthetic from a single fabricated company, rules were derived by a single author, train and test share the same six departments (so memorization of department priors cannot be separated from genuine policy learning), and *all traffic is benign* — the very adversarial prompt-injection threat model that motivates the work is never tested.

Potential Impact

Enterprise agent security and least-privilege scoping is a fast-emerging bottleneck (OWASP LLM06, the "lethal trifecta," AI control literature). The paper offers practitioners a concrete, cheap, self-hostable design pattern and a reusable metric. The scalable-oversight argument — that a weak, cheap classifier can supervise an arbitrarily strong agent because it judges *user input* rather than *agent behavior*, sidestepping the capability-gap constraint of AI control protocols — is the most portable idea here and could be cited beyond this specific system. The "serverless credential allocation" vision (per-task credential envelopes torn down after completion) is a plausible architectural direction. That said, impact is constrained: the evidence base is a single synthetic dataset, deployment is aspirational, and the work originates from an independent researcher without institutional replication.

Timeliness & Relevance

Very high. This directly targets a live problem as agentic deployments proliferate, and it engages current threads (alignment faking, in-context scheming, indirect prompt injection, AI control) coherently.

Strengths & Limitations

Strengths: intellectually honest reporting (fails its own bar), clean ablation methodology, strong reproducibility (code, data, policy doc, cached probability matrices, and the fine-tuned checkpoint all released), a genuinely useful conceptual reframing of task granularity for machine principals, and a well-motivated metric. Limitations: synthetic-only, small test split, no adversarial evaluation, single-author rule derivation with unmeasured fidelity, train/test department overlap, the runtime enforcement mode (identified as the *correct* deployment for prohibitions) designed but not evaluated, and heavy dependence on the author's own prior dataset and framing. Language is non-native with occasional awkward phrasing, though organization is clear.

Overall

A solid, honest, timely systems-evaluation paper that meaningfully advances a specific and increasingly relevant idea, but whose evidentiary weight is limited by synthetic data, small scale, and absence of adversarial testing. It is more likely to be a useful, cited reference point within the agentic-security niche than a field-shaping contribution.