Constraint Dependency Graph Retrieval for Compliance Detection on Long SKILL Documents
Shuaitao Zhao, Feng Ni, Lichao Ma, Yuchen Wei, Shuaike Li, Jiaye Lin, Fei Han, Yang Wei
Sensible, deployable closure-based retrieval plus joint RL/OPD training for compliance auditing, but incremental components, proprietary data, and evaluation inconsistencies limit confidence and impact.
Abstract
Large Language Model (LLM) agents carry out business operations under enterprise documents we refer to as SKILL documents, which take the form of standard operating procedures (SOPs) that state the conditions to verify and the actions to take before an operation proceeds. Long, densely cross-referenced SKILL documents exceed the context budget of small deployed auditors. Similarity-based retrieval then brings in irrelevant rules while omitting mandatory prerequisites, and the two-stage audit policy provides no trainable signal for the retrieval stage. We present SkillCDG, a compliance detection framework that reformulates each SKILL document as a constraint dependency graph, shifting rule retrieval from similarity-based matching to closure-based selection under the deployment's context budget. Each atomic rule is anchored to its source text span as a graph node, and mandatory prerequisite relations form the sole edge type. Policy-generated retrieval keywords activate seed rules, whose dependency closure yields a compact and conditionally complete rule context within that budget. The same graph also supports a joint optimization scheme. The retrieval stage is trained with group-relative policy optimization using a verifiable reward that combines output-format validity with downstream verdict correctness, with closure non-emptiness serving as a validity gate, while selective on-policy distillation concentrates teacher supervision on high-entropy reasoning tokens, with verdict and termination tokens always included, to improve verdict accuracy. Experiments on three enterprise datasets and two public benchmarks show that SkillCDG improves accuracy and F1 over the full-document and retrieval-based baselines.
AI Impact Assessments
(1 model)Core Contribution
SkillCDG targets compliance auditing of LLM agent responses against long enterprise SOP documents ("SKILL documents") using a small (4B) judge. The paper has two ideas. (1) Compile each document offline into a constraint dependency graph. Nodes are atomic rules (condition, action, verbatim source span). The only edge type is `require`, a mandatory prerequisite. At inference, policy-generated keywords activate seed rules, and the judge sees the transitive closure of the seeds. This is "closure-based" rather than similarity-based retrieval, with claimed tightness and conditional completeness. (2) Train retrieval and judgment jointly. The retrieval keywords are trained with GRPO, with a reward of format validity plus averaged downstream verdict correctness. The judge is trained with entropy-gated on-policy distillation from a Qwen3-30B-A3B teacher, using the top 20% of high-entropy tokens plus verdict and termination tokens.
The framing is sensible and practically motivated: relevance differs from sufficiency, and prerequisites can be lexically unrelated to the scenario. Individually, the components are mostly recombinations of known pieces: graph-structured RAG, BFS closure, GRPO with a downstream reward, and high-entropy-token-gated OPD. The novelty is the problem-specific integration and the "closure as completeness guarantee" framing, not any new algorithm. The completeness guarantee is only relative to the extracted graph, which the paper acknowledges ("conditionally complete"), and extraction quality is never evaluated.
Methodological Rigor
Strengths: five datasets (three proprietary, two public), an ablation, a controlled length-scaling experiment (conversations fixed, inapplicable clauses added), and a cost/latency analysis. The closure algorithm has a correct proof sketch and sensible edge-case handling (empty seed, span error, budget exceeded, no truncation).
Concerns are substantial:
Potential Impact
Practical impact is in enterprise agent auditing (customer service, refunds, access control), where small on-premise auditors and traceable, source-anchored verdicts matter. The graph-compilation approach is deployable and the latency and token savings (76–86% prompt reduction, 1.1–1.3× latency) are attractive. For the broader research community, the idea of using structural closure as a verifiable validity signal for training an intermediate retrieval step is mildly interesting, but the signal used is only non-emptiness, which is weak. The paper itself lists prerequisite-chain-completeness rewards as future work. The proprietary datasets limit community follow-up, and the code/data release status is unstated.
Timeliness
The topic is very timely: agent skills/SOPs, policy compliance benchmarks (CompliBench, PROBE, PolicyGuide), and RLVR with GRPO plus OPD are active. The paper fits the trend but also competes with many concurrent works. Its "SKILL document" terminology links to the currently popular agent-skills paradigm.
Strengths
Limitations
Overall this is a competent applied-systems paper with a sensible idea, moderate novelty, and evaluation weaknesses that limit confidence in how much the closure contributes to the reported gains. Expected impact is moderate within enterprise-agent auditing and RAG-for-policy niches.
Generated Nov 3, 2026
Comparison History (0)
Sensible, deployable closure-based retrieval plus joint RL/OPD training for compliance auditing, but incremental components, proprietary data, and evaluation inconsistencies limit confidence and impact.