Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost
Simon Dennis, Rivaan Patil, Kevin Shabahang, Hao Guo
Abstract
Agent orchestration frameworks have proliferated, collectively exceeding 290,000 GitHub stars across LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, Semantic Kernel, Strands, and LlamaIndex. All follow the same pattern: an external orchestrator above the LLM, injecting instructions and routing decisions every turn. Recent work has shown this architecture is dominated for procedural tasks by simply providing the procedure in a frontier model's system prompt [Dennis et al., 2026a], at the cost of consuming the context window, requiring a frontier model for every conversation, and exposing proprietary procedures to third-party providers. Compiling the procedure into the weights of a small fine-tuned model -- creating a subterranean agent -- should resolve all of these concerns, and prior work (SimpleTOD, FireAct, SynTOD, WorkflowLLM, Agent Lumos) has shown the technique works. Yet developer adoption has overwhelmingly favored orchestration. We identify three perceived barriers and address each empirically across travel booking (14 nodes), Zoom support (14 nodes, product-specific knowledge), and insurance claims (55 nodes, 6 decision hubs).
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper addresses the gap between academic work on compiling procedural knowledge into LLM weights and industry practice, which overwhelmingly favors external orchestration frameworks. The authors identify three perceived barriers to adoption—quality, cost, and flexibility—and systematically address each across three domains of increasing complexity (travel booking: 14 nodes, Zoom support: 14 nodes with domain knowledge, insurance claims: 55 nodes). The key claim is that fine-tuning small (3B–8B) models on synthetic conversations generated from procedure flowcharts yields agents that achieve 87–98% of frontier in-context quality at 128–462× lower cost, with a recompile cycle of 30–50 minutes.
The conceptual contribution—"persistent structure belongs in the weights, transient state belongs in the prompt"—is a clean architectural principle that could reshape how practitioners think about deploying procedural agents.
Methodological Rigor
Strengths. The experimental design has several commendable features:
Weaknesses. Several methodological concerns temper enthusiasm:
1. LLM-as-judge evaluation. All quality assessments rely on LLM judges. The authors use Claude as both data generator and primary judge, which creates a circularity concern despite the GPT-4.1 replication. No human evaluation is conducted, which is a notable gap for claims about "naturalness" and "graceful handling."
2. Synthetic-only evaluation. All scenarios are generated synthetically. There's no evaluation on real user interactions, which could reveal failure modes not captured by a simulated user that, by design, produces "contextually appropriate" responses.
3. User simulator limitations. The user simulator (Claude Sonnet 4.5) likely produces more cooperative, coherent users than real customers. The claim that compiled models handle edge cases well (87–98% of frontier) may not generalize to adversarial or genuinely confused users.
4. Cost analysis assumptions. The cost comparison assumes self-hosted inference on reserved A100s at $2.50/hr with optimal batching. Real deployment involves engineering overhead, monitoring, failover, and utilization inefficiencies not captured here.
5. The GPT-4.1 judge tells a somewhat different story. Under GPT-4.1, the LangGraph orchestrator leads the compiled model on more metrics, and the quality advantage is less clear-cut. The authors acknowledge this but frame both judges as supporting their conclusions.
Potential Impact
Practical impact could be substantial. The paper directly addresses practitioners deploying customer-facing conversational agents, a large and growing market. The 128–462× cost reduction is compelling for high-volume deployments (call centers, support chatbots). The 30–50 minute recompile cycle makes the approach viable for organizations with evolving procedures.
Conceptual impact is moderate. The idea of distilling agent capabilities into weights is not new (SimpleTOD, FireAct, etc.), and the authors acknowledge this. The contribution is primarily empirical—demonstrating the technique works at practical scale with comprehensive baselines—rather than introducing a novel method.
Influence on adjacent fields may include: (1) enterprise AI deployment patterns, shifting from API-dependent to self-hosted architectures; (2) data privacy applications where exposing procedures to third-party APIs is unacceptable; (3) edge deployment scenarios where latency matters.
Timeliness & Relevance
The timing is excellent. Agent frameworks are proliferating rapidly (290K+ GitHub stars cited), yet reliability remains poor (the paper cites 60% pass@1 agents showing only 25% consistency). There is clear industry demand for more reliable, cheaper agent deployment. The paper positions compilation as a practical alternative at precisely the moment when orchestration fatigue is setting in.
The paper also arrives as fine-tuning infrastructure has matured (vLLM, DeepSpeed ZeRO-3), making the proposed pipeline technically accessible. The framing as a "CI/CD cycle" rather than "retraining" is strategically effective for practitioner audiences.
Strengths & Limitations
Key strengths:
Notable limitations:
Overall Assessment
This is a well-executed empirical study that packages known techniques (synthetic data generation from flowcharts, full fine-tuning) into a compelling practical narrative. The main contribution is not methodological novelty but rather the systematic, barrier-by-barrier comparison that makes the case for adoption. The cost analysis is the strongest element—two orders of magnitude is hard to ignore. The quality analysis, while thorough statistically, is undermined by reliance on LLM judges and synthetic users. The paper would be significantly strengthened by even a small-scale human evaluation or real-world deployment study.
The work is most impactful as an engineering-oriented position paper backed by solid experiments, rather than as a fundamental scientific advance.
Generated May 22, 2026
Comparison History (23)
Paper 2 is more novel and broadly impactful: it introduces a topology-based, layerwise representation of ill-posedness via persistent homology and uses it for both detection and activation steering—bridging TDA, interpretability, and controllability. The method is model-agnostic, interpretable, and validated across multiple LLMs/datasets with sizable gains, suggesting methodological rigor and cross-field relevance. Paper 1 targets an important engineering direction (compiling workflows into weights) with clear applications and cost benefits, but is closer to an incremental consolidation/validation of existing fine-tuned agent approaches, with narrower scientific breadth.
Paper 2 identifies a fundamental and counterintuitive failure mode of LLMs—inverse scaling in forecasting under tail risk and superlinear growth—with broad implications across finance, epidemiology, and AI safety. It challenges the prevailing assumption that more capable models are uniformly better, provides mechanistic decomposition, and offers actionable recommendations for evaluation methodology. This has wider cross-disciplinary impact and addresses a critical blind spot in LLM evaluation. Paper 1, while practically useful, is more incremental—compiling workflows into weights is well-established, and the contribution is primarily engineering-focused with narrower scope.
Paper 1 addresses a highly practical problem with immediate industry relevance—compiling agentic workflows into LLM weights to reduce cost by 100x while maintaining quality. It targets a massive developer ecosystem (290K+ GitHub stars across frameworks) and provides empirical evidence across multiple real-world domains. Its practical applicability and clear cost-benefit proposition give it high near-term adoption potential. Paper 2, while intellectually ambitious in cataloguing impossibility results as design specifications, reads as a broad thesis with many claims spanning diverse subfields, making each individual result less deeply validated. Its impact is more speculative and theoretical.
Paper 1 is more novel by enabling autonomous agents to self-evolve via verified source-code rewriting, expanding adaptation beyond prompt/workflow layers to the actual harness (routing, invariants, hooks). It proposes a rigorous, safety-aware pipeline (evidence batching, deterministic stages, replay-based verification, gated rollout/rollback) with concrete performance gains, and has broad implications for reliable long-lived agent systems and software maintenance. Paper 2 is timely and practical, but “compiling workflows into weights” is a more established direction (several cited predecessors), making its incremental novelty and cross-field impact comparatively lower.
Paper 1 likely has higher scientific impact due to a more foundational contribution: it formalizes autonomous exploration for LLM agents with a verifiable metric (Exploration Checkpoint Coverage) and proposes a general training paradigm (Explore-then-Act) applicable across environments and agent embodiments. This advances methodology for robustness and generalization, with broad relevance to RL, embodied AI, and agent evaluation. Paper 2 is timely and practically important for cost/privacy in procedural workflows, but appears more applied/engineering-focused and task-domain bounded, building on an existing “compile workflows into weights” line rather than introducing a new general capability metric/paradigm.
Paper 1 is more scientifically novel and cross-disciplinary: it proposes a modular LLM augmentation framework tightly integrating topology-aware molecular perception, diffusion-based molecular generation, and reaction-aware reasoning via learned interfaces, directly addressing representation gaps between text and chemical structures. This enables broad, high-impact applications in drug discovery and synthesis, with stronger methodological breadth across multiple chemical tasks and an open-source 8B system. Paper 2 is timely and practically valuable for LLM deployment cost/efficiency, but it is more incremental relative to prior “compile workflows into weights” work and is narrower in domain generality despite strong engineering relevance.
Paper 1 addresses a highly practical and timely problem—replacing expensive agentic orchestration frameworks with fine-tuned small models—backed by empirical evidence across multiple real-world domains, offering two orders of magnitude cost reduction near frontier quality. Its direct relevance to the massive developer community using agent frameworks (290K+ GitHub stars) gives it enormous potential for real-world adoption. Paper 2 presents a solid incremental improvement to KV cache compression, but operates in a more crowded research space with narrower immediate impact. Paper 1's novelty in bridging the gap between agent architectures and model fine-tuning, combined with its practical implications, gives it higher estimated impact.
Paper 1 targets a broad, timely shift in agentic AI: replacing external orchestration with “compiled” procedures in model weights, promising major cost, privacy/IP, and deployment benefits. If validated, it could materially change how production agents are built across many domains, impacting both research and industry practice. It also directly addresses adoption barriers with multi-workflow evaluations (including a larger 55-node case). Paper 2 is a solid, practical efficiency method for long-video MLLMs, but is more incremental within an active line of token pruning/selection and may have narrower cross-field impact.
Paper 2 likely has higher scientific impact due to a more novel, generalizable algorithmic contribution (exploration-aware RL with variationally derived rewards and action grouping) that can transfer across many agent settings and benchmarks, with immediate relevance to test-time scaling and adaptive tool use. Its methodology appears more rigorous and broadly applicable (text + GUI agents, public code/models), enabling follow-on work. Paper 1 targets an important engineering/economics problem (compiling workflows into weights) with clear applications, but is more incremental relative to prior “workflow distillation” lines and may be narrower/less theoretically foundational.
Paper 2 proposes a foundational architectural shift for agentic systems by introducing event-sourced reactive graphs. This paradigm offers novel theoretical properties like deterministic replay, cheap forking, and causal lineage, which are critical for debugging, auditing, and self-improving agents. While Paper 1 provides a highly practical and cost-effective optimization for deploying existing workflows, Paper 2's fundamental rethinking of state management and coordination is more likely to inspire broader methodological changes and future research across the field of autonomous agents.