Formal Skill: Programmable Runtime Skills for Efficient and Accurate LLM Agents
Xi Zhang, Meijun Gao, Yuntian Zhao, Xinyu Tan, Yilun Yao, Feiyu Wang, Yanshu Wang, Dingsiyi
Abstract
Large Language Model (LLM) agents increasingly act inside real workspaces, where tools and skills determine whether model reasoning becomes reliable action. Existing skills remain largely informal: Markdown skills and instruction packs encode procedures as long natural-language documents, while function calling, Model Context Protocol (MCP) servers, and framework tools structure individual actions but usually leave workflow state, policy enforcement, and completion discipline outside the skill itself. We introduce Formal Skill, a runtime-native abstraction that represents reusable capability with JSON metadata and action schemas, reliable Python executors, hook-governed control logic, Formal Skill routing, and skill-local runtime state. By moving reusable procedure from repeated prompt text into executable state machines and hook policies, Formal Skill gives agents a token-efficient and enforceable control surface. We implement the abstraction in FairyClaw, an open-source event-driven runtime for executable, observable, and composable Formal Skills. On Harness-Bench, FairyClaw obtains highly competitive average scores while using substantially fewer tokens, with especially strong results on tasks that expose the role of Formal Skill.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Formal Skill — Programmable Runtime Skills for Efficient and Accurate LLM Agents
1. Core Contribution
The paper introduces Formal Skill, an abstraction that converts reusable agent capabilities from natural-language prompt documents into structured, executable runtime objects. The five components—JSON metadata/schemas, Python executors, lifecycle hooks, skill-local runtime state, and routing metadata—collectively form a "runtime contract" that shifts procedural invariants from soft prompt instructions to enforceable code. The implementation, FairyClaw, is an event-driven agent runtime, and CODEREPAIROPS serves as a concrete instantiation for code repair tasks.
The core problem addressed is well-articulated: existing agent skills (Markdown skill files, prompt-based procedures) are token-expensive, weakly enforced, and lack explicit state for recovery. The solution—formalizing skills as state machines with hook-governed control logic—is conceptually clean and practically motivated. The division of labor (model reasons, runtime enforces) is a sound architectural principle.
2. Methodological Rigor
Strengths in design exposition: The paper is thorough in describing the hook pipeline, state machine transitions, and the CODEREPAIROPS case study. Table 1 (phase policy), Table 2 (hook program), and the algorithms provide concrete implementation details that go beyond hand-waving.
Weaknesses in evaluation: This is where the paper is most vulnerable:
3. Potential Impact
The conceptual contribution—formalizing agent skills as executable state machines rather than prompt documents—addresses a genuine pain point in production agent systems. If adopted, this could influence:
However, the practical adoption barrier is notable and the authors acknowledge it: authoring Formal Skills requires writing JSON schemas, Python executors, hooks, and state machines—a significant step up from writing Markdown. The suggested bootstrapping path (LLMs compiling informal skills into formal ones) is speculative and unvalidated.
4. Timeliness & Relevance
The paper is highly timely. The rapid proliferation of coding agents (Claude Code, Codex CLI, Devin, etc.) and the emergence of standards like MCP create a real need for more principled skill abstractions. The observation that existing skills are "informal" and that this informality causes token waste, weak enforcement, and brittle recovery is well-aligned with current practitioner pain points.
The framing against Anthropic's Agent Skills specification and MCP is strategically relevant, positioning Formal Skill as a complementary layer rather than a replacement.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations:
Summary
This paper makes a timely and conceptually clean contribution to LLM agent architecture by formalizing skills as executable runtime objects. The implementation is concrete and the token efficiency gains are notable. However, the evaluation is insufficient to strongly support the claims: it lacks ablations, uses a narrow and potentially non-independent benchmark, reports no statistical measures, and relies heavily on a single detailed task example. The contribution is more of a well-motivated engineering framework than a rigorously validated scientific advance.
Generated May 20, 2026
Comparison History (23)
Paper 2 introduces a fundamental abstraction for LLM agent architecture, improving token efficiency, reliability, and workflow execution. Because it addresses core infrastructural challenges in AI agents, it has broad applicability across many domains. Paper 1 offers a valuable and rigorous application for clinical psychology, but its impact is more domain-specific compared to the foundational AI system improvements in Paper 2.
Spreadsheet-RL addresses a highly practical and widely-applicable problem (spreadsheet automation) with a concrete RL-based methodology, new benchmark datasets, and clear quantitative improvements. Its combination of a scalable data collection pipeline, domain-specific benchmarks, and a realistic Excel environment provides substantial contributions with broad real-world impact. Paper 2 introduces an interesting abstraction for LLM agent skills but is more incremental in nature—primarily an engineering/architectural contribution with less rigorous evaluation (no clear quantitative baselines) and narrower scope of impact.
Paper 1 has higher scientific impact potential because it introduces a cross-domain benchmark with explicit baselines/ablations and identifies general operating regimes for when coordinated AI agents help scientific inference under partial evidence—an insight that can guide evaluation and deployment across multiple scientific fields. Its applications span disparate domains (disease emergence, exoplanets, scientometrics, representation), increasing breadth and relevance. Paper 2 is practically valuable for LLM agent engineering (runtime skills, token efficiency) but is more incremental/system-focused with narrower scientific generalization beyond agent tooling.
Paper 2 addresses a fundamental gap in evaluating uncertainty-augmented systems across all of ML/AI, proposing a principled family of metrics grounded in proper scoring rules. This has broad applicability across classification, generation, and high-stakes decision-making domains. Its theoretical rigor (proper scoring rules) and generality give it wider cross-field impact. Paper 1, while practically useful, is more of an engineering contribution focused on LLM agent runtime optimization—an important but narrower systems contribution with less fundamental scientific novelty.
Paper 1 introduces a principled, theoretically grounded family of evaluation metrics (proper scoring rules) for uncertainty-augmented systems, addressing a fundamental gap in how such systems are assessed across multiple domains. Its contributions are broadly applicable to any high-stakes decision-making setting involving uncertainty. Paper 2 presents an engineering framework for LLM agent skills—useful but more incremental and narrowly scoped to the LLM agent tooling ecosystem, with less theoretical depth and narrower cross-field impact.
Paper 1 introduces a fundamental architectural abstraction ('Formal Skill') for LLM agents, addressing critical challenges in token efficiency, reliability, and state management. Its open-source runtime framework has broad applicability across all domains utilizing AI agents. In contrast, Paper 2 is a domain-specific empirical evaluation of existing LLMs in healthcare. While Paper 2 has strong real-world relevance, Paper 1's foundational methodological innovations provide a higher potential for broad, cross-disciplinary scientific impact in the rapidly evolving field of autonomous AI.
Paper 2 introduces a generalizable framework for LLM agents, addressing critical bottlenecks like token efficiency, reliable execution, and state management. Its broad applicability to AI agents in real-world workspaces promises significant cross-domain impact. In contrast, Paper 1 is a narrow case study focusing on a single mathematical problem in Lean 4, which, while useful for the theorem-proving niche, lacks the broader methodological innovation and widespread practical applications of Paper 2.
Paper 2 presents a counterintuitive and thought-provoking finding—that noisy observations can improve embodied LLM agent performance by breaking repetitive action loops—which challenges fundamental assumptions about observation fidelity in robotics/AI. This insight has broad implications for evaluation methodology, embodied AI system design, and understanding LLM reasoning failures. Its novelty and cross-disciplinary relevance (robotics, cognitive science, AI evaluation) give it higher potential impact. Paper 1, while practically useful, is primarily an engineering contribution (a runtime framework) with more incremental value to the agent tooling ecosystem.
Paper 2 offers a fundamental theoretical insight—that volatility and stochasticity drive exploration in opposite directions—with formal proofs extending the Gittins index framework and a novel closed-form exploration bonus (CAUSE). It bridges computational neuroscience, decision theory, AI exploration strategies, and psychiatric modeling, giving it broad interdisciplinary impact. Paper 1 is a solid engineering contribution (a runtime abstraction for LLM agents) but is more incremental and narrowly scoped to the LLM tooling ecosystem, with impact likely limited to agent framework design rather than foundational science.
Paper 1 introduces a broadly applicable, runtime-native abstraction for LLM agents (executable skills with state, routing, and policy hooks) that targets a central bottleneck in deploying agents: reliability, enforceability, and token efficiency. If adopted, it could impact many domains using agents (automation, tooling ecosystems, safety/policy, evaluation) and influence standards similar to function-calling/MCP. The open-source runtime plus benchmark evidence strengthens practicality. Paper 2 is timely and useful for multimodal ERC, but its impact is narrower (a specific task) and more incremental within a crowded VLM steering space.