Harnessing LLM Agents with Skill Programs
Hongjun Liu, Yifei Ming, Shafiq Joty, Chen Zhao
Abstract
Equipping LLM agents with reusable skills derived from past experience has become a popular and successful approach for tackling complex and long-horizon tasks. However, such lessons are often encoded as textual guidance that remains largely advisory, lacking explicit mechanisms for when and how to intervene in the agent loop. To bridge the gap, we introduce HASP(Harnessing LLM Agents with Skill Programs), a new framework that upgrades skills into executable Program Functions (PFs). Rather than offering passive advice, PFs act as executable guardrails that activate on failure-prone states and modify the next action or inject corrective context. HASP is highly modular: it can be applied at inference time for direct agent-loop intervention, during post-training to provide structured supervision, or for self-improvement by evolving validated, teacher-reviewed PFs. Empirically, HASP drives substantial gains compared to both training-free and training-based methods on web-search, math reasoning, and coding tasks. For example, on web-search reasoning, inference-time PFs alone improve the average performance by 25% compared to (multi-loop) ReAct Agent, while post-training and controlled evolution achieve a 30.4% gain over Search-R1. To provide deeper insights into HASP, our mechanism analysis reveals how PFs trigger and intervene, how skills are internalized, and the requirement for stable skill library evolution.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Harnessing LLM Agents with Skill Programs (HASP)
1. Core Contribution
HASP introduces a conceptual shift in how LLM agent skills are represented and deployed. Instead of encoding skills as textual advice injected into prompts (which models frequently ignore), HASP transforms them into executable Program Functions (PFs) — typed Python objects with explicit `should_activate` predicates and `intervene` methods that directly modify agent actions or inject corrective context at runtime. This moves from advisory to interventional skill representation.
The framework operates at three levels: (1) inference-time guardrails that intercept and repair agent decisions, (2) post-training supervision derived from PF intervention records (supporting SFT, rejection sampling, and on-policy distillation), and (3) self-improving library evolution where residual failures generate new candidate PFs that undergo executable validation and teacher review before admission.
The key insight is that skills should specify *when* they activate and *how* they change behavior, not merely *what* the agent should do. This is operationalized through a state-action intervention interface that separates the base policy's proposal from the harness's correction.
2. Methodological Rigor
The experimental design is thorough, spanning three diverse domains (web-search reasoning, math, coding) with a systematic 2×3 grid of post-training configurations (fixed/evolving library × SFT/RS/OPD). The ablation studies in Table 5 are particularly well-structured, isolating contributions of individual components (PF vs. teacher), supervision signals (timing, mode, correctness, outcome), and filtering mechanisms.
Strengths in rigor:
Weaknesses in rigor:
3. Potential Impact
Direct applications: The framework's modularity is genuinely useful — the same PF interface works across inference-time intervention, training data generation, and library evolution. This could enable practical deployment patterns where PFs serve as safety guardrails or quality control mechanisms in production agent systems.
Broader influence: The paper advances the programmatic control of LLM agents, connecting to the growing literature on constrained decoding, structured generation, and agent guardrails. The idea that skills should be executable rather than advisory could influence how agent memory and experience replay systems are designed.
Limitations on impact: The heavy reliance on domain-specific PF design (16 web-search skills, 11 math skills, 12 code skills, each with hand-crafted activation predicates) limits out-of-the-box transferability. The self-improvement loop partially addresses this but still requires careful filtering infrastructure.
4. Timeliness & Relevance
The paper addresses a genuine and timely problem: LLM agents frequently exhibit recognizable failure patterns that textual prompting cannot reliably prevent. The positioning against Search-R1, AgentFlow, and other recent agent training methods (many from 2025) demonstrates awareness of the rapidly moving frontier. The emphasis on structured intervention rather than end-to-end RL provides a complementary approach that could be combined with existing methods.
5. Strengths & Limitations
Key strengths:
Notable limitations:
Summary
HASP presents a well-engineered framework with a sound conceptual contribution (executable vs. advisory skills) and strong empirical results, particularly on web-search reasoning. The ablation studies are thorough and informative. However, the heavy engineering overhead, reliance on teacher models for strongest results, and limited novelty in the underlying learning mechanisms (SFT/RS/OPD are standard) temper the fundamental contribution. The work is best understood as a sophisticated systems contribution that demonstrates the value of structured intervention in agent loops, rather than a fundamental advance in agent learning.
Generated May 19, 2026
Comparison History (19)
HASP introduces a broadly applicable framework for converting learned skills into executable program functions that intervene in LLM agent loops, with demonstrated gains across multiple diverse task domains (web-search, math, coding). Its modular design applicable at inference, post-training, and self-improvement stages gives it wider applicability and methodological contribution. Paper 2 addresses a narrower (though important) problem of quantization failures in reasoning models with targeted fixes. While practically useful, its scope is more limited to efficiency optimization of specific model configurations, whereas HASP's skill program paradigm has broader potential to influence agent design across the field.
Paper 2 likely has higher impact due to broader applicability and nearer-term real-world deployment: executable “skill programs” that intervene in agent loops can improve performance across web search, math, and coding, and can be used at inference, post-training, and self-improvement. This makes it timely for agentic LLM systems and widely transferable across domains. Paper 1 is novel and rigorous in mechanistic interpretability for CoT faithfulness, but its impact is narrower (focused on unfaithful CoT detection) and depends on circuit-tracing infrastructure and specific benchmarks.
Paper 1 addresses a fundamental question about why code in pretraining improves reasoning, providing controlled experiments at scale (10T tokens) that challenge a widely-held assumption. Its findings—that structured reasoning traces, not executable code per se, drive reasoning gains—have broad implications for data-centric optimization across the entire foundation model training community. Paper 2 presents a useful but more incremental framework for LLM agents. Paper 1's mechanistic insights and practical guidance for training data composition will likely influence a wider range of future research and industry practices.
Paper 2 likely has higher scientific impact: it introduces a mechanistic, head-level causal account of modality-conflict hallucinations across multiple MLLMs (strong novelty + rigor via path patching and ablations), yielding broadly relevant insights for interpretability and multimodal safety. Its intervention (MACI) is targeted, inference-time, and shown to transfer zero-shot across models and benchmarks, increasing real-world applicability. The findings generalize across architectures and connect to wider work on causal interpretability, making its impact broader than Paper 1’s primarily engineering-oriented agent-skill framework.
Paper 2 likely has higher impact due to timeliness and broad applicability: agentic LLMs are a rapidly expanding area, and a modular framework for executable, reusable skills can transfer across many tasks (web, math, coding) and deployment settings (inference-time guardrails, post-training supervision, self-improvement). The reported large empirical gains on strong baselines suggest practical relevance. Paper 1 addresses an important XAI evaluation gap and proposes a novel metric plus adapter, but XAI impact is often constrained by domain-specific validation needs and slower adoption compared to agent frameworks.
Paper 1 addresses a critical bottleneck in AI safety and evaluation by providing rigorous, anytime-valid statistical guarantees for adaptive auditing. Its strong theoretical foundation and applicability to AI compliance give it broader, more foundational impact compared to Paper 2's architectural improvements for LLM agents.
WorldString proposes a fundamentally new representation paradigm for physical world models—unified actionable object state manifolds learned from point clouds/RGB-D—addressing a core gap in world modeling. Its differentiable architecture enabling integration with policy learning and neural dynamics has broad implications for robotics, simulation, and digital twins. While Paper 2 (HASP) presents solid engineering contributions for LLM agent skill execution with strong empirical gains, it is more incremental within the crowded LLM agent framework space. Paper 1's foundational contribution to physical world representation has greater long-term scientific impact potential across multiple fields.
Paper 2 (HASP) likely has higher scientific impact due to its broad, timely relevance to LLM agent reliability and long-horizon task performance, with clear, reusable mechanism design (executable skill programs) and multiple integration pathways (inference-time, post-training, self-improvement). It reports substantial empirical gains across diverse benchmarks (web-search, math, coding), suggesting strong real-world applicability and cross-domain impact. Paper 1 is novel for black-box vision interpretability and ontology induction, but its applicability is more specialized to vision/XAI and may see narrower uptake compared to general agent frameworks.
Paper 2 likely has higher scientific impact due to broader cross-domain applicability and timeliness: executable skill programs for LLM agents can improve reliability across many tasks (web, math, coding) and can be adopted widely at inference/post-training/self-improvement. Its modular framework and reported large empirical gains suggest immediate real-world utility and influence across AI research and tooling. Paper 1 is innovative and potentially high-impact in clinical decision support, but its impact is narrower (cardiology/ECG), with heavier deployment/regulatory barriers and a smaller affected research community.
Paper 2 introduces a highly relevant framework for improving LLM agents, a rapidly growing and impactful area in AI. Its method of upgrading skills into executable programs demonstrates significant performance gains across multiple domains (search, math, coding). This technical innovation offers broader and more immediate applicability to the wider AI community compared to Paper 1, which primarily provides a mapping and repository tool with a more specialized impact footprint in policy and ethics.