Trace2Skill: Verifier-Guided Skill Evolution for Long-Context EDA Agents
Zijian Du, Nathaniel Pinckney
Abstract
Complex Verilog Design Problems (CVDP) challenge hardware LLM agents because solving them requires localizing verifier-relevant RTL, testbenches, include paths, and build dependencies inside large repository snapshots, making precise edits, and recovering from sparse hidden-verifier failures. We present Trace2Skill, a test-time scaling framework that improves a hardware agent without RTL-specialized model fine-tuning. Rather than training a new model or only sampling more candidate solutions, Trace2Skill treats the agent's natural-language skill as an evolvable policy. It mines repeated rollout traces for success and failure modes, converts them into dense diagnostics and oracle lessons, and uses an oracle, mutator, and selector loop to produce task-specific skills that guide later search, editing, validation, and recovery. Because final pass/fail labels are often too coarse for hard failures, Trace2Skill also supports bounded runtime dense verifier feedback that returns sanitized functional observations while keeping hidden harnesses and reference solutions inaccessible to the agent. This feedback helps guide skill evolution and agent execution by connecting skill text, verifier evidence, and downstream behavior. Across hard CVDP tasks that defeat the seed CVDP agent, including tasks that also defeat frontier coding agents, Trace2Skill with dense verifier feedback substantially improves task pass rates and produces breakthrough passes on previously unsolved tasks, without requiring high-quality fine-tuning data, specialized RTL model training, or model weight updates. The same framework provides a general test-time scaling strategy that can extend beyond digital design to other verifiable EDA tasks.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Trace2Skill
Core Contribution
Trace2Skill proposes a training-free, test-time scaling framework that evolves natural-language "skills" (prompt-level policy documents) injected into a fixed hardware design agent. The key idea is to treat the agent's instruction prompt as an evolvable artifact: after each batch of rollouts on a Complex Verilog Design Problem (CVDP), the system mines execution traces for success/failure patterns, uses an oracle LLM to distill lessons, and applies evolutionary mutation and selection to produce improved task-specific skills for subsequent generations. An optional "dense verifier feedback" mechanism provides bounded, sanitized functional observations from a hidden test harness, offering richer signal than binary pass/fail.
The framework solves 6/8 hard CVDP tasks that defeated the baseline agent (and many that also defeated Claude Code and Codex), achieving a 33.6% pass rate versus 0% for the seed agent—without any model weight updates.
Methodological Rigor
Strengths in experimental design: The paper presents a clean four-configuration ablation (C1–C4) that isolates the contributions of dense feedback alone (C2), sparse skill evolution (C3), and their combination (C4). The progression C1→C2→C3→C4 clearly demonstrates that skill evolution and dense feedback are complementary, not redundant.
Metric design is thoughtful: The decomposition into SkillQ (content quality), AgentProgressQ (execution quality with LCB aggregation), AgentVarianceQ (stability), and SelectQ (combined survivor score) addresses a real problem—binary pass/fail is too coarse for hard tasks where most rollouts fail. The point-biserial correlation of 0.90 between AgentQ and verifier outcomes provides reasonable validation of the proxy signal.
Concerns about rigor:
Potential Impact
Domain-specific value: For EDA/hardware design automation, this work addresses a genuine pain point—LLM agents struggle with long-context repository navigation, RTL editing, and recovery from sparse verification failures. The idea of evolving prompt-level guidance without retraining is practically valuable for organizations that cannot easily fine-tune frontier models on proprietary RTL data.
Broader applicability: The general framework—mining execution traces, evolutionary prompt optimization, dense proxy metrics for sparse-reward tasks—could transfer to other verifiable engineering domains (formal verification, synthesis, timing closure). However, the paper provides no evidence of such transfer.
Limitations on impact: The approach is computationally expensive (640 rollouts per task configuration), uses multiple frontier models, and requires access to a verification infrastructure (NeMo Gym). This limits immediate adoption. The task-specific (non-transferable) nature of evolved skills also limits scalability—each new task requires its own evolutionary run.
Timeliness & Relevance
The paper is timely. Test-time compute scaling is an active research frontier, and applying it to hardware design agents addresses a real gap. The CVDP benchmark and the emergence of coding agents (Claude Code, Codex) as baselines make this relevant to both the EDA and LLM agent communities. The framing as complementary to ACE-RTL (which focuses on model fine-tuning) is appropriate.
Strengths
1. Clean ablation structure that isolates feedback and evolution contributions.
2. Rich traceability artifacts—the paper provides unusually detailed trace examples (Tables 5-7) showing exactly how skills evolve and how dense feedback changes agent behavior.
3. Practical architecture that requires no model fine-tuning, making it applicable to black-box API-served models.
4. Honest limitations section acknowledging compute costs, rollout variance, and benchmark narrowness.
5. Dense metric design that addresses the real challenge of learning from sparse binary rewards in hard verification tasks.
Weaknesses
1. Very small evaluation set (8 tasks for the main ablation) limits statistical confidence in the findings.
2. Missing compute-matched baselines: No comparison against simple best-of-N sampling with 640 rollouts but no evolution, which would isolate the value of the evolutionary skill optimization loop.
3. No generalization evaluation: Skills are task-specific, and the paper explicitly notes weak cross-task transfer but provides no formal analysis of what makes skills transferable or not.
4. Expensive and complex pipeline: Three frontier LLMs, evolutionary loop, custom metric computation, and NeMo Gym infrastructure create significant barriers to reproduction.
5. Metric design opacity: Many hand-tuned weights and thresholds without ablation or sensitivity analysis. The correlation validation (r=0.90) is on seed-skill rollouts only, not on evolved-skill rollouts where the distribution may shift.
6. Limited novelty in individual components: Evolutionary prompt optimization, trace mining, and dense reward shaping are each known techniques; the contribution is primarily their combination and application to EDA.
Overall Assessment
Trace2Skill presents a well-engineered system that demonstrates meaningful improvements on hard hardware design tasks through test-time skill evolution. The ablation is clean and the traceability artifacts are exemplary. However, the evaluation is narrow (8 tasks), lacks compute-matched baselines, and the framework's complexity and cost raise questions about practical scalability. The work is a solid applied contribution to EDA agent research but would benefit from broader evaluation and stronger baselines to establish the specific value of evolutionary skill optimization over simpler scaling strategies.
Generated May 22, 2026
Comparison History (21)
AgentFugue addresses a fundamental and broadly applicable question—whether scaling out peer agents via collective reasoning can yield capability gains—introducing a general-purpose framework applicable across diverse long-horizon tasks. Its contributions (shared reasoning hub, RL-trained communication layer, ecology of reusable reasoning) are domain-agnostic and relevant to the rapidly growing multi-agent systems field. Trace2Skill, while rigorous and valuable, targets a narrower domain (EDA/Verilog design) with more specialized applicability. AgentFugue's breadth of potential impact across AI agent research gives it higher estimated scientific impact.
Trace2Skill presents a novel test-time scaling framework with a concrete skill evolution mechanism (oracle, mutator, selector loop) that addresses a well-defined and important problem in EDA/hardware design. It demonstrates breakthrough results on previously unsolved tasks without requiring model fine-tuning, offering a generalizable methodology. Paper 2 introduces a valuable benchmark but is primarily an evaluation resource. While benchmarks are important, Trace2Skill's methodological contribution—evolvable skill policies from rollout traces with dense verifier feedback—has broader applicability beyond EDA and represents a more fundamental algorithmic advance in LLM agent reasoning.
Trace2Skill addresses a broader and more impactful problem: improving LLM agents for complex hardware design tasks through a novel test-time scaling framework that avoids fine-tuning. Its skill evolution approach (mining rollout traces, oracle/mutator/selector loop) is more generalizable across verifiable domains beyond EDA. While MemAudit makes a solid contribution to LLM agent security via post-hoc memory auditing, it addresses a narrower problem (memory poisoning defense) with a more incremental methodological advance. Trace2Skill's potential to transform hardware design automation and its transferable test-time scaling paradigm give it broader cross-field impact.
Paper 1 introduces a novel test-time scaling paradigm (skill evolution from rollout traces) coupled with bounded dense verifier feedback, addressing a hard, high-stakes domain (EDA/RTL) where hidden-verifier sparsity and repository-scale localization are key bottlenecks. It reports breakthrough solves on previously unsolved tasks without fine-tuning or weight updates, suggesting strong methodological and practical significance. While Paper 2 is timely and useful for agent serving, it is more incremental systems work (parallelizing summarization) with narrower scientific novelty and impact relative to the broader implications of verifier-guided skill evolution for verifiable tasks.
Paper 1 addresses a fundamental challenge in LLM post-training (on-policy distillation with privileged context), proposing a principled method (EDGE-OPD) with broad applicability across many LLM training scenarios. The evidence masking and guided rollout techniques offer general insights for knowledge transfer. Paper 2, while valuable for EDA/hardware design, targets a narrower domain (Verilog design) with a test-time scaling framework. Paper 1's contributions to understanding distillation dynamics and preventing side-effect learning have wider cross-field impact for the LLM training community.
Paper 1 tackles a highly complex, high-value problem (automated hardware design) by introducing an innovative test-time scaling and skill evolution framework. Overcoming the limits of long-context EDA tasks without model fine-tuning offers immense real-world applications in chip design. While Paper 2 provides a valuable benchmarking tool for text-to-image prompting, Paper 1's methodological novelty and its potential to unlock breakthroughs in specialized, high-stakes engineering domains give it a higher potential for broad scientific and industrial impact.
Paper 1 explores the highly timely and impactful intersection of LLM agents, test-time scaling, and hardware design automation. Its novel approach of evolving natural-language skills via verifier feedback without fine-tuning addresses critical bottlenecks in complex reasoning tasks. While Paper 2 provides a useful benchmark for knowledge graphs, Paper 1's methodological innovation in agentic workflows and potential to significantly accelerate chip design offer a higher broader impact and technological relevance.
Paper 1 is more likely to have higher scientific impact due to its novel test-time “skill evolution” framework that leverages verifier traces and optional dense, bounded feedback to systematically improve agent behavior without fine-tuning or weight updates. This is timely for LLM-agent reliability and scales to high-value, real-world EDA workflows where verification is the ground truth. Its methodology directly targets a hard industrial bottleneck (long-context repo localization + sparse verifier signals) and proposes a generalizable verifier-guided scaling paradigm beyond hardware. Paper 2 is solid and relevant, but hierarchical residual world models are a more crowded space.
Paper 1 documents a fundamental and counterintuitive finding—inverse scaling in LLM forecasting on critical tasks involving tail risk and superlinear growth—with broad implications across AI safety, finance, epidemiology, and evaluation methodology. It challenges prevailing assumptions that more capable models are universally better, proposes actionable evaluation recommendations, and spans multiple domains. Paper 2, while technically solid, addresses a narrower problem (Verilog design agents) with more limited cross-field impact. Paper 1's findings are more likely to reshape how the community evaluates and deploys LLMs in high-stakes forecasting settings.
Paper 1 addresses a broadly impactful problem—efficient video understanding with MLLMs—relevant across computer vision, NLP, and multimedia. Its training-free framework with a novel dual perspective (similarity for redundancy, difference for key events) and spatio-temporal graph modeling offers wide applicability. Paper 2, while technically interesting, targets a narrow domain (EDA/Verilog agents) with limited cross-field impact. Paper 1's approach is more generalizable, timely given the rapid growth of MLLMs, and likely to influence a larger research community.