FORGE: Fault-Oriented Rework Code Generation for Executable Interlocking Assembly

Fuqiang Wang, Jiahao Fu, Ren Hefeng, Yining Qian, Chuanting Zhang, Cheng Tan, Yuwen Liu, Jingxuan Wei

Nov 2, 2026arXiv:2611.02536v1

cs.AI

Scorecard· 16/16

4.0/10 impact

Reasonable benchmark and verifier-based task framing, but the 'framework' is essentially SFT with ordered targets, and the evidence for its modules is weak.

Abstract

Large language models have demonstrated strong capabilities in structured reasoning and code generation, yet identifying an assembly fault does not guarantee the ability to generate executable rework programs. In interlocking assemblies, a locally plausible repair program may violate blocking dependencies, omit required recovery operations, or terminate in an incorrect configuration. We identify this challenge as fault-conditioned executable rework generation, where models must jointly reason about structural failures and generate programs that restore constrained assembly states. To benchmark this capability, we introduce FORGE-Bench, containing 4,000 tasks across 26 interlocking assemblies with five assemblies held out for evaluating generalization to unseen structures. We further propose FORGE (Fault-Oriented Rework Code Generation for Executable Interlocking Assembly), a structured state-to-program framework that decomposes rework generation into Rework Structuring and Action Grounding. Rework Structuring captures fault-induced recovery dependencies, while Action Grounding maps these dependencies into state-conditioned executable actions. Experiments show that locally reasonable repair code is insufficient for reliable assembly recovery, and that connecting structural recovery reasoning with executable action realization is essential for verified target restoration.

AI Impact Assessments

(1 model)

Core Contribution

The paper defines "fault-conditioned executable rework generation": given an observed faulty state, a target state, and structural graphs (assembly, blocking, contact), a model must decide whether rework is needed, diagnose the fault, and output a legal action program that restores the target. It contributes (i) FORGE-Bench, 4,000 tasks over 26 Luban-lock interlocking assemblies, with an assembly-disjoint 21/5 split, 27.9% no-fault cases, 160 counterfactual pairs, and a symbolic verifier that replays actions; and (ii) FORGE, a "framework" fine-tuned from Qwen3.5-9B with LoRA.

The method is thinner than the framing suggests. Per Sections 3.2–3.4, Rework Structuring and Action Grounding are not separate architectural modules. Rework Structuring is a dependency-closure procedure used to order the reference rework plan in the supervision targets. Action Grounding is symbolic-state-conditioned supervision. There are no extra heads, no auxiliary losses, and no intermediate output. Everything is serialized into one JSON target trained with standard SFT. FORGE is therefore essentially SFT on a well-constructed, verifier-checked dataset. The "framework" label oversells the technical novelty.

Methodological Rigor

Strengths:

The verifier is deterministic and checks schema, action legality, state transitions, and target restoration.

The metric suite is detailed (SSE, SAPF, VMRE, DGTR, MVE, HRQ) and macro-averaged across assemblies.

The split is assembly-disjoint.

Failure modes are categorized.

Human-judgment correlation is reported.

Weaknesses:

The headline comparison is unfair. FORGE is fine-tuned on 3,000 in-distribution tasks, while the baselines are zero-shot, with no few-shot examples, no fine-tuned open-model baseline (for example SFT on a different backbone), and no tool or verifier feedback loops. The 72.75 versus 46.44 (Gemini) gap mostly shows that task-specific fine-tuning helps. It does not show that the proposed method is superior.

The most informative comparison is the ablation, where vanilla SFT scores 67.53 and full FORGE 72.75. This is a ~5-point gain from a single seed, with no variance, confidence intervals, or significance tests.

The ablations are mixed. Removing Action Grounding actually improves SAPF (66.67 vs 61.11) and slightly raises Overall-Core over vanilla SFT. The "w/o Rework Structuring" result is odd because the module only affects target ordering.

The test set is small (1,000 tasks, 5 assemblies), so macro-averaging over five assemblies is high-variance.

The reference is "sequence-minimal" only relative to a prescribed disassembly–repair–restoration order. SSE and MVE therefore penalize valid alternative solutions, which biases toward imitation of one reference.

Baseline scores of ~20–25 on several metrics, and identical values across VMRE/MVE for some models (for example 22.50, 22.50), suggest many models mostly fail on format or legality. Details of prompt design for baselines are thin ("same public task representation").

Several metrics assign zero to missing entries, which affects the aggregate.

The LLM-as-judge uses only 60 samples. The Pearson r = 0.9887 is computed across model-level means, including FORGE variants, so it is inflated and not very informative.

The "Gemini 3.5 Flash" and "Claude Sonnet 4.6" baselines are hard to verify and cannot be assessed here.

Potential Impact

The benchmark could be a modest resource for LLM-for-assembly, neuro-symbolic planning, and stateful tool-use evaluation. The task framing, which stresses diagnosis, dependency-aware recovery, abstention on no-fault inputs, and verified end-state restoration, is reasonable and relevant to manufacturing and robotics. However, the environment is purely symbolic, built from one family of puzzles (Luban locks), with discrete actions and no perception, continuous physics, or execution feedback. Real-world transfer is speculative, and the authors acknowledge this. The method contributes little reusable technique beyond "structure your supervision by dependency closure and train on verified symbolic traces." Adoption by others is likely to be limited to the benchmark, if released. The paper cites a repository, but no explicit release statement is visible.

Timeliness & Relevance

The paper fits the current interest in agentic code generation, state-changing tool-use benchmarks, and LLM-based embodied planning, and it correctly notes that plausibility is not executability. The observation is not new: similar results are well known from planning benchmarks (Blocksworld-style dependency failures, PlanBench) and from code-as-policies work. The paper does not engage with that literature deeply. Most of the cited related work is peripheral (VLA models), while classic AI planning and repair/replanning work is missing.

Strengths

A clean, verifiable task with explicit no-fault abstention and counterfactual pairs.

Assembly-disjoint evaluation.

A thorough, well-documented metric and failure taxonomy.

Detailed training configuration and an appendix with the schema and actions.

Limitations

Limited method novelty, since the method is SFT on structured targets.

Unequal baselines (fine-tuned versus zero-shot).

Single seed and small test set.

Synthetic, one-family environment with a finite assembly set. Symbolic verification means the "physical" claim rests on a hand-coded transition model.

Reference-bound metrics penalize alternative valid solutions.

Writing is generally readable but has some inflated terminology, a few typos, and an inconsistency in Figure 4 (it appears to show GUI-agent categories such as "Click, Type Text, Hotkey" unrelated to this task, suggesting recycled material).

The main-text claim of "Certificate-guided" and "Typed Obligation Certificate" components in Figure 2 does not appear in the method text.

Overall

This is a modest benchmark-plus-SFT paper in a narrow domain. Its contribution is the task and verifier-based dataset, not an algorithmic advance. The evidence supports "fine-tuning on this benchmark helps a lot," but only weakly supports the claimed benefits of the two modules. Expected impact is low to moderate.

Rating:4/ 10

Significance 3.5Rigor 4Novelty 3.5Clarity 5

Generated Nov 3, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 16/16

4.0/10 impact

Reasonable benchmark and verifier-based task framing, but the 'framework' is essentially SFT with ordered targets, and the evidence for its modules is weak.