This assessment is based on version 1 of this paper. Version 3 is now available on arXiv — the authors may have revised their methods, results, or conclusions.
EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design
Gioele Molinari, Florian Felten, Soheyl Massoudi, Mark Fuge
Abstract
Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturing preparation. We introduce a benchmark suite with three evaluation dimensions: (1) a workflow benchmark with seven prompt styles targeting distinct cognitive demands-including direct tool use, semantic disambiguation, conditional branching, and working-memory tasks; (2) a Retrieval-Augmented Generation (RAG) benchmark with gated scoring isolating retrieval contributions to parameter selection; and (3) an High Performance Computing (HPC) benchmark evaluating end-to-end ML training orchestration on a SLURM cluster. Alongside the benchmark we present EngiAI, a Multi-Agent System (MAS) reference implementation built on LangGraph that operationalizes the benchmark by coordinating seven specialized agents through a supervisor architecture, unifying topology optimization, document retrieval, HPC job orchestration, and 3D printer control. Across four LLM backends and two EngiBench problems, proprietary models achieve 96-97% average task completion on Beams2D, while open-source 4B-parameter models reach 55-78%, with clear generational improvement. Conditional branching proves most challenging, with task completion dropping to 20-53% for the conditional style on Photonics2D. RAG gating confirms near-perfect retrieval-augmented scores () versus near-zero without retrieval, validating the evaluation design. On HPC orchestration, one model completes all pipeline steps in 100% of runs while another drops to 50%, revealing that multi-step instruction following degrades over long-running workflows.
AI Impact Assessments
(1 model)Scientific Impact Assessment: EngiAI
1. Core Contribution
EngiAI addresses a genuine gap at the intersection of LLM-based multi-agent systems and engineering design automation. The paper makes two intertwined contributions: (1) a benchmark suite with three evaluation dimensions—workflow execution under varying cognitive demands, gated RAG scoring for parameter selection, and HPC training orchestration—and (2) a reference multi-agent implementation built on LangGraph that coordinates seven specialized agents (engineering, RAG, ArXiv, search, HPC, CLI, and 3D printer control) through a supervisor architecture.
The benchmark design is the more novel contribution. The seven prompt styles (FULL, NATURAL, W-RAND, W-DERIVED, W-DISTRACT, W-COND, W-MULTI) systematically probe distinct failure modes: numerical fidelity, semantic grounding under competing values, conditional branching on simulation outputs, and working-memory demands across sequential tool calls. These go beyond standard tool-calling benchmarks by requiring agents to chain numeric outputs from one tool into downstream decisions—a realistic demand in engineering workflows that existing benchmarks do not capture.
2. Methodological Rigor
Strengths in evaluation design: The gated RAG scoring mechanism is well-conceived—it prevents agents from receiving credit for parameters guessed from parametric memory by requiring that `search_documents` be invoked. The three-condition ablation (RAG-on, RAG-off, Empty RAG) cleanly isolates retrieval contributions. The discovery that Gemini scores well on P0 even with an empty index (likely due to memorized defaults) demonstrates the mechanism's diagnostic value.
Weaknesses: The experimental scope is limited. Only two EngiBench problems (Beams2D, Photonics2D) are tested, with Photonics2D receiving only three of seven prompt styles. Four LLM backends is a modest sample, and the two open-source models come from the same family (Qwen3), limiting generalizability claims. The HPC benchmark is restricted to two proprietary models. The paper acknowledges these limitations but they substantially constrain the conclusions.
The analysis is entirely descriptive—no statistical significance tests, confidence intervals, or sensitivity analyses are provided. With 15 runs per cell (3 seeds × 5 samples), the standard deviations reported are sometimes large relative to mean differences (e.g., W-COND task completion for Qwen3.5-4B: 0.60 ± 0.49), making it difficult to draw firm conclusions about model comparisons in several cases.
The composite scoring methodology, while documented transparently with weights justified via AHP analysis, involves many degrees of freedom. The 3D watertightness sub-score is uniformly zero due to a pipeline limitation (non-manifold meshes from threshold-based STL extraction), effectively wasting 11% of the design quality weight. This is acknowledged but not corrected.
The absence of a single-agent baseline with identical tool access is a notable omission for validating the multi-agent supervisor architecture's value.
3. Potential Impact
Practical relevance: The system addresses a real need—making ML-driven engineering design accessible to non-experts through natural language. The integration of topology optimization, RAG, HPC orchestration, and 3D printer control in a unified framework is practically valuable, even if each component individually is not novel.
Benchmark utility: If released publicly, the benchmark could become a useful evaluation tool for the growing number of engineering agent systems. The prompt-style taxonomy (especially W-COND and W-DISTRACT) provides reusable evaluation patterns applicable beyond engineering to any domain requiring conditional reasoning over tool outputs.
Findings with broader implications: The conditional branching results are the paper's most interesting finding. W-COND performance drops sharply on Photonics2D (best model: 53% TC), and the failure mode is specifically *branch inversion*—agents parse the conditional structure correctly but swap the branches. This is a precise, actionable insight about LLM reasoning that could inform prompt engineering and model development broadly. The HPC orchestration finding that GPT-5-mini degrades from 70% to 50% on natural vs. explicit prompts at the final step similarly provides useful evidence about multi-step instruction following degradation over long-horizon workflows.
4. Timeliness & Relevance
The paper is highly timely. LLM-based agent systems are proliferating rapidly, but engineering-specific evaluation frameworks lag behind. The positioning against Table 1 (showing no prior system covers all six capability dimensions) is compelling, though the comparison is somewhat apples-to-oranges since the systems address different design phases and levels of complexity.
The inclusion of HPC orchestration as a benchmark dimension is forward-looking—as ML-augmented design becomes more common, the ability to autonomously manage training pipelines becomes increasingly important.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Overall Assessment
This is a competent systems paper that identifies a real evaluation gap and proposes a reasonable benchmark to fill it. The prompt-style taxonomy and gated RAG scoring are genuinely useful contributions. However, the experimental scope is limited, the agent architecture is not particularly novel, and the analysis lacks statistical rigor. The most impactful elements are the benchmark design patterns (especially conditional branching and semantic disambiguation), which could transfer to other domains. The findings, while interesting, are preliminary given the narrow model and problem coverage.
Generated May 20, 2026
Comparison History (22)
Paper 2 introduces a comprehensive benchmark suite and a multi-agent framework bridging LLMs with complex, real-world engineering workflows like HPC orchestration and 3D printing. Benchmarks in emerging intersections of AI and physical sciences typically drive significant future research and garner high citations. While Paper 1 presents a strong algorithmic contribution for ranking and explainability, Paper 2 offers broader interdisciplinary impact, higher potential for real-world application, and addresses the highly timely challenge of evaluating multi-agent systems in specialized domains.
Paper 2 likely has higher impact due to broader applicability and timeliness: it delivers an end-to-end multi-agent engineering design framework plus a benchmark spanning workflow reasoning, RAG isolation, and real HPC/SLURM orchestration with manufacturing integration. This supports real-world adoption in engineering pipelines and provides a standardized evaluation suite across models and tasks. Paper 1 is novel and methodologically useful for direct optimizer evaluation, but is narrower in scope (harness optimization) and its benchmark size/domain breadth is more limited, reducing cross-field impact.
Paper 1 likely has higher impact due to timeliness and breadth: it targets the rapidly expanding LLM-agent ecosystem and provides a benchmark suite plus reference multi-agent implementation spanning retrieval, simulation, HPC orchestration, and manufacturing control. Benchmarks and tooling can become community infrastructure with broad cross-domain adoption. The methodology includes gated RAG evaluation and varied prompt/agent workflow tests, supporting rigorous comparisons across models. Paper 2 offers strong algorithmic advances for OOP/POMDP fragments with large performance gains, but the niche scope and narrower applicability may limit overall impact despite higher theoretical rigor.
Paper 2 likely has higher scientific impact due to broader applicability and stronger real-world relevance: it delivers a benchmark suite spanning workflow prompting, RAG attribution via gated scoring, and HPC/SLURM orchestration, plus a reference multi-agent implementation integrating simulation, retrieval, and manufacturing (3D printing). This creates reusable infrastructure for evaluating and building engineering-design agents across ML, HCI, systems, and CAD/CAE domains. Paper 1 is novel for ToM/BDI persuasive dialogue and provides a dataset and reasoning framework, but its impact is more niche (persuasion/ToM) and less directly tied to deployable engineering workflows.
Paper 2 introduces both a novel multi-agent framework and a comprehensive benchmark suite for LLM-driven engineering design, combining simulation, retrieval, and HPC orchestration with rigorous empirical evaluation across multiple LLM backends. It addresses a clear gap in evaluating multi-agent systems for engineering tasks and provides reproducible quantitative results. Paper 1 presents a governance architecture for enterprise agents that, while practically useful, is primarily a demo/system paper describing policy enforcement mechanisms without deep empirical validation or novel algorithmic contributions. Paper 2's broader methodological contribution and benchmark utility give it higher potential impact.
Paper 1 offers a clearer methodological contribution: a structured benchmark suite with gated RAG scoring, multi-agent workflow dimensions, and an HPC orchestration benchmark, plus a reference MAS implementation and comparative results across multiple LLMs. This is timely for evaluating agentic LLM systems and is likely reusable across engineering, ML evaluation, and tool-using agent research. Paper 2 is relevant infrastructure for publishing and may have practical adoption impact, but the abstract suggests less technical novelty and rigor (evaluation metrics/controls unclear), making its scientific impact less certain.
Paper 2 introduces both a novel multi-agent framework and a comprehensive benchmark suite for LLM-driven engineering design, addressing a clear gap in evaluation methodology. It provides rigorous empirical results across multiple LLM backends, evaluation dimensions (workflow, RAG, HPC), and problem domains, offering reproducible scientific contributions. Paper 1, while practically valuable, is primarily a demo/system paper describing a governance architecture without substantial empirical evaluation or novel scientific methodology. Paper 2's benchmark contributions have broader impact potential across the engineering AI and multi-agent systems communities.
Paper 2 has higher likely scientific impact: it contributes a concrete multi-agent framework plus a benchmark suite with clearly defined evaluation dimensions, measurable results across multiple LLM backends, and methodological controls (e.g., gated RAG scoring) that strengthen rigor and reproducibility. Its applications span engineering design, simulation/RAG, HPC orchestration, and manufacturing workflows, making it broadly useful to both AI-systems and engineering communities. Paper 1 is timely and potentially transformative as infrastructure, but appears more platform/proposal-oriented with impact dependent on adoption and less on generalizable scientific evaluation.
Paper 1 addresses a fundamental gap in LLM scientific reasoning by systematically investigating and improving logical faithfulness, which is broadly applicable across scientific domains. Its methodology—assessment criteria and logicality-guided training—offers transferable contributions beyond physics. Paper 2, while practical and well-constructed, is more narrowly focused on engineering design benchmarks and multi-agent orchestration. Paper 1's focus on the essence of scientific reasoning (logicality) has broader implications for the growing field of LLM-based science, making it likely to influence more research directions and attract wider citation.
Paper 2 addresses a broader interdisciplinary problem (survey methodology + LLMs + disaster research) with novel theoretical contributions (PMT-constrained knowledge graph, A-TLM method, subgroup-stratified bias auditing). It introduces methodologically rigorous comparisons against established statistical baselines, proposes new reporting standards, and has clear real-world applications in disaster preparedness and survey science—fields with massive user bases. Paper 1, while technically solid, is more narrowly focused on engineering design benchmarking for LLM agents, serving a smaller community. Paper 2's contributions to imputation methodology and bias auditing have broader cross-field applicability.