AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions
Minghao Chen, Xinyi Hu, Zhou Yu, Yufei Yin
Abstract
Large Language Model (LLM) based agents have demonstrated proficiency in multi-step interactions with graphical user interfaces (GUIs). While most research focuses on improving single-task performance, practical scenarios often involve repetitive GUI tasks for which invoking LLM reasoning repeatedly, i.e., the ReAct paradigm, is inefficient. Prior to LLMs, traditional Robotic Process Automation (RPA) offers runtime efficiency but demands significant manual effort to develop and maintain. To bridge this gap, we propose AutoRPA, a framework that automatically distills the decision logic of ReAct-style agents into robust RPA functions. AutoRPA introduces two core innovations: (1) A translator-builder pipeline, where a translator agent converts hard-coded ReAct actions into soft-coded procedures, and a builder agent synthesizes robust RPA functions via retrieval-augmented generation over multiple trajectories; (2) A hybrid repair strategy during code verification, combining RPA execution with ReAct-based fallback for iterative refinement. Experiments across multiple GUI environments demonstrate that RPA functions generated by AutoRPA successfully solve similar tasks while reducing token usage by 82% to 96%, significantly improving runtime efficiency and reusability.
AI Impact Assessments
(1 model)Scientific Impact Assessment: AutoRPA
1. Core Contribution
AutoRPA addresses a practical but underexplored problem: how to convert the expensive, per-instance reasoning of LLM-based GUI agents into reusable, low-cost RPA scripts for repetitive task types. The key insight is that many real-world GUI tasks are recurring (e.g., booking flights, filing reports), and invoking full LLM reasoning each time is wasteful. The framework introduces a three-phase pipeline: (1) exploration via a ReAct agent that collects trajectories, (2) a translator-builder pipeline that converts hard-coded actions into soft-coded, environment-resilient RPA functions via RAG over a tree-structured trajectory database, and (3) a hybrid repair strategy combining RPA execution with ReAct-based fallback for iterative code refinement.
The problem formulation is clean and well-motivated. The distinction between "hard-coded" actions (tied to specific UI indices/coordinates) and "soft-coded" actions (based on semantic element attributes) is a meaningful abstraction that enables generalization across task instances with different GUI layouts.
2. Methodological Rigor
The approach is methodologically sound with several well-designed components:
Translator Agent: Converting positional/index-based actions to semantic attribute-based element lookups is a practical and important step. The use of `find_element` with semantic kwargs (content descriptions, hint text, target descriptions) provides a reasonable abstraction layer.
Builder with Tree-structured RAG: The hierarchical trajectory database (interaction blocks → simplified trajectories → conclusions) with a `fetch_info` tool is a thoughtful design that balances context length constraints against the builder's need for detailed information.
Hybrid Repair: The combination of breakpoint analysis + ReAct continuation + builder refinement is more principled than simple retry-and-regenerate approaches. The analyzer agent's role in diagnosing failures and deciding whether to continue from the breakpoint or restart is a practical innovation.
Experimental Evaluation: The paper evaluates on three diverse benchmarks (AndroidWorld, WebArena, MiniWoB++) with multiple LLM backbones (GPT-4o, GPT-4.1, GPT-5, Claude-4.5-sonnet). The ablation study systematically removes key components. The results consistently show 82-96% token reduction with competitive or superior success rates. The scaling analysis (Figure 5) showing improvement with more building tasks is informative.
However, there are some methodological concerns:
3. Potential Impact
Practical Impact: The framework directly addresses a real deployment bottleneck. Enterprise RPA is a multi-billion dollar industry, and the manual effort required to create and maintain RPA scripts is a major pain point. AutoRPA could significantly reduce this barrier, making RPA accessible to users who can describe tasks in natural language rather than programming scripts.
Cost Reduction: The 82-96% token reduction is substantial and has direct financial implications for production deployments of LLM-based GUI automation. The "AutoRPA (code only)" variant is particularly compelling—it achieves near-ReAct performance with minimal token usage.
Research Direction: The paper opens a useful research direction at the intersection of LLM agents and traditional software automation. The idea of "distilling" agent behavior into executable code has broader applicability beyond GUI tasks—it could extend to API automation, database operations, or any domain where LLM agents perform repetitive structured tasks.
Limitations on Impact: The reliance on accessibility trees/DOM is a significant constraint, though the authors acknowledge this and point to OmniParser as a potential mitigation. The framework requires task type specification and multiple instances for building, which adds friction. The approach is also tightly coupled to specific LLM capabilities (GPT-4o+), limiting accessibility.
4. Timeliness & Relevance
This paper is highly timely. The field is experiencing rapid growth in LLM-based GUI agents (CUA from OpenAI, Computer Use from Anthropic, Gemini CU), but the cost-efficiency problem for deployment at scale is largely unaddressed. Most research optimizes single-task performance; AutoRPA is one of the first to systematically tackle the repetitive-task efficiency problem. The framing of "distilling" ReAct trajectories into code aligns with the broader trend of knowledge distillation from expensive models to cheaper execution pathways.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Other Observations:
Generated May 21, 2026
Comparison History (26)
Paper 2 addresses a critical bottleneck in LLM agent deployment: the high computational cost and latency of ReAct loops for repetitive tasks. By bridging LLM reasoning with traditional RPA systems, it offers massive, quantifiable efficiency gains (82-96% token reduction) for workflow automation. While Paper 1 introduces a valuable and rigorous benchmark for emotional intelligence, Paper 2's framework has broader, more immediate real-world economic and practical utility across software engineering and autonomous AI agents, directly enabling scalable, cost-effective AI automation.
Paper 1 offers higher potential impact due to its broader applicability and resolution of a fundamental bottleneck in LLM agents. While Paper 2 is highly innovative for mechanical engineering (topology optimization), Paper 1 addresses the prohibitive computational cost and latency of the ReAct paradigm in repetitive tasks. By distilling expensive LLM reasoning into reusable RPA scripts and reducing token usage by up to 96%, AutoRPA provides a highly scalable automation solution. This promises massive cross-disciplinary adoption, economic value, and establishes a critical bridge between modern generative AI and traditional enterprise automation.
AutoRPA addresses a more novel and practically impactful problem—bridging LLM-based agents with traditional RPA for efficient GUI automation. It introduces a genuinely new framework (translator-builder pipeline, hybrid repair) with significant practical benefits (82-96% token reduction). Paper 1, while solid, proposes incremental improvements to visual token pooling for Video LLMs using training-free techniques, which is a narrower contribution in a crowded space. AutoRPA has broader cross-field impact (software engineering, automation, HCI) and addresses the timely challenge of making LLM agents cost-efficient for real-world deployment.
Paper 1 offers a concrete, timely systems contribution: distilling ReAct-style GUI agents into reusable RPA code with measurable efficiency gains (82–96% token reduction) and a hybrid repair/verification loop. This is novel in bridging LLM agents and traditional RPA, has clear near-term real-world applications (enterprise automation), and is supported by experiments across GUI environments. Paper 2 provides an interesting conceptual framework and agenda for KG re-engineering, but is primarily theoretical with limited empirical validation, making impact more uncertain and longer-term.
Paper 2 (ExComm) likely has higher impact due to broader applicability and timeliness: it addresses a central bottleneck in agentic test-time scaling (error propagation) across math, reasoning, and tool-using settings, with a general communication-and-verification protocol that can be layered onto many multi-agent pipelines. Its methodology includes explicit conflict detection, tool-based verification, belief-update design, and diversity control, evaluated on widely watched benchmarks (AIME/GAIA) with clear gains and cost trade-offs. Paper 1 is practically valuable for GUI RPA, but its impact is narrower and more domain-specific.
LCGuard addresses a fundamental and emerging security concern in multi-agent LLM systems—information leakage through latent KV cache sharing—which is a novel problem formulation with broad implications as latent communication becomes more prevalent. It introduces a principled adversarial training framework with formal definitions of safety. Paper 2 (AutoRPA) offers practical efficiency gains for GUI automation but is more incremental, combining existing paradigms (ReAct + RPA). LCGuard's contribution to AI safety and privacy in multi-agent systems has broader cross-field impact and higher timeliness given rapid multi-agent deployment.
Paper 2 proposes a paradigm shift in autonomous agents by enabling source-level self-evolution, moving beyond the limitations of text-mutable artifacts (like prompts or schemas). This Turing-complete self-rewriting approach has profound implications for AGI and self-improving systems across multiple domains. Paper 1, while highly practical and effective for reducing costs in GUI automation, offers a more incremental optimization within a narrower scope.
Paper 2 has higher potential impact due to stronger novelty (first large-scale evaluation on genuinely open problems), higher scientific stakes (advancing formalized mathematics and reliable theorem proving), and broader cross-field reach (methods applicable across many math domains and beyond to verification-heavy sciences). The demonstrated real-world outcomes (solving open Erdős problems, proving OEIS conjectures, active deployment in multiple research areas) indicate immediate relevance and adoption potential. While Paper 1 offers practical efficiency gains for GUI automation, its impact is narrower and more engineering-focused.
AutoRPA addresses a practical and broadly applicable problem—bridging LLM-based agents and traditional RPA for efficient GUI automation. It introduces a concrete framework with measurable improvements (82-96% token reduction) and has clear real-world applications in enterprise automation. Paper 1, while offering interesting mechanistic insights into sycophancy as a persona-level property, is narrower in scope, focused on interpretability of steering vectors with incremental findings. Paper 2's combination of novelty, practical utility, and cross-field relevance (NLP, software engineering, HCI) gives it higher potential impact.
Paper 2 (AgentCo-op) likely has higher scientific impact due to broader novelty and cross-domain applicability: it introduces retrieval-based synthesis of interoperable multi-agent workflows with typed artifact handoffs and localized repair, addressing a key bottleneck in open-world scientific automation. Its demonstrated use in genomics case studies suggests strong real-world relevance and timeliness for scientific tool/agent integration, and the interoperability/typed interface idea can generalize across fields. Paper 1 is impactful for GUI RPA efficiency, but its scope is narrower (GUI task automation) and more application-specific.