AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees
Yuankai Li, Tinghui Zhu, Ha Min Son, Zhe Zhao, Xin Liu, Muhao Chen
Abstract
Large Multimodal Models (LMMs) have recently emerged as promising backbones for GUI-agent models, where high-resolution GUI screenshots are introduced to the prompts at each iteration step. However, these screenshots exhibit highly non-uniform spatial information density: large regions may carry little information and are visually homogeneous, while key text and icons may require high visual fidelity. Existing approaches to this problem either require additional training or rely on attention-based token compression, ignoring the structured layout and spatial redundancy of GUI screenshots. To fill the gap, this paper proposes AquaUI, a training-free inference-time token reduction method for GUI agent models that utilizes the non-uniform information density in screenshots. AQuaUI constructs an adaptive quadtree on each screenshot input and keeps one representative merged token per leaf of the quadtree. AQuaUI preserves the spatial positions of retained tokens throughout the pipeline to ensure that all position-encoding stages remain consistent. To further improve temporal consistency across multi-step GUI interactions, we propose a conditional quadtree algorithm that leverages the continuity between consecutive screenshots within a single request. Specifically, it refines the current quadtree using previous quadtrees as references, helping preserve fine-grained regions across static or mildly shifted GUI states. We implement AQuaUI on state-of-the-art GUI agent models and conduct experiments on standard grounding and navigational benchmarks. AQuaUI consistently shows improved accuracy-efficiency trade-offs over prior baselines. Notably, on GUI-Owl-1.5-32B-Instruct, AQuaUI achieves up to 13.22% speedup and 29.52% fewer visual tokens while retaining 99.06% of full-token performance, suggesting that the spatial redundancy of GUI screenshots can be exploited at inference without retraining.
AI Impact Assessments
(1 model)Scientific Impact Assessment: AQuaUI
1. Core Contribution
AQuaUI introduces a training-free, inference-time visual token reduction method specifically designed for GUI agent models. The key insight is that GUI screenshots have highly non-uniform information density—large homogeneous backgrounds coexist with small, detail-rich regions (text, icons)—and this structure can be exploited via adaptive quadtree partitioning. Each leaf of the quadtree retains a single representative token, effectively compressing visually redundant regions while preserving fine-grained detail where needed. The paper additionally proposes a conditional quadtree mechanism that leverages temporal continuity across consecutive screenshots in multi-step GUI interactions, classifying regions as static, shifted, or replaced to maintain consistency.
The problem addressed—reducing the computational burden of processing high-resolution GUI screenshots at each interaction step—is practically relevant as GUI agents scale to larger models and longer interaction horizons.
2. Methodological Rigor
Strengths in design: The quadtree formulation is well-motivated for GUI imagery. The grid alignment and boundary handling (§3.2) correctly addresses the non-power-of-two dimension issue, and the splitting criteria (variance-based and gradient-based) are straightforward and interpretable. The conditional quadtree mechanism (§3.3) with its static/shifted/replaced classification is a sensible heuristic for temporal consistency.
Experimental breadth: The evaluation spans five grounding benchmarks (ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G, UI-Vision, MMBench-GUI) and two navigational benchmarks (AndroidControl, AndroidWorld), across multiple model families (Qwen2-VL, Qwen3-VL, ShowUI, GUI-Owl, MAI-UI). This is commendably thorough.
Concerns:
3. Potential Impact
Practical value: The training-free nature is AQuaUI's strongest selling point. It can be applied to off-the-shelf GUI agent models without retraining, making it immediately deployable. The ~30% token reduction with <1 point accuracy drop on Qwen3-VL models is compelling for production settings where inference cost matters.
Scope of applicability: The approach is specifically tailored to GUI screenshots and may not generalize well to natural images or other visual domains. However, within the GUI agent space—a rapidly growing area—this is a useful contribution. The method's dependence on model-specific position encoding modifications (detailed in Appendix A.5) limits plug-and-play deployment to new architectures.
Adjacent field influence: The quadtree-based approach could inspire similar structured compression methods for other domains with non-uniform information density (e.g., document understanding, chart/diagram processing). The conditional quadtree idea for temporal consistency could be useful in video understanding pipelines.
4. Timeliness & Relevance
GUI agents are an active research frontier, with models like UI-TARS, GUI-Owl, and Claude's computer use pushing boundaries. The efficiency bottleneck of processing high-resolution screenshots at each step is well-recognized. This paper addresses a genuine current need, particularly as context windows become the practical constraint for multi-step interactions. The timing is appropriate, though the field is moving rapidly—the paper references works from 2026, indicating a very current research context.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Overall Assessment
AQuaUI is a solid engineering contribution that addresses a real efficiency bottleneck in GUI agents through a well-motivated spatial compression approach. The training-free nature and comprehensive evaluation are commendable. However, the novelty is incremental (quadtrees are classical; the concurrent QLIP work explores the same idea), the speedups are modest and model-dependent, and the implementation burden partially contradicts the simplicity narrative. The paper represents useful applied research rather than a breakthrough contribution.
Generated May 20, 2026
Comparison History (21)
Paper 1 offers a highly novel, training-free algorithmic contribution (adaptive quadtrees) to solve spatial redundancy in LMMs for GUI agents. This methodological innovation advances the fundamental understanding of efficient token representation in vision-language models. In contrast, while Paper 2 provides valuable open-source models and details an impressive industrial engineering pipeline, it represents an engineering scaling effort rather than a fundamental scientific or algorithmic breakthrough.
Paper 1 addresses a fundamental, systemic flaw in how production LLM performance is benchmarked, exposing widespread measurement bias caused by client-side bottlenecks. Because evaluation metrics drive hardware and software optimization across the entire LLM infrastructure industry, correcting this methodology has an immense and immediate cross-disciplinary impact. Paper 2 presents a clever efficiency optimization for GUI agents, but its scope is limited to a specific sub-domain of multimodal models, making Paper 1 more broadly impactful.
ScenePilot addresses a critical safety challenge in autonomous driving with a novel boundary-driven framework that combines physical feasibility constraints with adversarial scenario generation. Its formulation as constrained multi-objective RL with RSS-derived feasibility scoring is methodologically innovative and addresses a fundamental gap (physically valid yet challenging scenarios). The real-world safety implications for AV deployment give it broader impact. While AQuaUI offers useful efficiency gains for GUI agents, it is primarily an engineering optimization (training-free token reduction) with incremental improvements, and its impact is narrower, limited to GUI-agent inference efficiency.
AQuaUI addresses a broadly relevant problem in LMM efficiency for GUI agents, offering a training-free, principled approach (adaptive quadtrees) that exploits spatial redundancy. Its applicability spans any GUI-agent system using multimodal models, giving it wider cross-field impact. The method is elegant, well-motivated by information-theoretic principles, and achieves strong efficiency gains with minimal accuracy loss. ScenePilot, while rigorous and valuable for autonomous driving safety testing, targets a narrower domain. AQuaUI's training-free nature and general applicability to the rapidly growing LMM ecosystem give it higher potential for broad adoption and citation impact.
Paper 2 addresses AI sycophancy, a critical issue in AI alignment and safety, offering fundamental insights into model behavior by showing it as a persona-level property. Its conceptual contribution to LLM steerability has broader theoretical and practical implications across the field compared to Paper 1, which provides a valuable but domain-specific engineering optimization for GUI agents.
Paper 2 likely has higher impact due to timeliness and broad applicability: inference-time visual token reduction addresses a pressing bottleneck for LMM-based GUI agents and can transfer to other vision-language settings with structured screens (mobile, web, RPA). The adaptive/conditional quadtree is a simple, training-free, model-agnostic method with clear efficiency gains on standard benchmarks, improving deployability. Paper 1 is novel and important for strategic ML in tabular domains, but its impact is narrower (strategic manipulation settings) and depends more on assumptions about agent behavior and priors.
Paper 2 introduces a genuinely novel methodology—using CNNs to discover structural patterns in enumerated solutions and translating them into constraint streamliners via LLM synthesis. This bridges machine learning and constraint programming in an innovative way, with dramatic speedups (100-1000x) on hard benchmarks. The approach has broad applicability across combinatorial optimization. Paper 1, while solid engineering, offers incremental improvements (13% speedup) on a narrower problem (GUI token reduction) using relatively straightforward quadtree decomposition, representing more of an optimization than a conceptual advance.
While Paper 1 offers a highly practical and timely engineering optimization for GUI agents, Paper 2 provides stronger foundational contributions to its field. By establishing a theoretically grounded mechanism using the graph information bottleneck to dictate both topology and communication capacity in MARL, Paper 2 offers mathematical guarantees and novel principles (like water-filling capacity allocation) that are likely to drive deeper, long-lasting theoretical advancements across multi-agent systems.
Paper 1 offers a fundamental theoretical breakthrough by providing the first finite-sample guarantees for neural Q-learning under decentralized partial observability. Its mathematically rigorous approach addresses the highly relevant problem of multi-agent LLM coordination across trust boundaries. In contrast, Paper 2 presents a practical but narrower engineering optimization for token reduction in GUI agents. Paper 1's combination of strong methodological rigor, broad applicability to multi-agent workflows, and significant algorithmic innovation gives it higher potential for widespread scientific impact.
GeoX introduces a novel self-play framework for geospatial reasoning that combines executable programs, verifiable rewards, and reinforcement learning without requiring large-scale human annotations. This represents a more fundamental methodological innovation with broader impact: it addresses the core challenge of data scarcity in geospatial AI, introduces a new reasoning paradigm (abduction/deduction/induction over spatial primitives), and releases a benchmark. Paper 1, while practical, offers an incremental engineering contribution (training-free token reduction for GUI agents) with narrower scope. GeoX's approach could generalize to other spatial reasoning domains beyond geospatial applications.