Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation
Heng Qu, Yike Liu, Renren Jin, Wenzong Zhang, Pengzhi Gao, Wei Liu, Jian Luan
Abstract
Vision-Language Models (VLMs) have shown rapid progress in mobile GUI navigation. This paper presents a systematic study of data scaling, benchmarking, and reasoning for VLM-based agents in this domain. To facilitate rigorous evaluation, we introduce HyperTrack, a large-scale dataset with over 16000 real-world tasks across more than 650 Chinese mobile applications, along with GUIEvalKit, an open-source toolkit for unified benchmarking of VLMs on offline GUI navigation tasks. Using HyperTrack, we analyze the effects of training data scale on both supervised and reinforcement-based finetuning. Our results show that reinforcement-based finetuning consistently outperforms supervised finetuning, particularly in out-of-domain settings, highlighting the synergy between data scaling and reinforcement learning. Leveraging GUIEvalKit, we further benchmark state-of-the-art (SOTA) VLMs and analyze how interaction history and reasoning capabilities influence task completion. Together, HyperTrack and GUIEvalKit provide a comprehensive platform for developing and evaluating VLM agents in mobile GUI navigation tasks.
AI Impact Assessments
(1 models)Scientific Impact Assessment
1. Core Contribution
This paper makes three interconnected contributions to the mobile GUI navigation domain: (1) HyperTrack, a large-scale dataset of 16,000+ real-world tasks across 650+ Chinese mobile apps; (2) GUIEvalKit, an open-source evaluation toolkit for unified offline and semi-online benchmarking; and (3) a systematic empirical study of data scaling laws under SFT vs. DAPO-style reinforcement learning, along with novel decision-level analysis of reasoning behavior.
The paper's most distinctive contribution is the semi-online evaluation (SOEval) protocol, which bridges the gap between static offline evaluation and expensive online evaluation by conditionally incorporating on-policy decision artifacts into the evaluation history. The decision-level analysis framework—introducing decision diversity and decision stability as principled behavioral metrics—represents a genuinely novel analytical lens for understanding GUI agent behavior beyond standard accuracy metrics.
2. Methodological Rigor
Data scaling analysis is well-executed, spanning training sizes from 16 to 8,192 episodes with both SFT and RL finetuning on UI-TARS-1.5-7B. The approximately log-linear scaling relationship is convincingly demonstrated across four test splits (in-domain, unseen app, unseen device, unseen app & device). Complementary experiments with Qwen3-VL-8B-Thinking and Gaussian spatial rewards strengthen generalizability claims.
SOEval validation correlates metrics with AndroidWorld online success across six models, showing improved Spearman correlation (0.771 vs. 0.657) and R² (0.624 vs. 0.482) compared to offline evaluation. However, the correlation is computed over only six data points, limiting statistical confidence. The authors acknowledge this is an approximation rather than a substitute for online evaluation.
Decision-level analysis is methodologically thorough. The clustering-based execution-to-decision abstraction is validated through extensive sensitivity analysis across DBSCAN parameters (ε values from 5 to 140), distance metrics (L1 vs. L2), and cross-metric Wasserstein distances. The 512 rollouts per step task (8 rounds × 64 trajectories) provide substantial statistical support. The reasoning-execution consistency detector is validated on a manually curated benchmark of 648 examples with Wilson confidence intervals reported.
Potential concerns: The full HyperTrack dataset is not publicly available at submission time (only a preview subset), which limits immediate reproducibility. The RL reward formulation is relatively simple (binary action-type + parameter matching), and it's unclear how well findings generalize to more sophisticated reward designs.
3. Potential Impact
Dataset contribution: HyperTrack fills an important gap—Chinese-language mobile GUI datasets are scarce, and the 674-app, 17-category coverage with hierarchical UI documents, bounding boxes, and screen descriptions is comprehensive. The multi-split design enabling out-of-domain evaluation (unseen apps, devices, and tablets) is well-structured for generalization studies.
Toolkit contribution: GUIEvalKit's integration of 5 benchmarks and 30+ VLMs with standardized inference, evaluation, and parallelized execution provides immediate practical utility. The ABCModel abstraction and StepTaskModel encapsulation enable reproducible comparison across diverse model families.
Analytical frameworks: The decision diversity/stability analysis and the stability-diversity tradeoff finding (reasoning improves PASS@8 but hurts PASS@1) offer actionable insights for practitioners. The finding that RL consistently outperforms SFT in out-of-domain settings, with the gap widening at scale, has direct implications for training pipeline design.
Broader influence: The semi-online evaluation concept could influence evaluation methodology beyond GUI navigation—any sequential decision-making domain where online evaluation is expensive could benefit from context-aligned offline approximations.
4. Timeliness & Relevance
This paper addresses a highly active research area at the intersection of VLMs and agentic AI. The timing is excellent—multiple GUI agent systems (UI-TARS, AgentCPM-GUI, MagicGUI) have recently appeared, creating urgent need for standardized evaluation. The Chinese mobile ecosystem focus is commercially significant and underserved by existing English-centric benchmarks.
The paper's investigation of when reasoning helps vs. hurts is particularly timely given the proliferation of "thinking" models (Qwen3-VL-Thinking, GLM-4.1V-Thinking, etc.) and the common assumption that explicit reasoning uniformly improves performance.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
6. Additional Observations
The reasoning-execution consistency analysis (Section 5.4.1) with its two-stage majority voting scheme and failure taxonomy (61% action-type mismatch, 37% action-target mismatch) provides concrete diagnostic value. The odds ratio of 12.50 between R-E consistency and execution success is a strong signal for future work on improving reasoning-execution alignment.
The paper's breadth is both its strength and weakness—it covers substantial ground but sometimes sacrifices depth for coverage.
Generated May 27, 2026
Comparison History (21)
Paper 2 likely has higher scientific impact due to its novel mechanistic contribution: layer-wise causal analyses (probes, layer-skipping interventions, effective-depth) on multi-turn agent trajectories, yielding general insights about how depth is recruited during agentic reasoning. These findings can influence model design, efficiency, interpretability, and evaluation across many agent domains and model families. Paper 1 provides valuable resources (dataset/toolkit) and empirical scaling/finetuning results for mobile GUI navigation, but its impact is more application- and domain-specific (Chinese apps, offline benchmarks) and less likely to generalize broadly than Paper 2’s mechanistic conclusions.
Paper 2 introduces a large-scale dataset (HyperTrack, 16K+ tasks) and an open-source benchmarking toolkit (GUIEvalKit), which are reusable community resources likely to drive adoption and citations. It provides systematic insights on data scaling and RL vs. supervised finetuning for GUI agents—a rapidly growing application area. Paper 1 addresses an important efficiency problem with a solid engineering contribution (CIVIC), but its impact is narrower, being architecture-specific (Qwen3-VL) and focused on inference optimization. Paper 2's broader applicability, resource contributions, and relevance to the booming autonomous agent field give it higher potential impact.
Paper 2 addresses a fundamental and highly debated topic (LLM reasoning) by exposing methodological flaws in a prominent benchmark. Its push for rigorous statistical evaluation has broad implications across AI evaluation, likely shifting community practices and generating higher impact than the domain-specific GUI dataset in Paper 1.
Paper 2 addresses a critical and fundamental issue in AI safety—vulnerabilities to cooperative attacks in multi-agent systems. While Paper 1 provides a valuable dataset and toolkit for GUI navigation, Paper 2's focus on adversarial coordination and sentence-level defense mechanisms offers broader conceptual novelty and theoretical impact across the rapidly growing field of LLM-based multi-agent systems.
Paper 2 introduces a large-scale dataset (HyperTrack) and an open-source evaluation toolkit (GUIEvalKit) for VLM-based mobile GUI agents. By providing critical benchmarking infrastructure in the highly relevant and rapidly growing field of agentic AI, it is likely to serve as a foundational resource, driving broader follow-up research and yielding a higher scientific impact compared to the specific algorithmic improvements in federated learning presented in Paper 1.
Paper 2 introduces a large-scale dataset (HyperTrack, 16K+ tasks) and an open-source benchmarking toolkit (GUIEvalKit) for mobile GUI navigation, providing broadly reusable infrastructure for the rapidly growing VLM agent community. Its findings on RL vs supervised finetuning scaling laws have wide applicability. Paper 1 addresses a narrower problem—risk-controlled use of Lean as a judge for math reasoning—with important but specialized contributions. The negative results (sparse signal, low faithfulness) limit immediate practical impact, and the method's utility is contingent on autoformalization coverage improvements. Paper 2's breadth and resource contributions give it higher impact potential.
Paper 2 likely has higher impact due to broader applicability and stronger community leverage: mobile GUI navigation benchmarks and tooling (HyperTrack + GUIEvalKit) can drive standardized evaluation across many VLM/agent methods, enabling reproducible comparisons and accelerating progress. Its focus on scaling laws and RL vs supervised finetuning is timely for agentic VLMs and relevant across HCI, embodied/interactive AI, and RL. Paper 1 is novel and rigorous in addressing leakage and attribution for LLM trading, but its domain is narrower and constrained by market-specific evaluation and real-world deployment barriers.
Paper 2 introduces concrete, reusable research artifacts (HyperTrack dataset with 16K+ tasks, GUIEvalKit toolkit) and provides systematic insights into data scaling and reinforcement learning for VLM-based mobile GUI agents—a rapidly growing research area. Its contributions (dataset, benchmark toolkit, training methodology comparisons) are directly actionable and broadly applicable across the VLM and mobile AI communities. Paper 1 provides an interesting empirical analysis of A2A network flaws but is more descriptive and focused on a single platform's issues, with narrower applicability and less methodological novelty.
Paper 1 likely has higher scientific impact due to stronger novelty and broader, reusable infrastructure: a large-scale real-world dataset (16k tasks, 650 apps) plus an open-source benchmarking toolkit that can standardize evaluation and accelerate progress in VLM-based GUI agents. It also provides actionable empirical insights (scaling laws; RL vs supervised; OOD behavior; role of history/reasoning) with direct real-world application to mobile automation and accessibility. Paper 2 is timely and important as a characterization/audit of an A2A ecosystem, but it is more diagnostic and potentially platform-specific, with narrower methodological contributions beyond the studied network.
Claw-Anything addresses a more novel and forward-looking problem—always-on personal assistants with broad digital world access—which is a less explored but increasingly important research direction. It introduces a benchmark spanning multiple dimensions (long-horizon histories, backend services, multi-device interaction) that goes beyond existing narrow evaluations. The finding that GPT-5.5 achieves only 34.5% highlights significant open challenges, motivating future research. While Paper 1 makes solid contributions to mobile GUI navigation with scaling analysis and benchmarking tools, it operates in a more established domain with incremental advances. Paper 2's broader scope and novel evaluation paradigm give it higher potential impact.
MiniMax-M2 represents a more impactful contribution: it introduces a full frontier-tier MoE language model system with novel agent-native RL training (Forge), self-evolution capabilities, and demonstrates state-of-the-art performance across multiple benchmarks with only 9.8B activated parameters. Its architectural innovations (windowed-FIFO scheduling, prefix-tree merging, agent-driven data pipelines) and the self-evolution paradigm have broader implications for the field. Paper 2, while valuable, is more narrowly focused on mobile GUI navigation with dataset and toolkit contributions that serve a specific subdomain.
Paper 2 likely has higher impact due to timely relevance (VLM agents), strong real-world applicability (mobile GUI navigation), and broad utility from releasing a large dataset (16k tasks, 650 apps) plus an open benchmarking toolkit—assets that can catalyze follow-on work across ML, HCI, and agent evaluation. Its systematic scaling and finetuning analysis also improves methodological rigor and comparability. Paper 1 is novel within offline hierarchical RL and useful, but its impact may be narrower (HRL benchmarks/skills) and less immediately transferable than widely adopted datasets/benchmarks in the fast-moving VLM agent space.
Paper 1 offers a more comprehensive and rigorous scientific contribution with a large-scale dataset (16K+ tasks, 650+ apps), an open-source evaluation toolkit, and systematic analysis of data scaling and reinforcement learning vs supervised finetuning. These methodological insights into training paradigms and the reusable infrastructure (HyperTrack, GUIEvalKit) provide foundational tools likely to be widely adopted. Paper 2 introduces an interesting human-centered benchmarking philosophy, but its 130-task benchmark is smaller in scale and its primary contribution is more conceptual (reframing AI evaluation from replacement to enhancement) than methodological.
Paper 2 addresses long-term memory, personalization, and proactive behavior in AI agents, which are fundamental and widely applicable challenges across all human-AI interaction domains. While Paper 1 is highly rigorous, its focus is more narrowly constrained to mobile GUI navigation in Chinese applications. Paper 2's focus on the next frontier of agent capabilities—proactivity and long-term user alignment—gives it higher potential for broad scientific impact and general real-world application.
MuCRASP addresses a fundamental and broadly applicable problem—preserving chain-of-thought reasoning during structured pruning of VLMs. Its novelty lies in identifying pivot tokens and cross-modal activation differences, providing a principled pruning framework with strong empirical results across multiple models and benchmarks. This has wide applicability to model compression across the VLM community. Paper 2, while valuable for its dataset and benchmark contributions to mobile GUI navigation, addresses a narrower application domain (Chinese mobile apps) and is more incremental in its technical contributions (scaling analysis, benchmarking toolkit).
Paper 1 addresses the development of vision-language agents for mobile GUI navigation, a rapidly growing field with massive real-world applications in human-computer interaction and automation. Its contribution of a large-scale dataset (16,000 tasks) and rigorous analysis of data scaling and reinforcement learning will likely serve as a foundational benchmark for future agent research. While Paper 2 tackles a timely issue in academic publishing, its focus on peer review evaluation is relatively niche, giving Paper 1 a broader potential impact across both academia and industry.
MemFail addresses a fundamental and broadly applicable problem—understanding failure modes of LLM memory systems—with a novel diagnostic framework that decomposes memory into canonical operations and provides actionable insights for system design. This methodology is highly generalizable across many LLM agent applications. Paper 2, while valuable, is more narrowly focused on mobile GUI navigation with a Chinese-app-specific dataset, limiting its broader impact. MemFail's systematic failure-mode analysis fills a clear gap in the literature and is likely to influence how memory systems are designed and evaluated across the field.
Paper 2 likely has higher impact due to its creation of substantial community infrastructure (a large real-world dataset across hundreds of apps plus an open benchmarking toolkit), enabling broad, reproducible progress in an active area (VLM agents, mobile UI automation). Its findings on scaling and RL vs supervised finetuning are directly actionable and relevant to both academia and industry, with cross-field reach (HCI, RL, multimodal learning, agent evaluation). Paper 1 is a valuable metric insight for LM training diagnostics, but is narrower in scope and may see more incremental adoption.
Paper 2 likely has higher scientific impact due to stronger novelty and cross-disciplinary breadth: it introduces a psychometrically grounded EI benchmark (FACET) anchored in an established theory, reveals a conceptually important finding (EI fragmentation across perception/cognition/interaction), and links results to alignment mechanisms (RLHF) with clear safety implications. Its applications span AI safety, HCI, mental health, and policy. Paper 1 is valuable and rigorous (large dataset + toolkit) but is more domain-specific (mobile GUI navigation) and its core insights (scaling + RL helps) are less broadly paradigm-shifting.
Paper 2 likely has higher impact due to broad relevance (vision-language agents, benchmarking, RL finetuning), strong timeliness, and immediate real-world applicability to mobile automation and accessibility. Its main contributions—a large-scale dataset (16k tasks, 650 apps) and an open-source evaluation toolkit—can become community standards that enable reproducible comparisons and accelerate progress across many labs and model families. Paper 1 is novel and technically strong but more specialized to brick/assembly generation, limiting breadth despite clear applications in design and robotics.