MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research

Paper 1 likely has higher impact due to its substantial infrastructure contribution: a verifiable, scalable, browser-hosted mobile GUI simulation platform with deterministic state-based judging, parallel RL rollouts, and a sizable benchmark (416 task templates over 28 apps) enabling reproducible research. Its real-world applicability to mobile agents and evaluation (plus demonstrated sim-to-real transfer) broadens impact across RL, HCI, systems, and benchmarking. Paper 2 is a neat, timely algorithmic tweak for RLVR exploration, but narrower in scope and likely incremental relative to the platform-and-benchmark advance of Paper 1.

Paper 1 (MobileGym) likely has higher scientific impact due to a more technically novel and enabling infrastructure: verifiable, deterministic state-based judging, structured full-state capture/fork/compare, and highly parallel RL rollouts for mobile GUI agents—capabilities that can accelerate algorithmic research and reproducibility across the field. It also provides a sizable, parameterized benchmark with deterministic evaluation and shows sim-to-real transfer, increasing real-world applicability. Paper 2 (JobBench) is timely and socially important, but is primarily a benchmark/reframing with less methodological/technical innovation and narrower direct impact on agent training pipelines.

Paper 1 offers a concrete, technically novel platform enabling verifiable, deterministic evaluation and massively parallel RL for mobile GUI agents, plus a sizeable benchmark and evidence of sim-to-real transfer—likely to be broadly adopted and to accelerate empirical research. Its methodological contribution (structured state, deterministic judges, scalable rollouts) has clear real-world applications and cross-field utility (HCI, RL, agents, evaluation). Paper 2 is timely and important conceptually, but as a position paper its impact depends on downstream adoption of proposed frameworks/benchmarks and is less directly enabling than a widely usable, validated experimental platform.

Paper 2 (MobileGym) is likely to have higher impact due to a broadly useful, verifiable, scalable benchmark + simulation platform for mobile GUI agents, enabling reproducible evaluation and high-throughput RL—key bottlenecks in the field. Its deterministic state-based judging and large task suite can become shared infrastructure across labs, driving standardization and follow-on work. Paper 1 is novel in integrating planning/grounding/verification for multi-agent systems, but is more framework-specific and may be harder to standardize or reproduce broadly without a common substrate/benchmarks.

MobileGym addresses a critical bottleneck in mobile GUI agent research—verifiable evaluation and scalable RL training—with concrete empirical results showing sim-to-real transfer (95.1% retention of training gains). It provides a large benchmark (416 tasks, 28 apps), enables scalable parallel training, and is timely given the surge in LLM-based autonomous agents. Paper 1 presents an incremental tool for interactive ontology construction using SOMs, a well-established technique, with narrower applicability and limited empirical validation of broader impact.

While Paper 1 makes strong contributions to AI-driven formal mathematics, Paper 2 provides foundational infrastructure for the booming field of mobile GUI agents. By solving critical bottlenecks in agent research—namely, the lack of scalable, verifiable, and highly parallel simulation environments for RL—MobileGym has the potential to catalyze widespread follow-on research. Simulation platforms historically yield massive scientific impact by democratizing and accelerating training (e.g., OpenAI Gym). The demonstrated sim-to-real transfer further cements its immediate, broad real-world applicability for developing autonomous device-control LMs.

MobileGym introduces a novel simulation platform enabling verifiable RL training for mobile GUI agents—a rapidly growing research area. It provides infrastructure (416 task templates, 28 apps, scalable parallel rollouts) that can accelerate an entire subfield, with strong sim-to-real transfer (95.1% retention). Paper 1 offers a useful but incremental diagnostic insight (median vs. mean CE), which is a practical recommendation rather than a new capability. MobileGym's broader applicability to GUI agents, RL research, and mobile automation gives it significantly higher potential impact across multiple communities.

MobileGym provides a verifiable, scalable simulation platform enabling RL training for mobile GUI agents with strong sim-to-real transfer (95.1% retention). Its contributions—deterministic state-based judging, parallel rollouts, and demonstrated RL improvements—address fundamental infrastructure needs for the field. While Claw-Anything introduces an important benchmark for always-on assistants, MobileGym's combination of a reusable training platform, verifiable rewards enabling online RL, and validated sim-to-real transfer represents a more foundational contribution with broader methodological impact for the agent research community.

Paper 1 likely has higher impact: it introduces a broadly reusable, scalable, and verifiable simulation platform plus a substantial benchmark (416 task templates, 28 apps) addressing a major bottleneck in mobile GUI agent evaluation/training (deterministic judging, parallel RL). Its methodology is concrete and infrastructure-level, enabling downstream work across RL, LLM agents, HCI, and evaluation. Paper 2 is novel and timely for agent harness evolution, but is demonstrated on narrower domains (Liar’s Dice, Balatro) and may have more limited immediate generality and ecosystem-building compared with a new platform/benchmark.

Paper 1 presents a novel framework connecting classical A* search with LLM reasoning through post-training, demonstrating that small models (1-3B) can outperform much larger ones. This addresses a fundamental challenge in LLM reasoning with broad implications across many applications. The insight about A*-informed process reward models balancing accuracy and efficiency, and the finding about imperfect heuristics improving accuracy in larger search spaces, offer deep theoretical contributions. Paper 2, while practical and well-engineered, is more of an infrastructure/benchmark contribution for a specific domain (mobile GUI agents) with narrower impact scope.

MobileGym addresses a fundamental infrastructure gap in mobile GUI agent research by providing a verifiable, parallelizable simulation platform that enables scalable RL training. Its contributions—deterministic evaluation, sim-to-real transfer (95.1% gain retention), 416 task templates across 28 apps, and open-source availability—have broader impact across the rapidly growing fields of GUI agents, embodied AI, and RL. Paper 1, while practical for industrial maintenance, presents incremental multi-agent architecture improvements in a narrower domain with less generalizable contributions.

Paper 2 addresses a critical vulnerability in state-of-the-art MLLMs, introducing a novel structural attack that bypasses safety filters with a 92% success rate. Given the urgent global focus on AI safety and alignment, uncovering fundamental flaws in frontier models has broader, more immediate implications across the AI community than the domain-specific simulation platform for mobile GUI agents presented in Paper 1.

Paper 1 likely has higher impact due to stronger novelty and broader applicability: it introduces a verifiable, deterministic, highly parallel mobile GUI simulation platform enabling scalable RL and rigorous evaluation—capabilities that generalize across many agent/automation and HCI/ML benchmarks. Its methodological contribution (structured state, deterministic judging, large task suite, sim-to-real evidence) can become infrastructure used by many groups. Paper 2 is timely and clinically relevant, but is more domain-specific and appears more incremental (prompting/multi-agent voting) with narrower cross-field reuse and higher sensitivity to dataset/labeling constraints.

Paper 1 likely has higher scientific impact: it contributes a concrete, scalable, and verifiable simulation platform plus a sizable benchmark (416 task templates, 28 apps) enabling reproducible evaluation and high-throughput RL training—immediately useful for many labs and likely to become infrastructure. Its deterministic state-based judging and parallel rollouts address major pain points (evaluation noise, data collection cost) and it includes sim-to-real evidence. Paper 2 offers valuable taxonomies and evaluation diagnostics, but is primarily a protocol/analysis demonstration with a small run and no benchmark/platform release, limiting near-term adoption and downstream work.

Paper 1 addresses a critical bottleneck in mRNA delivery and therapeutics by improving lipid nanoparticle design. The combination of domain-specific LLM multi-agent frameworks with actual wet-lab validation offers direct, high-impact clinical applications. While Paper 2 provides valuable infrastructure for AI agent research, Paper 1's interdisciplinary approach and immediate relevance to life-saving medical technologies grant it a higher potential for profound scientific and societal impact.

Paper 1 introduces a novel architecture (PCSP) addressing a fundamental challenge in game AI—scalable persona-conditioned NPC behavior via a single shared RL policy. Its contributions span multiple dimensions: novel conditioning mechanism using frozen LLM embeddings, a training objective (InfoNCE + PPO + KL) proven load-bearing through ablations, validation across multiple environments including commercial UE5 deployment, and 22x inference speedup over LLM baselines. Paper 2, while valuable as an engineering platform for mobile GUI agents, is more incremental—providing a simulation environment and benchmark rather than a fundamentally new method. Paper 1's broader applicability across gaming, multi-agent systems, and persona modeling gives it higher potential impact.

Paper 1 presents a highly practical, rigorously tested simulation platform that directly solves a major bottleneck in GUI agent research: the high cost and lack of verifiability in training agents via RL. Its introduction of deterministic state-based judging, parallel rollouts, and demonstrated Sim-to-Real transfer offers immediate, high-impact utility to the AI community. While Paper 2 offers a valuable conceptual framework for agent systems, Paper 1 provides a concrete, empirically validated tool and benchmark that will likely catalyze a wide range of new experiments and advancements in agentic AI.

Paper 1 demonstrates that agentic LLM systems can achieve near-perfect performance on program verification benchmarks, a result with profound implications for formal methods and software engineering. Its finding that existing benchmarks are now inadequate challenges the field to develop harder evaluations, and the tight compiler-in-the-loop paradigm it validates could transform how verified software is produced. Paper 2, while a solid engineering contribution for mobile GUI agent training with good sim-to-real transfer, addresses a narrower problem (mobile agent simulation) with more incremental impact compared to the fundamental shift Paper 1 signals in formal verification capabilities.

MobileGym addresses a critical infrastructure gap in mobile GUI agent research by providing a verifiable, scalable simulation platform enabling reproducible evaluation and online RL training. The 95.1% sim-to-real transfer retention is particularly compelling, demonstrating practical utility. Its 416 parameterized tasks across 28 apps create a reusable community benchmark. While VeriTrace presents interesting cognitive-graph ideas for deep research agents, it offers more incremental improvements on narrower benchmarks. MobileGym's broader applicability as foundational infrastructure for the rapidly growing mobile agent field gives it higher potential impact.