iOSWorld: A Benchmark for Personally Intelligent Phone Agents

iOSWorld introduces a novel benchmark addressing a significant gap in mobile agent evaluation—personalization and persistent user identity. This is highly timely given the rapid development of LLM-based agents and has broad practical implications for the AI agent community. Paper 2 makes a solid contribution to understanding feedback alignment's scaling limitations, but it addresses a relatively niche problem in biologically plausible learning with incremental improvements on established benchmarks. iOSWorld's open-source release and practical relevance to the booming agent ecosystem give it higher potential impact.

GRAFT addresses a fundamental challenge in brain-computer interfaces—cross-day neural recording stability—with a novel architecture that separates reusable temporal dynamics from neuron-specific interfaces. It achieves state-of-the-art results on an established benchmark (NLB'21) and demonstrates practical recalibration with minimal parameter updates, directly advancing long-term BCI usability. While iOSWorld is a solid benchmark contribution for mobile AI agents, benchmarks typically have narrower lasting impact compared to methodological advances. GRAFT's innovation in neural population modeling has deeper scientific implications for both neuroscience and clinical BCI applications.

iOSWorld introduces a highly timely and novel benchmark for personalized, multimodal autonomous phone agents, a rapidly expanding frontier in AI. By simulating persistent user identities across interconnected apps, it addresses a critical gap in evaluating agentic systems. Benchmarks in nascent but popular areas often generate substantial scientific impact by standardizing evaluation and driving future research directions, whereas the tiny time series model, while practically useful, enters a more established and crowded subfield.

iOSWorld introduces a novel benchmark addressing a significant gap in mobile agent evaluation—personalization and persistent user identity. This fills an important need as phone agents become increasingly relevant, and the open-source release of 26 apps, tasks, and evaluation infrastructure will likely drive substantial follow-up research. While FlowBP makes solid technical contributions to reward backpropagation for flow matching models, it represents incremental improvements within a narrower subfield. iOSWorld's broader applicability across AI agent research, human-computer interaction, and mobile computing gives it higher potential impact.

Paper 1 introduces a highly timely and novel benchmark for mobile AI agents, a rapidly expanding field, providing a fully functional interactive simulator that will likely see immediate, broad adoption for training and evaluating LLMs. In contrast, while Paper 2 addresses a critical medical domain (lung cancer resistance), its own results demonstrate that the current dataset's input modality fails to perform better than chance, serving more as a negative result that sets up future work. Therefore, Paper 1 promises much higher immediate utility, citation potential, and broader technological impact.

Paper 2 (iOSWorld) likely has higher scientific impact because it introduces a broadly usable, open-source benchmark infrastructure for personalized, persistent-identity mobile agents—an area with strong real-world relevance and timely demand. Benchmarks often shape research directions across multiple communities (agents, HCI, multimodal models, evaluation), giving wide cross-field impact and strong practical applicability. Paper 1 is novel and useful (test-time prompt learning across heterogeneous streams), but its impact is more method-specific and may depend on adoption within a narrower subarea, whereas iOSWorld can become a standard evaluation platform.

iOSWorld introduces a novel benchmark addressing a significant gap in mobile agent evaluation—personalization and persistent user identity. This is highly timely given the rapid development of LLM-based agents and has broad implications for AI assistants, HCI, and industry applications. Paper 2 (COGENT) presents a solid technical contribution combining Neural ODEs with graph networks for physical forecasting, but it is more incremental, combining existing techniques (graph networks, Neural ODEs) and is evaluated on a single domain (ice-sheet simulations). iOSWorld's open-source benchmark has higher potential to catalyze research across the AI agent community.

Paper 1 addresses a fundamental challenge in deep reinforcement learning—training instability due to non-stationarity—with both theoretical grounding (provable advantages of isotropic Gaussian embeddings) and a practical, computationally inexpensive regularization method. It tackles representation collapse and neuron dormancy, which are broadly relevant problems. Paper 2 introduces a useful benchmark for phone agents with personalization, but benchmarks tend to have more transient impact and are narrower in scope. Paper 1's theoretical contributions and broad applicability across RL domains give it higher long-term scientific impact.

Paper 1 addresses a fundamental architectural limitation in Transformers regarding symbol invariance, proposing a theoretically grounded mechanism with provable guarantees. This has broad implications across multiple fields (formal reasoning, program synthesis, mathematics, logic) and introduces a principled inductive bias that could influence future neural architecture design. Paper 2, while practically valuable as a benchmark for phone agents, is more incremental—it extends existing mobile agent evaluation with personalization. Benchmarks have impact but are more transient, whereas architectural innovations with theoretical foundations tend to have deeper, longer-lasting influence.

Paper 1 investigates fundamental optimization mechanisms in deep learning, demonstrating that the new Muon optimizer outperforms the ubiquitous Adam optimizer in feature robustness and transferability. Because optimization is central to all deep learning models (LLMs, CNNs), a proven improvement here has a massive, field-wide impact. Paper 2 introduces a valuable but more narrowly focused benchmark for smartphone AI agents, which, while highly relevant for applied AI, does not match the fundamental breadth and foundational theoretical contribution of Paper 1.