GTA: Generating Long-Horizon Tasks for Web Agents at Scale
Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu
Abstract
Web agents, which couple language models with browsing and tool-use capabilities, show promise as open web assistants. Yet progress is increasingly limited by the lack of scalable, process-level supervision. Existing benchmarks are largely manually constructed, providing only coarse start-goal annotations without intermediate trajectories, while recent automatic generation efforts remain expensive, biased, and shallow. These limitations prevent reliable training and evaluation of agents that must generalize to realistic, multi-hop, cross-page tasks. We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control to produce realistic tasks paired with executable trajectories. This design decouples crawling from generation for greater efficiency, grounds tasks in the site graph to enforce compositionality, and ensures dense supervision through deterministic replays and systematic validation. We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage. The resulting benchmark reveals a significant human-agent performance gap and enables detailed diagnostics. Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
AI Impact Assessments
(1 models)Scientific Impact Assessment: GTA: Generating Long-Horizon Tasks for Web Agents at Scale
1. Core Contribution
GTA presents a scalable framework for automatically generating multi-hop web navigation tasks paired with executable ground-truth trajectories. The core insight is to decouple crawling from task generation: first build a site graph via breadth-first crawling, then use retrieval-augmented prompting to synthesize compositional tasks grounded in the graph structure. The pipeline has four stages: (1) crawling to construct a directed web graph G=(V,E), (2) retrieval-based seeding to identify candidate page pairs, (3) in-context generation of multi-hop queries requiring cross-page reasoning, and (4) multi-stage quality control (multi-hop validation, answer correctness, ambiguity detection, solvability checks).
The key problem addressed is the lack of scalable, process-level supervision for web agents. Existing benchmarks either rely on expensive manual annotation or automatic generation methods that produce shallow, single-hop tasks with strong exploration bias. GTA claims to be the first framework combining automatic generation, multi-hop reasoning, executable ground-truth trajectories, dynamic expansion, and multilingual coverage.
2. Methodological Rigor
The methodology has several commendable aspects but also notable gaps:
Strengths in evaluation design: The paper includes multiple analytical dimensions — a search-baseline difficulty test (showing only 14% solvability on GTA vs. 95-100% on prior benchmarks), page-coverage analysis (77-81% vs. 11-24% for competitors), human evaluation across four dimensions, and cost analysis. The search-baseline comparison is particularly effective at demonstrating that GTA tasks genuinely require multi-hop reasoning.
Weaknesses: The experimental evaluation of agents is thin. Only two agent architectures (Browser Use and AgentOccam) are evaluated, which the authors acknowledge but justify by citing precedent. The human evaluation uses only 20 tasks per dataset with 3 annotators and achieves a Cohen's Kappa of only 0.3 (fair agreement), which is relatively low and undermines confidence in the human judgment scores. The 87% correctness rate on 100 sampled tasks means roughly 13% of automatically generated tasks have incorrect answers — a non-trivial error rate for a benchmark.
The formal definition of multi-hop tasks is clean but somewhat simplistic — it counts pages providing evidence without capturing the depth or type of reasoning required. The quality control pipeline relies heavily on LLM-based validators, creating a circular dependency: LLMs validate tasks designed to test LLM-based agents. The paper acknowledges this limitation but doesn't quantify how often the validators fail.
The page-coverage comparison (Fig. 1) uses different websites for different benchmarks (espn.com for AgentTrek, underarmour.com for NNetNav, casper.com for WebDS), making direct comparison problematic — the differences could partly reflect site-specific characteristics rather than methodological superiority.
3. Potential Impact
Benchmark contribution: If the pipeline proves robust, the "self-evolving benchmark" framing is genuinely valuable. Static benchmarks suffer from contamination and obsolescence; a framework that generates fresh tasks from live web content addresses both problems. The coverage of 50+ websites across multiple domains and languages is substantial.
Training data generation: Beyond evaluation, the executable trajectories with step-level attribution could serve as training data for web agents, addressing the process-level supervision gap. This could be particularly impactful for reinforcement learning or imitation learning approaches.
Diagnostic value: The error analysis revealing that 90% of failures stem from inability to reach all required pages, 40% from early stopping, and 40% from over-reliance on search boxes provides actionable insights for agent development.
Adjacent fields: The crawl-retrieve-generate paradigm could transfer to other agent benchmarks (e.g., API agents, desktop agents) where compositional task generation is needed.
4. Timeliness & Relevance
This paper addresses a genuine and current bottleneck. The web agent field has seen rapid growth (Browser Use, AgentOccam, various commercial offerings), but benchmark saturation is a recognized problem — the paper shows agents achieving up to 82% on WebVoyager while struggling below 20-30% on GTA. The multilingual evaluation is particularly timely given the global deployment ambitions of web agents. The cross-website task setting (requiring integration across domains like healthcare sites) represents a realistic and underexplored challenge.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Reproducibility concerns: While code is released, dependence on live websites means exact reproduction may be difficult due to content drift — the paper acknowledges this but doesn't provide mitigation strategies beyond periodic re-crawling.
Overall Assessment
GTA makes a solid contribution to the web agent evaluation ecosystem by introducing a principled, efficient, and scalable task generation framework. The graph-based approach to ensuring compositionality is its strongest methodological contribution. However, the low realism scores, limited agent evaluation, weak inter-annotator agreement, and reliance on LLM-based validation temper enthusiasm. The work is more impactful as a framework/tool than as a definitive benchmark.
Generated May 29, 2026
Comparison History (19)
Paper 1 likely has higher scientific impact due to its scalable, broadly usable data-generation framework and release of a dynamic benchmark with dense trajectories across 50+ real websites. This directly addresses a major bottleneck (process-level supervision) for web agents and can catalyze progress across training, evaluation, and diagnostics for long-horizon tool/browsing agents, with wide applicability across academia and industry. Paper 2 is timely and practically valuable for reliability engineering, but its contribution is more system-specific and evaluated on a controlled fault-injection setup, which may limit broad adoption and cross-field impact relative to a large benchmark/dataset release.
AutoMedBench addresses a critical gap in medical AI research by introducing workflow-aware evaluation of autonomous agents across diverse medical tasks, with novel stage-level analysis revealing specific failure modes. Its domain (medical AI) has enormous real-world impact potential, and the granular diagnostic framework (5-stage scoring, error taxonomy) offers actionable insights for improving medical AI agents. While GTA makes solid contributions to web agent benchmarks with scalable task generation, its domain is narrower in societal impact. AutoMedBench's combination of medical domain relevance, methodological rigor, and novel workflow-level evaluation gives it higher potential impact.
Paper 2 addresses a critical bottleneck in the rapidly growing field of LLM-based web agents: the lack of scalable, process-level supervision for long-horizon tasks. By introducing an automated, scalable pipeline for generating realistic tasks and executable trajectories, it enables significant advancements in training and evaluating open web assistants. This has broader immediate real-world applications and cross-field impact compared to Paper 1, which, while methodologically innovative, is more narrowly focused on continuous control and robotics.
Paper 1 introduces a highly novel theoretical framework using entropy dynamics to understand fundamental mechanisms and failure modes in Multi-Agent Systems. Identifying the 'Reasoning Trap' offers profound scientific insights that can fundamentally shift how AI architectures are designed. While Paper 2 provides a practical and valuable benchmark pipeline, Paper 1's theoretical depth, methodological innovation, and potential to explain structural fragility in complex AI systems promise a deeper, longer-lasting scientific impact.
GTA addresses a fundamental bottleneck in web agent research—the lack of scalable, realistic benchmarks with process-level supervision. It provides a reusable framework and public benchmark that can benefit the broader community working on autonomous web agents, a rapidly growing field. While MIRA presents a solid contribution to mid-training data selection with clever rubric discovery, it addresses a more niche optimization problem within LLM training pipelines. GTA's combination of a novel formalization, a scalable pipeline applicable across 50+ websites, and a released dynamic benchmark gives it broader impact potential across multiple research communities (agents, evaluation, web understanding).
Paper 2 likely has higher impact due to its scalable, general-purpose infrastructure for generating long-horizon web-agent tasks with executable trajectories and validation. It enables broad real-world applications (training/evaluating web agents), offers a reusable benchmark spanning many sites and languages, and addresses a major bottleneck (process-level supervision) in a timely area. Its contributions can influence multiple subfields (agent evaluation, data generation, RL/IL, web automation). Paper 1 is novel mechanistic analysis but is narrower in direct applicability and downstream ecosystem effects.
Paper 2 likely has higher impact: it addresses a major bottleneck (scalable, process-level supervision) for web agents, offers a broadly usable data-generation and validation pipeline, and produces a benchmark spanning many real websites, languages, and multi-hop tasks—enabling training, evaluation, and diagnostics across the community. Its real-world applicability and breadth (agents, RL, benchmarking, data-centric AI) are high and timely. Paper 1 is novel mechanistic/safety work with clever efficiency gains for jailbreak search, but its applications are narrower and more security-specific.
GTA addresses a fundamental bottleneck in web agent research—the lack of scalable, high-quality benchmarks with process-level supervision. It introduces a complete framework for automatic task generation with executable trajectories across 50+ websites, enabling both training and evaluation. This has broad impact across the rapidly growing web agent and autonomous agent communities. While MIRA makes a solid contribution to LLM mid-training data selection (a narrower topic), GTA's benchmark release, formalization of multi-hop task generation, and applicability to the burgeoning agent ecosystem give it wider potential impact and timeliness.
Paper 2 (GTA) likely has higher impact due to broader applicability and timeliness: scalable, validated generation of long-horizon web-agent tasks with executable trajectories addresses a key bottleneck (process-level supervision) for training and evaluating tool-using agents. The released dynamic benchmark across many real websites can become shared infrastructure, enabling progress and diagnostics across the community. Paper 1 (CIVIC) is a strong systems contribution with clear efficiency benefits for VLM inference, but its impact is narrower (specific architectural/hardware efficiency optimizations) and may generalize less broadly than a widely adopted dataset/benchmark pipeline.
Paper 2 likely has higher scientific impact due to strong real-world applicability (privacy- and bandwidth-constrained speech translation deployment), clear timeliness (edge-cloud LLM/MLLM inference and privacy), and broad cross-lingual relevance across 45 languages and many translation directions. The split-inference design with compressed intermediate features plus curriculum/data-balancing addresses concrete deployment and bias issues and reports SOTA results with released code/models, suggesting methodological maturity and immediate adoption potential. Paper 1 is valuable for web-agent benchmarking/data generation, but its impact is more tool/benchmark-centric and may be narrower than privacy-preserving multilingual speech translation.