WorkstreamBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance
Thomson Yen, Julian Poeltl, Harshith Srinivas Gear, Yilin Meng, Joshua Fan, Adam Shen, Yili Liu, Ali Bauyrzhan
Abstract
LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions. To meet enterprise needs, frontier AI labs have developed agents that can construct entire spreadsheets from scratch. This is especially relevant in finance, where core workflows such as financial modeling, forecasting, and scenario analysis are commonly conducted through spreadsheets. Yet, existing spreadsheet benchmarks do not measure this advanced capability, focusing instead on question-answering or single-formula edits. To address this gap, we provide one of the first evaluations of agents on end-to-end spreadsheet tasks, focusing on economically critical financial workflows such as modeling and scenario analysis. Since deliverables therein are routinely reviewed and revised by multiple stakeholders, judging their quality necessarily involves high-level criteria such as readability or ease of modification. To reflect the multidimensional nature of solution quality, we develop an evaluation taxonomy comprising three dimensions: Accuracy, Formula, and Format, each comprising fine-grained criteria that reflect professional standards. The Claude family leads the benchmark and produces the most professional-looking outputs in our qualitative review, but even the strongest agents frequently fall short of professional finance standards and degrade sharply as the difficulty increases beyond a few chained calculations. This suggests that current agents are not yet able to reliably produce professional-quality spreadsheets at the level of complexity real-world workflows demand.
AI Impact Assessments
(1 model)Scientific Impact Assessment: WorkstreamBench
1. Core Contribution
WorkstreamBench addresses a genuine gap in LLM agent evaluation by shifting from atomic spreadsheet tasks (single-formula edits, QA) to end-to-end financial spreadsheet construction. The benchmark asks agents to produce complete, multi-sheet workbooks for tasks like financial modeling, DCF valuation, and scenario analysis — deliverables that mirror actual professional finance workflows. The key innovation is twofold: (a) the benchmark itself, curated from Financial Modeling World Cup (FMWC), ModelOff, and Wall Street Prep sources, and (b) a multi-dimensional evaluation taxonomy (Accuracy, Formula, Format) validated through an LLM-as-judge pipeline. The taxonomy captures professionally relevant quality dimensions — readability, formula transparency, error handling — that go beyond numerical correctness. This is a meaningful conceptual advance over SpreadsheetBench's exact-match paradigm, as the paper convincingly argues that professional spreadsheet quality is inherently multidimensional.
2. Methodological Rigor
Strengths in evaluation design: The three-tier taxonomy is well-motivated with concrete examples (e.g., the monolithic vs. decomposed formula comparison in Figure 3). The LLM judge validation is reasonably thorough: synthetic perturbations test per-subdimension sensitivity, while agent attempt annotations (408 expert annotations, 0.92 accuracy, 0.85 F1) demonstrate alignment with human judgment. The paper also documents compelling cases where the judge catches subtle errors (off-by-one in aggregation windows, hardcoded-but-numerically-correct values) that exact-match verification would miss.
Weaknesses: The weighting scheme across dimensions (Accuracy 50%, Formula 35%, Format 15%) is acknowledged as subjective but receives no sensitivity analysis — it would strengthen the paper to show how rankings change under alternative weightings. The expert annotation effort (700+ hours from 5 annotators) is substantial but the inter-annotator agreement is never reported, which is a significant omission for a paper arguing about subjective quality criteria. The 408 expert annotations, while meaningful, cover only a fraction of the total evaluation space (12 agents × ~228 tasks × 17 subdimensions).
The difficulty annotation scheme, while detailed in the appendix, relies on manual classification by a small team. The paper lacks formal inter-rater reliability metrics here as well. Task type annotation was delegated to Gemini-2.5-pro with only 10 randomly verified examples, which is thin validation for a benchmark paper.
3. Potential Impact
Practical relevance: The benchmark directly addresses a commercially active space — both Anthropic and OpenAI have shipped spreadsheet agents. The finding that even the best agent (Claude Web at 69.1/100) falls significantly short of professional standards, with sharp degradation at higher difficulty levels, provides a concrete and useful signal for the industry. The Playwright-based pipeline for evaluating proprietary GUI agents is a practical contribution that could see adoption beyond this specific benchmark.
Broader methodological influence: The paper's move toward artifact-level, multi-dimensional evaluation (rather than exact-match) represents an important direction for agent benchmarking more generally. The validated LLM-as-judge approach for inherently subjective quality criteria could inform evaluation frameworks in other domains where deliverable quality cannot be reduced to binary correctness (e.g., document drafting, slide creation, report generation).
Limitations on impact: The benchmark is narrowly scoped to finance spreadsheets, and some evaluation criteria (number notation conventions, sign consistency) are domain-specific. The paper acknowledges this but doesn't demonstrate how the framework transfers. The lack of harder tasks (few Level 5, no Level 6) limits the benchmark's longevity as agents improve.
4. Timeliness & Relevance
The paper is exceptionally well-timed. With Claude for Excel and ChatGPT for Excel both launched recently, and enterprise adoption of AI agents accelerating, there is acute need for benchmarks that go beyond toy tasks. The paper fills a specific gap identified relative to GDPval (which requires opaque internal expert annotations) and SpreadsheetBench (which focuses on atomic tasks). The quantitative comparison showing WorkstreamBench tasks involve 33× more cells and 93× more functions than SpreadsheetBench effectively establishes the complexity gap.
5. Strengths & Limitations
Key strengths:
Notable limitations:
Missing analyses: A per-dimension breakdown of scores by difficulty would strengthen the narrative. The paper could also benefit from analyzing which specific financial task types (DCF vs. 3-statement vs. scenario analysis) pose the greatest challenges, rather than only analyzing by difficulty level.
Overall Assessment
WorkstreamBench makes a timely and practically relevant contribution by formalizing the evaluation of end-to-end spreadsheet agent capabilities in finance. The multi-dimensional evaluation taxonomy is well-motivated and the LLM judge validation is reasonable, though not exhaustive. The findings — that current agents degrade sharply with complexity and fall short of professional standards — are valuable for both researchers and practitioners. The benchmark's primary limitation is its narrow domain focus and the inherent subjectivity in its evaluation framework, partially mitigated by expert validation.
Generated May 22, 2026
Comparison History (30)
Paper 1 presents a novel technical framework (LGSPF) that addresses fundamental challenges in combining LLMs with GNNs for fraud detection, introducing soft prompts to bridge graph structure and semantic space without text dependency. It offers methodological innovation with broader applicability beyond fraud detection to graph-based learning generally. Paper 2, while valuable as a benchmark for evaluating LLM agents on spreadsheet tasks in finance, is more narrowly scoped as an evaluation study rather than a methodological contribution, and benchmarks tend to have impact primarily within their specific application domain.
Paper 1 presents a novel algorithmic framework (IAD) for human-AI coordination with broad applicability across multi-agent systems, robotics, and HCI. Its hierarchical disentanglement approach is methodologically innovative and addresses a fundamental challenge in adaptive AI. Paper 2, while practically relevant, is primarily a benchmark/evaluation study for LLM spreadsheet agents in finance—a narrower contribution. Benchmarks can be impactful, but Paper 1's algorithmic novelty, theoretical depth, and demonstrated effectiveness with real human partners give it greater potential for lasting scientific influence across multiple research communities.
FrontierOR has higher potential impact due to several factors: (1) It addresses the broader and more fundamental challenge of LLM-based algorithm design for large-scale optimization, which spans many fields beyond finance; (2) Operations research optimization is applicable across logistics, manufacturing, healthcare, and many industries, giving it wider breadth of impact; (3) The benchmark methodology (180 tasks from top-tier OR venues with expert-verified evaluation) is more rigorous and scalable; (4) The gap it identifies (LLMs struggling to design efficient algorithms beyond correct formulations) points to a deeper scientific challenge. WorkstreamBench, while valuable, is more domain-specific to finance spreadsheets.
Paper 2 demonstrates a more significant scientific contribution by showing that agentic systems can achieve near-human-level performance on program verification (98.1% success rate), revealing that existing benchmarks are becoming insufficient. This has broader impact across formal verification, software engineering, and AI safety. It introduces a novel paradigm (compiler-in-the-loop agentic proving) with implications for reliable software development. Paper 1, while valuable for finance applications, is primarily a benchmark evaluation showing current limitations rather than advancing capabilities, and has a narrower domain focus.
WorkstreamBench provides a concrete, reproducible benchmark with empirical evaluations of current LLM agents on real-world financial spreadsheet tasks. It fills a clear gap in evaluation methodology, offers actionable findings about agent limitations, and will likely be cited by researchers developing and benchmarking LLM agents. Paper 1 proposes a high-level coordination protocol for multi-agent systems, but is largely conceptual/architectural without empirical validation. While ambitious in scope, protocol proposals without demonstrated adoption or experimental results tend to have lower near-term scientific impact than well-designed benchmarks that the community can immediately build upon.
Paper 2 introduces a novel, formalizable concept (semantic recoverability) and a runtime system (DART) addressing a fundamental reliability gap in structured tool/LLM agents under commitment-sensitive execution. Its methodological rigor includes formal criteria, modular design, multi-domain evaluation, external substrate validation, and a safety audit, suggesting broader, reusable impact across agent runtimes, workflow engines, and distributed systems. Paper 1 is timely and useful as a benchmark for finance spreadsheets, but is more domain-specific and primarily evaluative; its scientific contribution is narrower than DART’s general recovery framework.
Paper 2 tackles a fundamental challenge in AI—automated formal mathematical reasoning—by introducing a novel neurosymbolic framework and self-improvement pipeline. Its methodological innovations and demonstration that small models can match frontier models in complex proof optimization offer broader implications for AI development. While Paper 1 provides a useful practical benchmark for finance, Paper 2's contributions to model training, neurosymbolic scaffolding, and formal logic are likely to drive more significant downstream algorithmic research and cross-disciplinary impact.
Paper 1 establishes foundational, mathematically proven limits on LLM capabilities, translating theoretical impossibility results into actionable AI design rules. Its broad implications for AI architecture, scaling, and theoretical computer science offer a much higher potential for foundational scientific impact compared to Paper 2, which presents an applied, domain-specific benchmark.
Paper 1 introduces a novel conceptual framework (epistemic miscalibration in planning) for understanding a fundamental failure mode in LLM-based multi-agent systems, along with a principled methodology (EPC-AW) to address it. This contributes broadly to multi-agent AI systems research with theoretical depth and practical improvement. Paper 2, while useful, is primarily a benchmark contribution for a specific domain (finance spreadsheets), which has narrower impact scope and lower conceptual novelty. Paper 1's identification of a general, previously uncharacterized problem class gives it greater potential to influence future research directions.
Paper 2 introduces a novel conceptual framework (Grounded Personality Reasoning) that addresses a fundamental question about whether MLLMs truly understand or merely pattern-match—a question with broad implications across AI safety, fairness, and deployment. Its contributions (new task formalization, large dataset, comprehensive evaluation with novel failure-mode metrics, and benchmarking 27 models) are methodologically rigorous and reveal a striking 'Prejudice Gap' finding. This has broader cross-field impact (social AI, psychology, fairness/bias) compared to Paper 1, which provides a valuable but more domain-specific benchmark for spreadsheet tasks in finance.