SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents
Han Li, Vibhor Malik, Zahra Zanjani Foumani, Alberto Castelo, Shuang Xie, Ailin Fan, Keat Yang Koay, Yuanzheng Zhu
Abstract
A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience. We present SimGym, a framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents operating in a live browser. The framework comprises three key components: (a) a traffic-grounded persona generation pipeline that derives per-shop buyer archetypes and intents from production clickstream data; (b) a live-browser agent architecture that combines multimodal perception over visual and browser-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and (c) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior. We validate SimGym on A/B tests of visually driven UI theme changes from a major e-commerce platform across diverse storefronts and product categories. Empirical results show that SimGym agents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add-to-cart shifts observed across interface variants in real-buyer traffic. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants.
AI Impact Assessments
(1 model)Scientific Impact Assessment: SimGym
1. Core Contribution
SimGym presents an end-to-end framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents that interact with live browser environments. The framework integrates three components: (a) a traffic-grounded persona generation pipeline that derives buyer archetypes from production clickstream data, (b) a multimodal live-browser agent architecture with episodic memory and guardrails, and (c) an evaluation protocol that validates simulated outcomes against real human behavioral shifts.
The central problem addressed is the cost, latency, and risk of traditional A/B testing—which requires weeks of live traffic diversion and can degrade user experience. SimGym proposes to pre-screen interface variants synthetically, reducing experimental cycles from weeks to minutes. The key novelty lies in the integration of these three components into a unified, validated system rather than any single component being fundamentally new. Traffic-grounded persona generation, browser-based agents, and A/B evaluation have been explored individually, but their composition with validation against real production A/B tests is the primary contribution.
2. Methodological Rigor
Strengths in methodology:
Weaknesses in methodology:
3. Potential Impact
The practical implications are significant for the e-commerce industry. If synthetic A/B testing can reliably pre-screen interface variants, it could:
However, the current accuracy levels suggest SimGym is better positioned as a screening/prioritization tool rather than a replacement for real A/B tests. The paper acknowledges this in the conclusion.
For the research community, SimGym contributes a concrete methodology for validating synthetic agents against real intervention effects—a gap the authors correctly identify in prior work. The modular architecture could serve as a testbed for improved persona generation, agent architectures, or evaluation protocols.
4. Timeliness & Relevance
This work is highly timely. The convergence of capable VLMs (Gemini 3, GPT-5), robust browser automation tools (Stagehand), and growing interest in LLM-based agents creates a natural opportunity. The paper addresses a genuine industry pain point, and the timing relative to concurrent work (AgentA/B, SimAB, UXAgent, PAARS) positions it well in an emerging research area. The key differentiation from concurrent work is the validation against real production A/B test outcomes rather than proxy metrics or synthetic benchmarks.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations:
The paper is well-written and structured, with comprehensive appendices including agent trace examples. The industry authorship (Shopify) provides unique access to production data but may also introduce bias toward demonstrating commercial viability. The lack of a public dataset or reproducible benchmark limits community adoption for comparative research.
Overall, SimGym represents a solid applied contribution that demonstrates the feasibility of VLM-based A/B test simulation with real-world validation, but the moderate accuracy levels, narrow evaluation scope, and single-platform setting temper the strength of its conclusions.
Generated May 20, 2026
Comparison History (21)
Paper 1 presents a highly innovative application of VLM agents to simulate human behavior for A/B testing, solving a major real-world bottleneck in e-commerce. Its strong empirical validation (77% alignment with real traffic) and immediate practical utility demonstrate higher potential for widespread industry adoption and impact compared to Paper 2, which introduces a useful benchmark but leaves its core algorithmic challenge unsolved.
Paper 2 addresses a fundamental methodological problem—parametric look-ahead bias—that affects all LLM-based financial backtesting research. It introduces a novel, retraining-free technique (FinCAD) with rigorous evaluation across multiple models and assets. This problem generalizes beyond finance to any domain where LLMs are evaluated on historical data they were trained on, giving it broader methodological impact. Paper 1, while practically useful for e-commerce A/B testing simulation, is more application-specific and incremental in its VLM agent approach. Paper 2's contribution to research methodology integrity gives it higher scientific impact potential.
Paper 1 introduces a principled theoretical framework connecting recursive neural architectures to approximate Bayesian inference over latent reasoning trajectories, with practical diagnostics and significant performance gains (85.9% to 98.0% on Sudoku) without retraining. This has broad implications for understanding and improving neural reasoning systems. Paper 2 presents a useful engineering framework for simulating A/B tests using VLM agents, but is more application-specific to e-commerce with narrower scientific contributions. Paper 1's methodological novelty, theoretical depth, and cross-domain applicability give it higher potential scientific impact.
SimGym presents a novel, end-to-end framework addressing a significant practical problem in e-commerce A/B testing, combining VLM agents with real traffic data and validated against real-world outcomes (77% directional alignment). It offers clear real-world applications—reducing A/B test cycles from weeks to under an hour—with strong methodological rigor through empirical validation on a major platform. Paper 2 (AgentAtlas) contributes useful taxonomies and evaluation methodology for LLM agents, but is explicitly positioned as a 'measurement-protocol demonstration, not a benchmark release,' limiting its immediate impact. SimGym's concrete, validated system with direct industry applicability gives it higher potential impact.
SimGym addresses a high-value practical problem in e-commerce A/B testing with a novel framework combining VLMs, browser agents, and real traffic grounding. Its 77% directional alignment with real outcomes and reduction from weeks to under an hour represents significant practical impact. Paper 2 advances ToM benchmarking, a narrower NLP subfield, with incremental improvements on existing benchmarks. While both are competent works, SimGym's combination of methodological novelty (VLM agents for realistic simulation), broad commercial applicability, and validation against real production data gives it higher potential for cross-disciplinary impact in AI, HCI, and e-commerce.
SimGym addresses a well-defined, high-impact problem (A/B testing in e-commerce) with a complete framework validated against real-world data, achieving measurable results (77% directional alignment). It has clear practical applications, reducing experimental cycles from weeks to under an hour. Paper 1, while addressing an interesting problem in data-system composition, is self-described as an 'early prototype' and 'proof of life,' suggesting limited validation. SimGym's combination of VLM agents, traffic-grounded personas, and rigorous empirical validation gives it broader appeal across HCI, ML, and e-commerce research communities.
SimGym presents a more complete and validated framework with strong empirical results (77% directional alignment with real A/B tests), addressing a well-defined, high-value problem in e-commerce experimentation. It combines novel VLM agent architecture with real-world validation against production data. Paper 2, while addressing an interesting problem in data-system composition, is self-described as an 'early prototype' and 'proof of life,' with limited validation on a single workload. SimGym's broader applicability, methodological rigor, and immediate practical value for the large e-commerce industry give it higher impact potential.
Paper 2 (SimGym) has higher potential impact due to strong real-world applicability and timeliness: it targets a major bottleneck in industry experimentation (A/B test cost, latency, risk) and proposes an end-to-end, traffic-grounded VLM-agent simulation framework validated against real platform outcomes. Its methodology includes grounding personas in production clickstream data and an evaluation protocol comparing simulated vs. observed shifts, supporting practical adoption. Breadth is also larger (ML, HCI, causal/experimentation, e-commerce systems). Paper 1 is novel and important for strategic ML on tabular PFNs, but is narrower and more niche in immediate deployment.
Paper 1 presents a comprehensive, methodologically rigorous framework addressing fundamental challenges in survey research (declining response rates, missing data, AI fraud) with novel contributions including theory-constrained LLM imputation, subgroup-stratified bias auditing, and hallucination-managed chatbots. It tackles a broadly applicable methodological problem relevant across social sciences, public health, and disaster research. Paper 2, while innovative in applying VLM agents to A/B test simulation, addresses a narrower commercial application with 77% directional alignment—promising but domain-specific. Paper 1's breadth of impact, methodological contributions, and cross-disciplinary relevance give it higher scientific impact potential.
SimGym introduces a novel framework for simulating A/B tests using VLM agents in live browsers, addressing a significant practical problem in e-commerce. Its cross-disciplinary impact (HCI, ML, e-commerce) and immediate real-world applicability—reducing A/B test cycles from weeks to under an hour with 77% directional alignment—represent substantial practical value. While VISAFF makes solid contributions to multimodal ERC with its tuning-free approach, it is more incremental within an established research area. SimGym's novelty in combining traffic-grounded personas with browser-based VLM agents opens a new research direction with broader industry implications.