Generative AI and the Productivity Divide: Human-AI Complementarities in Education
Lihi Idan, Bharat Anand
Abstract
Generative Artificial Intelligence (GenAI) is transforming how firms create, process, and apply knowledge, yet little is known about the heterogeneity of its productivity effects across users. We report results from a randomized controlled experiment in which participants-analogs of early-career knowledge workers-were assigned to self-study a technical domain using either traditional resources or large-language-model (LLM) assistance. On average, GenAI access significantly increased task performance, but the distribution of gains was highly uneven. Improvements were not predicted by GPA or prior knowledge, but by \textit{AI Interaction Competence (AIC)} -- the ability to elicit, filter, and verify model outputs. High-AIC participants realized outsized gains; low-AIC participants saw limited or even negative marginal returns. A scaffolding intervention (conceptual maps) reduced outcome variance, indicating that standardized workflows can mitigate inequality in AI-mediated performance. We interpret these findings through the lens of human-AI complementarities: GenAI raises mean productivity while introducing a new axis of capability inequality. Managerially, firms should pair GenAI access with short AIC micro-training and simple standard operating procedures to capture value consistently and avoid uneven adoption outcomes.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper investigates the heterogeneous productivity effects of generative AI (GenAI) access in a knowledge-work learning context. The central claim is that while LLM access raises average task performance, the gains are unevenly distributed—not along traditional academic dimensions (GPA, prior knowledge) but along a newly proposed construct called AI Interaction Competence (AIC), defined as the ability to effectively prompt, filter, and verify LLM outputs. The paper additionally tests whether lightweight managerial interventions (scaffolding via conceptual maps, additional study time, peer collaboration) can reduce outcome variance.
The introduction of AIC as a moderating construct is the paper's most distinctive conceptual contribution. The finding that traditional academic markers (GPA, prior domain knowledge) do not moderate treatment effects while AIC does is provocative and, if robust, has significant implications for workforce development and organizational design around AI tools.
Methodological Rigor
The study has notable methodological strengths but also significant limitations that temper confidence in its conclusions.
Strengths:
Weaknesses:
1. Sample size and power concerns. With 179 participants total and 29 dropouts, the effective sample is ~150, distributed across at least 6 treatment arms. Several sub-conditions (e.g., scaffolding, peer, time) likely contain very small cell sizes (perhaps 15-25 per arm). The paper reports several results at p < 0.10 rather than p < 0.05, suggesting underpowered tests. The three-way interaction analysis (Treatment × Pre-intervention score × AIC) with this sample size is particularly concerning for reliability.
2. AIC measurement is poorly specified. The paper never clearly operationalizes how AIC was measured at baseline. It is described as "inferred from behavioral performance" rather than directly assessed, yet it is used as a baseline moderator. If AIC is derived from post-treatment behavior or correlated with treatment outcomes by construction, this introduces serious endogeneity. The paper acknowledges the weak correlation between self-assessed AIC and performance (ρ = 0.46) but does not resolve the measurement question.
3. Ecological validity. Using students studying LLMs via LLMs is a recursive design the authors acknowledge, but it introduces confounds: familiarity with the tool and familiarity with the content domain are entangled. The generalizability to "knowledge work" more broadly is asserted but not demonstrated.
4. Selective reporting and effect magnitude. The paper emphasizes a "17% productivity lift" and "47% increase in variance" in the Discussion, but these specific numbers do not appear in the Results section with accompanying standard errors or confidence intervals. Key regression tables are absent—the paper reports p-values but rarely coefficients, standard errors, or effect sizes in a systematic way. No regression tables are provided at all.
5. The domain choice (studying LLMs) likely inflates engagement and motivation in the LLM condition relative to what would occur in a typical workplace learning context, potentially biasing the average treatment effect upward.
6. Attrition. Differential attrition (20 vs. 9 dropouts) between conditions is acknowledged but not formally addressed through bounds analysis or inverse probability weighting. This could bias treatment effect estimates, particularly if dropouts from the baseline condition were lower-performing.
Potential Impact
The paper addresses a genuinely important question: who benefits from GenAI and why? The conceptual framing—that AI adoption is a "capability design" problem rather than a procurement problem—resonates with practical organizational challenges. The finding that scaffolding reduces variance without lowering means is actionable and, if replicated, could influence how firms structure AI-assisted workflows.
However, the impact is limited by the preliminary nature of the evidence. The construct of AIC, while intuitively appealing, lacks psychometric validation, a clear measurement protocol, and discriminant validity analysis against related constructs (e.g., digital literacy, metacognition, critical thinking). Without this, AIC risks being a label for "people who perform well with AI perform well with AI."
The managerial recommendations (micro-trainings, SOPs, prompt templates) are sensible but not strongly grounded in the experimental evidence. The scaffolding intervention showed only weakly significant effects, and the micro-training recommendation appears to be extrapolated rather than tested.
Timeliness & Relevance
The paper is highly timely. The question of how GenAI affects productivity inequality is one of the most pressing in both management research and public policy. There is genuine demand for causal evidence on this topic, and the number of rigorous experiments remains limited. The paper positions itself well relative to recent high-profile studies (Brynjolfsson et al., 2023; Dell'Acqua et al., 2025).
Strengths & Limitations Summary
Key Strengths:
Key Limitations:
Overall Assessment
This paper tackles an important and timely question with an appropriate experimental design, but the execution falls short of what would be needed for confident causal claims. The underpowered analyses, absent statistical tables, unclear AIC operationalization, and differential attrition are substantial concerns. The conceptual contribution (AIC as a new axis of inequality) is compelling as a hypothesis but requires much stronger empirical grounding to be influential. As a working paper, it offers a useful framework and suggestive evidence, but the findings should be treated as preliminary.
Generated May 19, 2026
Comparison History (22)
Paper 1 has higher likely scientific impact due to strong methodological novelty (formalizing Continual Model Routing, introducing a large-scale benchmark with >2,000 models, and proposing a new contrastive routing method) and broad applicability across ML systems that increasingly rely on model hubs and mixture-of-experts. Its contributions are reusable infrastructure (CMRBench) plus an algorithm (CARvE), enabling follow-on work and adoption in many domains. Paper 2 is timely and practically relevant, but its impact may be narrower to education/management contexts and depends on external validity beyond the experimental setting.
Paper 2 likely has higher scientific impact due to its novel, field-defining benchmark for Text-to-CAD that targets real industrial constraints (manufacturability, functionality, assemblability) and introduces a scalable evaluation protocol with validated VLM-based judging. Benchmarks and leaderboards often become widely adopted infrastructure, catalyzing progress across ML, CAD/CAE, robotics, and manufacturing. Paper 1 is timely and methodologically strong (RCT) with clear managerial implications, but its impact is narrower to education/knowledge-work productivity and may be less reusable as a community standard than a benchmark/dataset.
Paper 1 addresses a highly timely and cross-disciplinary issue (AI productivity inequality) with broad implications for economics, management, and education. Its introduction of 'AI Interaction Competence' provides immediate real-world value for workforce training. While Paper 2 offers strong technical innovation in embodied AI memory systems, its impact is largely confined to the AI research community, giving Paper 1 significantly greater breadth of impact and real-world relevance.
Paper 2 addresses a highly timely and universally relevant problem across multiple disciplines (economics, management, HCI, education) by empirically measuring GenAI's productivity impacts. Its introduction of 'AI Interaction Competence' and actionable insights for reducing inequality offer broader real-world applications and societal relevance compared to Paper 1, which, while technically sound, focuses on a narrower methodological advancement within reinforcement learning and AI safety.
Paper 2 is likely higher impact: it introduces a broadly usable diagnostic benchmark for a timely, high-stakes problem (privacy/intent-following in LLM agents) with clear real-world applicability for deployed systems and regulation. The methodology (adversarial two-model setup, policy dimensions, deterministic scoring, large multi-domain dataset) is readily extensible and can become a standard evaluation tool across academia and industry, influencing model training, red-teaming, and procurement. Paper 1 is rigorous and valuable, but its scope is narrower (education/early-career analogs) and its core contribution (heterogeneous gains + interaction competence) is less likely to become a cross-field infrastructure artifact.
Paper 2 addresses a fundamental question about how GenAI affects productivity inequality—a topic with enormous breadth of impact across economics, education, management, and policy. Its RCT methodology is rigorous, and the novel construct of AI Interaction Competence (AIC) as a predictor of differential gains is highly citable and actionable. The finding that scaffolding reduces variance has immediate practical implications for firms and educators. Paper 1, while technically sound, addresses a narrower optimization problem within LLM agent engineering with more limited cross-disciplinary relevance.
Paper 1 addresses a fundamental question about how GenAI affects productivity inequality—a topic with broad implications across economics, education, management, and policy. Its RCT methodology, novel construct of AI Interaction Competence (AIC), and actionable finding that scaffolding reduces variance make it highly relevant and timely. Paper 2 is a solid engineering contribution (benchmarking framework for computer-use agents) but has narrower impact, primarily within the AI agents community. Paper 1's findings about human-AI complementarities will likely influence organizational adoption strategies, workforce training, and educational policy at scale.
Paper 1 offers broader multi-disciplinary impact by addressing the critical socio-economic implications of GenAI adoption. Through a rigorous RCT, it introduces 'AI Interaction Competence' (AIC), providing actionable insights for education, economics, and management. While Paper 2 makes valuable technical contributions to LLM memory architectures, Paper 1's findings on workplace productivity inequality and actionable mitigation strategies possess wider real-world applicability and timeliness across diverse fields.
Paper 2 addresses a fundamental limitation of Chain-of-Thought reasoning in LLMs—attention degradation over long reasoning traces—and proposes InsightReplay, a novel, generalizable method with rigorous evaluation across 24 settings (multiple model scales, families, and benchmarks). It offers a mechanistic insight into why longer CoT can hurt performance and provides a practical solution applicable broadly to LLM reasoning. Paper 1, while methodologically sound (RCT design) and policy-relevant, addresses a narrower educational/organizational question with findings (AI skill matters, scaffolding helps) that are somewhat expected. Paper 2's technical contribution has broader impact potential across the rapidly growing LLM reasoning research community.
Paper 1 likely has higher scientific impact due to its methodological and technical contributions: it formalizes “full-scene household reasoning,” proposes a training-free, model-agnostic framework (TaskGround), and introduces a new human-validated benchmark (FullHome) that can standardize evaluation and accelerate follow-on research. It is timely for embodied/household agents and practical constraints (privacy, local compute), and its approach can generalize to other grounding-and-planning settings. Paper 2 is rigorous and societally relevant, but its impact may be narrower (education/workforce context) and less likely to seed reusable technical artifacts than Paper 1’s framework + benchmark.