The Impact of AI Usage and Informativeness on Skill Development in Logical Reasoning
Shang Wu, Hongyu Yao, Catarina Belem, Shuyuan Fu, Mark Steyvers, Padhraic Smyth
Abstract
Artificial intelligence (AI) is being increasingly integrated into human problem-solving, yet its effects on individual skill development remain unclear. We examine how both AI usage and informativeness can shape learning in the context of a controlled logical reasoning task with on-demand access to AI assistance. We find that greater AI usage is associated with weaker skill development: heavy AI users underperform relative to comparable peers, whereas light AI users perform similarly to matched users who do not use AI. We also find in our study that these patterns are mediated by AI informativeness. Low-information AI neither improves immediate performance nor preserves performance after AI assistance is removed, and is linked to weaker learning overall. On the other hand, high-information AI was found to improve short-run performance without reducing post-AI outcomes on average in our experiments, but with heterogeneous effects. Our findings in general suggest that AI can, depending on context, either complement human skill development by amplifying independent reasoning or can act as a substitute that undermines such reasoning, with the implication that regulating AI access and usage will be important for promoting skill development in the presence of AI assistance.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper investigates how AI usage intensity and informativeness affect human skill development in logical reasoning tasks, using a controlled pre-/post-assessment experimental design. The main novelty lies in the intersection of three factors: (1) measuring skill development *after* AI removal rather than during AI use, (2) experimentally manipulating AI informativeness while holding accuracy constant, and (3) examining individual heterogeneity in how users engage with AI assistance. The key finding is that heavy AI usage is associated with weaker skill development, and this relationship is moderated by AI informativeness—low-information AI uniformly harms learning while high-information AI produces heterogeneous effects that widen ability gaps.
Methodological Rigor
The experimental design has notable strengths but also significant limitations:
Strengths:
Weaknesses:
Potential Impact
The paper addresses a timely concern about AI's effects on human cognition and learning. The practical implications are relevant to:
However, the impact is tempered by the narrow experimental context. The logic puzzle paradigm, while controlled, is far removed from the educational and professional domains where these findings would matter most (e.g., medical diagnosis, programming, essay writing). The findings largely confirm intuitions rather than reveal surprising mechanisms—that heavy reliance on external tools reduces independent learning is well-established in educational psychology literature on scaffolding and desirable difficulties.
Timeliness & Relevance
The paper is highly timely. As LLMs and AI assistants become ubiquitous, understanding their impact on human skill development is urgent. The paper arrives amid growing concern about "cognitive offloading" and "deskilling" in AI-assisted environments. It speaks directly to debates in education about ChatGPT policies and in medicine about diagnostic deskilling with AI support. The HHAI 2026 venue is appropriate.
Strengths & Limitations
Key Strengths:
1. The pre-/post-AI assessment design is the paper's strongest methodological contribution, directly measuring what matters—performance after AI is removed.
2. The distinction between AI informativeness levels is a clean experimental manipulation that yields interpretable results.
3. The heterogeneity analysis revealing that high-information AI widens ability gaps is the most novel and policy-relevant finding.
4. The "solo share" metric captures a meaningful behavioral dimension of cognitive engagement.
Notable Weaknesses:
1. Causal claims exceed design: Usage intensity is not randomly assigned, yet much of the discussion implies causal relationships. The PSM approach mitigates but does not resolve this.
2. Limited statistical power: Key subgroup findings rely on small cells (splitting 43 high-info participants by ability level yields ~21 per group), and several results hover around conventional significance thresholds.
3. Task artificiality: Logic puzzles with deterministic solutions and perfect AI accuracy are far from real-world AI-assisted learning scenarios.
4. Short time horizon: 20 minutes of AI exposure is insufficient to study "skill development" in any meaningful sense—this is more accurately characterized as short-term performance transfer.
5. Missing controls and analyses: No analysis of learning curves within phases, no examination of which specific problems benefited from AI, and limited analysis of what strategies participants actually developed.
6. Self-selection bias: The finding that heavy AI users perform worse is confounded by the possibility that individuals who rely heavily on AI are systematically different in ways not captured by Phase 1 performance alone (e.g., motivation, self-regulation).
7. Inflated confidence finding: The observation about miscalibrated self-assessment among lower-ability participants is suggestive but based on single survey items without validated scales.
Overall Assessment
This paper makes a relevant and timely contribution to an important question, with a reasonably well-designed experiment. However, its impact is limited by the artificial task domain, modest sample sizes, short time horizons, and causal inference challenges. The findings are directionally interesting and policy-relevant but not definitive. The paper is best viewed as a well-motivated pilot study that points toward important dynamics requiring larger-scale, longer-term, and more ecologically valid investigation. It is appropriate for a workshop or short conference paper at HHAI but would need substantially more evidence to influence policy or practice.
Generated May 22, 2026
Comparison History (25)
Paper 1 addresses a critical bottleneck in AI agent deployment—computational inefficiency—by introducing a novel skill-distillation framework and new evaluation metrics. Its quantifiable improvements in success rates and token reduction offer immediate, highly impactful applications in AI engineering. While Paper 2 explores an important societal question regarding human-AI interaction, Paper 1's methodological rigor and direct technical innovations give it a higher potential for driving immediate progress in the fast-moving field of multimodal AI systems.
Paper 1 addresses a broadly relevant and timely question about AI's impact on human skill development and learning, with implications spanning education, workforce development, and AI policy. Its findings about AI usage undermining skill development have immediate real-world applications and will resonate across multiple disciplines (education, cognitive science, HCI, policy). Paper 2, while technically solid, introduces a benchmark for a narrower subfield (ICRL for ad-hoc teamwork) with primarily negative results showing baseline failures, limiting its immediate broader impact beyond the multi-agent RL community.
Paper 1 addresses a critical and highly timely societal question regarding how AI affects human cognitive skill development. Its findings have broad, cross-disciplinary implications for education, policy, and human-computer interaction. While Paper 2 presents a solid technical contribution to AI-assisted writing, Paper 1's insights into human learning in the age of AI offer wider potential real-world applications and societal impact.
Paper 1 addresses the broadly significant and timely question of how AI assistance affects human learning and skill development—a topic with immense societal implications as AI tools become ubiquitous in education and workplaces. Its findings on AI usage intensity and informativeness mediating learning outcomes are novel and policy-relevant, with potential impact across education, cognitive science, and AI policy. Paper 2 presents a solid security contribution (MemAudit) but addresses a narrower technical problem in LLM agent memory poisoning. While important, its impact is more confined to the AI security community, whereas Paper 1's insights have broader cross-disciplinary relevance.
Paper 1 presents a novel, actionable technical advancement (Co-ReAct) that directly improves the reasoning and search capabilities of LLM agents, a highly active and critical area of AI research. Its introduction of a trained rubric generator using a list-wise reward offers a reusable, drop-in component for future research. While Paper 2 addresses an important societal question regarding human-AI interaction, Paper 1 is likely to have a more immediate and measurable scientific impact through direct adoption and citations in the rapidly evolving field of autonomous AI agents.
Paper 1 likely has higher scientific impact due to methodological and technical novelty (adversarial latent-space robustification plus low-rank subspace alignment) addressing a timely, widely relevant bottleneck in multimodal LLM knowledge editing—reliability/generalization under distributional variation. It has clear downstream applicability to safer, updatable deployed MLLMs and can influence multiple ML subareas (model editing, robustness, multimodal representation learning). Paper 2 is important and timely for human-AI interaction and education, but its impact may be narrower and more context-dependent, hinging on task design and external validity.
Paper 1 addresses a timely, broadly relevant question about AI's impact on human skill development with empirical evidence from controlled experiments. Its findings on how AI usage intensity and informativeness affect learning have immediate implications for education, workforce training, and AI policy—topics of enormous current societal interest. The nuanced finding that AI can complement or substitute for human reasoning depending on context is novel and actionable. Paper 2 presents an interesting software architecture (ActiveGraph) for agentic systems, but it is more niche, lacks empirical validation of its claimed benefits, and primarily contributes to AI engineering rather than generating broadly impactful scientific insights.
Paper 1 addresses a critical technical bottleneck in machine learning (explainability) by introducing a novel integration of causal discovery and argumentation frameworks. This methodological advancement has broad, scalable applicability across numerous high-stakes domains requiring interpretable AI, offering foundational algorithmic tools for future AI development that typically yield higher cross-disciplinary citations than the behavioral insights presented in Paper 2.
Paper 2 addresses a fundamental question about how AI usage affects human skill development and learning, with broad implications for education, workforce development, and AI policy. Its controlled experimental design examining causal mechanisms (AI informativeness as mediator) provides rigorous evidence on a timely topic relevant across multiple fields. Paper 1, while practically useful, is an engineering contribution (a Python framework reducing boilerplate) with narrow impact limited to a specific developer community and lacking scientific novelty beyond software engineering convenience.
Paper 1 addresses a fundamental question about AI's impact on human skill development with rigorous experimental methodology, producing generalizable insights about AI-as-complement vs AI-as-substitute that have broad implications across education, workforce development, and AI policy. Its findings on AI usage intensity and informativeness mediating learning outcomes are novel and timely, relevant to nearly every domain where AI assistance is deployed. Paper 2, while creative in its pedagogical approach, is more niche—focused on a specific classroom practice and benchmark artifact with narrower applicability and less generalizable scientific contributions.