Large-scale semantic mapping of learner agency and autonomy reveals what measurement and generative AI research overlook
Fei Qin, Xiaobo Liu, Yaowen Zhang, Xuming Li, Fei Wang, Mutlu Cukurova, Jingjing Chen, Yu Zhang
Abstract
Learner agency and autonomy are foundational to personal development, yet a pervasive "jingle-jangle" fallacy (i.e. identical terms denoting different constructs, distinct terms denoting identical ones) has substantially hindered cumulative knowledge. Treating meaning as a phenomenon constituted through use in linguistic practice, we extracted 8,954 definitions and 2,700 scale items from over 14,000 publications, to investigate how researchers actually used learner agency and autonomy with a semantic analysis pipeline. The definitional landscape of two constructs resolves into three dimensions: regulation and control of learning (task), intrinsic motivation and internal decision-making (person), and social-relational action (sociocultural), thereby empirically quantifying the jingle-jangle fallacy. Existing scales, however, systematically underrepresent the sociocultural dimension. Critically, current generative AI research in education concentrates on learning regulation and control, narrowing the behavioral repertoire that AI-mediated learning environments are designed to cultivate. Beyond conceptual clarification, this work carries direct implications for conceptualization, measurement, and practice towards supporting the multidimensional learner agency and autonomy.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper tackles the "jingle-jangle fallacy" — where identical terms denote different constructs and distinct terms denote the same one — in the educational constructs of learner agency and learner autonomy. The authors assembled a corpus of 14,611 full-text articles, extracted 8,954 definitions and 2,700 scale items, and used LLM-based semantic embeddings to empirically map the conceptual landscape. The key finding is that the field's definitional space organizes not along the expected binary agency-autonomy divide, but into three latent dimensions: (C1) regulation and control of learning (task-oriented), (C2) intrinsic motivation and internal decision-making (person-oriented), and (C3) social-relational action (sociocultural). This three-cluster structure achieves substantially higher silhouette scores than the conventional binary grouping, demonstrating that the traditional labels are poor organizers of conceptual content. The paper further reveals that existing psychometric scales systematically underrepresent C3, and that generative AI research disproportionately concentrates on C1, potentially narrowing the scope of what AI-mediated educational environments are designed to support.
Methodological Rigor
The methodology is impressively thorough and well-validated at multiple stages. The pipeline—from corpus construction through definition extraction, embedding computation, clustering, and cross-domain comparison—is carefully documented with extensive supplementary materials.
Strengths in rigor:
Methodological concerns:
Potential Impact
This work has several important implications:
1. Measurement reform: The finding that existing scales fail to capture C3 and cannot discriminate agency from autonomy within C1/C2 is a direct challenge to current measurement practices. This could catalyze new scale development anchored to semantic dimensions rather than construct labels.
2. AI education design: The demonstration that GenAI research concentrates on regulation-and-control framings is timely and actionable. As educational AI systems proliferate, this finding could redirect attention toward designing systems that also support volitional and sociocultural dimensions of learner development.
3. Methodological template: The replicable LLM-based pipeline for large-scale construct synthesis—requiring no custom model training—is immediately transferable to other fields facing similar jingle-jangle problems (e.g., engagement, self-regulation, grit/resilience).
4. Interdisciplinary communication: By providing "empirical coordinates" for constructs that have resisted theoretical resolution across education, psychology, and sociology, this work could facilitate cross-disciplinary dialogue.
Timeliness & Relevance
The paper is exceptionally timely on two fronts. First, the proliferation of generative AI in education has made the question of what "learner agency" means in AI-mediated contexts urgent—design decisions depend on definitional clarity. Second, the recent methodological advances in semantic embedding analysis (Wulff & Mata, 2025b; Dorison & Charlesworth, 2025) have created a window for this type of large-scale computational conceptual analysis, and this paper is among the first to apply it at this scale in education.
Strengths & Limitations
Key strengths:
Notable limitations:
Overall Assessment
This is a well-executed, large-scale computational study that makes a genuine empirical contribution to a long-standing theoretical problem. Its methodological innovation, scale of analysis, and practical relevance to both measurement and AI-in-education design make it a notable contribution. The main limitations—modest absolute clustering metrics, English-only corpus, and relatively small GenAI subsample—are honestly acknowledged and do not undermine the central findings.
Generated Jun 10, 2026
Comparison History (17)
Paper 2 has higher likely scientific impact due to its unusually large-scale synthesis (14k+ publications, 8,954 definitions, 2,700 items) that directly addresses a field-wide construct-validity bottleneck (jingle-jangle fallacy) with broad implications for theory, measurement, and educational AI design. Its results are immediately actionable (scale development, evaluation, and AI intervention goals) and timely given rapid adoption of generative AI in education. Paper 1 is technically novel and rigorous but is narrower (adversarial robustness for summarization under specific submodular/DR-submodular settings), likely impacting a more specialized community.
Paper 1 likely has higher scientific impact due to its large-scale, field-spanning synthesis (14,000+ publications) that addresses a foundational construct-validity problem (jingle-jangle) affecting theory, measurement, and downstream interventions across education, psychology, and AI-in-education. Its methodological scope (definition + item mining at scale, semantic mapping) can reshape how agency/autonomy are operationalized and evaluated, and it directly flags systematic measurement bias and misalignment in generative AI research. Paper 2 is timely and practically valuable for AEC, but its impact is more domain-specific and incremental relative to broader conceptual reformation.
Paper 1 demonstrates higher potential scientific impact through its large-scale empirical analysis (14,000+ publications, 8,954 definitions, 2,700 scale items) that addresses a fundamental conceptual problem in education research. It provides actionable insights across multiple domains—measurement, AI in education, and learning theory—revealing systematic gaps in how learner agency/autonomy are conceptualized and measured. Its findings about AI research narrowing behavioral repertoires have timely implications for educational technology design. Paper 2, while practically relevant for EU AI Act compliance, is more narrowly focused on regulatory interpretation of a specific legal definition, limiting its broader scientific reach.
Paper 2 addresses foundational conceptual and measurement issues across a massive corpus (14,000+ publications) in education and psychology. By resolving the 'jingle-jangle' fallacy and critiquing current AI research directions, it offers profound, field-shaping implications for both educational theory and AI-mediated learning design, granting it broader cross-disciplinary impact than Paper 1's domain-specific data generation framework.
Paper 2 addresses a fundamental challenge in AI reasoning—rigorous mathematical proof verification—with a novel step-level verification framework that yields actionable methodological advances. Its contributions (context poisoning identification, strict deductive constraints, failure taxonomy analysis) have broad implications for automated reasoning, formal verification, and agentic AI systems. The work is timely given rapid LLM advancement and has clear extensibility to other domains requiring logical rigor. Paper 1, while valuable for educational research, addresses a narrower audience with semantic/bibliometric analysis of construct definitions, offering more incremental conceptual clarification than methodological breakthrough.
Paper 1 addresses a fundamental conceptual problem (jingle-jangle fallacy) across learner agency/autonomy research using a massive corpus analysis (14,000+ publications), with direct implications for measurement, AI in education, and educational practice. Its interdisciplinary reach spanning education, psychology, AI, and measurement science, combined with its timely critique of how generative AI narrows learning constructs, gives it broader impact potential. Paper 2 makes a solid but incremental contribution extending ReMax to continuous action spaces, achieving performance comparable to existing methods (SAC) rather than surpassing them, limiting its transformative potential.
Paper 2 has higher potential scientific impact due to its novel, actionable framework for pre-deployment assurance of enterprise AI agents (operational envelope, ontology-to-scenario generation, and machine-verifiable trust certificates) and its clear real-world applicability in regulated industries. It includes a pilot across multiple sectors and jurisdictions, quantitative comparisons with baselines, and cross-validation across LLM families, indicating methodological rigor and timeliness amid rapid agent deployment. Paper 1 is valuable for conceptual/measurement clarification in education research, but its impact is more domain-specific and less directly operationalizable across fields.
Paper 1 offers a mechanistically grounded insight into self-distillation for LLMs—showing that structural alignment between feedback and the model's reasoning trace is key—with direct implications for training methods (GRPO, RLHF variants) and broad applicability across LLM research. The per-token advantage analysis provides novel, actionable understanding. Paper 2 provides valuable conceptual clarification in education research but addresses a narrower community. Given the current pace and breadth of LLM/AI research, Paper 1's methodological contribution is likely to be more widely cited and built upon.
Paper 1 tackles a foundational theoretical issue in educational psychology and AI design by analyzing a massive corpus (14,000+ publications) to resolve conceptual ambiguities. Its findings broadly impact measurement, conceptualization, and the future development of generative AI in education. In contrast, Paper 2 presents a valuable but narrower technical contribution and a relatively small dataset (270 plans) for the specific applied task of floor plan furnishing, making Paper 1's scientific impact broader and more significant.
Paper 1 addresses a fundamental conceptual bottleneck across education, psychology, and AI using a novel large-scale semantic pipeline. Its insights into the limitations of current measurement scales and its timely implications for designing generative AI in education give it a broader, more transformative cross-disciplinary impact compared to the domain-specific manufacturing cybersecurity improvements presented in Paper 2.