The Power of Power Law: Asymmetry Enables Compositional Reasoning
Zixuan Wang, Xingyu Dang, Jason D. Lee, Kaifeng Lyu
Abstract
Natural language data follows a power-law distribution, with most knowledge and skills appearing at very low frequency. While a common intuition suggests that reweighting or curating data towards a uniform distribution may help models better learn these long-tail skills, we find a counterintuitive result: across a wide range of compositional reasoning tasks, such as state tracking and multi-step arithmetic, training under power-law distributions consistently outperforms training under uniform distributions. To understand this advantage, we introduce a minimalist skill-composition task and show that learning under a power-law distribution provably requires significantly less training data. Our theoretical analysis reveals that power law sampling induces a beneficial asymmetry that improves the pathological loss landscape, which enables models to first acquire high-frequency skill compositions with low data complexity, which in turn serves as a stepping stone to efficiently learn rare long-tailed skills. Our results offer an alternative perspective on what constitutes an effective data distribution for training models.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper challenges the prevailing intuition that balancing training data toward a uniform distribution improves learning of rare skills. The authors demonstrate that across multiple compositional reasoning tasks (multi-step arithmetic, state tracking on S₅, multi-hop QA, and grade-school math), training under power-law distributions consistently outperforms uniform distributions—sometimes dramatically, turning unlearnable tasks into learnable ones without requiring chain-of-thought or curriculum learning.
The key theoretical contribution is a minimalist *k-multiplicative composition* task that captures the essence of skill composition. The authors prove a sharp separation: under uniform distribution, any CSQ learner requires Ω̃(d^{k/2}) samples, while under power-law distribution with α > 1, gradient descent succeeds with Õ(d^{2α}) samples—independent of k in the exponent. This is a meaningful exponential-to-polynomial reduction for compositional tasks.
Methodological Rigor
Theoretical framework: The CSQ lower bound under uniform distribution is clean and follows standard techniques (Szörényi, 2009; Damian et al., 2022), leveraging the symmetry of the function class. The upper bound under power law is more involved and well-structured: the proof proceeds through population gradient analysis, establishes a PL condition enabled by the asymmetric initialization signal, then extends to finite-sample minibatch SGD via concentration arguments. The three-stage characterization (escape flat region → head skills accelerate tail → long-tail convergence) is elegant and provides genuine mechanistic insight.
Potential concerns: The theoretical model (k-multiplicative composition with ±1 scalars) is quite stylized compared to real compositional reasoning. The connection to transformers trained on state tracking or multi-hop QA is mechanistic/empirical rather than formal. The learner class in the upper bound is restricted to functions of the same multiplicative form as the target—the paper does not analyze whether transformers with standard architectures provably benefit similarly. The assumption α > 1 is acknowledged as sufficient but not necessary, and the gap between theory and practice (e.g., α = 1 works empirically in arithmetic) is not fully resolved.
Experimental design: The experiments are well-controlled. The random shuffling of skill indices across experiments is crucial—it rules out trivial explanations like implicit curriculum from natural ordering. The loss landscape visualizations (Figure 2) and gradient norm measurements (Figure 3) provide compelling evidence for the three-stage mechanism. However, all experiments are on synthetic tasks with relatively small models (≤0.6B parameters), leaving the question of whether these findings transfer to realistic pretraining scenarios open.
Potential Impact
This work has several avenues for impact:
1. Data curation strategies: The finding directly challenges the common practice of rebalancing training data toward uniformity. It suggests that natural power-law distributions in language data may be a feature, not a bug, for compositional reasoning—potentially influencing how practitioners approach data mixing and curation.
2. Understanding scaling laws: The connection between power-law data distributions and compositional learning provides a new lens for understanding neural scaling laws, complementing the "quanta" hypothesis of Michaud et al. (2023).
3. Theoretical foundations: The separation result between uniform and power-law distributions extends the growing literature on how distribution structure affects learnability (Daniely & Malach, 2020; Cornacchia & Mossel, 2023; Mousavi-Hosseini et al., 2023), generalizing from single-index models and parity to compositional tasks.
4. Practical implications for synthetic data: For domains requiring compositional reasoning (agentic AI, tool use, mathematical reasoning), this suggests that deliberately engineering power-law distributions in synthetic data could be more effective than balanced sampling.
Timeliness & Relevance
The paper is highly timely. There is intense current interest in improving reasoning capabilities of LLMs, and data distribution/curation is a first-order concern for training. The finding that uniform distributions can be actively harmful for compositional reasoning is directly relevant to ongoing debates about data mixing strategies, the role of chain-of-thought, and scaling laws. The connection to implicit reasoning (without CoT) is particularly relevant given the community's interest in internalizing reasoning chains.
Strengths
Limitations
Overall Assessment
This is a thought-provoking paper that identifies a genuine and underappreciated phenomenon—the beneficial role of distributional asymmetry in compositional learning. The combination of theory, mechanistic analysis, and diverse experiments is compelling, though the gap between the stylized theory and practical implications remains significant. The work opens productive research directions in data distribution design for reasoning tasks.
Generated May 5, 2026
Comparison History (40)
Paper 1 presents a highly innovative framework blending generative models with physical search, significantly accelerating molecular and materials discovery. Its ability to effectively explore rare but physically relevant structures and extrapolate beyond training data provides massive potential for real-world applications in drug discovery and novel materials, offering broader cross-disciplinary impact compared to the theoretical machine learning focus of Paper 2.
Paper 1 has significantly higher potential impact because it addresses a fundamental problem in foundation model training: the effect of data distribution on reasoning. Its counterintuitive theoretical finding that power-law distributions are superior to uniform distributions for learning long-tail skills challenges prevailing data curation paradigms. This insight broadly impacts NLP, ML theory, and large-scale model training. In contrast, Paper 2 presents a solid but niche neuro-symbolic pipeline specific to Wi-Fi-based human activity recognition, which has a much narrower scope and limited generalizability across different scientific fields.
Paper 1 represents a paradigm shift by demonstrating the first end-to-end autonomous AI scientific discovery on a physical platform, yielding a novel, unreported physical mechanism. This has transformative implications for accelerating research across multiple disciplines. Paper 2 provides valuable theoretical insights into LLM training dynamics, but its impact is narrower, primarily affecting machine learning data curation strategies rather than revolutionizing the scientific method itself.
While Paper 1 offers valuable theoretical insights into LLM training distributions, Paper 2 presents a breakthrough in predictive healthcare. By successfully building a multimodal generative model (HealthFormer) that acts as a 'clinical digital twin', it accurately forecasts physiological trajectories and simulates clinical interventions in silico. Validated across independent cohorts and published clinical trials, Paper 2 promises immediate, profound impacts on personalized medicine, clinical trial design, and disease risk stratification, giving it a much broader and more tangible real-world scientific impact.
Paper 1 represents a paradigm-shifting milestone in AI for Science by demonstrating the first end-to-end autonomous scientific discovery on a physical platform. Its ability to discover and validate a new physical mechanism has profound implications for accelerating scientific research across disciplines. While Paper 2 provides valuable theoretical insights into LLM training data distributions, Paper 1's tangible demonstration of autonomous discovery and its potential to fundamentally revolutionize the scientific method give it a significantly higher potential for broad, transformative impact.
MIMIC presents a fundamentally new multimodal foundation model for biomolecules that unifies sequence, structure, evolution, and regulatory modalities in a single generative framework, achieving state-of-the-art results across multiple tasks and enabling practical applications in RNA editing and protein design. Its breadth of impact spans computational biology, drug design, and genomics. While Paper 2 provides valuable theoretical insights into power-law distributions and compositional reasoning, its scope is narrower, focused on understanding training data distributions. MIMIC's practical applications and paradigm-shifting multimodal approach give it higher potential impact.
MIMIC presents a comprehensive multimodal foundation model for biomolecules that unifies sequence, structure, evolution, regulation, and context across nucleic acids and proteins. It demonstrates state-of-the-art results on multiple tasks and enables both prediction and constrained design (RNA editing, protein binder design), with clear biomedical applications. While Paper 2 provides valuable theoretical insights on power-law distributions for compositional reasoning, its scope is narrower and more theoretical. MIMIC's breadth of impact across computational biology, drug design, and genomics, combined with its practical applications, gives it substantially higher potential scientific impact.
Paper 2 likely has higher scientific impact due to its broad, timely relevance to autonomous AI research and scientific reliability, with a large-scale empirical evaluation (25,000+ runs) across eight domains and a clear methodological decomposition (base model vs scaffold) plus behavioral epistemic analysis. Its findings have immediate implications for how scientific-agent systems are evaluated, deployed, and trained, potentially influencing AI safety, meta-science, and agent design. Paper 1 is novel and theoretically grounded but is narrower in scope and nearer-term impact mainly within data curriculum/distribution design for compositional reasoning.
Paper 2 is likely to have higher impact due to a clear, broadly actionable and counterintuitive finding about training data distributions, backed by both empirical results across compositional tasks and a theoretical explanation with provable sample-complexity benefits. It directly informs dataset design and training strategy for many ML systems, making it widely applicable and timely. Paper 1 is important for evaluation/AI-science safety, with large-scale empirical evidence, but its main conclusion is largely diagnostic and may have more indirect downstream leverage than Paper 2’s prescriptive, theory-grounded guidance.
Paper 2 addresses a fundamental problem in machine learning regarding data distributions and compositional reasoning. Its counterintuitive finding about power-law distributions challenges existing paradigms and has broad implications for training large models across various domains. Paper 1, while innovative, is highly domain-specific (Wi-Fi based Human Activity Recognition) and thus has a much narrower scope of potential impact.
Paper 1 presents a highly innovative 'health world model' with immediate, transformative applications in personalized medicine, clinical trial simulation, and disease prediction. Its extensive validation across diverse cohorts and real-world trials demonstrates exceptional methodological rigor. While Paper 2 offers valuable theoretical insights into AI training methodologies, Paper 1's profound potential to revolutionize healthcare, optimize medical interventions, and directly impact human longevity gives it a significantly higher overall scientific and societal impact.
Paper 1 challenges fundamental assumptions about data curation in machine learning, demonstrating theoretically and empirically that power-law distributions outperform uniform ones for compositional reasoning. This offers broad, paradigm-shifting implications for training foundation models across various domains. While Paper 2 provides a strong, specialized framework for computational biology with real-world utility, Paper 1's findings have a significantly wider breadth of impact and address foundational methodologies in the rapidly advancing field of AI.
Paper 2 offers a broadly applicable, counterintuitive principle about training-data distributions—power-law sampling improving compositional reasoning—backed by both empirical results across tasks and a provable minimalist setting explaining why. This combination of generality plus theory can influence dataset design, curriculum learning, and scaling laws across many model families and domains, making its impact potentially wide and lasting. Paper 1 is timely and practically valuable for scientific agents, but appears more system/benchmark-driven and may generalize less beyond the frontier-science agent setting.
Paper 2 likely has higher scientific impact due to its direct, timely push toward automated scientific discovery: an end-to-end agentic framework for frontier-science data construction plus demonstrated SOTA gains on multiple challenging benchmarks, implying immediate real-world applicability and broad relevance across AI, scientific NLP, and tool-using agents. Paper 1 offers a novel and rigorous theoretical/empirical insight about power-law training distributions for compositional reasoning, but its impact is more foundational and narrower in application compared to a scalable system that advances scientific research agents.
Paper 2 challenges fundamental assumptions about data curation in AI, offering theoretical and empirical evidence that power-law distributions improve compositional reasoning. Its insights into data complexity and loss landscapes have broad implications for training foundation models across domains. Paper 1 presents a highly effective but domain-specific framework for single-cell genomics, making Paper 2's potential impact much broader and more foundational.
Paper 1 provides a fundamental theoretical insight about why power-law data distributions benefit compositional reasoning, challenging conventional wisdom about data curation. Its rigorous theoretical analysis with provable guarantees, combined with broad empirical validation across compositional reasoning tasks, offers deep understanding applicable across machine learning. Paper 2, while presenting a solid engineering framework for multi-agent visual systems, is more incremental and narrowly scoped. Paper 1's counterintuitive finding about data distributions has broader implications for training methodology across the field.
Paper 1 presents a fundamental and counterintuitive theoretical insight about data distributions for training, showing that power-law distributions are actually beneficial for compositional reasoning. It combines rigorous theoretical analysis with empirical validation across multiple tasks, offering a paradigm-shifting perspective on data curation. Its breadth of impact is high—affecting foundation model training, data science, and learning theory broadly. Paper 2 proposes an engineering framework for multi-agent visual systems that, while competent, is more incremental and narrower in scope, addressing a specific system design problem rather than revealing a fundamental principle.
Paper 1 is likely higher impact: it proposes a novel multi-agent, symbolic+metaheuristic framework that directly targets a major scientific bottleneck—discovering interpretable governing equations with strong extrapolation—enabling broad real-world applications across physical, biological, and engineered systems. If results hold, the reported gains (orders-of-magnitude extrapolation improvements and extreme parameter compression) suggest substantial practical and cross-disciplinary value. Paper 2 offers timely, rigorous theory about data distributions for compositional reasoning in ML, but its primary impact is more specialized to training dynamics, whereas Paper 1 could reshape AI-for-science workflows broadly.
Paper 2 addresses a fundamental question about data distributions and compositional reasoning in language models, providing both theoretical proofs and empirical evidence for a counterintuitive finding—that power-law distributions aid learning over uniform distributions. This has broad implications for pretraining data curation across all of NLP/ML. Paper 1, while interesting, addresses a narrow intersection of SLMs, DAOs, and decentralized governance with a single-model ablation study (Qwen-3.5-9B), limiting generalizability. Paper 2's theoretical depth, broader applicability, and foundational nature give it significantly higher potential impact.
Paper 1 offers a broadly applicable, theoretically grounded insight into why power-law data distributions can improve compositional reasoning, with a minimalist task, provable sample-complexity advantages, and an explanatory mechanism (beneficial asymmetry improving optimization landscape). This can influence data curation, training strategy, and theory across ML/NLP, making it high-breadth and timely. Paper 2 is timely and impactful for AI safety/healthcare evaluation, with pre-registration and clinician validation, but its impact is narrower, more benchmark/policy-oriented, and may depend on proprietary model behaviors and rapidly changing safety stacks.