MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
Shaden Alshammari, Kevin Wen, Abrar Zainal, Mark Hamilton, Navid Safaei, Sultan Albarakati, William T. Freeman, Antonio Torralba
Abstract
Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts. MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that retrieval-augmented generation performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mathnet.mit.edu.
AI Impact Assessments
(3 models)Scientific Impact Assessment: MathNet
1. Core Contribution
MathNet introduces a large-scale, multilingual, multimodal benchmark of 30,676 Olympiad-level math problems spanning 47 countries, 17 languages, and 143 competitions over four decades. The paper's distinguishing contribution beyond being "yet another math benchmark" lies in three aspects: (a) the scale and provenance of data sourced from official national booklets rather than community platforms like AoPS, (b) the introduction of a math-aware retrieval task that evaluates embedding models' ability to identify structurally equivalent problems, and (c) the construction of a retrieval-augmented problem solving (RAG) benchmark with expert-curated problem pairs.
The taxonomy of mathematical similarity—Invariance, Resonance, and Affinity—provides a useful conceptual framework for understanding different levels of mathematical relatedness, though its operationalization in the benchmark is limited primarily to the Invariance level (MathNet-Retrieve) and Resonance level (MathNet-RAG).
2. Methodological Rigor
Data collection pipeline: The three-stage extraction pipeline (OCR → LLM-based alignment → multi-stage verification) is well-designed for the heterogeneous nature of scanned national booklets across languages and formats. The triple verification (rule-based, GPT-4.1 judge, human review) provides reasonable quality assurance, though the paper lacks quantitative reporting on inter-annotator agreement or the proportion of problems rejected at each stage.
Evaluation protocol: The scoring protocol (GPT-5 as judge, binarized at 6/7) follows established practices from IMO-Bench. The paper commendably includes human expert grading for MathNet-RAG and provides a detailed comparison between LLM and human grading in Table 11, showing generally good but imperfect alignment. However, for MathNet-Solve (6,400 test problems), relying entirely on automated grading without systematic human validation of a significant sample is a weakness given the difficulty of grading Olympiad proofs.
Retrieval benchmark construction: MathNet-Retrieve uses GPT-5 to generate equivalent positives and hard negatives from anchor problems. While this is a practical approach at scale, the synthetic nature of this benchmark raises concerns about whether the equivalences and near-misses fully capture the subtlety of real mathematical equivalence. The expert-curated MathNet-RAG (only 35 pairs) is more convincing but very small.
Statistical reporting: The paper includes standard errors throughout, which is appreciated. However, some confidence intervals are quite wide (especially for language-specific and RAG results), making it difficult to draw strong conclusions about certain comparisons.
3. Potential Impact
Immediate impact: MathNet fills a genuine gap as the largest curated Olympiad-level dataset with official solutions. The multilingual coverage (17 languages vs. typically 1-2) enables cross-lingual evaluation that was previously impossible. The public release at mathnet.mit.edu should facilitate broad adoption.
Math-aware retrieval as a new task: This is arguably the most novel contribution. The demonstration that embedding models achieve only ~5% Recall@1 on mathematically equivalent problems (Table 4) while placing non-equivalent problems at higher similarity scores (Figure 6) reveals a fundamental limitation of current embeddings. This finding could catalyze research into structure-aware mathematical representations.
RAG for mathematical reasoning: The finding that RAG performance is "highly sensitive to retrieval quality" (up to 12% gains with expert-paired problems vs. marginal or negative gains with embedding-based retrieval) has practical implications for building mathematical reasoning systems.
Training data: The 23,776-problem training split could be valuable for fine-tuning mathematical reasoning models, though the paper focuses on evaluation rather than training.
4. Timeliness & Relevance
The paper is highly timely. With frontier models claiming IMO gold-medal performance and mathematical AI advancing rapidly, there is urgent need for larger, more diverse, and less contamination-prone benchmarks. The use of official national booklets rather than AoPS provides some protection against data contamination, though problems from well-known competitions (IMO shortlists, CMO) may still appear in training data. The paper does not systematically address contamination detection, which is a notable gap.
The math-aware retrieval angle addresses an emerging need as mathematical AI systems move toward more sophisticated tool use and reasoning chains that could benefit from retrieving relevant prior results.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Comparison to prior art: The paper clearly positions itself against existing benchmarks (Table 1) and offers genuine advantages in scale, language coverage, and task diversity. The retrieval component is genuinely novel in this space.
Overall Assessment
MathNet is a solid benchmark paper that makes a meaningful contribution to mathematical AI evaluation. Its primary strengths lie in scale, diversity, and the introduction of math-aware retrieval as a task. The retrieval findings—showing that embeddings fundamentally fail at capturing mathematical structure—are the most intellectually compelling results. The dataset will likely see broad adoption given its public availability and ICLR 2026 publication. However, the synthetic nature of the retrieval benchmark, small size of the RAG evaluation, and lack of contamination analysis temper enthusiasm somewhat.
Generated Apr 21, 2026
Comparison History (52)
While Paper 1 provides valuable insights and practical diagnostics for prompt engineering, Paper 2 introduces a massive, multilingual, and multimodal benchmark for mathematical reasoning. High-quality benchmarks like MathNet historically drive broad progress in AI model development and receive extensive citations, indicating a higher long-term scientific impact across the field of machine learning.
Paper 1 introduces a novel evaluation paradigm for rule-governed AI, reframing moderation assessment from label agreement to policy-grounded defensibility with new metrics (DI/AI) and a practical signal (PDS) derived from model logprobs. It is methodologically rich (formalization, large-scale validation, ablations on rule specificity, variance attribution) and directly targets a pressing real-world need: auditable, safe automation under explicit governance. Its ideas generalize beyond moderation to any compliance/regulatory setting, giving broader cross-field impact than Paper 2, which is mainly a valuable but more incremental benchmark release.
MathNet provides a large-scale, publicly available benchmark resource (30,676 problems, 47 countries, 17 languages) that will likely be widely adopted by the AI research community for evaluating mathematical reasoning and retrieval capabilities. Benchmarks of this quality and scale tend to accumulate high citations and drive research agendas. Paper 1 presents an intellectually interesting formal framework for AI governance with Coq proofs, but its highly theoretical nature, narrow focus on effect governance, and lack of empirical validation limit its near-term adoption and breadth of impact across the research community.
MathNet introduces a large-scale, multimodal benchmark for evaluating advanced reasoning in foundation models. Such datasets frequently become standard evaluation metrics, driving widespread adoption and high citations across the broader AI community. While Paper 1 offers a rigorous methodological contribution to human-AI collaboration in medical imaging, its highly specialized focus limits its breadth of impact compared to a global reasoning and RAG benchmark that directly addresses a central challenge in modern AI development.
Paper 2 (MathNet) likely has higher scientific impact because it releases a large, multilingual, multimodal, expert-curated dataset and benchmark that can become shared infrastructure for the community. Its applications span model evaluation, retrieval research, RAG, and multilingual/multimodal reasoning, enabling broad reuse across labs and industries. Benchmarks/datasets typically drive sustained citations and standardization. Paper 1 proposes a valuable alignment method, but its impact may be narrower (preference-optimization niche) and more contingent on adoption versus competing alignment techniques.
High-quality, large-scale benchmarks like MathNet historically drive rapid empirical progress and accumulate massive citations in the AI community. Its multimodal, multilingual, and retrieval-augmented aspects address critical bottlenecks in evaluating LLM mathematical reasoning. While Paper 2 offers a valuable theoretical perspective on world models for robotics, Paper 1 provides a concrete, immediately usable dataset and benchmark that will likely be adopted broadly and immediately across the foundation model research ecosystem.
Paper 1 likely has higher scientific impact due to a more novel methodological contribution (multi-agent symbolic/metaheuristic equation discovery) with direct, high-value real-world applications across scientific modeling, interpretability, and extrapolation—key bottlenecks in AI-for-science. If validated broadly, autonomous recovery of governing equations can influence multiple domains (physics, biology, engineering) and reshape scientific workflows. Paper 2 is timely and valuable infrastructure (large multilingual multimodal benchmark), with broad community utility, but benchmarks typically yield incremental impact compared to a new paradigm that demonstrably improves extrapolatable, interpretable scientific discovery.
PRTS introduces a fundamentally novel paradigm for VLA pretraining by reformulating it through goal-conditioned reinforcement learning with contrastive representations, addressing a core limitation in robot foundation models. It demonstrates SOTA across multiple benchmarks and real-world tasks, with broad implications for robotics. While MathNet is a valuable large-scale benchmark for mathematical reasoning, benchmarks generally have less transformative impact than methodological innovations. PRTS's contribution—injecting goal-reachability awareness into VLMs—represents a deeper conceptual advance with wider downstream applications in embodied AI.
While Paper 1 offers a valuable efficiency improvement for reasoning models, Paper 2 introduces a massive, multilingual, and multimodal benchmark for advanced mathematical reasoning. High-quality benchmarks of this scale typically become foundational evaluation standards, driving widespread progress, exposing vulnerabilities in state-of-the-art models, and garnering broader scientific impact and citations across the entire AI community.
MathNet introduces a large-scale, multilingual, multimodal benchmark spanning 47 countries with 30,676 Olympiad-level problems, addressing critical gaps in evaluating mathematical reasoning and retrieval for LLMs. Its breadth of impact is substantial—it serves the rapidly growing AI reasoning community, provides a public resource, and benchmarks state-of-the-art models on multiple tasks. Paper 2 offers elegant information-theoretic analysis of explanation methods but addresses a narrower audience (XAI theory). MathNet's timeliness, practical utility as a benchmark, and relevance to the massive LLM evaluation ecosystem give it higher potential impact.
Paper 1 likely has higher impact due to its large, high-quality, multilingual, multimodal benchmark and public release, which can become a standard evaluation substrate for many labs and accelerate progress in reasoning, retrieval, and RAG. Its methodological contribution (dataset + retrieval benchmark + evaluations) is broadly usable across model families and tasks, with clear real-world applications in education, tutoring, and scientific tooling. Paper 2 is novel and timely for agent-based social emergence, but its impact may be narrower and more dependent on simulation assumptions and validation complexity.
MathNet establishes a foundational, large-scale multimodal benchmark for mathematical reasoning. In AI, comprehensive evaluation datasets for frontier reasoning capabilities typically drive widespread adoption and high citation counts across the entire foundation model community, leading to broader scientific impact than system-level optimizations for specific GUI agent architectures like those proposed in Paper 1.
Paper 2 likely has higher scientific impact because it releases a large, multilingual, multimodal benchmark and retrieval dataset (30k+ Olympiad problems; 17 languages; curated equivalence pairs) that can become a widely used community standard, enabling reproducible evaluation and driving progress across LLM reasoning, multimodal learning, embeddings, and retrieval-augmented generation. Its real-world applications (education, search, tutoring, model benchmarking) and breadth across fields are broad and timely. Paper 1 is novel and important for AI peer-review security, but its immediate impact may be narrower and constrained by access to submission data and adoption pathways.
Paper 1 introduces a fundamentally novel conceptual framework arguing that AI safety in multi-agent systems depends on interaction topology rather than individual model alignment. This challenges core assumptions in the AI safety community and has broad implications for regulation, evaluation, and deployment of agentic AI systems—a rapidly growing area. Its cross-disciplinary relevance (complex systems, policy, safety) and timeliness give it high impact potential. Paper 2, while valuable as a comprehensive benchmark, is more incremental, extending existing mathematical reasoning evaluation with greater scale and multilinguality, and benchmarks tend to have diminishing marginal impact in a crowded space.
Paper 2 likely has higher impact due to immediate, broad real-world applicability and adoption potential: a large, public, multilingual multimodal benchmark and retrieval suite can become a standard evaluation resource across ML, NLP, education, and information retrieval. Its scale, task diversity, and released data enable rapid follow-on work and reproducibility. Paper 1 is highly novel and rigorous (machine-checked formal results) with long-term importance for AI governance and semantics, but its impact is narrower and adoption-dependent on specialized formal methods and tooling.
MathNet introduces a large-scale, multilingual, and multimodal benchmark for mathematical reasoning. High-quality, comprehensive benchmarks historically exert massive scientific impact by becoming standard evaluation criteria for all new foundation models. Its unique inclusion of retrieval and RAG tasks addresses critical gaps in current evaluations, ensuring broad, long-term adoption across the AI community, whereas inference decoding algorithms (Paper 2) typically see narrower or more transient adoption.
MathNet introduces a large-scale, multimodal, multilingual benchmark (30,676 problems, 47 countries, 17 languages) addressing a significant gap in mathematical reasoning evaluation. Its breadth—spanning problem solving, retrieval, and RAG tasks—gives it wide utility across multiple research communities (NLP, math AI, information retrieval, education). Large benchmarks tend to become community standards with high citation impact. Paper 1 provides useful practical insights on zero-shot confidence estimation for LLM routing but addresses a narrower engineering problem with incremental contributions over existing methods.
Paper 2 likely has higher impact due to broad, timely relevance to foundation-model evaluation, a large publicly released multimodal/multilingual dataset, and a new benchmark spanning solving, retrieval, and RAG—enabling widespread use across NLP, IR, multimodal learning, and education. Its scale and standardized evaluation can drive rapid follow-on work and comparability across models. Paper 1 is novel and rigorous within stream reasoning/CER, but its applications and audience are narrower, making field-wide impact likely smaller.
MathNet introduces a large-scale, multilingual, multimodal benchmark spanning 47 countries and 30,676 Olympiad-level problems, addressing significant gaps in mathematical reasoning evaluation. It defines three novel tasks including the first benchmark for mathematical problem retrieval. Its breadth of impact is substantial—serving the LLM reasoning, retrieval, and RAG communities simultaneously. Paper 2 (PARA) proposes an incremental improvement to LoRA compression via post-hoc SVD-based rank pruning, which, while useful, represents a more narrow and incremental contribution in an already crowded parameter-efficient fine-tuning space.
MathNet introduces a comprehensive, large-scale, and multilingual benchmark for a critical bottleneck in AI (mathematical reasoning and retrieval). High-quality benchmarks in this space often become foundational evaluation standards, driving widespread field advancement and accumulating massive citations. While Paper 2 offers a valuable and practical optimization technique for LoRA, its scope is more narrowly focused on parameter efficiency, whereas MathNet impacts model evaluation, reasoning, and retrieval across the broader AI community.