FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Pith reviewed 2026-05-17 00:37 UTC · model grok-4.3
The pith
FrontierMath shows that current AI models solve under 2% of hundreds of original expert-level mathematics problems.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI 1.
What carries the argument
The FrontierMath benchmark of original unpublished problems paired with automated verification to test advanced mathematical reasoning.
If this is right
- Supplies a contamination-resistant testbed for measuring AI progress toward expert-level mathematical abilities.
- Shows that present models remain far below the performance of human mathematicians on problems across number theory, analysis, algebraic geometry, and category theory.
- Allows consistent tracking of improvements as AI systems develop better reasoning methods.
- Sets evaluation standards that match the multi-hour or multi-day effort typical for human experts.
Where Pith is reading between the lines
- Persistent low scores may suggest that current AI training approaches require new components to reach expert mathematical performance.
- The benchmark could support comparisons with other scientific reasoning tasks to identify where math presents unique difficulties.
- High performance on FrontierMath problems might eventually link to an AI system's capacity for producing original mathematical results.
Load-bearing premise
The problems are genuinely original and unpublished with no data contamination risk, and automated verification reliably measures true mathematical reasoning ability.
What would settle it
A current leading AI model achieving success on more than 10 percent of the FrontierMath problems without prior exposure to them would challenge the reported performance gap.
read the original abstract
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FrontierMath, a benchmark of hundreds of original, expert-vetted mathematics problems spanning computational areas like number theory and real analysis as well as abstract topics in algebraic geometry and category theory. Problems are designed to require hours to days of expert effort, and the manuscript reports that current state-of-the-art AI models solve under 2% of them using automated verification on unpublished problems to reduce contamination risk.
Significance. If the evaluation methodology proves robust, FrontierMath would provide a valuable, high-bar testbed for tracking progress toward expert-level mathematical reasoning in AI systems. The emphasis on original problems and broad coverage of modern mathematics branches is a positive feature that could help quantify the claimed gap between AI performance and human expertise.
major comments (2)
- [Abstract and verification description] The automated verification procedure for abstract problems is under-specified. The abstract states that problems include 'abstract questions in algebraic geometry and category theory' and relies on 'automated verification,' yet no details are given on answer formats, checker implementation, handling of multi-step proofs or constructions, or acceptance criteria for equivalent but non-canonical solutions. This directly affects the reliability of the <2% solve-rate claim.
- [Benchmark design and problem selection] Details on problem curation and vetting for originality are insufficient. The central claim depends on the problems being genuinely new and unpublished to avoid data contamination, but the manuscript provides limited information on the expert review process or safeguards against prior publication.
minor comments (2)
- [Abstract] The exact number of problems and their distribution across subfields should be reported more precisely rather than as 'hundreds' to allow better assessment of statistical power.
- [Introduction] Consider adding a table or section comparing FrontierMath to existing benchmarks (e.g., MATH, GSM8K) in terms of difficulty and verification approach for context.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback and for recognizing the potential value of FrontierMath as a high-bar benchmark. We address each major comment below, providing clarifications and committing to specific revisions that will strengthen the manuscript without altering its core claims.
read point-by-point responses
-
Referee: [Abstract and verification description] The automated verification procedure for abstract problems is under-specified. The abstract states that problems include 'abstract questions in algebraic geometry and category theory' and relies on 'automated verification,' yet no details are given on answer formats, checker implementation, handling of multi-step proofs or constructions, or acceptance criteria for equivalent but non-canonical solutions. This directly affects the reliability of the <2% solve-rate claim.
Authors: We agree that the manuscript's description of automated verification is high-level and would benefit from greater specificity, particularly for abstract problems. In the revised version we will add a dedicated subsection on verification procedures. This will specify answer formats (e.g., explicit algebraic objects, invariants, or canonical representatives), the use of computer-algebra and formal-verification libraries for checking equivalence or isomorphism, and the fact that problems are constructed so that a final verifiable output (rather than a full multi-step proof) can be checked automatically. We will also clarify acceptance criteria for non-canonical but equivalent solutions. These additions will directly support the reliability of the reported solve rates. revision: yes
-
Referee: [Benchmark design and problem selection] Details on problem curation and vetting for originality are insufficient. The central claim depends on the problems being genuinely new and unpublished to avoid data contamination, but the manuscript provides limited information on the expert review process or safeguards against prior publication.
Authors: The referee correctly notes that the current text gives only a brief account of curation. We will expand the relevant section to describe the process in more detail: problems were proposed by domain experts, reviewed internally for correctness and difficulty, and checked for novelty against recent literature and standard databases. All problems were developed specifically for this benchmark and have not been previously published. To preserve the benchmark's utility we will not release the full problem set at this stage, but the added description of the vetting workflow and contamination safeguards will better substantiate our claims. revision: yes
Circularity Check
No circularity: benchmark introduction with direct empirical evaluation
full rationale
The paper introduces FrontierMath as a new collection of original, unpublished mathematics problems and reports an empirical result that current SOTA models solve under 2% of them. No equations, fitted parameters, or derivations are present. The central claim rests on direct model evaluation against the benchmark rather than any self-referential definition, renamed known result, or load-bearing self-citation chain. The automated verification procedure is described at a high level in the abstract but does not reduce the performance statistic to an input by construction. This is a standard benchmark paper whose claims are falsifiable by external replication and therefore self-contained.
Axiom & Free-Parameter Ledger
Forward citations
Cited by 46 Pith papers
-
Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Soohak is a new 439-problem mathematician-authored benchmark showing frontier LLMs reach only 30% on research math and fail to exceed 50% on refusing ill-posed questions.
-
Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Soohak is a 439-problem mathematician-curated benchmark where frontier LLMs reach at most 30.4% on research math challenges and no model exceeds 50% on refusal for ill-posed problems.
-
Co-Designing Quantum Codes with Transversal Diagonal Gates via Multi-Agent Systems
A Lean-verified multi-agent system produces a catalogue of 14,116 quantum codes with transversal diagonal gates for small parameters, extracts infinite families, and resolves specific distance-3 cases with constructio...
-
Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
CritPt benchmark shows state-of-the-art LLMs reach only 5.7% average accuracy on full-scale unpublished physics research tasks, rising to about 10% with coding tools.
-
IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs
ISOSCI benchmark finds 91.3% of reasoning-mode accuracy gains in LLMs on science problems depend on domain knowledge rather than invariant logical structure.
-
Beyond Shapley: Efficient Computation of Asymmetric Shapley Values
Polynomial-time exact ASV computation for rooted trees via topological equivalence classes, plus sampling approximation for arbitrary causal DAGs.
-
Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search
Fine-tuned 0.6B LLMs with beam search achieve 85% success on 60 test Shannon entropy inequalities (n=10-15), outperforming GPT-5.5 (1.7%) and Psitip (33.3%).
-
LeanMarathon: Toward Reliable AI Co-Mathematicians through Long-Horizon Lean Autoformalization
LeanMarathon uses four contract-scoped agents on an evolving blueprint coordinated by a two-stage orchestrator to formalize seven theorems from Erdős problems in Lean, proving 258 lemmas with no sorry across three runs.
-
CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions
CrowdMath is a new dataset of annotated collaborative math proof discussions where frontier LLMs achieve 83-88% on next-post prediction but only 0.42 macro-F1 on identifying contribution roles.
-
GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory
GTBench is a new curriculum-grounded benchmark showing GPT-5 performs strongly on basic graph theory tasks but all models, including it, struggle more on advanced proofs with notable evaluator disagreements.
-
FVSpec: Real-World Property-Based Tests as Lean Challenges
A new benchmark of 9,415 Lean 4 specifications derived from 2,772 scraped Python property-based tests, plus a three-agent LLM transpilation pipeline and proof-generation baselines.
-
Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
The paper introduces a multi-turn interactive benchmark using 474 executable games to evaluate LLMs on evidence acquisition, belief updating, contextual robustness, and metacognitive adaptation, revealing large perfor...
-
Forecasting Scientific Progress with Artificial Intelligence
Introduces the CUSP benchmark across 4760 events and finds frontier AI models can pick plausible directions but fail to predict whether or when scientific advances will occur, with performance varying by domain and in...
-
Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
Formal Conjectures is a Lean 4 benchmark containing 2615 formalized problems with 1029 open conjectures, designed to evaluate automated mathematical reasoning and proof discovery.
-
MathDuels: Evaluating LLMs as Problem Posers and Solvers
Self-play between LLMs for problem authoring and solving, scored via Rasch modeling, shows that authoring and solving skills are partially decoupled and that the benchmark difficulty evolves with new models.
-
Fine-Tuning Small Reasoning Models for Quantum Field Theory
Small 7B reasoning models were fine-tuned on synthetic and curated QFT problems using RL and SFT, yielding performance gains, error analysis, and public release of data and traces.
-
Problem Reductions at Scale: Agentic Integration of Computationally Hard Problems
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
-
$k$-server-bench: Automating Potential Discovery for the $k$-Server Conjecture
k-server-bench formulates potential-function discovery for the k-server conjecture as a code-based inequality-satisfaction task; current agents fully solve the resolved k=3 case and reduce violations on the open k=4 case.
-
DeonticBench: A Benchmark for Reasoning over Rules
DEONTICBENCH is a new benchmark of 6,232 deontic reasoning tasks from U.S. legal domains where frontier LLMs reach only ~45% accuracy and symbolic Prolog assistance plus RL training still fail to solve tasks reliably.
-
Data and Evaluation Closed-Loop for Model Capability Enhancement
Proposes capability slices with dual taxonomies and mapping rules to form a closed loop converting benchmark failures into targeted data interventions, validated via two opposing case studies on BBH and math reasoning.
-
Theorist Toolbox: Tools for Agent Based LLM-assisted economic theory Research
External verification structures, not model capability, determine the reliability of LLM-assisted economic theory, as shown in attempts to design an incentive mechanism for a grade inflation model where adversarial ch...
-
Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model
Boundary-aware Curriculum RL raises average pass@256 by 9.8 points over base models and 10.3 points over vanilla RLVR on Qwen, Llama, and DeepSeek families.
-
Learning the ARTS of Search for Automated Discovery
ARTS improves automated scientific discovery by using reasoning LMs with test-time training to separate hypothesis merit from execution quality in tree search, achieving 15.3% relative gains on 22 MLGym and MLEBench tasks.
-
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
LRMs show a large production-evaluation gap on the VAIR dataset with valid answers but invalid reasoning, driven by answer confirmation bias as evidenced by CoT analysis, linear probes, and causal patching.
-
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B p...
-
Self-Improving Language Models with Bidirectional Evolutionary Search
Bidirectional Evolutionary Search augments autoregressive expansion with evolutionary recombination operators and dense backward subgoal feedback to produce better candidates than standard best-of-N or tree search for...
-
Lean-GAP: A Dataset of Formalized Graduate Algebra Problems
Lean-GAP is a dataset of 430 graduate algebra problems formalized in Lean 4 from Dummit and Foote, with a described pipeline for autoformalization and verification plus analysis of challenges.
-
RMA: an Agentic System for Research-Level Mathematical Problems
RMA, a multi-agent system with structured memory and iterative feedback loops, solves 8 out of 10 research-level math problems on the new First Proof benchmark and outperforms GPT-5.2R and Aletheia according to expert...
-
Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs
OPT-BENCH trains LLMs on NP-hard optimization via quality-aware RLVR, achieving 93.1% success rate and 46.6% quality ratio on Qwen2.5-7B while outperforming GPT-4o and transferring gains to other domains.
-
Agentic Frameworks for Reasoning Tasks: An Empirical Study
An empirical evaluation of 22 agentic frameworks on BBH, GSM8K, and ARC benchmarks shows stable performance in 12 frameworks but highlights orchestration failures and weaker mathematical reasoning.
-
Automated Conjecture Resolution with Formal Verification
An AI framework combining informal reasoning and formal verification resolves an open commutative algebra problem and produces a Lean 4-checked proof with minimal human input.
-
Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
XpertBench provides 1,346 rubric-scored expert tasks showing leading LLMs achieve a maximum ~66% success rate and ~55% mean score across domains.
-
CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics
CFDLLMBench is a new benchmark suite with CFDQuery, CFDCodeBench, and FoamBench to evaluate LLMs on graduate-level CFD knowledge, numerical reasoning, and context-dependent code implementation.
-
Sakana Fugu Technical Report
Sakana Fugu trains LLM orchestrators using fine-tuning, evolutionary algorithms, and RL to build query-adaptive multi-agent scaffolds, claiming SOTA results on benchmarks including SWE-Bench Pro and GPQA-Diamond.
-
How reliable are LLMs when it comes to playing dice?
LLMs score 0.96 on standard probability exercises but 0.59 on counterintuitive ones and drop further with biased wording or misleading cues, indicating they are not genuine probabilistic reasoners.
-
Trust Region On-Policy Distillation
TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.
-
STAR-P\'olyaMath: Multi-Agent Reasoning under Persistent Meta-Strategic Supervision
STAR-PólyaMath introduces a multi-agent framework with meta-strategic supervision and state-machine orchestration that reports state-of-the-art and perfect scores on eight top math competition benchmarks.
-
Riemann-Bench: A Benchmark for Moonshot Mathematics
Riemann-Bench is a private benchmark of 25 research-level math problems on which all tested frontier AI models score below 10%.
-
Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis
A hypothesis-driven pipeline generates targeted hard math problems that drop Llama-3.3-70B-Instruct accuracy from 77% on MATH to as low as 45%.
-
Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
The paper unifies perspectives on Long CoT in reasoning LLMs by introducing a taxonomy, detailing characteristics of deep reasoning and reflection, and discussing emergence phenomena and future directions.
-
Humanity's Last Exam
Humanity's Last Exam is a new 2,500-question benchmark at the frontier of human knowledge where state-of-the-art LLMs show low accuracy.
-
MadEvolve: Evolutionary Optimization of Trading Systems with Large Language Models
MadEvolve uses LLMs for evolutionary optimization of trading strategies and reports significant backtest improvements on Bitcoin tasks including signal feature evolution and joint strategy optimization.
-
Artificial Intelligence and the Structure of Mathematics
AI agents exploring Platonic mathematical structures via proof hypergraphs may reveal the overall architecture of formal mathematics and what makes parts of it human-accessible.
-
AI for Mathematics: Progress, Challenges, and Prospects
AI for math combines task-specific architectures and general foundation models to support research and advance AI reasoning capabilities.
-
Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
An integrated survey organizing AI mathematical reasoning into informal, formal, discovery, and technique axes while cataloging benchmarks and assessing failure modes.
-
A Survey of Reinforcement Learning for Large Reasoning Models
A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.
Reference graph
Works this paper leans on
-
[1]
MSC2020 Mathematics Subject Classification System , author =
-
[2]
Training verifiers to solve math word problems, 2021 , author =
work page 2021
-
[3]
Advances in neural information processing systems , volume=
Mathematical capabilities of chatgpt , author=. Advances in neural information processing systems , volume=
-
[4]
Measuring mathematical problem solving with the math dataset , author =
-
[5]
Math Olympiad Hardness Scale (MOHS) , author =
-
[6]
MOHS was a mistake , author =
-
[7]
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models , author =
-
[8]
Are We Done with MMLU? , author =
-
[9]
Mathematical discoveries from program search with large language models , author =. Nature , publisher =
-
[10]
Minif2f: a cross-system benchmark for formal olympiad-level mathematics , author =
-
[11]
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author =
-
[12]
Arb: Advanced reasoning benchmark for large language models , author =
-
[13]
PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition , author =
-
[14]
Solving olympiad geometry without human demonstrations , author =. Nature , publisher =
-
[15]
Investigating data contamination in modern benchmarks for large language models , author =
-
[16]
Benchmark Data Contamination of Large Language Models: A Survey , author =
-
[17]
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, In- bar Mosseri, Michal Irani, and Tali Dekel
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks , author =. 2103.14749 , archiveprefix =
-
[18]
Learning to Reason with LLMs , author =
-
[19]
OpenAI o1-mini , author =
-
[20]
Introducing OpenAI o1-preview , author =
-
[21]
Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku , author =
-
[22]
Grok-2 Beta Release , author =
-
[23]
Release notes , author =
-
[24]
Claude 3.5 Sonnet Model Card Addendum , author =
-
[25]
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts , author =. 2310.02255 , archiveprefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[26]
Measuring Massive Multitask Language Understanding
Measuring Massive Multitask Language Understanding , author =. 2009.03300 , archiveprefix =
work page internal anchor Pith review Pith/arXiv arXiv 2009
-
[27]
Missing undergraduate mathematics in mathlib , author =
-
[28]
Theory of Groups of Finite Order , author =
-
[29]
Equality of orders of a set of integers modulo a prime , author =. 1912.02554 , archiveprefix =
-
[30]
Curves over Finite Fields Attaining the Hasse-Weil Upper Bound
Garcia, Arnaldo. Curves over Finite Fields Attaining the Hasse-Weil Upper Bound. European Congress of Mathematics. 2001
work page 2001
-
[31]
The Weil Conjectures for Curves , author =
-
[32]
Aryan Gulati and Brando Miranda and Eric Chen and Emily Xia and Kai Fronsdal and Bruno de Moraes Dumont and Sanmi Koyejo , booktitle=. Putnam-. 2024 , url=
work page 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.