Humanity's Last Exam
Pith reviewed 2026-05-10 18:36 UTC · model grok-4.3
The pith
A benchmark of 2500 expert-level questions shows state-of-the-art LLMs still perform poorly on hard academic problems.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors assembled 2500 multi-modal questions across dozens of subjects, each carrying a known, unambiguous solution that is easily verified yet not quickly retrievable from the internet. State-of-the-art LLMs achieve low accuracy and poor calibration on this collection, in contrast to their near-ceiling performance on saturated earlier benchmarks, thereby exposing a measurable distance between present model abilities and the expert human frontier on closed-ended academic questions.
What carries the argument
The Humanity's Last Exam benchmark itself, a fixed set of 2500 expert-developed questions with verifiable answers that resist rapid retrieval.
If this is right
- The benchmark supplies a durable yardstick for measuring gains in reasoning and knowledge on genuinely difficult problems.
- Model developers gain a concrete signal that current approaches leave substantial headroom before expert-level closed-ended performance.
- Policymakers receive a clearer view of the distance between deployed systems and human-expert capability on academic tasks.
- Subsequent evaluation efforts can adopt the same global-expert, verifiable-answer design for other domains.
Where Pith is reading between the lines
- Strong performance on this set may correlate with competence on complex real-world expert workflows that mix facts and reasoning.
- The multi-modal format points to a need for joint advances in text and visual understanding at frontier difficulty.
- Repeated use of the same questions over time will let researchers quantify whether gains are genuine or partly due to data leakage.
- Similar coordinated expert efforts could produce parallel tests for fields where knowledge moves faster than static benchmarks allow.
Load-bearing premise
The questions have clear solutions that cannot be quickly found through internet searches and sit at the current edge of what human experts know.
What would settle it
An independent check that shows many of the questions can be answered correctly by standard web search or that top LLMs reach above 60 percent accuracy on the full set without additional training.
read the original abstract
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces Humanity's Last Exam (HLE), a multi-modal benchmark of 2,500 closed-ended questions (multiple-choice and short-answer) spanning mathematics, humanities, and natural sciences. Questions were developed globally by subject-matter experts and are asserted to have unambiguous, verifiable solutions that cannot be quickly answered via internet retrieval. The paper claims that existing benchmarks like MMLU are saturated (>90% LLM accuracy) and positions HLE as a frontier benchmark on which state-of-the-art LLMs exhibit low accuracy and poor calibration, revealing a substantial gap to expert human performance. The benchmark is released publicly at lastexam.ai.
Significance. If the questions are rigorously validated as non-retrievable and frontier-level, HLE would be a valuable contribution by supplying a non-saturated, broad-coverage benchmark for tracking LLM progress on expert academic tasks. The global expert curation and multi-modal design are strengths, and the public release supports reproducibility. However, the claimed significance of the LLM capability gap rests on unshown validation evidence, limiting its current impact for research and policy.
major comments (2)
- [Abstract and question development section] Abstract and the section describing question development: The assertion that 'each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval' is load-bearing for interpreting low LLM accuracy as evidence of a true capability frontier rather than training-data gaps or leakage. No concrete methodology is supplied (e.g., expert search audits, originality checks, or quantitative retrievability tests), directly addressing the central claim.
- [Results and evaluation sections] Results and evaluation sections: The abstract states that SOTA LLMs 'demonstrate low accuracy and calibration' on HLE, yet the provided information contains no quantitative results, specific model accuracies, baselines, calibration metrics, or statistical details. This absence makes it impossible to assess the magnitude or robustness of the reported gap.
minor comments (1)
- [Abstract] Abstract: Including one or two concrete accuracy figures (with model names) would make the 'low accuracy' claim more precise and informative for readers.
Simulated Author's Rebuttal
We thank the referee for their constructive feedback on our manuscript introducing Humanity's Last Exam. We address each major comment point by point below, with clear indications of planned revisions to improve clarity and rigor.
read point-by-point responses
-
Referee: [Abstract and question development section] Abstract and the section describing question development: The assertion that 'each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval' is load-bearing for interpreting low LLM accuracy as evidence of a true capability frontier rather than training-data gaps or leakage. No concrete methodology is supplied (e.g., expert search audits, originality checks, or quantitative retrievability tests), directly addressing the central claim.
Authors: We agree that explicit validation details are essential to support the non-retrievability claim and distinguish capability gaps from data leakage. The manuscript describes global expert curation and the requirement for verifiable solutions, but we acknowledge the need for greater specificity. In the revised version, we will add a dedicated subsection under question development that outlines the concrete procedures: expert-conducted web searches for each question, checks against academic databases and prior benchmarks for originality, and any quantitative thresholds or audit logs used to confirm that solutions cannot be quickly retrieved. Examples of such checks for representative questions will be included where feasible without compromising the benchmark. revision: yes
-
Referee: [Results and evaluation sections] Results and evaluation sections: The abstract states that SOTA LLMs 'demonstrate low accuracy and calibration' on HLE, yet the provided information contains no quantitative results, specific model accuracies, baselines, calibration metrics, or statistical details. This absence makes it impossible to assess the magnitude or robustness of the reported gap.
Authors: We apologize that the quantitative results were not presented with sufficient prominence or completeness in the version under review. The manuscript does contain an evaluation section reporting model performance, but we will revise it to include explicit tables with per-model accuracies (e.g., for GPT-4o, Claude 3.5 Sonnet, and others), direct comparisons to human expert baselines, calibration metrics such as expected calibration error, and basic statistical details including confidence intervals or variance across question subsets. This will enable readers to evaluate the scale and reliability of the observed gap. revision: yes
Circularity Check
No circularity: benchmark dataset release without derivations or fits
full rationale
The paper introduces Humanity's Last Exam as a new multi-modal benchmark consisting of 2,500 expert-authored questions. It contains no mathematical derivations, model equations, parameter fittings, or predictions derived from internal computations. The central claims—that questions are unambiguous, verifiable, and not quickly retrievable via internet, and that current LLMs show low accuracy—rest on the empirical construction and release of the dataset itself rather than any self-referential reduction of outputs to inputs. No self-citation chains, ansatzes, or renamings of known results are used to justify load-bearing steps. The work is therefore self-contained as a benchmark contribution with no derivation chain to inspect for circularity.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Questions have known, unambiguous, and easily verifiable solutions that cannot be quickly answered via internet retrieval.
Forward citations
Cited by 60 Pith papers
-
PCB-QA: Evaluating LLMs over the First Printed Circuit Board Design Question-Answer Dataset
PCB-QA is the first QA benchmark for LLMs on printed circuit board designs, with Gemini 3 Flash Preview reaching 93% accuracy on a JSON textual representation.
-
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?
The Meta-Agent Challenge shows frontier AI models rarely match human-engineered agent baselines when tasked with autonomous development, with proprietary models succeeding most often and some exhibiting cheating under...
-
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
Introduces BonaFide benchmark of 3,066 ground-truth labeled CoTs showing most faithfulness metrics perform near chance with biases and poor scaling to longer chains.
-
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
AI model builders mostly highlight unique benchmarks that act as flexible narrative tools for market positioning rather than standardized scientific measurements.
-
Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Soohak is a 439-problem mathematician-curated benchmark where frontier LLMs reach at most 30.4% on research math challenges and no model exceeds 50% on refusal for ill-posed problems.
-
Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Soohak is a new 439-problem mathematician-authored benchmark showing frontier LLMs reach only 30% on research math and fail to exceed 50% on refusing ill-posed questions.
-
neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing
neuralCAD-Edit benchmark shows even the best foundation model (GPT 5.2) scores 53% lower than human CAD experts in acceptance trials for multimodal-instructed 3D model edits.
-
PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
Only two of seven LLMs produce positive returns on live Polymarket data, with MiMo-V2-Flash at 17.6% CWR and Gemini-3-Flash at 6.2% CWR while the other five lose money.
-
Evaluating Large Language Models in Scientific Discovery
The SDE benchmark shows LLMs lag on scientific discovery tasks relative to general science tests, with diminishing scaling returns and shared weaknesses across models.
-
Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
CritPt benchmark shows state-of-the-art LLMs reach only 5.7% average accuracy on full-scale unpublished physics research tasks, rising to about 10% with coding tools.
-
Meta-Benchmarks for Financial-Services LLM Evaluation
A meta-benchmarking framework organizes 452 LLM benchmarks into 41 O*NET Generalized Work Activities and 38 BIAN domains, using discrimination-coverage-recency weights to scale K-factors in an Elo tournament for compa...
-
Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents
Ko-WideSearch is a new Korean breadth-search benchmark spanning 16 categories and three difficulty tiers that evaluates web agents on full set membership plus per-item attributes, showing consistent gaps between set r...
-
Mat-Pref: Verifiable-Reward Training Improves Compositional Reasoning in Inorganic Materials
Mat-Pref benchmark shows GRPO after SFT lets Qwen3-8B reach 65-72% on compositional materials reasoning tasks, exceeding zero-shot 235B models on held-out structure families and cross-property transfer.
-
Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients
ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...
-
Representing Time Series as Structured Programs for LLM Reasoning
T2SP converts time series into structured programs for trends, periods, and events, enabling off-the-shelf LLMs to perform better on editing, captioning, and QA tasks than raw string inputs.
-
Can AI Agents Synthesize Scientific Conclusions?
A new benchmark and clean-room harness show frontier AI agents reach only 0.337 factual F1 when synthesizing conclusions from scientific evidence.
-
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
ResearchClawBench supplies 40 grounded tasks and expert rubrics to measure autonomous research agents, with the strongest systems scoring only 21.5 and 20.7 on average.
-
LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
LiveBrowseComp shows search agents rely on intrinsic knowledge on standard benchmarks, with scores dropping 25-40 points and closed-book accuracy below 2% on questions about facts from the prior 90 days.
-
RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data
ATLAS traces RLVR data to 20 atomic sources, most datasets are variants, and DAPO++ curated with SCA improves RLVR performance while Q predicts training effectiveness.
-
Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
The paper introduces a multi-turn interactive benchmark using 474 executable games to evaluate LLMs on evidence acquisition, belief updating, contextual robustness, and metacognitive adaptation, revealing large perfor...
-
Trust but Verify: Prover-Verifier Deliberation for Selective LLM Prediction
Prover-verifier deliberation yields a high-confidence subset of LLM answers with ~30pp higher precision than the complement on GPQA Diamond by using defender-challenger dialogues.
-
IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents
IdleSpec improves LLM agent accuracy by generating and aggregating speculative plans during idle time between tool calls and observations using complementary drafting strategies.
-
Evaluating Cognitive Age Alignment in Interactive AI Agents
The paper presents ChildAgentEval as the first psychometrically grounded benchmark comparing MLLM-based agents' reasoning performance to age-specific human cognitive stages.
-
Argus: Evidence Assembly for Scalable Deep Research Agents
Argus coordinates a Navigator and multiple Searchers via an evidence graph to assemble complete, source-traced answers, yielding benchmark gains up to 12.7 points with 8 parallel agents and 86.2 on BrowseComp with 64 agents.
-
TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
TRIAGE evaluates LLMs on prospective metacognitive control by requiring a single plan for task selection, sequencing, and token allocation under a calibrated budget, revealing substantial gaps in current models across...
-
Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
Formal Conjectures is a Lean 4 benchmark containing 2615 formalized problems with 1029 open conjectures, designed to evaluate automated mathematical reasoning and proof discovery.
-
AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents
AssayBench is a new gene-ranking benchmark for phenotypic CRISPR screens that shows zero-shot generalist LLMs outperform both biology-specific LLMs and trainable baselines on adjusted nDCG.
-
Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents
A new image-bank harness and closed-loop on-policy data evolution method raises multimodal agent performance on visual search benchmarks from 24.9% to 39.0% for an 8B model and from 30.6% to 41.5% for a 30B model.
-
MaD Physics: Evaluating information seeking under constraints in physical environments
MaD Physics is a new benchmark for evaluating AI agents on constrained information-seeking, model inference, and prediction in three physical environments with altered laws to avoid knowledge contamination.
-
LLM-Guided Monte Carlo Tree Search over Knowledge Graphs: Composing Mechanistic Explanations for Drug-Disease Pairs
TESSERA combines LLMs as local policy and evaluator with MCTS on knowledge graphs to compose mechanistic drug-disease explanations.
-
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
DiagnosticIQ benchmark shows frontier LLMs perform similarly on standard rule-to-action tasks but lose substantial accuracy under distractor expansion and condition inversion, pointing to calibration as the key deploy...
-
AcademiClaw: When Students Set Challenges for AI Agents
AcademiClaw is a new benchmark of 80 student-sourced academic tasks where the best frontier AI agents achieve only a 55% pass rate.
-
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
The Reward Hacking Benchmark shows RL post-training raises exploit rates in tool-using LLM agents from 0.6% to 13.9%, with environmental hardening cutting exploits by 87.7% relative without lowering task success.
-
Super Apriel: One Checkpoint, Many Speeds
A single 15B supernet checkpoint supports runtime switching between attention mixer placements for multiple decode speed presets while retaining 77-96% quality relative to the teacher model.
-
Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints
Stargazer benchmarks AI agents on physics-constrained model fitting for astrophysical data, revealing that agents achieve statistical fits but often fail to recover correct physical parameters.
-
Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints
Stargazer benchmark shows frontier AI agents achieve statistical fits to radial velocity data but frequently fail to recover correct physical planetary system parameters.
-
PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models
PokeGym is a new benchmark that tests VLMs on long-horizon tasks in a complex 3D game using only visual observations, identifying deadlock recovery as the primary failure mode.
-
GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces
GeoBrowse is a two-level geolocation benchmark combining visual cue composition with knowledge-intensive multi-hop queries, paired with the GATE agent workflow that outperforms no-tool, search-only, and image-only baselines.
-
The limits of bio-molecular modeling with large language models : a cross-scale evaluation
LLMs perform adequately on bio-molecular classification tasks but remain weak on regression, with hybrid architectures outperforming others on long sequences and fine-tuning hurting generalization.
-
Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests
Large-scale log study of 14M+ agentic searches finds short sessions, intent-specific repetition patterns, and that 54% of new query terms trace to prior retrieved evidence.
-
MemEvolve: Meta-Evolution of Agent Memory Systems
MemEvolve jointly evolves agent experiential knowledge and memory architectures via a modular codebase, delivering up to 17% gains on agent benchmarks with cross-task and cross-model generalization.
-
Scaling Latent Reasoning via Looped Language Models
Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.
-
Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
Evaluations of 53 LLMs on 14 basic math tasks show reasoning models use ~18x more tokens with sometimes lower accuracy, non-monotonic gains from extended budgets, and sharp performance drops under token constraints.
-
EduArt: An educational-level benchmark for evaluating art history knowledge in large language models
EduArt is a new benchmark of 871 educational questions that reveals multimodal LLMs perform near ceiling on multiple-choice art history items but drop sharply on open completion and error identification tasks.
-
Subliminal Clocks: Latent Time Modelling in Diffusion Language Models
DLMs encode a decodable latent timestep signal in residual activations that can be steered to predictably change model confidence and entropy.
-
Theoria: Rewrite-Acceptability Verification over Informal Reasoning States
Theoria rewrites solutions into auditable typed state transitions with justifications, certifying 105 of 185 HLE problems at 91.4% precision and outperforming holistic judges on adversarial poisoned proofs by catching...
-
ECHO: Prune to act, trace to learn with selective turn memory in agentic RL
ECHO is a selective turn-memory framework for agentic RL that compresses turns into indexed records, selects them for bounded contexts, and uses source indices to assign outcome credit to supporting evidence, reaching...
-
ACE: Pluggable Adaptive Context Elasticizer across Agents
ACE is a pluggable module that elastically orchestrates historical agent steps as raw, abstract, or dropped to maintain compact yet recoverable context for LLM agents handling long trajectories.
-
You Don't Need to Run Every Eval
The benchmark score matrix of 84 models on 133 tasks is approximately rank-2; BenchPress recovers held-out scores to within 4.6 points and identifies 5-benchmark subsets that predict the full scorecard to within 3.93-...
-
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
MINCE shrinks IFEVAL by 54%, MMLU by 89%, and GSM8K by 70% via few-model Monte Carlo calibration while keeping maximum drift at or below 2.62 percentage points.
-
How Inference Compute Shapes Frontier LLM Evaluation
Frontier LLM scores on challenging benchmarks rise substantially with increased inference compute, showing that fixed-budget evaluations increasingly understate capability.
-
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
SEAGym turns existing benchmarks into multi-view evaluation sources for measuring reusable improvements in LLM agent harnesses, revealing complementary signals missed by single-curve or isolated-task tests.
-
ICBCBench: An Industry Consortium Benchmark for Financial Deep Research
ICBCBench is a new consortium-built benchmark that jointly measures retrieval-reasoning accuracy and end-to-end report quality for deep research agents in finance.
-
MiniMax Sparse Attention
MiniMax Sparse Attention is a GQA-based block-sparse attention mechanism that selects top-k blocks independently per group and delivers 28.4x per-token compute reduction at 1M context with on-par performance plus 14.2...
-
The Illusion of Multi-Agent Advantage
Automatically generated multi-agent systems underperform CoT-SC on benchmarks and a new diagnostic dataset, exposing architectural bloat that fails to deliver functional utility.
-
APPO: Agentic Procedural Policy Optimization
APPO refines branching and credit assignment in agentic RL via a Branching Score and procedure-level scaling, improving baselines by nearly 4 points on 13 benchmarks.
-
Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation
Deep research agents exhibit widespread search-time contamination on six public benchmarks, with three defined leakage types inflating performance by up to 4%.
-
Quantifying Faithful Confidence Expression in Large Reasoning Models
A new framework quantifies faithful confidence expression in large reasoning models by comparing linguistic decisiveness to token probabilities, hidden states, and response consistency, revealing it as a persistent challenge.
-
Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses
Harness-1 uses a state-externalizing harness for RL-trained search agents and reports 0.730 average curated recall, outperforming the next open subagent by 11.4 points.
-
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
ResearchClawBench is a new benchmark that evaluates autonomous AI research agents on 40 tasks grounded in published papers using expert rubrics, finding that top systems score only 20-26 out of 100.
Reference graph
Works this paper leans on
-
[1]
C. Alberti, K. Lee, and M. Collins. A bert baseline for the natural questions, 2019. URL https: //arxiv.org/abs/1901.08634
-
[2]
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y . Gal, and X. Davies. Agentharm: A benchmark for measuring harmfulness of llm agents, 2024. URLhttps://arxiv.org/abs/2410.09024
work page internal anchor Pith review arXiv 2024
-
[3]
The claude 3 model family: Opus, sonnet, haiku, 2024
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://api. semanticscholar.org/CorpusID:268232499
work page 2024
-
[4]
Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 son- net, 2024
Anthropic. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 son- net, 2024. URL https://assets.anthropic.com/m/1cd9d098ac3e6467/original/ Claude-3-Model-Card-October-Addendum.pdf
work page 2024
-
[5]
Responsible scaling policy updates, 2024
Anthropic. Responsible scaling policy updates, 2024. URL https://www.anthropic.com/ rsp-updates
work page 2024
-
[6]
R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URLhttps://arxiv.org/abs/2505.08775
work page internal anchor Pith review arXiv 2025
- [7]
-
[8]
Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. Mc- Candlish, C. Olah, B. Mann, and J. Kaplan...
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[9]
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, and T. Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URLhttps://arxiv.org/abs/1611.09268
work page internal anchor Pith review arXiv 2018
-
[10]
Purple Llama CyberSecEval: A secure coding benchmark for language models,
M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Ascher- mann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y . Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V . V ontimitta, S. Whitman, and J. Saxe. Purple llama cyberseceval: A secure coding benchmark for language models, 2023. URLhttps://arxiv...
-
[11]
J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. M ˛ adry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2024. URLhttps://arxiv.org/abs/2410.07095
work page Pith review arXiv 2024
-
[12]
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[13]
Arc prize 2024: Technical report.arXiv preprint arXiv:2412.04604, 2024
F. Chollet, M. Knoop, G. Kamradt, and B. Landers. Arc prize 2024: Technical report, 2024. URL https://arxiv.org/abs/2412.04604
-
[14]
Training Verifiers to Solve Math Word Problems
K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[15]
Deepseek-v3 technical report, 2024
DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://github.com/deepseek-ai/ DeepSeek-V3/blob/main/DeepSeek_V3.pdf
work page 2024
-
[16]
D. Dua, Y . Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https://arxiv.org/abs/1903. 00161. 10
work page 2019
-
[17]
A. Dubey et al. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[18]
B. Gao, F. Song, Z. Yang, Z. Cai, Y . Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y . Zhang, X. Ren, T. Liu, and B. Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024. URL https://arxiv.org/abs/2410.07985
work page internal anchor Pith review arXiv 2024
-
[19]
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. de Oliveira Santos, O. Järviniemi, M. Barnett, R. Sandler, J. Sevilla, Q. Ren, E. Pratt, L. Levine, G. Barkley, N. Stewart, B. Grechuk, T. Grechuk, and S. V . Enugandla. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai,...
work page internal anchor Pith review arXiv 2024
-
[20]
C. He, R. Luo, Y . Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y . Huang, Y . Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024. URLhttps://arxiv.org/abs/2402.14008
work page internal anchor Pith review arXiv 2024
-
[21]
Measuring Coding Challenge Competence With APPS
D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021. URL https: //arxiv.org/abs/2105.09938
work page internal anchor Pith review arXiv 2021
-
[22]
Measuring Massive Multitask Language Understanding
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021. URLhttps://arxiv.org/abs/2009.03300
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[23]
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103. 03874
work page 2021
-
[24]
D. Hendrycks, A. Zou, M. Mazeika, L. Tang, B. Li, D. Song, and J. Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures, 2022. URLhttps://arxiv.org/abs/2112.05135
-
[25]
A. Hosseini, A. Sordoni, D. Toyama, A. Courville, and R. Agarwal. Not all llm reasoners are created equal,
- [26]
-
[27]
A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. W. andMadhu Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, R. Sims, Z. Zhang, S. Goldshtein, and D. Das. Facts leaderboard. https://kaggle.com/facts-leaderb...
work page 2024
-
[28]
C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URLhttps://arxiv.org/abs/2310.06770
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[29]
Dynabench: Rethinking benchmarking in NLP
D. Kiela, M. Bartolo, Y . Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in nlp, 2021. URLhttps://arxiv.org/abs/2104.14337
-
[30]
P. Kumar, E. Lau, S. Vijayakumar, T. Trinh, S. R. Team, E. Chang, V . Robinson, S. Hendryx, S. Zhou, M. Fredrikson, S. Yue, and Z. Wang. Refusal-trained llms are easily jailbroken as browser agents, 2024. URLhttps://arxiv.org/abs/2410.13886
-
[31]
J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques. Lab-bench: Measuring capabilities of language models for biology research,
-
[32]
URLhttps://arxiv.org/abs/2407.10362
work page internal anchor Pith review arXiv
-
[33]
N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-V oss, C. B. Breuer, S. Marks, O. Patel, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Lin, A. A. Hunt,...
work page internal anchor Pith review arXiv 2024
-
[34]
P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255
work page internal anchor Pith review arXiv 2024
- [35]
- [36]
-
[37]
OpenAI. Openai o1 system card, 2024. URLhttps://cdn.openai.com/o1-system-card-20240917. pdf
work page 2024
-
[38]
Openai and los alamos national laboratory announce bio- science research partnership, 2024
OpenAI. Openai and los alamos national laboratory announce bio- science research partnership, 2024. URL https://openai.com/index/ openai-and-los-alamos-national-laboratory-work-together/
work page 2024
-
[39]
Introducing swe-bench verified, 2024
OpenAI. Introducing swe-bench verified, 2024. URL https://openai.com/index/ introducing-swe-bench-verified/
work page 2024
-
[40]
OpenAI et al. Gpt-4 technical report, 2024. URLhttps://arxiv.org/abs/2303.08774
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[41]
S. Ott, A. Barbosa-Silva, K. Blagec, J. Brauner, and M. Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence.Nature Communications, 13(1):6793, 2022
work page 2022
- [42]
-
[43]
Discovering Language Model Behaviors with Model-Written Evaluations
E. Perez, S. Ringer, K. Lukoši ¯ut˙e, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. Mueller, J. Hyun, J. Landau, K. Ndousse, L. Goldberg, L. ...
work page internal anchor Pith review arXiv 2022
-
[44]
M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V . Krakovna, D. Lindner, M. Rahtz, Y . Assael, S. Hodkinson, H. Howard, T. Lieberum, R. Kumar, M. A. Raad, A. Webson, L. Ho, S. Lin, S. Farquhar, M. Hutter, G. Deletang, A. Ruoss, S. El-Sayed, S. Brown, A. Dragan, R. Shah, A. Dafoe, and T. Shevlane. Evaluating frontier models for dangerous capabil...
work page 2024
-
[45]
SQuAD: 100,000+ Questions for Machine Comprehension of Text
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URLhttps://arxiv.org/abs/1606.05250
work page internal anchor Pith review arXiv 2016
-
[46]
Know What You Don't Know: Unanswerable Questions for SQuAD
P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for squad, 2018. URLhttps://arxiv.org/abs/1806.03822
work page Pith review arXiv 2018
-
[47]
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URLhttps://arxiv.org/abs/2311.12022
work page internal anchor Pith review arXiv 2023
-
[48]
K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023
work page 2023
-
[49]
M. Skarlinski, J. Laurent, A. Bou, and A. White. About 30% ofHumanity’s Last Exam chemistry/biology answers are likely wrong, July 2025. URL https://www.futurehouse.org/ research-announcements/hle-exam
work page 2025
-
[50]
V . K. Srinivasan, Z. Dong, B. Zhu, B. Yu, H. Mao, D. Mosk-Aoyama, K. Keutzer, J. Jiao, and J. Zhang. Nexusraven: A commercially-permissive language model for function calling. InNeurIPS 2023 F oun- dation Models for Decision Making Workshop, 2023. URL https://openreview.net/forum?id= 5lcPe6DqfI
work page 2023
-
[51]
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slone, A. Rahane, A. S. Iyer, A. Andreassen, A. Madotto, A. S...
work page internal anchor Pith review arXiv 2023
- [52]
-
[53]
G. Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,
-
[54]
URLhttps://arxiv.org/abs/2403.05530
work page internal anchor Pith review Pith/arXiv arXiv
-
[55]
arXiv preprint arXiv:2407.11214 , year =
G. Tsoukalas, J. Lee, J. Jennings, J. Xin, M. Ding, M. Jennings, A. Thakur, and S. Chaudhuri. Putnambench: Evaluating neural theorem-provers on the putnam mathematical competition, 2024. URLhttps://arxiv. org/abs/2407.11214
-
[56]
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https://arxiv.org/abs/1804. 07461
work page 2019
-
[57]
A. Wang, Y . Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2020. URL https://arxiv.org/abs/1905.00537
work page internal anchor Pith review arXiv 2020
-
[58]
Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (published at neurips 2024 track datasets and benchmarks), 2024. URL https://arxiv.org/abs/2406.01574
work page internal anchor Pith review arXiv 2024
-
[59]
J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models, 2024. URLhttps://arxiv.org/abs/2411.04368
work page internal anchor Pith review arXiv 2024
-
[60]
H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts, 2024. URLhttps://a...
-
[61]
xAI. Grok-2 beta release, 2024. URLhttps://x.ai/blog/grok-2
work page 2024
-
[62]
F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function call- ing leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_ leaderboard.html, 2024
work page 2024
-
[63]
Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/ 1809.09600
work page internal anchor Pith review arXiv 2018
-
[64]
S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URLhttps://arxiv.org/abs/2406.12045
work page internal anchor Pith review arXiv 2024
-
[65]
A. K. Zhang, N. Perry, R. Dulepet, J. Ji, J. W. Lin, E. Jones, C. Menders, G. Hussein, S. Liu, D. Jasper, P. Peetathawatchai, A. Glenn, V . Sivashankar, D. Zamoshchin, L. Glikbarg, D. Askaryar, M. Yang, T. Zhang, R. Alluri, N. Tran, R. Sangpisit, P. Yiorkadjis, K. Osele, G. Raghupathi, D. Boneh, D. E. Ho, and P. Liang. Cybench: A framework for evaluating ...
-
[66]
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
W. Zhong, R. Cui, Y . Guo, Y . Liang, S. Lu, Y . Wang, A. Saied, W. Chen, and N. Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https://arxiv.org/abs/ 2304.06364. 13 A Authors We offered optional co-authorship to all question submitters with an accepted question in HUMANITY’SLAST EXAM(including both public and private...
work page internal anchor Pith review arXiv 2023
-
[67]
Independent Researcher
-
[68]
University of California, Berkeley
-
[69]
Massachusetts Institute of Technology
-
[70]
University of Cambridge
-
[71]
University of Oxford
-
[72]
Princeton University
-
[73]
Carnegie Mellon University
-
[74]
University of Chicago
-
[75]
University of Michigan
-
[76]
École Polytechnique Fédérale de Lausanne
-
[77]
University of Toronto
-
[78]
University of Illinois Urbana-Champaign
-
[79]
Washington University
-
[80]
University of Wisconsin-Madison
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.