ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
Pith reviewed 2026-07-04 00:28 UTC · model grok-4.3
The pith
Current AI research agents and LLMs score below 27 on a benchmark of 40 real scientific papers they must rediscover from data and literature.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
ResearchClawBench shows that current autonomous research agents and LLMs achieve average scores of only 21.5 and 20.7 on tasks that require rediscovering the scientific contributions of 40 real papers from supplied literature and data, with the overall frontier mean reaching just 26.5. Error patterns concentrate in mismatches between generated experimental protocols and the original ones, misalignment of evidence, and omission of the scientific core.
What carries the argument
ResearchClawBench, a set of 40 tasks each grounded in a published paper with related literature and raw data provided while the target paper is hidden, scored by expert-curated multimodal rubrics that decompose the original scientific artifacts into weighted criteria.
If this is right
- Reproducible measurement of progress toward autonomous scientific research is now possible across multiple domains.
- Development efforts should focus on improving protocol generation, evidence alignment, and identification of scientific cores.
- The benchmark framework supports evaluation of both rediscovery and novel contributions within the same rubric structure.
- Direct comparison between full agents and native LLMs is enabled under a single evaluation protocol.
Where Pith is reading between the lines
- If the low scores persist across future models, architectures that chain multiple experimental steps may be needed before reliable autonomous research emerges.
- The task format could be extended to measure cumulative research progress over sequences of related papers rather than isolated rediscoveries.
- Domains with stronger multimodal data requirements may show even larger gaps than the current average.
Load-bearing premise
The expert-curated rubrics correctly identify whether a system has recovered the scientific essence of each paper without systematic bias for or against particular agent behaviors.
What would settle it
An agent or LLM that scores above 80 on the full set of 40 tasks while independently reproducing the quantitative results and core conclusions of the hidden papers would falsify the claim that current systems remain far from reliable re-discovery.
Figures
read the original abstract
AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces ResearchClawBench, a benchmark of 40 tasks drawn from real published papers across 10 scientific domains. Each task supplies related literature and raw data while withholding the target paper; expert-curated multimodal rubrics decompose the target artifacts into weighted criteria. The authors evaluate seven autonomous research agents and seventeen LLMs under a unified protocol, reporting that the strongest agent (Claude Code) averages 21.5 and the strongest LLM (Claude-Opus-4.7) averages 20.7 on the 40-task set, with an LLM frontier mean of 26.5. They conclude that current systems remain far from reliable re-discovery, with failures concentrated in experimental protocol mismatch, evidence mismatch, and missing scientific core.
Significance. If the rubrics prove reliable and free of systematic bias, the benchmark supplies a concrete, reproducible frontier for tracking progress toward end-to-end autonomous scientific research. The provision of external task sources and the separation between agent and LLM evaluations are positive features that avoid circularity.
major comments (2)
- [Abstract / Evaluation] Abstract and Evaluation section: the headline performance numbers (21.5 for the best agent, 20.7 for the best LLM) are presented without any reported inter-rater reliability statistics, blinded re-scoring by independent experts, or sensitivity analysis on rubric weightings. Because the central claim that systems are “far from reliable re-discovery” rests on these scores faithfully reflecting scientific-core recovery, the absence of such validation directly undermines interpretability of the quantitative results.
- [Methods / Evaluation] Rubric construction paragraph (Methods/Evaluation): the description states that rubrics are “expert-curated and weighted” but supplies no protocol for how partial credit is assigned, how multimodal criteria are aggregated, or how alternative valid scientific approaches that differ from the original paper’s protocol are scored. This omission leaves open the possibility that low scores reflect rubric mismatch rather than capability gaps.
minor comments (1)
- [Results] The paper should clarify the exact number of tasks per domain and whether domain-level aggregates are reported, to allow readers to assess whether performance is uniformly low or driven by a few domains.
Simulated Author's Rebuttal
We are grateful to the referee for their insightful comments regarding the validation of our rubrics and the transparency of the evaluation protocol. We respond to each major comment in turn and describe the changes we will implement in the revised manuscript.
read point-by-point responses
-
Referee: [Abstract / Evaluation] Abstract and Evaluation section: the headline performance numbers (21.5 for the best agent, 20.7 for the best LLM) are presented without any reported inter-rater reliability statistics, blinded re-scoring by independent experts, or sensitivity analysis on rubric weightings. Because the central claim that systems are “far from reliable re-discovery” rests on these scores faithfully reflecting scientific-core recovery, the absence of such validation directly undermines interpretability of the quantitative results.
Authors: We concur that the lack of reported inter-rater reliability and sensitivity analyses limits the strength of our quantitative claims. To address this, we will perform a blinded re-scoring of a representative subset of tasks by independent domain experts and report the inter-rater reliability statistics. We will also conduct a sensitivity analysis varying the rubric weightings and include the results in the revised Evaluation section. These additions will provide stronger support for the conclusion that current systems are far from reliable re-discovery. revision: yes
-
Referee: [Methods / Evaluation] Rubric construction paragraph (Methods/Evaluation): the description states that rubrics are “expert-curated and weighted” but supplies no protocol for how partial credit is assigned, how multimodal criteria are aggregated, or how alternative valid scientific approaches that differ from the original paper’s protocol are scored. This omission leaves open the possibility that low scores reflect rubric mismatch rather than capability gaps.
Authors: We agree that a more detailed description of the rubric protocol is necessary to rule out rubric mismatch as an explanation for the observed scores. In the revised manuscript, we will elaborate on the rubric construction process, specifying the guidelines for partial credit assignment, the method for aggregating multimodal criteria, and the approach to scoring alternative valid scientific protocols that differ from the original paper. This will enhance the clarity and defensibility of the benchmark. revision: yes
Circularity Check
Empirical benchmark with external tasks; no derivations or self-referential reductions
full rationale
This paper introduces a benchmark consisting of 40 tasks grounded in real published papers, with evaluation via expert-curated rubrics and direct empirical scoring of agents and LLMs. No mathematical derivations, first-principles predictions, fitted parameters, or uniqueness theorems appear in the text. All claims rest on external task sources and observed performance numbers rather than any chain that reduces to the paper's own inputs or self-citations. The work is therefore self-contained as a measurement instrument with no circularity.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Expert-curated rubrics can be applied consistently to agent outputs to measure rediscovery of scientific contributions.
- domain assumption Hiding the target paper while providing related literature and raw data creates a fair test of autonomous capability.
Forward citations
Cited by 2 Pith papers
-
Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements
Closed-loop LM-agent auto research finds some transferable gains on molecular property prediction benchmarks via external data but shows non-transfer for model and feature edits selected on validation.
-
EurekAgent: Agent Environment Engineering is All You Need For Autonomous Scientific Discovery
EurekAgent achieves new state-of-the-art results on mathematics, kernel engineering, and machine learning tasks by engineering agent environments for autonomous scientific discovery, including a 26-circle packing resu...
Reference graph
Works this paper leans on
-
[1]
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Gpqa: A graduate-level google-proof q&a benchmark , author=. arXiv preprint arXiv:2311.12022 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[2]
Humanity's last exam , author=. arXiv preprint arXiv:2501.14249 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[3]
Advances in Neural Information Processing Systems , volume=
Scicode: A research coding benchmark curated by scientists , author=. Advances in Neural Information Processing Systems , volume=
-
[4]
International Conference on Learning Representations , volume=
Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery , author=. International Conference on Learning Representations , volume=
-
[5]
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
The ai scientist: Towards fully automated open-ended scientific discovery , author=. arXiv preprint arXiv:2408.06292 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[6]
PaperBench: Evaluating AI's Ability to Replicate AI Research
PaperBench: Evaluating AI's Ability to Replicate AI Research , author=. arXiv preprint arXiv:2504.01848 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[7]
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=
Scienceworld: Is your agent smarter than a 5th grader? , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=
work page 2022
-
[8]
MLGym : A new framework and benchmark for advancing AI research agents, 2025
Mlgym: A new framework and benchmark for advancing ai research agents , author=. arXiv preprint arXiv:2502.14499 , year=
-
[9]
Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=
Crowdsourcing multiple choice science questions , author=. Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=
-
[10]
MLAgentBench : Evaluating language agents on machine learning experimentation, 2023
Mlagentbench: Evaluating language agents on machine learning experimentation , author=. arXiv preprint arXiv:2310.03302 , year=
-
[11]
International Conference on Learning Representations , volume=
Mle-bench: Evaluating machine learning agents on machine learning engineering , author=. International Conference on Learning Representations , volume=
-
[12]
Advances in Neural Information Processing Systems , volume=
Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents , author=. Advances in Neural Information Processing Systems , volume=
-
[13]
AIRS -bench: a suite of tasks for frontier AI research science agents, 2026
AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents , author=. arXiv preprint arXiv:2602.06855 , year=
-
[14]
Advances in Neural Information Processing Systems , volume=
Mlr-bench: Evaluating ai agents on open-ended machine learning research , author=. Advances in Neural Information Processing Systems , volume=
-
[15]
Researchers’ perceptions of automating scientific research , author=. AI & SOCIETY , volume=. 2025 , publisher=
work page 2025
-
[16]
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
From generation to judgment: Opportunities and challenges of llm-as-a-judge , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
work page 2025
-
[17]
Evoscientist: Towards multi-agent evolving ai scientists for end-to-end scientific discovery , author=. arXiv preprint arXiv:2603.08127 , year=
-
[18]
Journal of Systems and Software , volume=
Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents , author=. Journal of Systems and Software , volume=. 2025 , publisher=
work page 2025
-
[19]
WebArena: A Realistic Web Environment for Building Autonomous Agents
Webarena: A realistic web environment for building autonomous agents , author=. arXiv preprint arXiv:2307.13854 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[20]
Advances in Neural Information Processing Systems , volume=
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=
-
[21]
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Scibench: Evaluating college-level scientific problem-solving abilities of large language models , author=. arXiv preprint arXiv:2307.10635 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[22]
arXiv preprint arXiv:2506.12958 , year=
Domain specific benchmarks for evaluating multimodal large language models , author=. arXiv preprint arXiv:2506.12958 , year=
-
[23]
Chembench: a cheminformatics workbench , author=. Bioinformatics , volume=. 2010 , publisher=
work page 2010
-
[24]
Advances in neural information processing systems , volume=
What can large language models do in chemistry? a comprehensive benchmark on eight tasks , author=. Advances in neural information processing systems , volume=
-
[25]
The Fourteenth International Conference on Learning Representations , year=
Earthse: A benchmark evaluating earth scientific exploration capability for large language models , author=. The Fourteenth International Conference on Learning Representations , year=
-
[26]
MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs
MSEarth: A Multimodal Scientific Dataset and Benchmark for Phenomena Uncovering in Earth Science , author=. arXiv preprint arXiv:2505.20740 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[27]
Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark , author=. arXiv preprint arXiv:2409.11363 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[28]
AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage
Autoreproduce: Automatic ai experiment reproduction with paper lineage , author=. arXiv preprint arXiv:2505.20662 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[29]
ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration
ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration , author=. arXiv preprint arXiv:2605.03042 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[30]
arXiv preprint arXiv:2512.16969 , year=
Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows , author=. arXiv preprint arXiv:2512.16969 , year=
-
[31]
arXiv preprint arXiv:2511.14366 , year=
ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning , author=. arXiv preprint arXiv:2511.14366 , year=
-
[32]
Kimi K2.5: Visual Agentic Intelligence
Kimi K2. 5: Visual Agentic Intelligence , author=. arXiv preprint arXiv:2602.02276 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[33]
arXiv preprint arXiv:2602.09132 , year=
SciDataCopilot: An Agentic Data Preparation Framework for AGI-driven Scientific Discovery , author=. arXiv preprint arXiv:2602.09132 , year=
-
[34]
MinerU: An Open-Source Solution for Precise Document Content Extraction
Mineru: An open-source solution for precise document content extraction , author=. arXiv preprint arXiv:2409.18839 , year=
work page internal anchor Pith review Pith/arXiv arXiv
- [35]
-
[36]
Towards an AI co-scientist , author=. arXiv preprint arXiv:2502.18864 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[37]
Ai-researcher: Autonomous scientific innovation , author=. arXiv preprint arXiv:2505.18705 , year=
-
[38]
InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discovery , author=. 2026 , eprint=
work page 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.