pith. sign in

arxiv: 2606.07591 · v4 · pith:IIHFIAMAnew · submitted 2026-05-28 · 💻 cs.LG · cs.AI· cs.CL

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

Pith reviewed 2026-07-04 00:28 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords autonomous scientific researchbenchmarkAI agentslarge language modelsre-discoveryscientific discoveryevaluation
0
0 comments X

The pith

Current AI research agents and LLMs score below 27 on a benchmark of 40 real scientific papers they must rediscover from data and literature.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces ResearchClawBench to test whether autonomous systems can perform end-to-end scientific research by recovering the results and contributions of existing published papers. Each of the 40 tasks supplies background literature and raw data but conceals the target paper, and expert rubrics score how well a system matches the original scientific artifacts. Seven autonomous research agents and seventeen native LLMs were run under one protocol. The strongest agent reached an average of 21.5 and the strongest LLM 20.7, with a frontier mean of 26.5. These numbers establish a concrete performance floor and show that failures concentrate in protocol design, evidence handling, and capture of the scientific core.

Core claim

ResearchClawBench shows that current autonomous research agents and LLMs achieve average scores of only 21.5 and 20.7 on tasks that require rediscovering the scientific contributions of 40 real papers from supplied literature and data, with the overall frontier mean reaching just 26.5. Error patterns concentrate in mismatches between generated experimental protocols and the original ones, misalignment of evidence, and omission of the scientific core.

What carries the argument

ResearchClawBench, a set of 40 tasks each grounded in a published paper with related literature and raw data provided while the target paper is hidden, scored by expert-curated multimodal rubrics that decompose the original scientific artifacts into weighted criteria.

If this is right

  • Reproducible measurement of progress toward autonomous scientific research is now possible across multiple domains.
  • Development efforts should focus on improving protocol generation, evidence alignment, and identification of scientific cores.
  • The benchmark framework supports evaluation of both rediscovery and novel contributions within the same rubric structure.
  • Direct comparison between full agents and native LLMs is enabled under a single evaluation protocol.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the low scores persist across future models, architectures that chain multiple experimental steps may be needed before reliable autonomous research emerges.
  • The task format could be extended to measure cumulative research progress over sequences of related papers rather than isolated rediscoveries.
  • Domains with stronger multimodal data requirements may show even larger gaps than the current average.

Load-bearing premise

The expert-curated rubrics correctly identify whether a system has recovered the scientific essence of each paper without systematic bias for or against particular agent behaviors.

What would settle it

An agent or LLM that scores above 80 on the full set of 40 tasks while independently reproducing the quantitative results and core conclusions of the hidden papers would falsify the claim that current systems remain far from reliable re-discovery.

Figures

Figures reproduced from arXiv: 2606.07591 by Bin Wang, Bo Zhang, Chaofan Hu, Chunfeng Song, Dongzhan Zhou, Fangchen Yu, Fenghua Ling, Guangtao Zhai, Haoxiang Yin, Haoxuan Li, Haoyu Zhou, Hengjian Gao, Jiamin Wu, Koutian Wu, Kun Li, Lei Bai, Lixue Cheng, Lu Mi, Mao Su, Mianxin Liu, Peng Ye, Qi Li, Qinglong Cao, Ruizhe Chen, Shengdu Chai, Sheng Xu, Shengyuan Xu, Shixiang Tang, Shuo Li, Siqi Sun, Tianfan Fu, Tianlin Ye, Wanghan Xu, Wanli Ouyang, Weijie Ma, Wenlong Zhang, Xiangyu Zhao, Xingjian Guo, Xinyu Gu, Xue Yang, Xuming He, Xuxuan Xie, Yifan Zhou, Yiheng Wang, Yixin Chen, Yuhao Zhou, Yuqiang Li, Zhangrui Zhao, Zhenfei Yin, Zhiwang Zhou, Zijie Guo.

Figure 1
Figure 1. Figure 1: Overview of ResearchClawBench. (a) ResearchClawBench spans 10 domains and 40 end-to-end tasks, covering diverse scientific questions and data modalities. (b) Overall scores of agents and LLMs; the 50-point line marks target-paper-level re-discovery, and scores above it indicate the discovery regime. arXiv:2606.07591v3 [cs.LG] 17 Jun 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Pump-off, pump-on at thetap = 0 deg, and pump-induced difference maps. The cyan marker denotes the processed replica target region used for raw-window validation. 3. Methods 3.1 Replica-band energy test For each processed replica entry with order n = +-1, I computed an inferred parent energy 𝐸𝑝𝑎𝑟𝑒𝑛𝑡 = 𝐸𝑟𝑒𝑝𝑙𝑖𝑐𝑎 − 𝑛ℏ𝜔, using the pump energy stored in the processed feature file, pump_energy = 0.248 eV. A Floq… view at source ↗
Figure 1
Figure 1. Figure 1: Pump-off, pump-on at thetap = 0 deg, and pump-induced difference maps. The cyan marker denotes the processed replica target region used for raw-window validation. 3. Methods 3.1 Replica-band energy test For each processed replica entry with order n = +-1, I computed an inferred parent energy 𝐸𝑝𝑎𝑟𝑒𝑛𝑡 = 𝐸𝑟𝑒𝑝𝑙𝑖𝑐𝑎 − 𝑛ℏ𝜔, using the pump energy stored in the processed feature file, pump_energy = 0.248 eV. A Floq… view at source ↗
Figure 2
Figure 2. Figure 2: Left: extracted Dirac-cone dispersion and identified replica features. Right: order-averaged replica-parent separations compared with the 0.248 eV pump photon energy. 4.2 Raw pump-induced signal near the replica region The raw HDF5 maps support the presence of a pump-induced feature near the processed target re￾gion. Averaging pump-on minus pump-off intensity in the target window gives positive values for … view at source ↗
Figure 2
Figure 2. Figure 2: Left: extracted Dirac-cone dispersion and identified replica features. Right: order-averaged replica-parent separations compared with the 0.248 eV pump photon energy. 4.2 Raw pump-induced signal near the replica region The raw HDF5 maps support the presence of a pump-induced feature near the processed target re￾gion. Averaging pump-on minus pump-off intensity in the target window gives positive values for … view at source ↗
Figure 3
Figure 3. Figure 3: Pump-induced difference maps for thetap = 0 deg and 90 deg, an energy distribution curve through the target momentum, and comparison of raw-window signal with mean-subtracted processed polarization intensity. 4.3 Polarization dependence and Volkov final-state interpretation The polarization CSV shows a weak but structured intensity variation. The fitted pi-periodic model gives: • model: I(theta)=c+a cos(2t… view at source ↗
Figure 3
Figure 3. Figure 3: Pump-induced difference maps for thetap = 0 deg and 90 deg, an energy distribution curve through the target momentum, and comparison of raw-window signal with mean-subtracted processed polarization intensity. 4.3 Polarization dependence and Volkov final-state interpretation The polarization CSV shows a weak but structured intensity variation. The fitted pi-periodic model gives: • model: I(theta)=c+a cos(2t… view at source ↗
Figure 4
Figure 4. Figure 4: Replica intensity versus pump polarization angle with a pi-periodic fit, shown both on linear and polar axes. 5. Validation and traceability 5.1 Directly verified from workspace data • The raw HDF5 axes, spectra shapes, and intensity ranges are summarized in outputs/data_ overview.json. • The processed replicas are photon-spaced from their inferred parent energy by 0.248 eV for both first-order sidebands; … view at source ↗
Figure 4
Figure 4. Figure 4: Replica intensity versus pump polarization angle with a pi-periodic fit, shown both on linear and polar axes. 5. Validation and traceability 5.1 Directly verified from workspace data • The raw HDF5 axes, spectra shapes, and intensity ranges are summarized in outputs/data_ overview.json. • The processed replicas are photon-spaced from their inferred parent energy by 0.248 eV for both first-order sidebands; … view at source ↗
read the original abstract

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces ResearchClawBench, a benchmark of 40 tasks drawn from real published papers across 10 scientific domains. Each task supplies related literature and raw data while withholding the target paper; expert-curated multimodal rubrics decompose the target artifacts into weighted criteria. The authors evaluate seven autonomous research agents and seventeen LLMs under a unified protocol, reporting that the strongest agent (Claude Code) averages 21.5 and the strongest LLM (Claude-Opus-4.7) averages 20.7 on the 40-task set, with an LLM frontier mean of 26.5. They conclude that current systems remain far from reliable re-discovery, with failures concentrated in experimental protocol mismatch, evidence mismatch, and missing scientific core.

Significance. If the rubrics prove reliable and free of systematic bias, the benchmark supplies a concrete, reproducible frontier for tracking progress toward end-to-end autonomous scientific research. The provision of external task sources and the separation between agent and LLM evaluations are positive features that avoid circularity.

major comments (2)
  1. [Abstract / Evaluation] Abstract and Evaluation section: the headline performance numbers (21.5 for the best agent, 20.7 for the best LLM) are presented without any reported inter-rater reliability statistics, blinded re-scoring by independent experts, or sensitivity analysis on rubric weightings. Because the central claim that systems are “far from reliable re-discovery” rests on these scores faithfully reflecting scientific-core recovery, the absence of such validation directly undermines interpretability of the quantitative results.
  2. [Methods / Evaluation] Rubric construction paragraph (Methods/Evaluation): the description states that rubrics are “expert-curated and weighted” but supplies no protocol for how partial credit is assigned, how multimodal criteria are aggregated, or how alternative valid scientific approaches that differ from the original paper’s protocol are scored. This omission leaves open the possibility that low scores reflect rubric mismatch rather than capability gaps.
minor comments (1)
  1. [Results] The paper should clarify the exact number of tasks per domain and whether domain-level aggregates are reported, to allow readers to assess whether performance is uniformly low or driven by a few domains.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We are grateful to the referee for their insightful comments regarding the validation of our rubrics and the transparency of the evaluation protocol. We respond to each major comment in turn and describe the changes we will implement in the revised manuscript.

read point-by-point responses
  1. Referee: [Abstract / Evaluation] Abstract and Evaluation section: the headline performance numbers (21.5 for the best agent, 20.7 for the best LLM) are presented without any reported inter-rater reliability statistics, blinded re-scoring by independent experts, or sensitivity analysis on rubric weightings. Because the central claim that systems are “far from reliable re-discovery” rests on these scores faithfully reflecting scientific-core recovery, the absence of such validation directly undermines interpretability of the quantitative results.

    Authors: We concur that the lack of reported inter-rater reliability and sensitivity analyses limits the strength of our quantitative claims. To address this, we will perform a blinded re-scoring of a representative subset of tasks by independent domain experts and report the inter-rater reliability statistics. We will also conduct a sensitivity analysis varying the rubric weightings and include the results in the revised Evaluation section. These additions will provide stronger support for the conclusion that current systems are far from reliable re-discovery. revision: yes

  2. Referee: [Methods / Evaluation] Rubric construction paragraph (Methods/Evaluation): the description states that rubrics are “expert-curated and weighted” but supplies no protocol for how partial credit is assigned, how multimodal criteria are aggregated, or how alternative valid scientific approaches that differ from the original paper’s protocol are scored. This omission leaves open the possibility that low scores reflect rubric mismatch rather than capability gaps.

    Authors: We agree that a more detailed description of the rubric protocol is necessary to rule out rubric mismatch as an explanation for the observed scores. In the revised manuscript, we will elaborate on the rubric construction process, specifying the guidelines for partial credit assignment, the method for aggregating multimodal criteria, and the approach to scoring alternative valid scientific protocols that differ from the original paper. This will enhance the clarity and defensibility of the benchmark. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark with external tasks; no derivations or self-referential reductions

full rationale

This paper introduces a benchmark consisting of 40 tasks grounded in real published papers, with evaluation via expert-curated rubrics and direct empirical scoring of agents and LLMs. No mathematical derivations, first-principles predictions, fitted parameters, or uniqueness theorems appear in the text. All claims rest on external task sources and observed performance numbers rather than any chain that reduces to the paper's own inputs or self-citations. The work is therefore self-contained as a measurement instrument with no circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the assumption that the 40 selected papers and their expert rubrics form a representative and stable test of autonomous research capability. No free parameters or invented entities are described in the abstract.

axioms (2)
  • domain assumption Expert-curated rubrics can be applied consistently to agent outputs to measure rediscovery of scientific contributions.
    Invoked when the abstract states that rubrics enable evaluation of target-paper-level re-discovery.
  • domain assumption Hiding the target paper while providing related literature and raw data creates a fair test of autonomous capability.
    Stated in the task construction description.

pith-pipeline@v0.9.1-grok · 5922 in / 1362 out tokens · 16979 ms · 2026-07-04T00:28:23.381557+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements

    cs.AI 2026-06 unverdicted novelty 6.0

    Closed-loop LM-agent auto research finds some transferable gains on molecular property prediction benchmarks via external data but shows non-transfer for model and feature edits selected on validation.

  2. EurekAgent: Agent Environment Engineering is All You Need For Autonomous Scientific Discovery

    cs.AI 2026-06 unverdicted novelty 5.0

    EurekAgent achieves new state-of-the-art results on mathematics, kernel engineering, and machine learning tasks by engineering agent environments for autonomous scientific discovery, including a 26-circle packing resu...

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · cited by 2 Pith papers · 13 internal anchors

  1. [1]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    Gpqa: A graduate-level google-proof q&a benchmark , author=. arXiv preprint arXiv:2311.12022 , year=

  2. [2]

    Humanity's Last Exam

    Humanity's last exam , author=. arXiv preprint arXiv:2501.14249 , year=

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Scicode: A research coding benchmark curated by scientists , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    International Conference on Learning Representations , volume=

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery , author=. International Conference on Learning Representations , volume=

  5. [5]

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    The ai scientist: Towards fully automated open-ended scientific discovery , author=. arXiv preprint arXiv:2408.06292 , year=

  6. [6]

    PaperBench: Evaluating AI's Ability to Replicate AI Research

    PaperBench: Evaluating AI's Ability to Replicate AI Research , author=. arXiv preprint arXiv:2504.01848 , year=

  7. [7]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    Scienceworld: Is your agent smarter than a 5th grader? , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

  8. [8]

    MLGym : A new framework and benchmark for advancing AI research agents, 2025

    Mlgym: A new framework and benchmark for advancing ai research agents , author=. arXiv preprint arXiv:2502.14499 , year=

  9. [9]

    Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=

    Crowdsourcing multiple choice science questions , author=. Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=

  10. [10]

    MLAgentBench : Evaluating language agents on machine learning experimentation, 2023

    Mlagentbench: Evaluating language agents on machine learning experimentation , author=. arXiv preprint arXiv:2310.03302 , year=

  11. [11]

    International Conference on Learning Representations , volume=

    Mle-bench: Evaluating machine learning agents on machine learning engineering , author=. International Conference on Learning Representations , volume=

  12. [12]

    Advances in Neural Information Processing Systems , volume=

    Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents , author=. Advances in Neural Information Processing Systems , volume=

  13. [13]

    AIRS -bench: a suite of tasks for frontier AI research science agents, 2026

    AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents , author=. arXiv preprint arXiv:2602.06855 , year=

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Mlr-bench: Evaluating ai agents on open-ended machine learning research , author=. Advances in Neural Information Processing Systems , volume=

  15. [15]

    AI & SOCIETY , volume=

    Researchers’ perceptions of automating scientific research , author=. AI & SOCIETY , volume=. 2025 , publisher=

  16. [16]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    From generation to judgment: Opportunities and challenges of llm-as-a-judge , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

  17. [17]

    EvoScientist : Towards multi-agent evolving AI scientists for end-to-end scientific discovery, 2026 b

    Evoscientist: Towards multi-agent evolving ai scientists for end-to-end scientific discovery , author=. arXiv preprint arXiv:2603.08127 , year=

  18. [18]

    Journal of Systems and Software , volume=

    Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents , author=. Journal of Systems and Software , volume=. 2025 , publisher=

  19. [19]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Webarena: A realistic web environment for building autonomous agents , author=. arXiv preprint arXiv:2307.13854 , year=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models , author=. arXiv preprint arXiv:2307.10635 , year=

  22. [22]

    arXiv preprint arXiv:2506.12958 , year=

    Domain specific benchmarks for evaluating multimodal large language models , author=. arXiv preprint arXiv:2506.12958 , year=

  23. [23]

    Bioinformatics , volume=

    Chembench: a cheminformatics workbench , author=. Bioinformatics , volume=. 2010 , publisher=

  24. [24]

    Advances in neural information processing systems , volume=

    What can large language models do in chemistry? a comprehensive benchmark on eight tasks , author=. Advances in neural information processing systems , volume=

  25. [25]

    The Fourteenth International Conference on Learning Representations , year=

    Earthse: A benchmark evaluating earth scientific exploration capability for large language models , author=. The Fourteenth International Conference on Learning Representations , year=

  26. [26]

    MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs

    MSEarth: A Multimodal Scientific Dataset and Benchmark for Phenomena Uncovering in Earth Science , author=. arXiv preprint arXiv:2505.20740 , year=

  27. [27]

    CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

    Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark , author=. arXiv preprint arXiv:2409.11363 , year=

  28. [28]

    AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage

    Autoreproduce: Automatic ai experiment reproduction with paper lineage , author=. arXiv preprint arXiv:2505.20662 , year=

  29. [29]

    ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration

    ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration , author=. arXiv preprint arXiv:2605.03042 , year=

  30. [30]

    arXiv preprint arXiv:2512.16969 , year=

    Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows , author=. arXiv preprint arXiv:2512.16969 , year=

  31. [31]

    arXiv preprint arXiv:2511.14366 , year=

    ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning , author=. arXiv preprint arXiv:2511.14366 , year=

  32. [32]

    Kimi K2.5: Visual Agentic Intelligence

    Kimi K2. 5: Visual Agentic Intelligence , author=. arXiv preprint arXiv:2602.02276 , year=

  33. [33]

    arXiv preprint arXiv:2602.09132 , year=

    SciDataCopilot: An Agentic Data Preparation Framework for AGI-driven Scientific Discovery , author=. arXiv preprint arXiv:2602.09132 , year=

  34. [34]

    MinerU: An Open-Source Solution for Precise Document Content Extraction

    Mineru: An open-source solution for precise document content extraction , author=. arXiv preprint arXiv:2409.18839 , year=

  35. [35]

    2026 , howpublished =

    Mingxin Yang , title =. 2026 , howpublished =

  36. [36]

    Towards an AI co-scientist

    Towards an AI co-scientist , author=. arXiv preprint arXiv:2502.18864 , year=

  37. [37]

    Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou

    Ai-researcher: Autonomous scientific innovation , author=. arXiv preprint arXiv:2505.18705 , year=

  38. [38]

    2026 , eprint=

    InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discovery , author=. 2026 , eprint=