SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation

Anzhe Xie; Jianming Long; Jiaxin Mao; Qingyao Ai; Weihang Su; Xuanyi Chen; Yiqun Liu; Ziyi Ye

REVIEW 2 major objections 2 minor 13 cited by

The SurGE benchmark reveals a significant performance gap in how well large language models and agentic systems generate scientific surveys.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-18 21:36 UTC

load-bearing objection SurGE releases a practical benchmark and dataset for scientific survey generation, but the automated metrics need human validation to back the reported performance gaps. the 2 major comments →

arxiv 2508.15658 v5 submitted 2025-08-21 cs.CL cs.AIcs.IR

SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation

Weihang Su , Anzhe Xie , Qingyao Ai , Jianming Long , Xuanyi Chen , Jiaxin Mao , Ziyi Ye , Yiqun Liu This is my paper

classification cs.CL cs.AIcs.IR

keywords scientific survey generationbenchmarkevaluation frameworklarge language modelscitation accuracycomprehensivenessacademic literature

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Academic literature is growing so fast that writing surveys by hand has become impractical. Large language models appear able to automate the work, yet the lack of shared tests has made it hard to measure real progress. This paper supplies SurGE, a collection of computer science topics each paired with an expert survey and its references, together with a corpus of over one million papers. It also supplies an automated scorer that checks generated surveys on comprehensiveness, citation accuracy, structural organization, and content quality. When a range of current methods is run on the benchmark, even the strongest agent-based approaches show clear shortfalls against the expert references.

Core claim

SurGE consists of test instances that each contain a topic description, an expert-written survey, and the full set of cited references, plus a large-scale academic corpus of over one million papers; paired with this is an automated evaluation framework that scores generated surveys across four dimensions, and evaluations of diverse LLM-based methods on these resources expose a substantial performance gap.

What carries the argument

The SurGE benchmark together with its four automated evaluation dimensions that compare generated surveys against expert-written references using the supplied paper corpus.

Load-bearing premise

The four automated dimensions and the collected expert surveys together give a reliable enough picture of survey quality that the observed gaps will hold for new topics and new methods.

What would settle it

An experiment in which independent experts rate the best current LLM-generated surveys as equal or superior to the provided expert surveys on overall usefulness would directly challenge the reported performance gap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Standardized testing of this kind can now direct development of improved survey generation systems.
Future models will need targeted gains in literature coverage and citation precision to close the gap.
Agentic frameworks require additional capabilities to reach expert-level survey organization and content quality.
Releasing the benchmark data and code allows the community to track and compare future advances.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Literature synthesis tools may remain most useful when combined with human oversight for the foreseeable future.
The same structure of expert references plus automated scoring could be replicated for survey generation in biology or physics.
Repeated evaluation on SurGE over successive model releases would provide a concrete record of capability growth in automated research assistance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

SurGE releases a practical benchmark and dataset for scientific survey generation, but the automated metrics need human validation to back the reported performance gaps.

read the letter

Hey, the main thing to know is that this paper puts out SurGE, a benchmark with expert-written surveys, their full reference lists, a million-paper corpus, and a four-way automated scorer for LLM survey outputs in computer science. They run some current methods and show a clear gap even for agentic setups. The construction is the real addition here. Prior summarization or citation work does not directly give you expert targets plus the full cited papers for each topic, so this setup lets you check citation accuracy and coverage in a more grounded way than generic metrics. Releasing the code, data, and models openly is straightforward and useful for anyone who wants to test their own generation pipeline against it. The stress-test note is on point about the metrics. The abstract and available details do not include any correlation numbers between the automated scores on comprehensiveness, citation accuracy, structure, and content quality and independent human ratings of generated surveys. Without that, the size of the gap they highlight could partly trace back to how the dimensions were defined rather than the intrinsic hardness of the task. Sampling details for the test instances are also thin in what is shown. This is aimed at researchers working on long-form scientific generation, retrieval-augmented agents, or literature synthesis tools. A reader who needs a ready testbed to measure progress on survey-style output would get immediate use from the released collection. It should go to peer review. The dataset and protocol address a timely practical need, and the authors can add the missing validation steps if referees ask for them.

Referee Report

2 major / 2 minor

Summary. The paper introduces SurGE, a benchmark for scientific survey generation in computer science consisting of test instances (each with a topic description, expert-written survey, and cited references) plus a corpus of over one million papers. It proposes an automated evaluation framework across four dimensions—comprehensiveness, citation accuracy, structural organization, and content quality—and reports evaluations of diverse LLM-based methods, including advanced agentic frameworks, that reveal significant performance gaps.

Significance. If the automated metrics reliably proxy human-assessed survey quality, SurGE could provide a much-needed standardized resource for measuring progress on automated scientific survey generation, an increasingly important task given literature growth. The open-sourcing of code, data, and models is a clear strength that enables reproducibility and follow-on work. The reported gaps usefully highlight task complexity, but their interpretation depends on metric validity.

major comments (2)

[Evaluation Framework] Evaluation Framework section: the four automated dimensions are introduced without any reported quantitative validation (e.g., Pearson/Spearman correlation) against independent human ratings on a held-out set of generated surveys, nor inter-annotator agreement statistics for the expert reference surveys. This directly affects the central claim of a significant performance gap, because the gap could partly reflect choices in metric operationalization rather than intrinsic difficulty of survey synthesis.
[Benchmark Construction] Benchmark Construction (likely §3): no details are given on the sampling procedure used to select the test instances from the million-paper corpus. Without this, it is difficult to assess whether the observed gaps generalize or are influenced by selection bias in the collected instances.

minor comments (2)

[Abstract] The abstract states that 'diverse LLM-based methods' were evaluated but does not name the specific systems or agentic frameworks; adding this list would improve readability.
[Experiments] Figure and table captions should explicitly state the number of test instances and the exact LLM configurations used so that readers can interpret the numerical results without cross-referencing the text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below and outline the revisions we will make to strengthen the manuscript's clarity and rigor.

read point-by-point responses

Referee: [Evaluation Framework] Evaluation Framework section: the four automated dimensions are introduced without any reported quantitative validation (e.g., Pearson/Spearman correlation) against independent human ratings on a held-out set of generated surveys, nor inter-annotator agreement statistics for the expert reference surveys. This directly affects the central claim of a significant performance gap, because the gap could partly reflect choices in metric operationalization rather than intrinsic difficulty of survey synthesis.

Authors: We agree that explicit validation of the automated metrics against human judgments would strengthen the central claims. In the revised manuscript we will add a dedicated subsection reporting a human evaluation study: expert annotators rate a held-out sample of generated surveys on the same four dimensions, and we compute Pearson and Spearman correlations with the automated scores. We will also report inter-annotator agreement statistics for the expert reference surveys. These additions will directly address concerns about metric validity and support the interpretation of the observed performance gaps. revision: yes
Referee: [Benchmark Construction] Benchmark Construction (likely §3): no details are given on the sampling procedure used to select the test instances from the million-paper corpus. Without this, it is difficult to assess whether the observed gaps generalize or are influenced by selection bias in the collected instances.

Authors: We acknowledge the omission of sampling details. In the revised version we will expand the Benchmark Construction section to describe the full procedure: topics were drawn from a stratified sample of recent (2020–2024) high-impact computer-science papers, balanced across major sub-areas (e.g., NLP, vision, systems), with explicit criteria for topic scope and citation count. We will also document how the corresponding expert surveys were sourced and any filtering steps applied to the million-paper corpus. This added transparency will allow readers to evaluate potential selection effects. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark release with no derivational circularity

full rationale

The paper introduces SurGE as a benchmark consisting of topic descriptions, expert-written reference surveys, cited papers, and a large external corpus, then applies an automated evaluation framework across four explicitly defined dimensions. All reported results are direct measurements against these externally sourced references and corpus data; no equations, fitted parameters, or predictions are defined in terms of the paper's own outputs, and no load-bearing claim reduces to a self-citation chain or self-definitional construction. The work is therefore self-contained as an empirical contribution rather than a closed derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central contribution rests on the domain assumption that expert surveys constitute reliable ground truth and that the chosen automated metrics align with human notions of survey quality; no free parameters or invented entities are introduced.

axioms (1)

domain assumption Expert-written surveys constitute reliable ground truth for measuring generated survey quality.
The benchmark design treats these surveys as the reference standard against which LLM outputs are scored.

pith-pipeline@v0.9.0 · 5745 in / 1230 out tokens · 41381 ms · 2026-05-18T21:36:24.958926+00:00 · methodology

0 comments

read the original abstract

The rapid growth of academic literature makes the manual creation of scientific surveys increasingly infeasible. While large language models show promise for automating this process, progress in this area is hindered by the absence of standardized benchmarks and evaluation protocols. To bridge this critical gap, we introduce SurGE (Survey Generation Evaluation), a new benchmark for scientific survey generation in computer science. SurGE consists of (1) a collection of test instances, each including a topic description, an expert-written survey, and its full set of cited references, and (2) a large-scale academic corpus of over one million papers. In addition, we propose an automated evaluation framework that measures the quality of generated surveys across four dimensions: comprehensiveness, citation accuracy, structural organization, and content quality. Our evaluation of diverse LLM-based methods demonstrates a significant performance gap, revealing that even advanced agentic frameworks struggle with the complexities of survey generation and highlighting the need for future research in this area. We have open-sourced all the code, data, and models at: https://github.com/oneal2000/SurGE

discussion (0)

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
cs.CL 2026-06 unverdicted novelty 8.0

MetaSyn benchmark shows LLM pipelines recover at most 52.7% of ground-truth included studies due to screening failures on PI/ECO eligibility, despite 90.9% retrieval recall at K=200.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
cs.CL 2026-06 unverdicted novelty 7.0

LLM agents reach 90.9% retrieval recall at K=200 but recover at most 52.7% of ground-truth included studies because they cannot reliably apply PI/ECO eligibility criteria to topically similar distractors.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
cs.CL 2026-06 unverdicted novelty 7.0

MetaSyn benchmark shows LLM agents recover at most 52.7% of relevant studies in meta-analysis pipelines due to failures in PI/ECO-based screening despite strong retrieval.
Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation
cs.CL 2026-06 unverdicted novelty 7.0

Introduces LongJudgeBench benchmark showing LLM judges remain unstable for long-form output evaluation even with rubrics or references.
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
cs.DL 2026-05 unverdicted novelty 7.0

RWGBench is a citation-centric benchmark for related work generation built from 40k CS papers and a 100-paper test set, with multi-dimensional metrics that better match human expert judgment than standard similarity scores.
Skill Retrieval Augmentation for Agentic AI
cs.CL 2026-04 unverdicted novelty 7.0

Agents improve when they retrieve skills on demand from large corpora, yet current models cannot selectively decide when to load or ignore a retrieved skill.
Skill Retrieval Augmentation for Agentic AI
cs.CL 2026-04 unverdicted novelty 7.0

Introduces the SRA paradigm and SRA-Bench benchmark showing retrieval-based skill augmentation improves agent performance but skill incorporation remains a bottleneck regardless of retrieval quality.
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
cs.DL 2026-05 accept novelty 6.5

RWGBench evaluates related-work generation as citation decision-making (selection, placement, organization, discourse) rather than text similarity, exposing retrieval and generation failures that standard metrics miss.
Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness
cs.AI 2026-06 unverdicted novelty 6.0

Xcientist externalizes research synthesis and validation in AI scientists via contract-governed artifacts to maintain traceable trajectories and avoid claim drift across three domains.
DeepSurvey: Enhancing Analytical Depth and Citation Reliability in Automated Survey Generation
cs.AI 2026-05 unverdicted novelty 6.0

DeepSurvey introduces an agentic system for automated survey generation that improves depth through full-text keynotes, cross-paper clustering, and code analysis, while boosting citation reliability via graph expansio...
Enhancing Judgment Document Generation via Agentic Legal Information Collection and Rubric-Guided Optimization
cs.CL 2026-05 unverdicted novelty 6.0

Judge-R1 improves LLM judgment document generation by combining agentic legal information retrieval with GRPO-based rubric-guided optimization, outperforming baselines on the JuDGE benchmark.
Skill Retrieval Augmentation for Agentic AI
cs.CL 2026-04 unverdicted novelty 6.0

Introduces SRA paradigm and SRA-Bench benchmark (5,400 tasks, 26,262 skills) showing retrieval improves performance but LLMs fail to selectively incorporate retrieved skills.
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
cs.DL 2026-05 conditional novelty 5.5

RWGBench evaluates related-work generation as citation decision-making with multi-dimensional metrics that track expert judgment better than ROUGE/BERTScore.