REVIEW 2 major objections 1 minor
Encoding dense nuggets into a grid matrix allows end-to-end extraction of complex events from scientific documents.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-05-23 23:38 UTC pith:7VVCIJTL
load-bearing objection New SciEvents dataset and nugget-grid model for scientific event extraction, but no annotation stats or experimental numbers visible yet. the 2 major comments →
EXCEEDS: Extracting Complex Events via Nugget-based Grid Modeling in Scientific Domain
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By constructing a tailored dataset for scientific events and proposing an end-to-end framework that encodes dense nuggets into a grid matrix, the task of extracting complex, multi-event structures from documents is simplified to nugget-based grid modeling, leading to state-of-the-art results on the new benchmark.
What carries the argument
Nugget-based grid modeling, which encodes dense nuggets into a grid matrix to represent and extract complex events.
Load-bearing premise
The multi-stage manual annotation and quality control process yields reliable labels that faithfully capture the denser nuggets and more complex information forms in scientific documents.
What would settle it
Running EXCEEDS on a new scientific corpus annotated independently would show performance no better than existing methods, or the annotation labels would vary substantially across different annotator teams.
If this is right
- Event extraction in science can be performed in a single end-to-end model rather than staged pipelines.
- The SciEvents dataset serves as a public resource for developing and evaluating scientific event extraction systems.
- Complex information forms in documents become manageable through matrix-based representation of nuggets.
- Automated understanding of scientific literature gains a structured event layer.
Where Pith is reading between the lines
- Grid-based nugget encoding might transfer to other domains with high information density, such as legal or technical texts.
- The approach highlights the value of treating information nuggets as the basic units for modeling rather than full sentences or paragraphs.
- Document-level multi-event extraction could benefit from similar matrix simplifications in related NLP tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that scientific-domain event extraction is underserved due to two characteristics—denser nuggets/events and more complex information forms—relative to news/finance/biology. It introduces SciEvents (2,508 documents, 24,381 events) constructed via multi-stage manual annotation with a tailored schema, proposes EXCEEDS (an end-to-end grid-modeling framework that encodes nuggets into a matrix and reduces complex extraction to nugget-based grid modeling), and reports SOTA results on SciEvents, with public release of both dataset and framework.
Significance. If the dataset reliably encodes the stated domain characteristics and the grid-modeling gains are attributable to them rather than annotation artifacts, the work supplies a needed benchmark and method for scientific event extraction. The explicit public release of the dataset and framework is a concrete strength that supports reproducibility and follow-on research.
major comments (2)
- [Dataset construction] Dataset construction section: the multi-stage manual annotation and quality control process is described (yielding 2,508 documents and 24,381 events) but supplies no inter-annotator agreement figures, no nugget-density statistics relative to other domains, and no error analysis on complex information forms. This is load-bearing for the central claim that SciEvents faithfully captures the two motivating characteristics so that EXCEEDS gains can be attributed to domain handling rather than label artifacts.
- [Experiments] Experiments section: the SOTA claim is asserted without reported metrics, baselines, or ablation results on the grid-modeling components (e.g., nugget encoding or matrix formulation), so the data-to-claim link for the framework cannot be verified from the supplied experimental details.
minor comments (1)
- [Abstract] Abstract states SOTA performance but contains no quantitative results, baselines, or ablation details; this reduces immediate clarity even if the full experimental section supplies them.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback and the recommendation for major revision. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation of the dataset and experiments.
read point-by-point responses
-
Referee: [Dataset construction] Dataset construction section: the multi-stage manual annotation and quality control process is described (yielding 2,508 documents and 24,381 events) but supplies no inter-annotator agreement figures, no nugget-density statistics relative to other domains, and no error analysis on complex information forms. This is load-bearing for the central claim that SciEvents faithfully captures the two motivating characteristics so that EXCEEDS gains can be attributed to domain handling rather than label artifacts.
Authors: We agree that inter-annotator agreement figures, comparative nugget-density statistics, and error analysis are important to substantiate the dataset's domain characteristics. The multi-stage annotation process was intended to capture denser nuggets and complex forms, but these quantitative validations were not included. In the revised manuscript we will add IAA scores, nugget-density comparisons against existing datasets from other domains, and error analysis on complex information forms. revision: yes
-
Referee: [Experiments] Experiments section: the SOTA claim is asserted without reported metrics, baselines, or ablation results on the grid-modeling components (e.g., nugget encoding or matrix formulation), so the data-to-claim link for the framework cannot be verified from the supplied experimental details.
Authors: We acknowledge that the experimental details require expansion for full verifiability. While the manuscript reports state-of-the-art results on SciEvents, we will revise the Experiments section to explicitly list all evaluation metrics, the complete set of baselines, and ablation studies isolating the contributions of nugget encoding and the matrix formulation. revision: yes
Circularity Check
No circularity: empirical dataset construction plus model evaluation on that dataset
full rationale
The paper constructs SciEvents via multi-stage manual annotation (2,508 docs, 24,381 events) and proposes EXCEEDS as a grid-modeling framework, then reports experimental SOTA results on SciEvents. No equations, derivations, fitted-parameter predictions, or self-citation chains appear in the provided text. The central claims rest on direct empirical measurement rather than any reduction of outputs to inputs by construction. This is the standard self-contained pattern for dataset+model papers and receives the default non-circularity finding.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Scientific domain exhibits denser nuggets and more complex information forms than other domains, requiring tailored datasets and methods.
read the original abstract
It is crucial to understand a specific domain by events. Extensive event extraction research has been conducted in many domains such as news, finance, and biology. However, event extraction in scientific domain is still insufficiently supported by comprehensive datasets and tailored methods. Compared with other domains, scientific domain has two characteristics: (1) denser nuggets and events, and (2) more complex information forms. To solve the above problem, considering these two characteristics, we first construct SciEvents, a large-scale multi-event document-level dataset with a schema tailored for scientific domain. It consists of 2,508 documents and 24,381 events under multi-stage manual annotation and quality control. Then, we propose EXCEEDS, an end-to-end scientific event extraction framework by encoding dense nuggets into a grid matrix and simplifying complex event extraction as a nugget-based grid modeling task. Experiments on SciEvents demonstrate state-of-the-art performances of EXCEEDS. Both the SciEvents dataset and the EXCEEDS framework are released publicly to facilitate future research.
Figures
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.