pith. sign in

arxiv: 2604.08501 · v2 · pith:7WOBTVBZnew · submitted 2026-04-09 · 💻 cs.DL · cs.CL· cs.SE

sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing

Sergey V Samsonau This is my paper

Pith reviewed 2026-05-10 17:14 UTC · model grok-4.3

classification 💻 cs.DL cs.CLcs.SE
keywords scientific manuscript verificationcitation integrity checkinglocal AI linterreference support verificationretraction detectionopen-weights modelsAI-assisted scientific writingbibliographic analysis
0
0 comments X

The pith

A locally running open-source tool can verify that a scientific manuscript's citations exist, are unretracted, and support the claims made about them.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes sciwrite-lint as a third option for scientific quality assurance beyond journal gatekeeping or pure open science. It is an open-source pipeline that runs entirely on the researcher's machine using public databases, a consumer GPU, and open-weights models, without sending any manuscripts externally. The system checks reference existence and retraction status, aligns metadata with canonical records, downloads and parses cited papers to confirm they back the stated claims, and recurses one level into those papers' own bibliographies. Each reference receives an aggregated reliability score. This approach is evaluated on thirty unseen arXiv and bioRxiv papers that include injected errors, with false-positive analysis performed by an LLM adjudicator.

Core claim

sciwrite-lint provides a local verification pipeline that confirms references exist, checks retraction status, compares metadata against canonical records, downloads and parses cited papers to verify support for the manuscript's claims, extends the check one level deeper into the cited papers' bibliographies, and assigns each reference a per-reference reliability score that aggregates all signals.

What carries the argument

The sciwrite-lint verification pipeline, which uses open-weights models running locally to parse downloaded papers and test whether they support specific claims.

If this is right

  • Authors can run automated integrity checks on drafts before submission without relying on external services or peer review.
  • Fabricated citations that evade current gatekeeping can be detected at the author level through metadata, retraction, and claim-support checks.
  • Each reference receives a composite reliability score that combines existence, retraction status, metadata match, and claim verification.
  • Verification extends one bibliographic level deeper, revealing inconsistencies in the cited papers' own reference lists.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Embedding the pipeline into common writing environments could turn citation verification into a routine, real-time step during drafting.
  • Increasing the recursion depth beyond one level might expose longer chains of weak or fabricated citations.
  • The experimental contribution-scoring module, which draws on philosophy-of-science frameworks, could evolve into quantitative tests of argument structure if the integrity component proves reliable.

Load-bearing premise

Automated local parsing of cited papers using open-weights models can reliably determine whether those papers support the specific claims made in the manuscript under review.

What would settle it

A controlled test set of manuscripts containing fabricated or unsupported citations where the pipeline's claim-verification step fails to flag them at a high rate.

Figures

Figures reproduced from arXiv: 2604.08501 by Sergey V Samsonau.

Figure 1
Figure 1. Figure 1: Verification architecture. Top: per-paper operations (steps 1–6 above). Bottom: the pipeline fans out through the citation graph at three levels: the manuscript (all operations), cited papers (downloaded, parsed, consistency-checked), and their references (existence + metadata checked via API). Solid borders: full text available; dashed: API-verified only [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Contribution profiles for 3 of the 20 calibration papers. Left: Nobel Prize discovery (balanced). [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
read the original abstract

Scientific papers make claims about prior work backed by citations. Verifying those citations at scale (that each cited paper exists, says what the citation claims, and is itself reliable) is structurally beyond what human review can deliver: a typical paper has dozens of citations, and a careful reviewer reads at most a handful end-to-end. AI-assisted writing makes this gap even more urgent: LLMs hallucinate references and may fill in plausible details from titles or abstracts of papers they never read, worse for the smaller local-weights models that privacy-aware researchers must use. sciwrite-lint applies the linting paradigm from software engineering to citation verification: it runs entirely on the researcher's machine (free public databases, a single consumer GPU, and open-weights models), is fast enough to re-lint between revisions so authors catch problems at the source while drafting, and serves journals and reviewers as an automated first pass. The pipeline checks reference existence, metadata accuracy, retraction status, and claim support, traverses one level into cited papers' bibliographies, and produces per-reference reliability scores. We evaluate on 30 unseen papers (arXiv and bioRxiv) with error injection and LLM-adjudicated false-positive analysis. The same linting workflow extends to internal consistency: numbers in text vs. tables, abstract vs. body, figure captions vs. content, statistical results vs. their verbal interpretation, plus structural cross-references (dangling cites, orphan references). As a separate experimental contribution we also propose SciLint Score: citation-chain integrity combined with a contribution component operationalizing five philosophy-of-science frameworks (Popper, Lakatos, Kitcher, Laudan, Mayo).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces sciwrite-lint, an open-source local linter for scientific manuscripts that verifies reference existence, retraction status, metadata consistency against canonical records, downloads and parses cited papers to check claim support using open-weights models, and extends verification one level into the cited papers' bibliographies. Each reference receives an aggregated reliability score. An experimental SciLint Score is proposed that combines the integrity pipeline with a contribution metric derived from five philosophy-of-science frameworks (Popper, Lakatos, Kitcher, Laudan, Mayo). The evaluation consists of running the pipeline on 30 unseen arXiv and bioRxiv papers after error injection, followed by LLM-adjudicated false-positive analysis.

Significance. If the core verification steps can be shown to be reliable, the tool would provide a practical, privacy-preserving, and accessible method for authors to perform pre-submission integrity checks without transmitting manuscripts to external services. The fully local execution using free public databases, consumer hardware, and open-weights models is a clear strength that supports reproducibility and broad adoption. The open-source release and experimental contribution component also invite community extension. However, the absence of quantitative performance metrics and independent validation limits the immediate significance of the reported results.

major comments (2)
  1. [Evaluation (abstract and main text)] The evaluation on 30 papers (described in the abstract and evaluation section) reports no quantitative results, error rates, precision/recall figures, or detailed methodology for the error-injection protocol and LLM adjudication process. This is load-bearing for the central claim that the pipeline provides effective verification, as it leaves the actual performance of the claim-support step unquantified.
  2. [Pipeline description and evaluation] The claim-support verification component, which downloads cited papers and uses open-weights models to decide whether they support specific manuscript claims, relies on LLM adjudication for false-positive analysis without any human-labeled ground-truth subset. This setup can detect obvious mismatches but does not establish accuracy on nuanced cases of partial support, context omission, or interpretive disagreement, directly affecting the reliability of the per-reference scores.
minor comments (2)
  1. [Abstract] The abstract states that the contribution component is 'released as experimental code' but provides no link, repository details, or usage instructions in the provided text.
  2. [Pipeline description] The manuscript would benefit from a table summarizing the verification signals aggregated into the per-reference reliability score.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback on the manuscript. We address each major comment below with clarifications on the evaluation design and planned revisions to improve transparency around methodology and limitations.

read point-by-point responses

  1. Referee: [Evaluation (abstract and main text)] The evaluation on 30 papers (described in the abstract and evaluation section) reports no quantitative results, error rates, precision/recall figures, or detailed methodology for the error-injection protocol and LLM adjudication process. This is load-bearing for the central claim that the pipeline provides effective verification, as it leaves the actual performance of the claim-support step unquantified.

    Authors: We agree that quantitative metrics such as precision, recall, or error rates would provide stronger evidence for the pipeline's effectiveness. The reported evaluation is a proof-of-concept demonstration using error injection on 30 unseen papers followed by LLM adjudication for false-positive analysis, rather than a full benchmark study. We will revise the evaluation section to include a more detailed description of the error-injection protocol, the specific types of errors introduced, the adjudication prompts used, and any observed patterns in detected issues. We will also add an explicit statement that this constitutes an initial functional validation rather than a comprehensive performance benchmark. revision: partial

  2. Referee: [Pipeline description and evaluation] The claim-support verification component, which downloads cited papers and uses open-weights models to decide whether they support specific manuscript claims, relies on LLM adjudication for false-positive analysis without any human-labeled ground-truth subset. This setup can detect obvious mismatches but does not establish accuracy on nuanced cases of partial support, context omission, or interpretive disagreement, directly affecting the reliability of the per-reference scores.

    Authors: We acknowledge this limitation in the current evaluation design. The LLM adjudication serves as a practical, scalable method to surface potential mismatches in the absence of existing human-annotated ground-truth datasets for scientific claim support. We recognize that this approach is better suited to identifying clear discrepancies than to resolving nuanced interpretive cases. In the revised manuscript, we will expand the limitations discussion to emphasize that the per-reference reliability scores function as aggregated heuristic signals to guide author attention, not as definitive accuracy measures. We will also outline directions for future work, including community-driven creation of human-labeled subsets for more rigorous validation. revision: partial

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper describes a software pipeline and its evaluation on 30 papers using error injection plus LLM adjudication, with no mathematical derivations, equations, fitted parameters, or predictions. The core integrity verification is presented as a direct implementation of public databases and open models; the SciLint Score contribution component is explicitly labeled experimental and not used to support integrity claims. No self-citations are load-bearing for any central result, and no step reduces by construction to its own inputs. This is a self-contained tool description rather than a derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim depends on the reliability of external public databases for metadata and retractions plus the capability of open-weights models to parse and compare scientific claims accurately in a fully local setting; the experimental SciLint Score adds an unvalidated layer.

axioms (1)
  • domain assumption Public databases and open-weights models suffice to verify reference existence, retraction status, metadata, and claim support without external services or human intervention.
    Invoked throughout the pipeline description as the basis for local-only operation.
invented entities (1)
  • SciLint Score no independent evidence
    purpose: Combines integrity verification with a contribution component operationalizing five philosophy-of-science frameworks into computable structural properties.
    Introduced as an experimental extension released for community development, with no validation results provided.

pith-pipeline@v0.9.0 · 5591 in / 1534 out tokens · 67233 ms · 2026-05-10T17:14:24.319457+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.