Pith. sign in

REVIEW 2 major objections 2 minor 13 references

Strong performance on existing document parsing benchmarks does not transfer to expert-level documents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 16:59 UTC pith:75CUJ5YI

load-bearing objection Dr. DocBench introduces a failure-sampled document set across 52 domains but the sampling risks capturing current-model artifacts rather than intrinsic expert difficulty. the 2 major comments →

arxiv 2606.01393 v1 pith:75CUJ5YI submitted 2026-05-31 cs.CL cs.AIcs.CV

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Minglai Yang , Xinyan Velocity Yu , Pengyuan Li , Xinyu Guo , Zhenting Qi , Konwoo Kim , Longtian Ye , Xiaolong Luo
show 17 more authors
Jinhe Bi Henry Zhang Haris Riaz Xuan Zhang Yunze Xiao Bangya Liu Tom Tang Yunfei Zhao Qunshu Lin Zihan Wang Minghao Liu Michael Lingzhi Li Yilun Du Jesse Thomason Rogerio Feris Alex Pentland Zexue He
This is my paper
classification cs.CL cs.AIcs.CV
keywords document parsingbenchmarkOCRvision-language modelslayout analysisexpert-level documentsBISAC domainsparser failures
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Dr. DocBench, a benchmark for expert-level document parsing drawn from a large multilingual book corpus across 52 subject domains. It uses parser-failure-based sampling to select pages where current systems struggle, resulting in 4,514 annotated pages with detailed layout and content annotations. Evaluations show that pipeline-based parsers and general-purpose VLMs perform poorly on these cases despite good results on prior benchmarks. This highlights the need for more robust methods that handle complex structures like chemical formulas, music notation, and cross-page layouts. The work provides a testbed for advancing document intelligence beyond common genres.

Core claim

Dr. DocBench is constructed by sampling challenging documents from a large-scale multilingual book corpus using parser-failure-based selection across 52 BISAC domains, yielding 4,514 pages with 65k annotations for layout, reading order, hierarchical relations, and domain-specific contents. Evaluations demonstrate that strong results on existing benchmarks do not carry over to this expert-level set, with substantial failures observed across subjects, content types, and structural attributes.

What carries the argument

Parser-failure-based sampling that selects documents where multiple state-of-the-art parsers fail, targeting expert-domain structures.

Load-bearing premise

Parser-failure-based sampling from a large-scale multilingual book corpus produces a representative set of expert-level difficult documents across the 52 BISAC domains.

What would settle it

If randomly selected pages from the corpus prove equally difficult for parsers as the failure-sampled ones, or if new models achieve high accuracy on Dr. DocBench without improvement on the underlying sampling method.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pipeline-based parsers and VLMs show substantial failures on the new benchmark.
  • Performance gaps appear across 52 subject domains and various content types including chemical formulas and music notation.
  • The benchmark reveals issues with complex tables and cross-page layouts.
  • Existing benchmarks are insufficient for testing expert-level document intelligence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Future parser development may need to prioritize failure cases from diverse domains rather than uniform sampling.
  • Dr. DocBench could guide the creation of specialized training data for VLMs in professional fields.
  • The sampling method might apply to other AI evaluation tasks to find hard examples automatically.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper presents Dr. DocBench, a new benchmark for expert-level document parsing constructed from a large-scale multilingual book corpus spanning 52 BISAC domains. It uses parser-failure-based sampling to select 4,514 challenging pages (averaging ~100 pages per document) with 65k annotations covering layout, reading order, hierarchical relations, and domain-specific elements such as chemical formulas, music notation, complex tables, and cross-page layouts. Evaluations demonstrate that strong performance by pipeline parsers and VLMs on prior benchmarks does not transfer, with substantial failures across subjects and structures.

Significance. If the sampling procedure successfully isolates intrinsically expert-level structures rather than model-specific artifacts, Dr. DocBench would be a valuable addition as a diagnostic testbed for document intelligence, particularly given its scale, multilingual coverage, and focus on long documents with domain-specific visual content. The release of annotations and the observed transfer gap could usefully guide future VLM and parser development.

major comments (2)
  1. [§3.2] §3.2 (Parser-failure-based sampling): The central claim that Dr. DocBench measures expert-level difficulty across BISAC domains rests on the assumption that failures of current SOTA systems identify intrinsically hard expert content (chemical formulas, music notation, etc.) rather than current-model-specific edge cases. No quantitative validation is provided that the sampled pages are representative of domain-expert structures independent of the particular parsers used for filtering; this risks circularity where the benchmark simply amplifies existing weaknesses.
  2. [§4] §4 (Evaluation and transfer results): The reported lack of transfer is presented as evidence of benchmark difficulty, but without an ablation that compares failure rates on Dr. DocBench pages versus matched non-failure pages from the same corpus, it is unclear whether the gap is driven by the intended expert structures or by the sampling filter itself.
minor comments (2)
  1. [§3.3] The description of annotation quality control and inter-annotator agreement is brief; adding explicit numbers and protocol details would strengthen reproducibility claims.
  2. [Figure 2, Table 1] Figure 2 and Table 1 would benefit from clearer legends distinguishing pipeline parsers from VLMs and from explicit error bars on the reported metrics.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major comment below and indicate where revisions will be made to clarify the sampling and evaluation methodology.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (Parser-failure-based sampling): The central claim that Dr. DocBench measures expert-level difficulty across BISAC domains rests on the assumption that failures of current SOTA systems identify intrinsically hard expert content (chemical formulas, music notation, etc.) rather than current-model-specific edge cases. No quantitative validation is provided that the sampled pages are representative of domain-expert structures independent of the particular parsers used for filtering; this risks circularity where the benchmark simply amplifies existing weaknesses.

    Authors: We acknowledge the risk of circularity. Our sampling employed an ensemble of five architecturally diverse SOTA parsers (both traditional pipelines and VLMs) and retained only pages where multiple independent systems failed. The selected pages contain domain-specific elements (e.g., chemical formulas, music notation) whose intrinsic complexity is documented in the respective scientific literatures. In the revision we will expand §3.2 to detail the ensemble construction, report inter-parser agreement statistics on the filtered set, and include qualitative examples illustrating structures that remain challenging even under human expert inspection. revision: partial

  2. Referee: [§4] §4 (Evaluation and transfer results): The reported lack of transfer is presented as evidence of benchmark difficulty, but without an ablation that compares failure rates on Dr. DocBench pages versus matched non-failure pages from the same corpus, it is unclear whether the gap is driven by the intended expert structures or by the sampling filter itself.

    Authors: We agree that a matched ablation would strengthen causal attribution. Constructing precisely matched non-failure pages with equivalent domain, length, and structural distributions would require substantial new annotation. We will add a limitations paragraph in §4 and include preliminary statistics comparing error rates on Dr. DocBench pages against a random sample of 500 non-filtered pages from the same corpus, which exhibit markedly lower failure rates across the same parser suite. revision: partial

Circularity Check

1 steps flagged

Parser-failure sampling makes lack of transfer tautological by construction

specific steps
  1. fitted input called prediction [Abstract]
    "Built from a large-scale multilingual book corpus, Dr. DocBench spans 52 BISAC subject domains and selects challenging documents through parser-failure-based sampling, targeting cases where multiple state-of-the-art systems struggle. [...] Evaluations of pipeline-based parsers and general-purpose VLMs show that strong performance on existing benchmarks does not transfer to our expert-level document parsing."

    Documents are chosen precisely where the evaluated systems already struggle; the reported result that those systems struggle on the selected set is therefore forced by the sampling criterion rather than an independent test of expert-level difficulty.

full rationale

The paper constructs its benchmark by selecting pages where current parsers fail, then reports that parsers fail on the benchmark. This reduces the central evaluation claim to the selection method itself. No mathematical derivations, self-citation chains, or other patterns are present; the paper is an empirical benchmark release.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The benchmark rests on the domain assumption that failure-based sampling identifies expert-level cases; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Parser-failure-based sampling from a multilingual book corpus selects challenging expert-level documents across 52 domains
    Abstract states the benchmark targets cases where multiple state-of-the-art systems struggle.

pith-pipeline@v0.9.1-grok · 5848 in / 1115 out tokens · 25076 ms · 2026-06-28T16:59:51.213331+00:00 · methodology

0 comments
read the original abstract

Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Optical Character Recognition (OCR) and document parsing benchmarks are increasingly limited in coverage and difficulty: many focus on common document genres or uniformly sampled pages where modern parsers already perform strongly, while offering limited annotation for expert-domain structures such as chemical formula, music notation, complex tables, and cross-page layouts. We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing. Built from a large-scale multilingual book corpus, Dr. DocBench spans 52 BISAC subject domains and selects challenging documents through parser-failure-based sampling, targeting cases where multiple state-of-the-art systems struggle. It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality page- and block-level annotations for layout, reading order, hierarchical relations, and domain-specific visual contents. Evaluations of pipeline-based parsers and general-purpose VLMs show that strong performance on existing benchmarks does not transfer to our expert-level document parsing. Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.

Figures

Figures reproduced from arXiv: 2606.01393 by Alex Pentland, Bangya Liu, Haris Riaz, Henry Zhang, Jesse Thomason, Jinhe Bi, Konwoo Kim, Longtian Ye, Michael Lingzhi Li, Minghao Liu, Minglai Yang, Pengyuan Li, Qunshu Lin, Rogerio Feris, Tom Tang, Xiaolong Luo, Xinyan Velocity Yu, Xinyu Guo, Xuan Zhang, Yilun Du, Yunfei Zhao, Yunze Xiao, Zexue He, Zhenting Qi, Zihan Wang.

Figure 1
Figure 1. Figure 1: Overview of DR.DOCBENCH across 52 BISAC subject domains. Left: annotated pages per subject (4,514 pages total). Right: per-subject overall score for 5 frontier VLMs (Claude Opus 4.6, GPT-5.5, Gemini 3.1 Pro, Kimi, GPT-4o) across the 52 subjects, restricted to subjects with overall-metric coverage. Robust document understanding requires recover￾ing both content and structure from visually com￾plex pages, en… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DR.DOCBENCH. The benchmark spans diverse BISAC subject domains and provides fine-grained annotations for layout, recognition, and expert-domain structures, including chemistry diagrams, music notation, complex tables, formulas, and pseudo-code. las, chemical structures, rotated tables, or domain￾specific layouts). This creates an evaluation gap: although VLMs appear nearly saturated on stan￾dar… view at source ↗
Figure 3
Figure 3. Figure 3: Impact of sliding window size (n pages) on overall and per-metric performance across seven models. The top-left panel shows a normalized aggregate score; the remaining panels show individual metrics. Reading order degrades most consistently with increasing window size, while formula and table metrics are comparatively stable. 400 600 800 Output Tokens per Page 631 502 651 764 767 806 861 742 734 775 70 60 … view at source ↗
Figure 4
Figure 4. Figure 4: Token efficiency vs overall score. more blocks while separating page-local contents from cross-page continuations. Text extraction and table parsing also worsen. Formula metrics are comparatively stable, suggesting that formulas are more spatially local (details in Appendix H.5). 5.2 Token Efficiency [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Table case study. (5a) source image, (5b) ground-truth HTML rendering. (5c) Kimi in the current window collapses to plain text (5d). Kimi in the previous window produces correct content but ignores the HTML format instruction (5e). Doubao attends almost entirely to prior-page content and misses the target table. From full line to wireless, Kimi drops from 49.1 → 21.4 TEDS; Doubao collapses to 8.1, which is… view at source ↗
Figure 6
Figure 6. Figure 6: Music score → MusicXML transcription re￾sults (1-page window). Lower edit distance is better. The baseline is the mean pairwise edit distance across all 6 ground-truth documents. The subject of Optical Music Recognition (OMR) is a brand-new subject not present in prior work (Ouyang et al., 2025; Liu et al., 2024; Fu et al., 2025). We are the first to evaluate this subject on a variety of frontier VLMs and … view at source ↗
Figure 7
Figure 7. Figure 7: Number of annotated pages per BISAC subject in the full [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Mean and standard deviation of four-parser disagreement for each BISAC subject. The mean captures [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Unified inference prompt used in our evaluation (Part 1). [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Unified inference prompt used in our evaluation (Part 2). [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Left: the composition of the data elements per subject. Right: the composition of the layout types per [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Scaling per model size 2 4 6 8 10 12 14 Window size (n) 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Edit Distance Formula Edit Distance 2 4 6 8 10 12 14 Window size (n) 0.0 0.2 0.4 0.6 0.8 CDM Formula CDM claude-opus-4.6 gemini-3.1-pro gpt-4o gpt-5.5 paddleocr qwen3.5-122b qwen3.5-flash [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Impact of sliding window size n pages) on the formula metrics across seven models. ordering cues across more pages, and accuracy tends to fall as this span grows. Text edit distance shows a mixed early-window benefit. For text edit distance, GPT-5.5 and Gem￾ini both improve marginally from n=1 to n=2 (GPT-5.5: 0.287 → 0.285; Gemini: 0.335 → 0.322), suggesting a slight cross-page context benefit for resolv… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Vladimir I Levenshtein

    Kimi k2.5: Visual agentic intelligence.arXiv Preprint. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. 2...

    1966

  2. [2]

    P1-vl: bridging visual perception and scien- tific reasoning in physics olympiads.arXiv preprint arXiv:2602.09443. Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Yuanhong Zheng, Dongsheng Ma, and...

    work page arXiv 2025

  3. [3]

    MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale

    Mineru2. 5-pro: Pushing the limits of data- centric document parsing at scale.arXiv preprint arXiv:2604.04771. Bin Wang, Fan Wu, Linke Ouyang, Zhuangcheng Gu, Rui Zhang, Renqiu Xia, Botian Shi, Bo Zhang, and Conghui He. 2025. Image over text: Transform- ing formula recognition evaluation with character detection matching. InProceedings of the Computer Vis...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    - Transcribe text in its original language and script — do not translate

    Text Processing: - Accurately recognize all text content in the image without guessing or inferring. - Transcribe text in its original language and script — do not translate. This includes but is not limited to Chinese, English, Japanese, Russian, Arabic, and other languages.,→ - For degraded, blurry, or noisy scans: transcribe what is legible; do not inv...

  5. [5]

    - Enclose inline formulas with \( \)

    Mathematical Formula Processing: - Convert all mathematical formulas to LaTeX format. - Enclose inline formulas with \( \). For example: This is an inline formula \( E = mc^2 \) - Enclose block formulas with \[ \]. For example: \[ \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} \]

  6. [6]

    H$_2$O, CO$_2$), render them using \ce{} or standard LaTeX subscript/superscript notation

    Chemical Notation: - For chemical reaction equations and ionic equations, use the \ce{} syntax from the mhchem package.,→ Example inline: \( \ce{H2O} \) Example block: \[ \ce{2H2 + O2 -> 2H2O} \] - For 2D molecular structures given as SMILES strings, preserve the SMILES notation exactly inside a smiles block:,→ ```smiles CCO ``` - For chemical symbols, el...

  7. [7]

    Domain-Specific Notation: - Legal documents: preserve clause and sub-clause numbering exactly (e.g. Article 3.2(a)(i)).,→ - Biomedical text: preserve gene names, protein names, dosage units, and clinical notation without alteration.,→ - Financial documents: preserve ticker symbols, currency symbols, and numerical formats (e.g. $1,234.56, =C,¥).,→ - Code a...

  8. [8]

    - Wrap the entire table with <table> and </table>

    Table Processing: - Convert tables to HTML format. - Wrap the entire table with <table> and </table>. - For long tables that span multiple pages, output all rows in a single continuous HTML table.,→ - For tables with merged cells, use rowspan and colspan attributes

  9. [9]

    - Do not add speaker attributions, panel labels, or separators

    Comics and Dialog Balloons: - Extract all readable text in visual reading order: speech bubbles, thought bubbles, caption boxes, narration boxes, and any text visible within panels.,→ - Output each piece of extracted text as a separate paragraph. - Do not add speaker attributions, panel labels, or separators. Do not output placeholder text for panels with...

  10. [10]

    - For checkboxes or radio buttons, indicate the selection state: [x] for checked, [ ] for unchecked.,→ Figure 9: Unified inference prompt used in our evaluation (Part 1)

    Forms: - Render form fields and their labels as key–value pairs using a definition list or plain text, preserving the original field order.,→ - For filled-in values, transcribe the handwritten or typed content exactly. - For checkboxes or radio buttons, indicate the selection state: [x] for checked, [ ] for unchecked.,→ Figure 9: Unified inference prompt ...

  11. [11]

    Do not describe or caption them.,→ - Exception: extract any embedded text labels, axis labels, legend text, or annotations that are legible (see rule 4 above).,→

    Figure Handling: - Ignore purely visual figures (photographs, illustrations, charts) where no text is embedded. Do not describe or caption them.,→ - Exception: extract any embedded text labels, axis labels, legend text, or annotations that are legible (see rule 4 above).,→

  12. [12]

    P1"> <part-name>...</part-name> <part-abbreviation>...</part-abbreviation> </score-part> </part-list> <part id=

    Music: - If the image contains a music score (sheet music with staves, notes, clefs, time signatures, etc.), render it in MusicXML format inside a fenced code block:,→ ```musicxml <part-list> <score-part id="P1"> <part-name>...</part-name> <part-abbreviation>...</part-abbreviation> </score-part> </part-list> <part id="P1"> <measure number="1"> ... </measu...

  13. [13]

    overthinking penalty

    Output Format: - Ensure the output Markdown document has a clear structure with appropriate line breaks between elements.,→ - For complex layouts, maintain the original document's reading order as closely as possible. - Do not wrap the entire output in a code block. Do not add explanations, comments, or meta-commentary.,→ Figure 10: Unified inference prom...

    work page arXiv 2050