Pith. sign in

REVIEW 2 major objections 1 minor 8 references

A multi-agent harness generates scientific figures from diverse inputs and converts them to editable SVGs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 07:34 UTC pith:VN35GNOO

load-bearing objection Crafter adds a multi-agent harness plus SVG editor for varied scientific figures and releases a new benchmark, but the claim that structured layouts require the harness over a stronger single generator is not directly tested. the 2 major comments →

arxiv 2605.30611 v1 pith:VN35GNOO submitted 2026-05-28 cs.CV cs.AIcs.CL

Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputs

Haozhe Zhao , Shuzheng Si , Zhenhailong Wang , Zheng Wang , Liang Chen , Xiaotong Li , Zhixiang Liang , Maosong Sun
show 1 more author
Minjia Zhang
This is my paper
classification cs.CV cs.AIcs.CL
keywords scientific figure generationmulti-agent systemseditable SVGfigure benchmarkinput diversityvector graphics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that scientific figures consist of structured semantic components, so localized generation errors are better addressed by a coordinating harness of agents than by any single improved model. Crafter applies this harness to produce figures across types and input conditions without altering the base generator, while CraftEditor uses the same pattern to turn raster results into editable vector files. The authors also release CraftBench, a benchmark with human ratings across three figure types and four input settings. If the claim holds, researchers could automate more of figure creation and revision without retraining models for each new condition.

Core claim

Crafter is a multi-agent harness that coordinates specialized agents to generate publication-quality scientific figures from varied text, code, or mixed inputs across figure types, without any architectural modification to the underlying generator; the same harness pattern powers CraftEditor to convert outputs into locally editable SVGs. On PaperBanana-Bench and the new CraftBench, Crafter outperforms both standalone generators and prior agentic baselines, with ablations showing each agent component contributes independently, and CraftEditor produces SVGs that exceed all compared baselines in editability and quality.

What carries the argument

The multi-agent harness, which decomposes figure generation into coordinated subtasks handled by separate agents rather than a monolithic model.

Load-bearing premise

Localized errors on structured figure layouts require a multi-agent harness instead of further improvements to a single generator backbone.

What would settle it

A single generator model, after targeted improvements but without any multi-agent coordination, matching or exceeding Crafter's human-rated scores on CraftBench across all three figure types and four input conditions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Crafter works on any figure type and input condition without code changes to the base model.
  • Each agent component in the harness contributes measurably to final quality, as shown by ablations.
  • CraftEditor produces SVGs that remain faithful to the generated layout while allowing local edits.
  • Both systems surpass standalone generators and earlier agent baselines on the provided benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The harness pattern could extend to other structured visual tasks such as diagram or table creation.
  • Widespread adoption would cut the manual revision time researchers spend on figures.
  • Benchmarks like CraftBench could become standard for evaluating editable output quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces Crafter, a multi-agent harness for generating scientific figures from diverse inputs (text, code, data) across multiple figure types without architectural modifications, and CraftEditor, which applies a similar pattern to convert raster outputs into editable SVGs. It presents CraftBench, a new benchmark covering three figure types and four input conditions with human quality annotations. Experiments claim Crafter substantially outperforms standalone generators and an agentic baseline on PaperBanana-Bench and CraftBench, with ablations confirming each component's contribution; CraftEditor is reported to produce superior editable outputs compared to baselines. Code and benchmark are released.

Significance. If the empirical claims hold under scrutiny, the work addresses a practical bottleneck in scientific publishing by providing a generalizable system for figure generation and post-hoc editing. The introduction of CraftBench and open-sourcing of code/benchmark are positive contributions that could facilitate follow-up research. The multi-agent harness for handling structured figure compositions offers a potential architectural insight, provided the necessity of the harness (versus stronger single backbones) is substantiated.

major comments (2)
  1. [Abstract] Abstract: The central motivation states that 'localized errors generators produce on such layouts demand not a stronger backbone but a harness.' However, the experiments compare only against standalone generators and one agentic baseline plus ablations; no head-to-head evaluation is described against a single generator backbone given equivalent total compute, fine-tuning data volume, or added internal iteration/refinement modules. This leaves the claim that the harness architecture is required by the problem structure (rather than current generator limitations) without direct support.
  2. [Abstract] Abstract and experimental sections: The abstract asserts outperformance and that 'ablations confirming each component's independent contribution,' but the provided text does not detail the data splits, error bars, statistical significance tests, or full experimental protocol (e.g., how inputs were sampled across the four conditions). Without these, the support for the central empirical claim cannot be fully assessed.
minor comments (1)
  1. [Abstract] The abstract references PaperBanana-Bench without clarifying its relation to CraftBench or whether it is an existing or newly introduced resource.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments below, agreeing where additional clarity or details are warranted and providing our reasoning on the architectural claim.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central motivation states that 'localized errors generators produce on such layouts demand not a stronger backbone but a harness.' However, the experiments compare only against standalone generators and one agentic baseline plus ablations; no head-to-head evaluation is described against a single generator backbone given equivalent total compute, fine-tuning data volume, or added internal iteration/refinement modules. This leaves the claim that the harness architecture is required by the problem structure (rather than current generator limitations) without direct support.

    Authors: We acknowledge that a direct head-to-head with a single backbone under strictly matched total compute or fine-tuning volume would provide stronger support for the claim. The agentic baseline already includes internal iteration and refinement loops, serving as the closest available proxy. Our error analysis in the manuscript highlights localized component failures that persist even under refinement, which the harness addresses via explicit role separation. In revision we will expand the discussion section to clarify this distinction and the practical difficulties of equating compute across paradigms, without claiming the harness is the only possible solution. revision: partial

  2. Referee: [Abstract] Abstract and experimental sections: The abstract asserts outperformance and that 'ablations confirming each component's independent contribution,' but the provided text does not detail the data splits, error bars, statistical significance tests, or full experimental protocol (e.g., how inputs were sampled across the four conditions). Without these, the support for the central empirical claim cannot be fully assessed.

    Authors: We agree that these details are essential. The full data splits, sampling procedure across the four input conditions, annotation protocol, and benchmark release are described in the supplementary material and the open-sourced code. In the revised manuscript we will move key protocol elements into the main experimental section, add error bars from repeated runs, and report statistical significance (paired t-tests) for the primary comparisons. revision: yes

Circularity Check

0 steps flagged

No circularity: claims rest on experimental comparisons, not self-referential derivations

full rationale

The paper presents an empirical system (Crafter multi-agent harness and CraftEditor) evaluated on PaperBanana-Bench and CraftBench with ablations. No equations, fitted parameters, or derivation chains appear in the provided text. The central motivation—that structured figure layouts 'demand not a stronger backbone but a harness'—is stated as an assumption rather than derived from prior results or self-citations. All performance claims are supported by direct comparisons to baselines and component ablations, which are independent of the paper's own outputs. This satisfies the default expectation of no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an applied engineering contribution in AI; the abstract describes no mathematical axioms, free parameters, or newly invented physical entities.

pith-pipeline@v0.9.1-grok · 5773 in / 1031 out tokens · 29015 ms · 2026-06-29T07:34:54.889130+00:00 · methodology

0 comments
read the original abstract

Scientific figures are among the most effective means of communicating complex research ideas, yet producing publication-quality illustrations remains one of the most labor-intensive parts of paper preparation. Existing automated systems each target a single figure type under text-only input, leaving the diversity of types and conditions researchers actually use unaddressed; their raster outputs further cannot be locally revised. Because scientific figures are structured compositions of discrete semantic components, the localized errors generators produce on such layouts demand not a stronger backbone but a harness. We instantiate this harness in two complementary systems: Crafter, a multi-agent harness for figure generation that generalizes across figure types and input conditions without architectural changes, and CraftEditor, which applies the same pattern to convert raster outputs into editable SVGs. Moreover, we introduce CraftBench, a benchmark spanning three figure types and four input conditions with human quality annotation. Experiments show that Crafter substantially outperforms both standalone generators and the agentic baseline on PaperBanana-Bench and CraftBench, with ablations confirming each component's independent contribution; CraftEditor faithfully converts outputs into editable SVGs that surpass all baselines. Our code and benchmark are available at https://github.com/HaozheZhao/Crafter.

Figures

Figures reproduced from arXiv: 2605.30611 by Haozhe Zhao, Liang Chen, Maosong Sun, Minjia Zhang, Shuzheng Si, Xiaotong Li, Zheng Wang, Zhenhailong Wang, Zhixiang Liang.

Figure 1
Figure 1. Figure 1: CRAFTER architecture. Given context and Docs, the intent reasoner seeds S0. The plan generator D proposes K candidate plans; Image-Gen backend E renders each plan; the critic V emits directive diagnostics; the specification refiner R writes typed edits into S; and the convergence judge routes each round to accept, refine, or revert to Final output. (Figure is generated by CRAFTER.) of methodology figures [… view at source ↗
Figure 2
Figure 2. Figure 2: CRAFTEDITOR architecture. Three phases convert a raster a ∗ into an editable SVG v. Extraction: a VLM designer D authors a keep/delete plan, an image editor E executes it, and V verifies the cleaned canvas (§3.3.1). Processing: each element is captioned, grounded, and classified. Composition: D generates SVG skeletons, E injects assets, and a hybrid critic V (VLM + program￾matic checkers) drives iterative … view at source ↗
Figure 3
Figure 3. Figure 3: Representative CRAFTBENCH samples. Each column shows one task. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: CRAFTBENCH distribution. Inner: task types; outer: per-task style. CRAFTBENCH contains 279 samples spanning four tasks and three styles ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison across different input conditions. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 1 canonical work pages

  1. [1]

    URLhttps://arxiv.org/abs/2501.03936. ZhipuAI. GLM-Image: A native multimodal image generation model.Technical Report, 2025. ZhipuAI / Z.ai. Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, and Jinsung Yoon. Paper- Banana: Automating academic illustration for AI scientists.arXiv preprint arXiv:2601.23265, 2026a. Minjun Zhu, Zhen Lin, Yix...

    work page arXiv 2025

  2. [2]

    Caption keyword filter(G1): requires figure-type language and method-related keywords (e.g., overview,architecture,pipeline) in the caption

  3. [3]

    Photographs, statistical charts, screenshots, equation-only renders, and tables are rejected

    Strict content classifier(G2): a vision-language classifier assigns each figure to one of 15 fine- grained types; onlydiagram,illustration showing method,architecture, andteaserare accepted. Photographs, statistical charts, screenshots, equation-only renders, and tables are rejected

  4. [4]

    Complexity rescore(G3): a vision-language rubric verifies that the figure is worth recreating as a drawing, exhibits sufficient design richness (score ≥4/5 ), contains at least 8 distinct named components, and would take an estimated10+ minutes to recreate manually

  5. [5]

    Alignment verification(G4): a vision-language check verifies that at least 70% of authored visual claims match the figure content, at least 60% of proposed edit targets are feasible, and the caption alignment score reaches≥3/5

  6. [6]

    First-pass quality assurance(G5): a vision-language reviewer flags cropping artifacts, water- marks, low resolution, caption mismatch, and edit-target referent issues

  7. [7]

    Evidence-required quality assurance(G6): a stricter second pass in which every flag must cite direct pixel-region evidence with confidence≥4/5; unsupported flags are auto-discarded

  8. [8]

    the bottom-row icons were over-deleted; restore them; remove the page number instead

    Manual review(G7): human inspection of every flagged sample plus all reference-conditioned task inputs. The interface and acceptance rule are described in Section C.3. C.3 Reference-Conditioned Task Construction For each of the three reference-conditioned tasks, conditioning inputs are constructed from the source figures through a semi-automatic pipeline ...

    2026