pith. sign in

arxiv: 2305.20050 · v1 · pith:2CJ6UHSWnew · submitted 2023-05-31 · 💻 cs.LG · cs.AI· cs.CL

Let's Verify Step by Step

Hunter Lightman , Vineet Kosaraju , Yura Burda , Harri Edwards , Bowen Baker , Teddy Lee , Jan Leike , John Schulman
show 2 more authors
Ilya Sutskever Karl Cobbe
This is my paper

Pith reviewed 2026-05-10 15:30 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords process supervisionoutcome supervisionMATH datasetprocess reward modellarge language modelsreasoningactive learningPRM800K
0
0 comments X

The pith

Process supervision outperforms outcome supervision for training models to solve MATH problems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares outcome supervision, which rewards only a correct final answer, with process supervision, which rewards each correct intermediate reasoning step. The authors train large language models on the challenging MATH dataset and find that process supervision produces substantially more accurate solutions. Their best process-supervised model reaches 78 percent accuracy on a representative subset of the MATH test set. They further show that active learning makes the collection of step-level labels more effective and release the full set of 800,000 human step labels used in their experiments.

Core claim

Process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision.

What carries the argument

A process reward model trained on human-provided step-level correctness labels that scores each intermediate reasoning step rather than only the final answer.

Load-bearing premise

The collected human step-level labels are consistent and unbiased enough to produce a reward model that generalizes to unseen problems.

What would settle it

A head-to-head test on the full MATH test set in which the process-supervised model solves no more problems than a comparably trained outcome-supervised model.

read the original abstract

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript investigates process versus outcome supervision for training large language models on multi-step mathematical reasoning tasks from the MATH dataset. The central finding is that process supervision significantly outperforms outcome supervision, with the best process-supervised model solving 78% of problems on a representative subset of the MATH test set. The authors additionally demonstrate that active learning improves the efficacy of process supervision and release the PRM800K dataset of 800,000 step-level human feedback labels.

Significance. If the reported outperformance generalizes, the work provides valuable empirical evidence favoring process supervision for building more reliable LLM reasoners on challenging benchmarks. The 78% solve rate is a notable quantitative result, and the public release of PRM800K is a clear strength that will support reproducible follow-on research. The grounding in independent human labels on held-out problems avoids circularity and strengthens the evaluation.

major comments (2)
  1. [Abstract] Abstract: the 78% solve rate and the claim of significant outperformance over outcome supervision are both measured on a 'representative subset' of the MATH test set. No statistical confirmation is provided (e.g., Kolmogorov-Smirnov test or chi-squared comparison of difficulty levels 1-5 and category distributions) that the subset matches the full test distribution. This is load-bearing for the headline conclusion that process supervision is superior for the MATH dataset, as any post-hoc selection or skew could inflate both the absolute figure and the gap versus outcome supervision.
  2. [Experimental results] Experimental results section: the comparison between process and outcome supervision should explicitly document controls for model size and total training compute to rule out the possibility that observed differences arise from unequal resource allocation rather than the supervision method itself.
minor comments (1)
  1. [Abstract] The paper should clarify the exact selection procedure for the representative subset and include a table or figure comparing its statistics to the full MATH test set.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their positive evaluation, recommendation for minor revision, and constructive comments. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation of our results.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the 78% solve rate and the claim of significant outperformance over outcome supervision are both measured on a 'representative subset' of the MATH test set. No statistical confirmation is provided (e.g., Kolmogorov-Smirnov test or chi-squared comparison of difficulty levels 1-5 and category distributions) that the subset matches the full test distribution. This is load-bearing for the headline conclusion that process supervision is superior for the MATH dataset, as any post-hoc selection or skew could inflate both the absolute figure and the gap versus outcome supervision.

    Authors: We appreciate the referee's emphasis on rigorous validation of the subset. The subset was constructed by selecting problems to match the overall distribution of difficulty levels (1-5) and categories from the full MATH test set, based on the dataset's provided metadata. While we did not include formal statistical tests in the original submission, we agree this would bolster the claim. In the revised manuscript, we will add a dedicated paragraph and table in the Experimental Results section (or a new appendix) that compares the distributions using chi-squared tests for categories and difficulty levels, along with summary statistics. This will confirm the subset's representativeness and support the headline findings without altering the reported numbers. revision: yes

  2. Referee: [Experimental results] Experimental results section: the comparison between process and outcome supervision should explicitly document controls for model size and total training compute to rule out the possibility that observed differences arise from unequal resource allocation rather than the supervision method itself.

    Authors: We agree that explicit documentation of these controls is essential for a fair comparison. All models in the main experiments used identical base architectures and sizes (the 7B LLaMA model), the same training hyperparameters, batch sizes, and number of training steps, resulting in equivalent total compute for process-supervised and outcome-supervised variants. The only difference was the form of the supervision signal and associated reward model training. These details appear in the 'Models and Training' and 'Experimental Setup' sections, but we will revise the Experimental Results section to include a concise, dedicated statement (and possibly a small table) explicitly confirming equal model size and compute allocation to eliminate any ambiguity. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical results grounded in independent human labels

full rationale

The paper reports an empirical comparison of process versus outcome supervision on the MATH dataset, with the central 78% solve-rate claim obtained by direct evaluation of a trained model on a held-out representative subset using newly collected human step-level labels (PRM800K). No mathematical derivation chain exists that reduces any result to its inputs by construction. There are no self-definitional equations, no fitted parameters renamed as predictions, no load-bearing self-citations, and no uniqueness theorems imported from prior author work. The evaluation relies on external benchmarks (MATH) and independent human annotations rather than tautological reuse of training signals.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the reliability of human step-level annotations and the assumption that the selected MATH subset reflects the full test distribution; no new entities or free parameters are introduced beyond standard training hyperparameters.

axioms (1)
  • domain assumption Human step-level feedback is accurate and unbiased
    The process reward model is trained directly on these labels; any systematic bias would propagate to the reported performance gap.

pith-pipeline@v0.9.0 · 5504 in / 1160 out tokens · 30812 ms · 2026-05-10T15:30:13.237539+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse

    cs.LG 2026-06 conditional novelty 8.0

    GRPO's group-mean baseline assigns identical advantages to all tokens under output-only rewards, inducing gradient sparsity and an intrinsic rank-2 structure proven from the zero-sum constraint and confirmed by SVD on...

  2. DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

    cs.LG 2026-05 conditional novelty 8.0

    DualKV is a new FlashAttention variant that shares prompt KV across multiple rollouts in RL training, delivering 1.63-3.82x speedups on 8B-30B models while remaining mathematically identical to standard attention.

  3. The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits

    cs.LG 2026-05 unverdicted novelty 8.0

    Shared token budgets between visible chain-of-thought and answers create a coupling tax that makes non-thinking competitive on math benchmarks, with a truncation decomposition predicting the crossover and split budget...

  4. MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

    cs.CL 2026-04 unverdicted novelty 8.0

    MedPRMBench is the first fine-grained benchmark for process reward models in medical reasoning, featuring 6500 questions, 13000 chains, 113910 step labels, and a baseline that improves downstream QA accuracy by 3.2-6....

  5. Co-Designing Quantum Codes with Transversal Diagonal Gates via Multi-Agent Systems

    quant-ph 2025-10 accept novelty 8.0 full

    A Lean-verified multi-agent system produces a catalogue of 14,116 quantum codes with transversal diagonal gates for small parameters, extracts infinite families, and resolves specific distance-3 cases with constructio...

  6. ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

    cs.CR 2025-07 unverdicted novelty 8.0

    ExCyTIn-Bench is the first benchmark of 7542 questions from Microsoft Sentinel threat investigation graphs, where the best LLM agent achieves a reward of 0.606.

  7. ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

    cs.DC 2026-07 unverdicted novelty 7.0

    ELDR reduces median TPOT by 5.9-13.9% in PD-disaggregated MoE serving via expert signatures from prefill, K-means partitioning, and locality-band routing with KV-co-indexed signature cache.

  8. Flow Reasoning Models: Scaling Reasoning Through Iterative Self-Refinement

    cs.AI 2026-06 conditional novelty 7.0

    Flow models reach 99.2% Sudoku accuracy in 7 passes and 96.1% on out-of-distribution Sudoku-Extreme by selecting dynamically stable candidates and training with self-conditioning plus DPO to avoid failed outputs.

  9. Grounded Iterative Language Planning: How Parameterized World Models Reduce Hallucination Propagation in LLM Agents

    cs.AI 2026-06 unverdicted novelty 7.0

    GILP trains a parameterized backbone for valid actions and state predictions, then uses a consistency gate with LLM drafts to reduce hallucinated-state rate from 0.176 to 0.035 on GPT-4o-mini while raising success fro...

  10. InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy

    cs.AI 2026-06 unverdicted novelty 7.0

    InvestPhilBench is a new multi-layer benchmark for LLM procedural reasoning in investment philosophy, with BASP metrics showing composite scores saturate while gate reconstruction accuracy reveals procedural deficits.

  11. VCT: A Verifiable Transcript System for LLM Conversations

    cs.CR 2026-06 unverdicted novelty 7.0

    VCT abstracts non-linear LLM operations into authenticated state transitions via atomic Q&A hash chains, session Merkle roots, and account-level roots with joint signatures, plus protocols for deletions and concurrenc...

  12. A Verifiable Search Is Not a Learnable Chain-of-Thought

    cs.LG 2026-06 unverdicted novelty 7.0

    Verifiable search procedures cannot be learned as forward chain-of-thought by language models; they instead learn memorization, verification, or require precomputed catalogs.

  13. Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

    cs.AI 2026-06 unverdicted novelty 7.0

    ICT framework applies JS divergence to token logits to select critical tokens for selective RLVR updates, claiming 4.58% average pass@4 gains on Qwen2.5 models across seven reasoning benchmarks.

  14. Beyond Parallel Sampling: Diverse Query Initialization for Agentic Search

    cs.AI 2026-06 unverdicted novelty 7.0

    DivInit improves agentic search breadth scaling by selecting diverse first-turn queries from a single model generation, delivering 5-7 point gains on multi-hop QA across five models and eight benchmarks at matched compute.

  15. Failure Modes of Large Language Models on Research-Level Mathematics: A Taxonomy and an Empirical Characterisation

    cs.DL 2026-06 conditional novelty 7.0

    This paper introduces a taxonomy of four LLM failure modes on research math proofs and empirically shows premise smuggling in all eight audited Gemini outputs, with a new audit instrument achieving 100% precision.

  16. Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    SWITCH uses explicit <swi> and </swi> boundary tokens to make latent chain-of-thought compatible with on-policy RL (GRPO) and open to causal mechanistic probing, outperforming prior hidden-state recurrence methods.

  17. Agreement in Representation Space for Open-Ended Self-Consistency

    cs.CL 2026-06 unverdicted novelty 7.0

    EBA clusters sampled LLM generations in representation space to estimate agreement, outperforming random selection with stable scaling and showing that central positions correlate with higher generation quality.

  18. The Power of Test-Time Training for Approximate Sampling

    cs.DS 2026-06 unverdicted novelty 7.0

    Establishes a quadratic lower bound on query complexity for sampling from large classes of distributions given approximate density oracles, answers an open question on optimality of random walks, and shows circumventi...

  19. Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation

    cs.LG 2026-06 unverdicted novelty 7.0

    AR-OPD disentangles privileged supervision via anchored residual guidance to reduce hindsight leakage in on-policy distillation, reporting gains of 2.3 points over full privileged OPD and 7.9 over SFT on reasoning tasks.

  20. VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

    cs.CV 2026-06 unverdicted novelty 7.0

    A paired-image benchmark reveals that many MLLMs fail to update predictions when task-critical visual evidence changes, even when they answer individual images correctly.

  21. Cascading Hallucination in Agentic RAG: The CHARM Framework for Detection and Mitigation

    cs.AI 2026-06 unverdicted novelty 7.0

    Introduces CHARM framework that detects cascading hallucinations in agentic RAG at 89.4% rate with 5.3% false positives and reduces error propagation by 82.1% on multi-hop QA benchmarks.

  22. From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

    cs.AI 2026-06 unverdicted novelty 7.0

    ChemCoTBench-V2 is a new rule-verifiable benchmark with 5,620 samples across 18 tasks that evaluates LLM chemical reasoning traces using deterministic chemistry rules and reference traces rather than final answers alone.

  23. ResMerge: Residual-based Spectral Merging of Large Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    ResMerge improves merging of RL expert LLMs via a stable residual consensus backbone plus gated head correction, outperforming task-vector and spectral baselines in capability preservation.

  24. Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

    cs.CL 2026-06 unverdicted novelty 7.0

    Chunk-Level Guided Generation uses off-the-shelf large LLMs to score fixed-length chunks from small models via likelihoods, matching trained PRM performance on math benchmarks without reward-model training.

  25. PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

    cs.LG 2026-05 unverdicted novelty 7.0

    PEFT-Arena reveals distinct stability-plasticity profiles across PEFT methods, with orthogonal finetuning achieving the best Pareto frontier under comparable parameter budgets, supported by weight-space spectral and a...

  26. ARBITER: Reasoning Trajectory Basins and Majority Vote Failures in Test-Time Sampling

    cs.LG 2026-05 unverdicted novelty 7.0

    ARBITER models reasoning trajectory basins in test-time sampling and uses model-internal signals to correct majority-vote failures, recovering part of the oracle gap on math benchmarks.

  27. EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation

    cs.AI 2026-05 unverdicted novelty 7.0

    EDGE-OPD adds guided rollouts and evidence masking to on-policy self-distillation, enabling successful learning of target identities where standard OPSD and RLSD fail.

  28. GS-QA: A Benchmark for Geospatial Question Answering

    cs.DB 2026-05 unverdicted novelty 7.0

    GS-QA is a new benchmark of 2,800 QA pairs on 28 templates using OSM and Wikipedia data to evaluate LLMs on spatial predicates, multi-source reasoning, and diverse answer types including distances and counts.

  29. Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines

    cs.CL 2026-05 unverdicted novelty 7.0

    Causal diagnosis identifies the routing module as bottleneck in LLM agents but prompt patching there degrades results due to linguistic co-adaptation, while upstream patching improves them.

  30. The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    In 1-3B instruction-tuned LMs on GSM8K, arithmetic CoT readout is dominated by positional copying of the trailing number before the answer delimiter, accounting for 54-92 percentage points of accuracy.

  31. LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning

    cs.AI 2026-05 unverdicted novelty 7.0

    LinAlg-Bench shows LLMs switch from execution errors to computational abandonment and structured fabrication at 4x4 matrix scale, indicating a working memory limit rather than knowledge gaps.

  32. Dynamic Chunking for Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    DCDM replaces positional blocks with learnable semantic chunks via differentiable Chunking Attention, yielding consistent gains over block and unstructured diffusion baselines up to 1.5B parameters.

  33. Learning from Language Feedback via Variational Policy Distillation

    cs.LG 2026-05 unverdicted novelty 7.0

    VPD frames language feedback learning as variational EM so the teacher policy refines itself via trust-region updates on outcomes while the student learns dense token distributions on its own rollouts, outperforming f...

  34. Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces

    cs.AI 2026-05 unverdicted novelty 7.0

    Language models produce overcomplete reasoning traces where on average 46% of steps can be removed while preserving the answer in 86% of cases, with necessity concentrated in the top three steps.

  35. Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling

    cs.LG 2026-05 unverdicted novelty 7.0

    DRATS derives a minimax objective from a feasibility formulation of MTRL to adaptively sample tasks with the largest return gaps, leading to better worst-task performance on MetaWorld benchmarks.

  36. Where Does Reasoning Break? Step-Level Hallucination Detection via Hidden-State Transport Geometry

    cs.CL 2026-05 unverdicted novelty 7.0

    Hallucination is detected as a transport-cost excursion in hidden-state trajectories, localized via contrastive PCA in a teacher model and distilled to a BiLSTM student.

  37. Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization

    cs.LG 2026-05 unverdicted novelty 7.0

    RDPO applies magnitude-aware quantile normalization and Mahalanobis whitening to decorrelate heterogeneous rewards in multi-objective RL, improving instruction following and writing quality on LongCat-Flash post-train...

  38. Test-Time Hinting for Black-Box Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Test-Time Hinting trains a hint generator to prepend contextual guidance to VLM prompts, improving accuracy on natural-image VQA benchmarks with generalization to unseen tasks and models.

  39. Reward-Weighted On-Policy Distillation with an Open Property-Equivalence Verifier for NL-to-SVA Generation

    cs.AR 2026-05 unverdicted novelty 7.0

    Reward-Weighted On-Policy Distillation with an open property-equivalence verifier produces a 7B model that surpasses prior SOTA on NL-to-SVA generation across pass@1/5/10 metrics.

  40. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

    cs.LG 2026-05 unverdicted novelty 7.0

    Distillation signals align better with ideal updates on incorrect student rollouts than correct ones, with optimal teacher context depending on student capacity and task.

  41. Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

    cs.LG 2026-05 unverdicted novelty 7.0

    Presents a likelihood-based benchmark for equation-suffix prediction in technical papers with controls to detect shortcut vulnerabilities in model forecasts.

  42. Equilibrium Residuals Expose Three Regimes of Matrix-Game Strategic Reasoning in Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    LLMs rely on semantic cues for matrix-game equilibria but can acquire approximate computation via residual training on small instances, with a Lipschitz proof enabling transfer to larger anonymous games.

  43. Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

    cs.AI 2026-05 unverdicted novelty 7.0

    Frontier LLMs achieve 95-100% accuracy on AMC/AIME problems but recover far fewer distinct valid strategies than human references, while collectively generating 50 novel strategies.

  44. AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization

    cs.AI 2026-05 unverdicted novelty 7.0

    AgentPSO evolves reusable multi-agent reasoning skills via PSO-inspired natural-language updates, outperforming static agents and test-time multi-agent baselines on math and general reasoning tasks with cross-benchmar...

  45. A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Massive activations originate in a specific ME Layer across LLM families; reducing their token rigidity via a targeted method boosts performance and mitigates attention sinks.

  46. A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

    cs.CL 2026-05 conditional novelty 7.0

    Massive activations first appear in a single ME Layer due to RMSNorm and FFN, remain invariant thereafter, and a simple softening method raises LLM performance while reducing attention sinks.

  47. KL for a KL: On-Policy Distillation with Control Variate Baseline

    cs.LG 2026-05 unverdicted novelty 7.0

    vOPD stabilizes on-policy distillation gradients by subtracting a closed-form per-token negative reverse KL baseline as a detached control variate, preserving unbiasedness while lowering variance and matching expensiv...

  48. Mathematical Reasoning via Intervention-Based Time-Series Causal Discovery Using LLMs as Concept Mastery Simulators

    cs.LG 2026-05 unverdicted novelty 7.0

    CIKA uses LLM-based interventions to probe causal effects of concepts on math reasoning success, achieving competitive results on benchmarks like Omni-MATH and GSM8K with a frozen 7B model.

  49. LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification

    cs.CL 2026-05 unverdicted novelty 7.0

    LaTER reduces LLM token usage 16-33% on reasoning benchmarks by exploring in latent space then switching to explicit CoT verification, with gains like 70% to 73.3% on AIME 2025 in the training-free version.

  50. Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control

    cs.LG 2026-05 unverdicted novelty 7.0

    Star Elastic trains N nested submodels in a single post-training job on a parent reasoning LLM, supporting elastic budget control that matches or exceeds independent baselines while cutting training compute by up to 360x.

  51. Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning

    stat.ML 2026-05 unverdicted novelty 7.0

    InfoTree casts intermediate state selection in tree search as monotone submodular maximization under fixed rollout budgets, yielding closed-form UUCB terms and lifting mixed-outcome ratios while outperforming flat GRP...

  52. Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

    cs.AI 2026-05 unverdicted novelty 7.0

    TraceLift trains reasoning planners with executor-grounded rewards that multiply a rubric-based reasoning quality score by measured uplift on a frozen executor, outperforming execution-only training on math and code b...

  53. Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

    cs.AI 2026-05 conditional novelty 7.0

    TraceLift trains reasoning planners with executor-grounded rewards that multiply a rubric-based reasoning quality score by measured performance uplift on a frozen executor, outperforming outcome-only training on math ...

  54. Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    Decoding-time use of process reward models for bias mitigation raises fairness scores by up to 0.40 on a bilingual benchmark while preserving fluency across four LLMs and extends to open-ended generation with low overhead.

  55. NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles

    cs.AI 2026-05 accept novelty 7.0

    NeuroState-Bench is a human-calibrated benchmark with 144 tasks and 306 side-query probes showing that commitment integrity in LLM agent profiles diverges from task success, with 31 of 32 profiles changing rank under ...

  56. RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

    cs.CL 2026-05 unverdicted novelty 7.0

    RMGAP benchmark shows state-of-the-art reward models reach at most 49.27% Best-of-N accuracy when forced to select responses matching diverse preferences.

  57. Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards

    cs.LG 2026-05 unverdicted novelty 7.0

    SGAC replaces reward-variance heuristics with a multi-feature learnable selector emphasizing output entropy, yielding 68% accuracy on Hendrycks MATH with Qwen2.5-Math-1.5B versus 64-66% baselines.

  58. BoostLoRA: Growing Effective Rank by Boosting Adapters

    cs.LG 2026-04 unverdicted novelty 7.0

    BoostLoRA grows effective adapter rank linearly via iterative boosting on hard examples with orthogonal low-rank updates, outperforming both single-shot ultra-low-rank adapters and full fine-tuning on math and code ta...

  59. Training Computer Use Agents to Assess the Usability of Graphical User Interfaces

    cs.CL 2026-04 unverdicted novelty 7.0

    uxCUA is a trained computer use agent that assesses GUI usability more accurately than larger models by learning to prioritize and execute important user interactions on labeled interface datasets.

  60. Don't Make the LLM Read the Graph: Make the Graph Think

    cs.AI 2026-04 conditional novelty 7.0

    Under a governance-capability gap where more capable AI carries greater authority exposure, improvements in AI capability can reduce optimal deployment in high-loss environments.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 279 Pith papers · 15 internal anchors

  1. [1]

    A General Language Assistant as a Laboratory for Alignment

    A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 ,

    work page internal anchor Pith review arXiv

  2. [2]

    Sparks of Artificial General Intelligence: Early experiments with GPT-4

    S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 ,

    work page internal anchor Pith review arXiv

  3. [3]

    Training Verifiers to Solve Math Word Problems

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Selection-inference: Ex- ploiting large language models for interpretable logical reasoning,

    A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712,

    work page arXiv

  5. [5]

    Reinforcement Learning with a Corrupted Reward Channel

    T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417 ,

    work page Pith review arXiv

  6. [6]

    L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti- mization. arXiv preprint arXiv:2210.10760 ,

    work page internal anchor Pith review arXiv

  7. [7]

    Measuring Mathematical Problem Solving With the MATH Dataset

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Large Language Models are Zero-Shot Reasoners

    T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 ,

    work page internal anchor Pith review arXiv

  9. [9]

    Solving Quantitative Reasoning Problems with Language Models

    A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra- masesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858,

    work page internal anchor Pith review arXiv

  10. [10]

    Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336,

    work page arXiv

  11. [11]

    On faithfulness and factuality in abstractive summarization.arXiv preprint arXiv:2005.00661, 2020

    J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661 ,

    work page arXiv 2005

  12. [12]

    WebGPT: Browser-assisted question-answering with human feedback

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332 ,

    work page internal anchor Pith review arXiv

  13. [13]

    14 M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114,

    work page internal anchor Pith review arXiv

  14. [14]

    GPT-4 Technical Report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Training language models to follow instructions with human feedback

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 ,

    work page internal anchor Pith review arXiv

  16. [16]

    J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034,

    work page arXiv

  17. [17]

    Stuhlm¨ uller and J

    A. Stuhlm¨ uller and J. Byun. Supervise process, not outcomes.https://ought. org/updates/2022-04-06-process ,

    work page 2022

  18. [18]

    Solving math word problems with process- and outcome-based feedback

    J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human pref- erences. arXiv preprint arXiv:1909.08593 ,

    work page internal anchor Pith review arXiv 1909

  22. [22]

    15 A MathMix Similar to Lewkowycz et al. (2022) we construct a large-scale dataset of high- quality math-relevant tokens for use in a lightweight pretraining stage, before finetuning on comparably smaller datasets like MATH and PRM800K. This dataset, which we call MathMix, has two main differences compared to the one used to train Minerva. First, it is sm...

    work page 2022