pith. sign in

arxiv: 2302.01318 · v1 · pith:DCS3XIDBnew · submitted 2023-02-02 · 💻 cs.CL

Accelerating Large Language Model Decoding with Speculative Sampling

Charlie Chen , Sebastian Borgeaud , Geoffrey Irving , Jean-Baptiste Lespiau , Laurent Sifre , John Jumper This is my paper

Pith reviewed 2026-05-11 07:23 UTC · model grok-4.3

classification 💻 cs.CL
keywords speculative samplinglanguage model decodingtransformer accelerationrejection samplinginference optimizationlarge language modelsChinchilla
0
0 comments X

The pith

Speculative sampling generates multiple tokens per transformer call by drafting sequences from a smaller model and verifying them with rejection sampling that matches the target distribution exactly.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an algorithm called speculative sampling to speed up decoding in large language models without changing the model or its outputs. A faster but weaker draft model proposes several candidate tokens in advance, which the large target model then scores in parallel. A modified rejection sampling step decides which proposals to accept or reject so the final token probabilities stay identical to what the target model would have produced on its own. The central observation is that scoring a short batch of continuations takes roughly the same wall-clock time as generating one token from the big model. On a 70-billion-parameter model this yields measured speedups of two to two-and-a-half times while sample quality remains unchanged.

Core claim

Speculative sampling enables the generation of multiple tokens from each call to the target transformer by drafting continuations with a smaller model and accepting or rejecting them via a modified rejection sampling procedure that matches the target distribution exactly within numerical precision. When benchmarked on Chinchilla, this yields a 2-2.5x speedup in distributed setups.

What carries the argument

The speculative sampling procedure, which interleaves parallel scoring of short draft sequences with a rejection-sampling rule that preserves the exact token probabilities of the target model.

If this is right

  • Decoding throughput on existing hardware increases by a factor of two to two-and-a-half for large models.
  • No changes to model weights or architecture are required to obtain the speedup.
  • The output distribution remains identical to standard autoregressive sampling, so downstream applications see no quality change.
  • The technique applies directly in distributed training and inference setups without additional synchronization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Pairing each large model with a lightweight draft model could become a standard deployment pattern for latency-sensitive services.
  • The same parallel-verification idea might apply to other autoregressive generation tasks such as image or audio synthesis once suitable draft models exist.
  • If the draft model can be made even cheaper, the effective cost per generated token could fall further without retraining the target model.

Load-bearing premise

Scoring several short continuations from the draft model in parallel takes about as long as sampling one token from the much larger target model.

What would settle it

A timing measurement on the 70B model showing that the observed wall-clock speedup drops below 1.5x, or a statistical test showing that token distributions produced by speculative sampling differ from those of standard sampling from the target model.

read the original abstract

We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 3 minor

Summary. The paper introduces speculative sampling, an algorithm to accelerate transformer decoding by generating multiple tokens per target model call. A smaller draft model proposes short candidate continuations, which are scored in parallel by the target model; a modified rejection sampling step then accepts or rejects tokens to ensure the output distribution exactly matches the target model's distribution (within hardware numerics). The authors report 2-2.5x decoding speedups on the 70B Chinchilla model in a distributed setup, with no model modifications and no degradation in sample quality.

Significance. If the central algorithmic claim holds, the work is significant for large-model inference: it delivers practical speedups on a real 70B model while exactly preserving the target distribution and requiring no retraining or architectural changes. The self-contained construction, absence of free parameters, and direct empirical validation on Chinchilla address the key latency assumption (parallel draft scoring comparable to one target forward pass) and make the technique immediately deployable. This combination of theoretical correctness and measured wall-clock gains on a production-scale model is a clear strength.

major comments (1)
  1. [§3] §3 (Algorithm): The modified rejection sampling procedure is asserted to preserve the target distribution exactly. A concise derivation or proof sketch showing that the per-token acceptance probabilities (especially when the draft proposes a sequence of length >1) yield the correct marginals under the target would allow independent verification of edge cases such as zero-probability proposals or numerical underflow.
minor comments (3)
  1. [§4] The experimental section would benefit from an explicit statement of the draft-model architecture and size relative to Chinchilla-70B, as well as the precise hardware configuration used for the distributed timing measurements.
  2. [§4] Figure 2 (speedup curves) lacks error bars or multiple-run statistics; adding these would clarify the stability of the reported 2-2.5x factor across different prompt lengths.
  3. A short related-work paragraph contrasting the method with prior speculative decoding or speculative execution techniques would help readers situate the novelty of the rejection-sampling modification.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their positive evaluation and constructive feedback on the algorithmic section. The request for a proof sketch is well-taken; we provide a concise derivation below and will incorporate it into the revised manuscript to support independent verification of the distribution-preserving property.

read point-by-point responses

  1. Referee: [§3] §3 (Algorithm): The modified rejection sampling procedure is asserted to preserve the target distribution exactly. A concise derivation or proof sketch showing that the per-token acceptance probabilities (especially when the draft proposes a sequence of length >1) yield the correct marginals under the target would allow independent verification of edge cases such as zero-probability proposals or numerical underflow.

    Authors: We agree an explicit sketch aids verification and will add the following to §3 of the revised manuscript. Let q be the draft distribution and p the target. For a token x ~ q the acceptance probability is min(1, p(x)/q(x)). The probability of accepting x is min(p(x), q(x)). Upon rejection (probability max(0, 1 - p(x)/q(x))), we resample from the residual r(y) ∝ max(0, p(y) - q(y)). The total output probability for any z is therefore min(p(z), q(z)) + max(0, p(z) - q(z)) = p(z). This identity holds conditionally at each position given an accepted prefix, so sequential application over a draft sequence of length K > 1 yields the exact target marginal at every step. For zero-probability proposals: tokens with q(x) = 0 are never proposed by the draft (softmax support is full in practice); we add a small epsilon in code to avoid division issues. Numerical underflow is handled via log-space ratios with clamping to [0,1], preserving the distribution up to floating-point precision as stated in the paper. revision: yes

Circularity Check

0 steps flagged

No significant circularity in algorithmic construction

full rationale

The paper introduces speculative sampling as an algorithmic method combining a draft model for generating candidate tokens with a modified rejection sampling procedure to preserve the exact target distribution. Correctness follows directly from the rejection sampling analysis (which is a standard technique and not derived from the paper's own fitted values or self-referential equations). The central benchmark result is an empirical measurement of speedup under the stated latency assumption, not a 'prediction' that reduces to inputs by construction. No self-citations are used to justify uniqueness theorems, no ansatzes are smuggled via prior work, and no parameters are fitted then relabeled as predictions. The derivation chain is self-contained and externally verifiable via the rejection sampler's properties.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on one key domain assumption about relative latencies and on the correctness of the rejection sampling construction; no free parameters or new entities are introduced.

axioms (1)
  • domain assumption Latency of parallel scoring of short continuations from a faster draft model is comparable to single-token sampling from the target model.
    This observation is explicitly invoked to justify why the method yields net speedup.

pith-pipeline@v0.9.0 · 5416 in / 1359 out tokens · 115938 ms · 2026-05-11T07:23:19.309503+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 8.0

    Mistletoe introduces a stealthy attack on speculative decoding that collapses acceleration by reducing average accepted length while preserving output semantics.

  2. The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits

    cs.LG 2026-05 unverdicted novelty 8.0

    Shared token budgets between visible chain-of-thought and answers create a coupling tax that makes non-thinking competitive on math benchmarks, with a truncation decomposition predicting the crossover and split budget...

  3. Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

    cs.DC 2026-07 unverdicted novelty 7.0

    Lynx partitions KV cache bits into anchor and residual streams for progressive transfer, enabling speculative decoding on partial data followed by verification to match BF16 accuracy at 4-bit-like TTFT.

  4. Certified Speculative Execution for Untrusted AI Agents

    cs.CR 2026-06 unverdicted novelty 7.0

    CGPA enables certified speculative execution of untrusted AI proposals in constrained sequential decisions via verifier rejection, conformal boundary gating, and solver deferral, yielding zero violations and regret wi...

  5. When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

    cs.LG 2026-06 unverdicted novelty 7.0

    Develops theory for acceptance in speculative decoding under greedy/relaxed/tree criteria, with exact KL certificates and margin bounds, evaluated on Qwen3 models.

  6. Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing

    cs.CL 2026-06 unverdicted novelty 7.0

    LBR performs token-level test-time scaling via local branch routing on hidden states, enabling end-to-end RL training and improving Pass@1 and Pass@32 on math benchmarks over CoT and RLVR baselines.

  7. Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing

    cs.CL 2026-06 unverdicted novelty 7.0

    Local Branch Routing (LBR) is a token-level framework for test-time scaling in language models that uses local branch hidden states for routing and supports end-to-end RL, showing gains in Pass@1 and Pass@32 on math r...

  8. MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling

    cs.AI 2026-06 unverdicted novelty 7.0

    MARS is a margin-adversarial stopping rule for parallel LLM test-time scaling that saves 25-47% tokens while matching full-budget majority-vote accuracy by learning trace switch probabilities and applying adversarial bounds.

  9. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    cs.CL 2026-06 accept novelty 7.0

    Naive samplers beat published diffusion and flow models on gen-PPL with incoherent output, proving the metric unsound and motivating distributional evaluation suites.

  10. WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

    cs.LG 2026-06 unverdicted novelty 7.0

    WhiFlash introduces token-level cross-paradigm routing between autoregressive and diffusion drafting models, with cache optimizations, to raise acceptance lengths and deliver up to 69.6% throughput gains over EAGLE-3.

  11. Parallel Jacobi Decoding for Fast Autoregressive Image Generation

    cs.CV 2026-06 conditional novelty 7.0

    Parallel Jacobi Decoding accelerates autoregressive image models 4.8x-6.4x by using 2D spatial draft expansion and adjusted attention masks while keeping generation quality competitive.

  12. D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

    cs.DC 2026-06 unverdicted novelty 7.0

    D^2SD uses two diffusion drafters in a prefix tree structure with confidence scores to select and recover alternative draft sequences, achieving higher acceptance rates in speculative decoding.

  13. Speculative Sampling For Faster Molecular Dynamics

    cs.LG 2026-06 unverdicted novelty 7.0

    LSD extends speculative sampling to second-order Langevin dynamics, achieving 3-9x speedup in MD while exactly sampling from the target distribution without relative error.

  14. Cost-Aware Diffusion Draft Trees for Speculative Decoding

    cs.CL 2026-06 unverdicted novelty 7.0

    CaDDTree jointly selects tree structure and budget to maximize expected tokens per unit time in speculative decoding, proving unimodality under convex verification cost and matching oracle DDTree performance on Qwen models.

  15. Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

    cs.CL 2026-06 unverdicted novelty 7.0

    Chunk-Level Guided Generation uses off-the-shelf large LLMs to score fixed-length chunks from small models via likelihoods, matching trained PRM performance on math benchmarks without reward-model training.

  16. OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification

    cs.LG 2026-05 unverdicted novelty 7.0

    OmniOPD replaces token-level logit matching in on-policy distillation with Monte Carlo chunk-level semantic verification and a peak-entropy scheduler.

  17. TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding

    cs.AI 2026-05 unverdicted novelty 7.0

    TAPS converts diffusion marginal probabilities into path-conditioned acceptance estimates to select prefix-closed subtrees under a fixed verification budget, achieving up to 7.9x end-to-end speedup over autoregressive...

  18. EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

    cs.LG 2026-05 unverdicted novelty 7.0

    EST-PRM stress-tests five PRM models on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench using three label-preserving transformations and reports model-specific vulnerability patterns.

  19. Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting

    cs.LG 2026-05 unverdicted novelty 7.0

    BASTION is a budget-aware speculative decoding framework with adaptive tree-structured block diffusion drafting that reports up to 6.61x speedup and 39% improvement over block-diffusion baselines.

  20. Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

    cs.LG 2026-05 unverdicted novelty 7.0

    Graft combines pruning and retrieval in a sequential mechanism to build hybrid draft trees for speculative decoding, delivering up to 5.41× speedup and 21.8% better average speedup than EAGLE-3 on large models.

  21. SSV: Sparse Speculative Verification for Efficient LLM Inference

    cs.OS 2026-05 unverdicted novelty 7.0

    SpecSA is a sparse speculative-verification framework that integrates speculative decoding and dynamic sparse attention to achieve up to 3.49x end-to-end throughput and 6.86x kernel speedups on H100 GPUs for long-cont...

  22. SNLP: Layer-Parallel Inference via Structured Newton Corrections

    cs.LG 2026-05 unverdicted novelty 7.0

    SNLP enables layer-parallel Transformer inference by replacing sequential layer execution with structured Newton corrections and SNLP-aware training regularization, yielding up to 2.3x wall-clock speedup on 0.5B model...

  23. Skim: Speculative Execution for Fast and Efficient Web Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    Skim profiles website patterns offline to enable fast-path speculative execution for web agents, cutting median cost by 1.9x and latency by 33.4% with no accuracy loss on benchmarks.

  24. PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 7.0

    PSD is a training-free framework that jointly optimizes spatial unmasking and temporal speculative decoding in diffusion LLMs to reach up to 5.5x tokens per forward pass while preserving accuracy comparable to greedy ...

  25. Factorization-Error-Free Discrete Diffusion Language Model via Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 7.0

    FeF-DLLM achieves factorization-error-free generation in discrete diffusion language models via prefix-conditioned posterior factorization and speculative decoding, delivering 5.04 pp higher accuracy and 3.86x faster ...

  26. Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 7.0

    Mistletoe is a stealthy attack that collapses the speedup of speculative decoding by reducing average accepted length τ without changing output semantics or perplexity.

  27. SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

    cs.LG 2026-05 unverdicted novelty 7.0

    SlimSpec replaces the standard LM-head in draft models with a low-rank version to deliver 4-5x faster speculative decoding while preserving full vocabulary and competitive acceptance rates.

  28. Test-Time Speculation

    cs.CL 2026-05 unverdicted novelty 7.0

    Test-Time Speculation adapts draft models online via target-model verifications to sustain high acceptance lengths during long LLM generations.

  29. Future Validity is the Missing Statistic: From Impossibility to $\Phi$-Estimation for Grammar-Faithful Speculative Decoding

    cs.LG 2026-05 unverdicted novelty 7.0

    Speculative decoding under local grammar masking samples from the projected distribution μ^proj instead of the grammar-conditional μ*, and the future-validity function Φ corrects it via a Doob transform to achieve exa...

  30. SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

    cs.CL 2026-05 unverdicted novelty 7.0

    SpecBlock achieves 8-13% higher mean speedup than EAGLE-3 at 44-52% drafting cost via block-iterative drafting with hidden-state inheritance, dynamic rank-head branching, valid-prefix masking, and optional cost-aware ...

  31. Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL

    cs.LG 2026-05 conditional novelty 7.0

    A one-parameter early-termination gate based on mean pairwise prefix edit distance reduces wall-clock time by 10.7% and raises held-out success by 2.5 pp in GRPO on ALFWorld by cutting zero-advantage batch dilution.

  32. UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 7.0

    UniVer frames tree-based speculative decoding as conditional optimal transport, proving it is lossless with optimal acceptance rates and delivering 4.2-8.5% longer accepted sequences than standard rejection sampling.

  33. Component-Aware Self-Speculative Decoding in Hybrid Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Component-aware self-speculative decoding achieves high acceptance rates in parallel hybrid models like Falcon-H1 but fails in sequential ones like Qwen3.5, with the gap tied to how components are integrated.

  34. An Empirical Study of Speculative Decoding on Software Engineering Tasks

    cs.SE 2026-04 unverdicted novelty 7.0

    Speculative decoding accelerates LLM inference on SE tasks without accuracy loss, with model-based methods suiting code generation and model-free methods suiting repository-level repair and editing.

  35. FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving

    cs.DC 2026-04 unverdicted novelty 7.0

    FASER delivers up to 53% higher throughput and 1.92x lower latency in dynamic LLM serving by adjusting speculative lengths per request, early pruning of rejects, and overlapping draft/verification phases via frontiers.

  36. Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing

    cs.CL 2026-04 unverdicted novelty 7.0

    Copy-as-Decode recasts LLM editing as grammar-constrained decoding over copy and generate primitives, delivering closed-form upper-bound speedups of 13x pooled on editing benchmarks via parallel prefill without any training.

  37. WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference

    cs.IT 2026-04 unverdicted novelty 7.0

    WISV uses a channel-aware semantic acceptance policy on hidden representations to boost accepted sequence length by up to 60.8% and cut interaction rounds by 37.3% in distributed speculative decoding, with under 1% ac...

  38. Speculative Decoding for Autoregressive Video Generation

    cs.CV 2026-04 conditional novelty 7.0

    A training-free speculative decoding method for block-based autoregressive video diffusion uses a quality router on worst-frame ImageReward scores to accept drafter proposals, achieving up to 2.09x speedup at 95.7% qu...

  39. From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning

    cs.CL 2026-04 unverdicted novelty 7.0

    SpecGuard adds step-level verification to speculative decoding via attention grounding and log-probability scores, yielding 3.6% higher accuracy and 11% lower latency on reasoning benchmarks.

  40. MARS: Enabling Autoregressive Models Multi-Token Generation

    cs.CL 2026-04 unverdicted novelty 7.0

    MARS fine-tunes autoregressive models to predict multiple tokens per step via continued training on instruction data, achieving 1.5-1.7x throughput while matching baseline accuracy and supporting real-time speed adjustment.

  41. Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling

    cs.LG 2026-04 unverdicted novelty 7.0

    Cactus uses constrained optimization to guarantee bounded divergence from the verifier LLM distribution during speculative sampling, raising acceptance rates without the distortion seen in typical acceptance sampling.

  42. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    cs.LG 2026-03 conditional novelty 7.0

    EvoESAP uses evolutionary search guided by a speculative-decoding-inspired ESAP metric to discover non-uniform layer-wise sparsity allocations for MoE expert pruning, improving generation accuracy up to 19.6% at 50% sparsity.

  43. When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

    cs.LG 2026-02 conditional novelty 7.0

    Aurora unifies speculative decoder training and serving via asynchronous RL on inference traces, delivering 1.5x day-0 speedup on frontier models and 1.25x adaptation gains on distribution shifts.

  44. Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices

    cs.DC 2025-12 conditional novelty 7.0

    Vec-LUT delivers up to 4.2x speedup over prior LUT methods for parallel ultra-low-bit LLM inference on edge devices by unifying lookups across tokens and adding cache-aware tensor layouts.

  45. VVS: Accelerating Speculative Decoding for Visual Autoregressive Generation via Partial Verification Skipping

    cs.CV 2025-11 conditional novelty 7.0

    VVS accelerates visual AR image generation by partially skipping verifications in speculative decoding, achieving 2.8x fewer target forward passes while preserving competitive quality.

  46. Efficient Autoregressive Inference for Transformer Probabilistic Models

    stat.ML 2025-10 conditional novelty 7.0

    A causal autoregressive buffer enables efficient batched autoregressive sampling and joint density evaluation in set-based transformer models by caching context and attending to prior predictions.

  47. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

    cs.CL 2024-12 unverdicted novelty 7.0

    o1-like models overthink easy tasks; self-training reduces compute use without accuracy loss on GSM8K, MATH500, GPQA, and AIME.

  48. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

    cs.CV 2024-06 conditional novelty 7.0

    Scaled vanilla autoregressive models based on Llama achieve 2.18 FID on ImageNet 256x256 image generation, beating popular diffusion models without visual inductive biases.

  49. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    cs.LG 2024-01 conditional novelty 7.0

    Medusa augments LLMs with multiple decoding heads and tree-based attention to predict and verify several tokens in parallel, yielding 2.2-3.6x inference speedup via two fine-tuning regimes.

  50. Fast Inference from Transformers via Speculative Decoding

    cs.LG 2022-11 accept novelty 7.0

    Speculative decoding accelerates exact sampling from large autoregressive models by 2-3x on T5-XXL by running smaller approximation models in parallel to propose token sequences that the large model then verifies in b...

  51. Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters

    cs.AI 2026-07 unverdicted novelty 6.0

    Accept-Until-Fail training improves average accepted block length in speculative decoding from 2.40 to 2.61 by limiting cross-entropy support to the drafter's first predicted failure point.

  52. Diffusion-GR2: Diffusion Generative Reasoning Re-ranker

    cs.IR 2026-07 unverdicted novelty 6.0

    Diffusion-GR2 converts an AR reasoning re-ranker to block-diffusion via CFT, OPD, and RL stages, recovering near-parity accuracy on Amazon Beauty with 2.4-3.5x decode speedup.

  53. Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning

    cs.CL 2026-06 unverdicted novelty 6.0

    PRP introduces proactive routing via Draft Rating Learning and Joint Rating Learning to route queries early between draft and target models for efficient multimodal reasoning.

  54. Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path

    cs.LG 2026-06 unverdicted novelty 6.0

    Speculative pre-positioning decodes stateful sessions ahead with the target model to enable near-constant-time responses from cached distributions or pre-paid deltas at 87% precision for capable models.

  55. Depth Exploration for LLM Decoding

    cs.LG 2026-06 unverdicted novelty 6.0

    DEX replaces single-depth selection with parallel exploration over multiple candidate depths, committing the final-depth token while collapsing reusable states to reduce per-token computation.

  56. To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair

    cs.SE 2026-06 unverdicted novelty 6.0

    Empirical analysis of LLM repair agents shows execution provides concentrated benefits, with restrictions causing only a 1.25 pp non-significant drop in resolve rate while cutting token and time costs.

  57. HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction

    cs.LG 2026-06 unverdicted novelty 6.0

    HyperDFlash reports higher accepted draft lengths and speedups versus MTP and DFlash baselines by aligning drafting with MHC residual streams via gated reduction and KL distillation.

  58. Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off

    cs.DC 2026-06 unverdicted novelty 6.0

    Analytical bounds demonstrate edge-cloud DSD improves latency only in low-RTT regimes and primarily benefits multi-tenant throughput via draft offloading under client overlap.

  59. P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling

    cs.CV 2026-06 unverdicted novelty 6.0

    P-MTP uses progressive curriculum loss and confidence-gated dynamic drafting to scale look-ahead depth in multi-token prediction, claiming up to 5x speedup with negligible accuracy loss in document parsing.

  60. Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

    cs.CL 2026-06 unverdicted novelty 6.0

    Dustin reports 27.85x self-attention and 9.17x end-to-end speedups at 32k length on Qwen2.5-72B using draft-augmented sparse verification with negligible accuracy loss on PG-19 and LongBench.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · cited by 158 Pith papers · 7 internal anchors

  1. [1]

    Vivit: Avideovisiontransformer

    A.Arnab, M.Dehghani, G.Heigold, C.Sun, M.Lucic, andC.Schmid. Vivit: Avideovisiontransformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816–6826. IEEE Computer Society,

    work page 2021

  2. [2]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  3. [4]

    URLhttps://arxiv.org/abs/2107.03374. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [5]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339,

    work page internal anchor Pith review arXiv

  5. [6]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer,G.Heigold,S.Gelly,etal. Animageisworth16x16words: Transformersforimagerecognition at scale.arXiv preprint arXiv:2010.11929,

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  6. [7]

    T. Ge, H. Xia, X. Sun, S. Chen, and F. Wei. Lossless acceleration for seq2seq generation with aggressive decoding. ArXiv, abs/2205.10350,

    work page arXiv

  7. [8]

    Training Compute-Optimal Large Language Models

    8 Accelerating Large Language Model Decoding with Speculative Sampling J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J.Welbl, A.Clark, etal. Trainingcompute-optimallargelanguagemodels. arXivpreprint arXiv:2203.15556,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [9]

    X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. TinyBERT: Distilling BERT for natural language understanding. InFindings of the Association for Computational Linguis- tics: EMNLP 2020, pages 4163–4174, Online, Nov

    work page 2020

  9. [10]

    Generating radiology reports via memory-driven transformer

    Association for Computational Linguis- tics. doi: 10.18653/v1/2020.findings-emnlp.372. URL https://aclanthology.org/2020. findings-emnlp.372. Y. Kim and A. M. Rush. Sequence-level knowledge distillation.CoRR, abs/1606.07947,

    work page doi:10.18653/v1/2020 2020

  10. [11]

    URL http://arxiv.org/abs/1606.07947. Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. ArXiv, abs/2211.17192,

    work page Pith review arXiv

  11. [12]

    The depth-to-width interplay in self-attention, 2021

    Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua. The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467,

    work page arXiv 2006

  12. [13]

    Narayan, S

    S. Narayan, S. B. Cohen, and M. Lapata. Don’t give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium, Oct.-Nov

    work page 2018

  13. [14]

    Cohen, and Mirella Lapata

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1206. URL https://aclanthology.org/D18-1206. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference.arXiv preprint arXiv:2211.05102,

    work page doi:10.18653/v1/d18-1206

  14. [15]

    J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446,

    work page internal anchor Pith review arXiv

  15. [16]

    V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  16. [18]

    URL http://arxiv.org/abs/1911.02150. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053,

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  17. [20]

    URLhttp://arxiv.org/abs/1811.03115. A. Wiggers and E. Hoogeboom. Predictive sampling with forecasting autoregressive models. In H. D. III and A. Singh, editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 10260–10269. PMLR, 13–18 Jul

    work page Pith review arXiv

  18. [21]

    Advances in Neural Information Processing Systems , year =

    URL https://proceedings.mlr.press/v119/wiggers20a.html. 9 Accelerating Large Language Model Decoding with Speculative Sampling Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.arXiv preprint arXiv:2206.01861,

    work page arXiv