Pith. sign in

REVIEW 1 major objections 2 minor 60 cited by

For compute-optimal LLM training, scale model size and training tokens equally.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-10 15:55 UTC pith:BF2N5BFY

load-bearing objection The paper shows that LLMs have been undertrained on data and that equal scaling of parameters and tokens is compute-optimal, backed by a large sweep and a direct validation run. the 1 major comments →

arxiv 2203.15556 v1 pith:BF2N5BFY submitted 2022-03-29 cs.CL cs.LG

Training Compute-Optimal Large Language Models

Jordan Hoffmann , Sebastian Borgeaud , Arthur Mensch , Elena Buchatskaya , Trevor Cai , Eliza Rutherford , Diego de Las Casas , Lisa Anne Hendricks
show 14 more authors
Johannes Welbl Aidan Clark Tom Hennigan Eric Noland Katie Millican George van den Driessche Bogdan Damoc Aurelia Guy Simon Osindero Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre
This is my paper
classification cs.CL cs.LG
keywords compute optimal trainingscaling lawslarge language modelstransformerChinchillamodel sizetraining tokens
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines the best way to split a fixed compute budget between the number of model parameters and the amount of training data for transformer language models. Through experiments with over 400 models of varying sizes and data volumes, it concludes that current large models are undertrained due to keeping data fixed while increasing parameters. The central result is that optimal performance comes from scaling model size and tokens in tandem, doubling both when compute doubles. This approach yields models that outperform much larger ones on standard benchmarks while using fewer resources for inference.

Core claim

By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.

What carries the argument

The scaling relation where optimal model size N and data D are proportional under fixed compute budget, derived from fitting loss as a function of N and D.

Load-bearing premise

The parametric form of the scaling law fitted to models up to 16B parameters and 500B tokens holds for larger scales.

What would settle it

Training a model at a larger compute budget using the equal-scaling prediction and finding that its loss or downstream performance is worse than a model using a different N-to-D ratio.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Chinchilla achieves higher accuracy on benchmarks like MMLU with 70B parameters than larger models using the same compute.
  • Smaller optimal models reduce the compute needed for fine-tuning and inference.
  • Future training runs should increase data proportionally to model size rather than fixing data size.
  • Undertrained models can be improved by adding more data instead of just more parameters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Data collection efforts will need to grow in tandem with model scaling to maintain optimal performance.
  • Similar optimal scaling ratios may apply to other domains like vision or multimodal models.
  • This challenges the prior trend of ever-larger models trained on fixed amounts of data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper investigates the optimal allocation of compute between model size (N) and training tokens (D) for transformer language models. By training over 400 models spanning 70M to 16B parameters and 5B to 500B tokens, the authors fit a parametric scaling law for validation loss and conclude that compute-optimal training requires scaling N and D equally. They validate the prediction by training Chinchilla (70B parameters, ~1.4T tokens) under the same compute budget as Gopher (280B parameters, 300B tokens); Chinchilla outperforms Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG on a wide range of downstream tasks, including achieving 67.5% average accuracy on MMLU.

Significance. If the derived scaling laws and equal N/D scaling hold, the work is highly significant: it provides empirical evidence that many recent large models are undertrained and demonstrates a practical method for more efficient compute allocation that reduces inference and fine-tuning costs. The strength lies in the scale of the experimental sweep (>400 models) combined with direct out-of-sample validation via the Chinchilla training run, which moves the claim beyond pure curve-fitting.

major comments (1)
  1. [Scaling Laws section (around the derivation of optimal N and D)] The scaling-law fit is performed on models up to 16B parameters; the Chinchilla prediction extrapolates both in N (to 70B) and in D (to 1.4T tokens). While the successful Chinchilla run provides supporting evidence, the manuscript should quantify the uncertainty in the predicted optimum arising from variance in the fitted coefficients (A, B, α, β) and discuss whether alternative functional forms for L(N,D) would materially change the equal-scaling conclusion.
minor comments (2)
  1. [Abstract] Abstract contains a typographical error: '4× more more data' should be '4× more data'.
  2. [Figures 2–5 and associated text] Several figures (e.g., loss-vs-compute curves and downstream-task comparisons) would benefit from explicit error bars or shaded uncertainty regions to convey variability across the 400-model sweep.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their positive evaluation of the work and for the constructive suggestion regarding uncertainty quantification. We address the major comment below and will incorporate the requested analysis in the revised manuscript.

read point-by-point responses

  1. Referee: [Scaling Laws section (around the derivation of optimal N and D)] The scaling-law fit is performed on models up to 16B parameters; the Chinchilla prediction extrapolates both in N (to 70B) and in D (to 1.4T tokens). While the successful Chinchilla run provides supporting evidence, the manuscript should quantify the uncertainty in the predicted optimum arising from variance in the fitted coefficients (A, B, α, β) and discuss whether alternative functional forms for L(N,D) would materially change the equal-scaling conclusion.

    Authors: We agree that an explicit quantification of uncertainty in the extrapolated optimum would strengthen the presentation. In the revised manuscript we will add a short subsection to the Scaling Laws section that reports bootstrap confidence intervals on the fitted coefficients A, B, α, and β (obtained by resampling the >400 training runs with replacement). These intervals will be propagated through the closed-form expression for the optimal N*(C) and D*(C) to give a range of plausible optima at the compute budget used for Chinchilla. Regarding alternative functional forms, we will include a brief discussion showing that the equal-scaling conclusion is robust: the optimum arises from balancing the two power-law terms, so modest changes to the exponents or the use of a multiplicative interaction term leave the scaling exponents for N and D with respect to compute essentially unchanged (both remain close to 0.5). The successful Chinchilla training run, which lies well outside the fitted regime, already provides direct empirical support for the predicted allocation. revision: yes

Circularity Check

0 steps flagged

No significant circularity; out-of-sample validation confirms scaling prediction

full rationale

The derivation fits a parametric loss function L(N, D) to empirical results from over 400 models (70M–16B parameters, 5B–500B tokens), derives the compute-optimal relation by minimizing under the constraint C ≈ 6ND, and directly tests the resulting prediction by training Chinchilla (70B parameters, ~1.4T tokens) under Gopher's compute budget. Chinchilla's superior downstream performance constitutes independent falsification outside the fitting set. No step reduces to self-definition, fitted-input renaming, or load-bearing self-citation; the functional form and optimum are externally validated rather than tautological.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The optimal allocation is obtained by fitting a parametric loss function L(N,D) to the 400-model results and minimizing under a compute constraint; the functional form and fitted coefficients are the main unverified inputs.

free parameters (1)
  • Scaling law coefficients
    Parameters (A, B, alpha, beta, E) in the assumed loss scaling form L(N, D) = E + A/N^alpha + B/D^beta fitted to the empirical results of the 400 models.
axioms (1)
  • domain assumption Loss follows a power-law dependence on model size N and data D
    The functional form used to fit data and derive the equal-scaling optimum.

pith-pipeline@v0.9.0 · 5617 in / 1398 out tokens · 67828 ms · 2026-05-10T15:55:30.872186+00:00 · methodology

0 comments
read the original abstract

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. An Open-Source Training Dataset for Foundation Models for Black-box Optimization

    cs.LG 2026-05 unverdicted novelty 8.0

    BBO-Pile is the first large-scale open dataset of real optimization trajectories used to train and scale foundation models that imitate black-box optimization methods.

  2. The Economics of Model Collapse: Equilibrium, Welfare, and Optimal Provenance Subsidies in Synthetic Data Markets

    econ.GN 2026-05 unverdicted novelty 8.0

    Introduces the Synthetic Data Contamination Equilibrium and derives closed-form optimal provenance subsidies s* = KL(q||p)/(2 kappa) plus watermark strengths to mitigate model collapse, validated by OLS matching struc...

  3. The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

    cs.LG 2026-04 unverdicted novelty 8.0

    Transformer weight spectra exhibit transient compression waves that propagate layer-wise, persistent non-monotonic depth gradients in power-law exponents, and Q/K-V asymmetry, with the spectral exponent alpha predicti...

  4. Large Language Diffusion Models

    cs.CL 2025-02 unverdicted novelty 8.0

    LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.

  5. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders

    cs.IR 2024-03 unverdicted novelty 8.0

    BLaIR is a new benchmark and 570M-review dataset showing that LLM performance rankings on recommendation tasks have little correlation with rankings on general embedding benchmarks like MTEB.

  6. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

    cs.CL 2023-09 unverdicted novelty 8.0

    Promptbreeder evolves both task prompts and the mutation prompts that improve them using LLMs, outperforming Chain-of-Thought and Plan-and-Solve on arithmetic and commonsense reasoning benchmarks.

  7. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

    cs.CL 2023-05 conditional novelty 8.0

    Tiny language models under 10M parameters trained on a synthetic children's story dataset generate fluent, consistent, multi-paragraph English text with near-perfect grammar and reasoning.

  8. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    cs.CL 2023-04 accept novelty 8.0

    Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.

  9. Editing Models with Task Arithmetic

    cs.LG 2022-12 accept novelty 8.0

    Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.

  10. Teaching Models to Express Their Uncertainty in Words

    cs.CL 2022-05 unverdicted novelty 8.0

    GPT-3 can learn to express well-calibrated uncertainty about its answers using natural language phrases rather than logits.

  11. Will Scaling Improve Social Simulation with LLMs?

    cs.CL 2026-07 conditional novelty 7.0

    Scaling improves LLM social simulation fidelity in most opinion and behavior tasks but not for human cognitive bias calibration or low-resource domains.

  12. HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

    cs.LG 2026-07 unverdicted novelty 7.0

    HERMES provides a reusable hierarchical labeling substrate for pre-training data that reveals granularity-specific effects in data mixing rules during model training.

  13. SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    SemRF supplies fixed semantic anchors and pseudo-inverse tying to produce stable coordinates for residual dynamics, Voronoi traces, and minimum-action canonical paths that link to parameter efficiency under controlled...

  14. Automated High-Precision Extraction and Forensic Verification of Data-Bearing Vector Figures

    cs.CR 2026-06 unverdicted novelty 7.0

    An automated system extracts data from vector figures with bit-exact recovery for matplotlib markers, maps precision across renderers and formats, and provides injective verification via re-rendering certificates.

  15. CORTEX: High-Quality Cross-Domain Organization of Web-Scale Corpora through Ontological Corpus Graph

    cs.CL 2026-06 unverdicted novelty 7.0

    Cortex uses an Ontological Corpus Graph to structure web-scale corpora, creating a refined 24.14B-token corpus and a new benchmark validated on eight LLMs.

  16. Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

    cs.LG 2026-06 unverdicted novelty 7.0

    During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.

  17. The Inference-Compute Frontier and a Latency-Efficient Architecture for Limit Order Book Prediction

    cs.LG 2026-06 unverdicted novelty 7.0

    Empirical power-law frontier between predictive loss and structural forward work in LOB models extrapolates to held-out high-compute architectures with R²=0.941, motivating FastBiNLOB which exceeds SOTA macro-F1 at lo...

  18. Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

    stat.ME 2026-06 unverdicted novelty 7.0

    KCas transfers student-selected smoothing parameters to full-sample teacher models via asymptotic scaling laws in smoothing splines and kernel methods, cutting computation while retaining performance guarantees.

  19. Distilling a Modular Reservoir Through a Genomic Bottleneck

    cs.NE 2026-06 unverdicted novelty 7.0

    Hypernetworks distill modular reservoir connectivity via a genomic bottleneck to generate sparse recurrent networks solving difficult temporal tasks with minimal training and maintained robustness.

  20. The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders

    cs.IR 2026-06 unverdicted novelty 7.0

    Transformer recommenders amplify popularity bias via spectral collapse when scaled; SPRINT constrains attention column-sums and feed-forward spectral norms to improve fairness and scaling behavior.

  21. Dual Dimensionality for Local and Global Attention

    cs.CL 2026-06 unverdicted novelty 7.0

    Distance-Adaptive Representation (DAR) keeps full KV dimensionality inside a local window and reduces it to 1/4 outside, matching full-dimensional baselines on pretraining (70M-410M) and 1B-scale fine-tuning while uni...

  22. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    math.OC 2026-06 unverdicted novelty 7.0

    OptMuon combines orthogonalized momentum with closed-loop adaptation to achieve noise-adaptive convergence rates that automatically become near-optimal deterministic first-order rates without retuning when noise vanishes.

  23. Complexity-Balanced Diffusion Splitting

    cs.CV 2026-06 unverdicted novelty 7.0

    CBS partitions the diffusion timeline into segments of equal approximation burden via Dirichlet energy and trajectory acceleration monitors estimated by an auxiliary model, yielding higher synthesis quality at fixed p...

  24. Scaling Laws for Behavioral Foundation Models over User Event Sequences

    cs.LG 2026-06 unverdicted novelty 7.0

    Across 600 runs from 10^15 to 10^19 FLOPs, behavioral models show a 2% embedder is compute-optimal at all scales, training is data-heavy at low compute, and optimal negatives increase with budget until memory-limited.

  25. Spectral Scaling Laws of Muon

    cs.LG 2026-06 unverdicted novelty 7.0

    Muon momentum matrices show layer-dependent power-law scaling of stabilized singular value quantiles with model size from 77M to 2.8B parameters.

  26. Forget Attention: Importance-Aware Attention Is All You Need

    cs.AI 2026-06 unverdicted novelty 7.0

    SISA adds an SSM importance term inside the attention score and runs the full operation as one SDPA call on augmented Q/K vectors, reporting better LAMBADA and perfect NIAH at small scale.

  27. Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation

    cs.LG 2026-05 unverdicted novelty 7.0

    IRSL applies IRT to reduce scaling law estimation from O(M×N) to O(M+N) parameters, enabling reliable estimates with only 50 questions per benchmark after calibration and generalizable ability scores across related be...

  28. D$^3$: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training

    cs.CL 2026-05 unverdicted novelty 7.0

    D³ introduces a dynamic directional graph-constrained framework that models sample interactions via loss dependencies to derive an optimized training sequence for LLMs.

  29. How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions

    cs.LG 2026-05 unverdicted novelty 7.0

    Vendi Score and scaling-law objectives belong to the class of matrix spectral functions, which are submodular, enabling efficient greedy selection of training data that outperforms random subsets in predicting held-ou...

  30. Neural Scaling Laws for Jet Generation

    hep-ph 2026-05 unverdicted novelty 7.0

    Scaling laws hold logarithmically for model size in autoregressive jet generation, with next-token loss correlating to physical metrics via sliced Wasserstein distance, but show weaker scaling for dataset size and com...

  31. FTibSuite: A Comprehensive Resource Suite for Tibetan Vision-Language Modeling

    cs.CV 2026-05 unverdicted novelty 7.0

    FTibSuite provides human-verified multimodal corpora, Tibetan-adapted benchmarks with quality controls, and a baseline VLM showing gains on tasks like MMBench while preserving Chinese capabilities.

  32. LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    LLMForge is a NAS framework with Infinite-Head Attention, a Forge-Former surrogate, and Forge-DSE engine that discovers hardware-specific architectures for edge language models, yielding variants with improved accurac...

  33. Scale-Dependent Collective Adaptation in Self-Amending LLM Societies: A Cross-Family Study of Emergent Governance

    nlin.AO 2026-05 unverdicted novelty 7.0

    LLM societies in Nomic show non-monotonic collective adaptation peaking at mid-scales, with smaller models rule-inert and larger ones restrictive.

  34. Do Language Models Align with Brains? Prediction Scores Are Not Enough

    q-bio.NC 2026-05 unverdicted novelty 7.0

    Language model representations fail all L-PACT alignment gates once controls explain the apparent predictive and relational effects.

  35. MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving

    cs.RO 2026-05 unverdicted novelty 7.0

    MindVLA-U1 introduces a unified streaming VLA with shared backbone, framewise memory, and language-guided action diffusion that surpasses human drivers on WOD-E2E planning metrics.

  36. Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

    cs.CL 2026-05 unverdicted novelty 7.0

    Temporarily reducing the learning rate on upper-layer query and key projections during early GPT pretraining prevents premature attention specialization and improves model performance.

  37. Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

    stat.ML 2026-05 unverdicted novelty 7.0

    In extensive-width networks, features are recovered sequentially through sharp phase transitions, yielding an effective width k_c that unifies Bayes-optimal generalization error scaling as Θ(k_c d / n).

  38. Simply Stabilizing the Loop via Fully Looped Transformer

    cs.LG 2026-05 unverdicted novelty 7.0

    Fully Looped Transformer stabilizes looped training up to 12 iterations via distributed inter-loop signals and attention injection, improving downstream performance by up to 13.2%.

  39. Active Testing of Large Language Models via Approximate Neyman Allocation

    cs.AI 2026-05 unverdicted novelty 7.0

    Proposes surrogate semantic entropy stratification followed by approximate Neyman allocation for active testing of LLMs on generative benchmarks, reporting up to 28% MSE reduction and 22.9% average budget savings vers...

  40. How Much is Brain Data Worth for Machine Learning?

    cs.AI 2026-05 conditional novelty 7.0

    Brain data is worth a variable number of task samples depending on task-brain alignment, noise levels, and latent dimension, with conditions under which it also improves robustness to test distribution shift.

  41. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 unverdicted novelty 7.0

    MLS-Bench is a benchmark with 140 tasks that evaluates AI agents on inventing generalizable and scalable ML methods, finding they lag human performance especially in insight-driven invention rather than tuning.

  42. DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards

    cs.LG 2026-05 unverdicted novelty 7.0

    DUET improves RLVR by allocating tokens across both prompt selection and rollout length, outperforming full-budget baselines even when using only half the tokens.

  43. Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

    cond-mat.dis-nn 2026-05 unverdicted novelty 7.0

    A two-level DMFT predicts width-consistent outlier escape and hyperparameter transfer under μP in deep networks, with bulk restructuring dominating for tasks with many outputs.

  44. On the Invariance and Generality of Neural Scaling Laws

    cs.LG 2026-05 unverdicted novelty 7.0

    Neural scaling laws are invariant under bijective data transformations and change predictably with information resolution ρ under non-bijective transformations, enabling cross-domain transport of fitted exponents.

  45. Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

    cs.AI 2026-05 unverdicted novelty 7.0

    RL training on more expressive logical tasks follows a steeper power-law scaling with reasoning depth and transfers more efficiently to math and reasoning benchmarks.

  46. Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

    cs.AI 2026-05 unverdicted novelty 7.0

    RL training compute for logical reasoning follows a power law in proof depth whose exponent rises with logic expressiveness, and more expressive training yields larger gains on downstream benchmarks.

  47. Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination

    cs.AI 2026-05 unverdicted novelty 7.0

    Attractor basins in transformer hidden states unify conflict and hallucination as basin competition or absence, with geometric margin outperforming entropy for detection and a scaling law governing confident hallucina...

  48. Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination

    cs.AI 2026-05 unverdicted novelty 7.0

    Transformer hidden states encode facts as attractor basins; hallucinations occur from basin absence and conflicts from basin competition, detected cleanly by geometric margin rather than entropy.

  49. The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence

    cs.LG 2026-05 unverdicted novelty 7.0

    Predictive representation learning structurally favors encoding slower or less noisy environment modes over causal system modes, as shown by an impossibility theorem for linear-Gaussian dynamics and large-scale neural...

  50. RAG over Thinking Traces Can Improve Reasoning Tasks

    cs.IR 2026-05 unverdicted novelty 7.0

    Retrieving structured thinking traces as a corpus improves reasoning performance on AIME, LiveCodeBench, and GPQA over standard RAG or no retrieval.

  51. Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge

    cs.DC 2026-05 unverdicted novelty 7.0

    Tempus delivers 607 GOPS at 10.677 W using fixed 16 AIE cores on Versal AI Edge, with 211.2x better platform-aware utility than spatial SOTA ARIES and zero URAM/DSP utilization.

  52. CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbation

    q-bio.GN 2026-04 unverdicted novelty 7.0

    CellxPert uses inference-time MCMC steering on a multi-omics single-cell foundation model to predict genome-wide transcriptomic responses to gene perturbations and outperforms baselines on cell-type annotation, pertur...

  53. Quantifying Prior Dominance in RAG Systems

    cs.CL 2026-04 unverdicted novelty 7.0

    Introduces NCU metric using token log-probabilities and finds small language models match or outperform larger ones in strict factual RAG extraction, while commercial APIs show high prior dominance and negative transfer.

  54. The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate

    cs.MA 2026-04 unverdicted novelty 7.0

    Homogeneous multi-agent debate introduces sycophantic conformity, contextual fragility, and consensus collapse, leading to equal or lower accuracy than isolated self-correction at 2.1-3.4x higher token cost on GSM-Har...

  55. LoopCTR: Unlocking the Loop Scaling Power for Click-Through Rate Prediction

    cs.IR 2026-04 unverdicted novelty 7.0

    LoopCTR trains CTR models with recursive layer reuse and process supervision so that zero-loop inference outperforms baselines on public and industrial datasets.

  56. Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms

    cs.CL 2026-04 unverdicted novelty 7.0

    Single-agent systems with tools provide the optimal performance-efficiency trade-off for small language models, outperforming base models and multi-agent setups.

  57. Neural Garbage Collection: Learning to Forget while Learning to Reason

    cs.LG 2026-04 conditional novelty 7.0

    Language models learn to evict KV cache entries end-to-end via reinforcement learning from outcome reward alone, achieving 2-3x cache compression while maintaining accuracy on Countdown, AMC, and AIME tasks.

  58. Causal inference for social network formation

    econ.EM 2026-04 conditional novelty 7.0

    Random team assignments in a professional firm reveal that indirect ties strongly increase new direct tie formation, while effects of degree and local density are smaller and less robust.

  59. Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys

    cs.AI 2026-04 unverdicted novelty 7.0

    A method using predicted rectification difficulty for optimal human sample allocation in LLM-augmented surveys captures 61-79% of theoretical efficiency gains and reduces MSE by 11% on two datasets without pilot data.

  60. How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study

    cs.AI 2026-04 unverdicted novelty 7.0

    LLMs and VLMs encode viewpoint information in hidden states but fail to bind it to corresponding observations, resulting in hallucinations in final layers on text-only viewpoint rotation tasks.