pith. sign in

arxiv: 2403.08295 · v4 · submitted 2024-03-13 · 💻 cs.CL · cs.AI

Gemma: Open Models Based on Gemini Research and Technology

Gemma Team: Thomas Mesnard , Cassidy Hardin , Robert Dadashi , Surya Bhupatiraju , Shreya Pathak , Laurent Sifre , Morgane Rivi\`ere , Mihir Sanjay Kale
show 99 more authors
Juliette Love Pouya Tafti L\'eonard Hussenot Pier Giuseppe Sessa Aakanksha Chowdhery Adam Roberts Aditya Barua Alex Botev Alex Castro-Ros Ambrose Slone Am\'elie H\'eliou Andrea Tacchetti Anna Bulanova Antonia Paterson Beth Tsai Bobak Shahriari Charline Le Lan Christopher A. Choquette-Choo Cl\'ement Crepy Daniel Cer Daphne Ippolito David Reid Elena Buchatskaya Eric Ni Eric Noland Geng Yan George Tucker George-Christian Muraru Grigory Rozhdestvenskiy Henryk Michalewski Ian Tenney Ivan Grishchenko Jacob Austin James Keeling Jane Labanowski Jean-Baptiste Lespiau Jeff Stanway Jenny Brennan Jeremy Chen Johan Ferret Justin Chiu Justin Mao-Jones Katherine Lee Kathy Yu Katie Millican Lars Lowe Sjoesund Lisa Lee Lucas Dixon Machel Reid Maciej Miku{\l}a Mateo Wirth Michael Sharman Nikolai Chinaev Nithum Thain Olivier Bachem Oscar Chang Oscar Wahltinez Paige Bailey Paul Michel Petko Yotov Rahma Chaabouni Ramona Comanescu Reena Jana Rohan Anil Ross Mcilroy Ruibo Liu Ryan Mullins Samuel L Smith Sebastian Borgeaud Sertan Girgin Sholto Douglas Shree Pandya Siamak Shakeri Soham De Ted Klimenko Tom Hennigan Vlad Feinberg Wojciech Stokowiec Yu-hui Chen Zafarali Ahmed Zhitao Gong Tris Warkentin Ludovic Peran Minh Giang Cl\'ement Farabet Oriol Vinyals Jeff Dean Koray Kavukcuoglu Demis Hassabis Zoubin Ghahramani Douglas Eck Joelle Barral Fernando Pereira Eli Collins Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy
This is my paper

Pith reviewed 2026-05-10 15:50 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords Gemmaopen language modelsGeminiLLM benchmarksmodel releasesafety evaluationpretrained models
0
0 comments X

The pith

Gemma open models built from Gemini research outperform similar open models on 11 of 18 text tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Gemma as a family of lightweight open language models developed using the research and technology from the Gemini models. It releases both 2 billion and 7 billion parameter versions in pretrained and instruction-tuned forms, with evaluations showing they lead other open models on eleven of eighteen standard text-based benchmarks for understanding, reasoning, and safety. The authors argue that making such capable models openly available, together with detailed responsibility assessments, supports safer progress across the field and opens the door to further LLM innovations.

Core claim

Gemma is a family of lightweight, state-of-the-art open models built from the research and technology used to create Gemini models. The models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. Two sizes are released (2 billion and 7 billion parameters) with both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, accompanied by comprehensive safety and responsibility evaluations.

What carries the argument

The Gemma model family, which adapts Gemini research and technology to produce efficient open language models at 2B and 7B scales.

Load-bearing premise

The chosen academic benchmarks and safety metrics are representative of real-world capabilities and risks.

What would settle it

Independent tests on new tasks or external safety audits where the Gemma models fail to match or exceed the reported advantages on the majority of evaluations.

read the original abstract

This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript introduces the Gemma family of lightweight open language models (2B and 7B parameters) derived from Gemini research and technology. It reports strong performance on academic benchmarks for language understanding, reasoning, and safety, with the claim that Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks. The authors release both pretrained and instruction-tuned model checkpoints along with comprehensive safety and responsibility evaluations and a description of the model development process.

Significance. If the benchmark results hold, the work makes a meaningful contribution to open LLM research by releasing high-performing, accessible models with accompanying safety assessments. The provision of model weights enables direct verification of the performance claims and supports further community experimentation, which strengthens the paper's value for reproducibility.

minor comments (3)
  1. [Abstract and §1] The abstract and introduction reference outperformance on 11 of 18 tasks but would benefit from an early summary table or explicit list of the tasks and baseline models to improve immediate readability for readers scanning the paper.
  2. [Model Development and Evaluation sections] The description of model development provides a high-level overview of training but could clarify the exact evaluation protocols (e.g., few-shot settings, prompt templates) used for the 18 text-based tasks to facilitate precise replication by others.
  3. [Results tables and figures] Figure and table captions should explicitly state the source of any external baseline numbers (e.g., from original papers or reproduced runs) to avoid ambiguity in the comparisons.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive review of our manuscript and for recommending acceptance. We appreciate the recognition of the value in releasing high-performing open models with accompanying safety evaluations and detailed development descriptions.

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper is a model release report that describes Gemma as built from Gemini research technology and evaluates it empirically on public academic benchmarks. The central claim of outperforming similar open models on 11 of 18 tasks rests on externally verifiable benchmark scores and released checkpoints, not on any internal equations, fitted parameters renamed as predictions, or self-citation chains that reduce the result to its own inputs by construction. No uniqueness theorems, ansatzes, or self-definitional steps appear; the evaluation methodology follows standard practices and supplies artifacts for independent checking, making the derivation chain self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

Claims rest on standard transformer training practices and benchmark comparisons rather than new theoretical constructs; several training hyperparameters and data choices are fitted during development.

free parameters (2)
  • model scale and architecture details
    Specific layer counts, hidden dimensions, and attention heads chosen for the 2B and 7B sizes.
  • training data mixture and weighting
    Selection and proportions of pretraining data sources tuned to achieve reported performance.
axioms (1)
  • domain assumption Transformer-based language models trained on large text corpora can achieve strong benchmark performance
    Invoked throughout the model development and evaluation sections.

pith-pipeline@v0.9.0 · 5891 in / 1196 out tokens · 38486 ms · 2026-05-10T15:50:05.869074+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Vision-language models for chest radiography do not always need the image

    cs.CV 2026-06 accept novelty 8.0

    A causal audit with image interventions shows text-only models reach within 5.7 accuracy points of top multimodal VLMs on chest radiography, with some large multimodal models statistically indistinguishable from small...

  2. eCream-MedCorpus A Large-Scale Corpus of Clinical Notes for Italian

    cs.CL 2026-06 unverdicted novelty 8.0

    Introduces the largest freely available Italian clinical notes corpus with 4M notes and expert-annotated subset for a new CRF-filling benchmark.

  3. eCream-MedCorpus A Large-Scale Corpus of Clinical Notes for Italian

    cs.CL 2026-06 unverdicted novelty 8.0

    EDEN releases the largest freely available Italian clinical notes corpus (4M notes, 6k annotated) and proposes CRF-filling as a structured extraction benchmark with zero-shot baselines from Gemma models.

  4. Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation

    cs.AI 2026-06 accept novelty 8.0

    Across 30 LLMs and 205 TLA+ tasks, syntactic correctness reaches at most 26.6% and semantic correctness 8.6%, with all successes limited to progressive prompting and no advantage from larger models.

  5. Looped Transformers with Layer Normalization Provably Learn the Power Method

    cs.LG 2026-05 unverdicted novelty 8.0

    Looped linear transformers with LN provably converge via GD to implement the power method on principal component prediction.

  6. ViMU: Benchmarking Video Metaphorical Understanding

    cs.CV 2026-05 unverdicted novelty 8.0

    ViMU is the first benchmark for evaluating video models on metaphorical and subtextual understanding using hint-free questions grounded in multimodal evidence.

  7. CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

    cs.CL 2026-05 accept novelty 8.0

    CiteVQA requires models to cite specific document regions with bounding boxes alongside answers and finds that even the strongest MLLMs frequently cite the wrong region, with top SAA scores of only 76.0 for closed mod...

  8. Pretraining Exposure Explains Popularity Judgments in Large Language Models

    cs.CL 2026-05 unverdicted novelty 8.0

    LLM popularity judgments align more closely with pretraining data exposure counts than with Wikipedia popularity, with stronger effects in pairwise comparisons and larger models.

  9. Learning the Signature of Memorization in Autoregressive Language Models

    cs.CL 2026-04 accept novelty 8.0

    A classifier trained only on transformer fine-tuning data detects an invariant memorization signature that transfers to Mamba, RWKV-4, and RecurrentGemma with AUCs of 0.963, 0.972, and 0.936.

  10. Adaptive Stopping for Multi-Turn LLM Reasoning

    cs.CL 2026-04 unverdicted novelty 8.0

    MiCP is the first conformal prediction method for multi-turn LLM pipelines that allocates per-turn error budgets to enable adaptive stopping with an overall coverage guarantee, shown to reduce turns and cost on RAG an...

  11. The Linear Representation Hypothesis and the Geometry of Large Language Models

    cs.CL 2023-11 conditional novelty 8.0

    Linear representations of high-level concepts in LLMs are formalized via counterfactuals in input and output spaces, unified under a causal inner product that enables consistent probing and steering.

  12. OpenSafeIntent: Evaluating Intent-Calibrated Safe Completion Across Dual-Use Prompt Sets

    cs.CL 2026-07 unverdicted novelty 7.0

    OpenSafeIntent benchmark shows models fail to calibrate safety across intent shifts in matched dual-use prompts, indicating current evaluations are insufficient.

  13. Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

    cs.CL 2026-06 unverdicted novelty 7.0

    Introduces Indi-RomCoM benchmark for evaluating LLMs on Romanized code-mixed Indic-English instructions across seven tasks, four languages, and three mixing levels.

  14. MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

    cs.CV 2026-06 unverdicted novelty 7.0

    MuseBench shows state-of-the-art MLLMs achieve only 48.29% accuracy on intent-level audiovisual arts understanding versus 87.18% for human experts.

  15. FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks

    cs.CR 2026-06 unverdicted novelty 7.0

    FlipGuard perturbs LLM weights prior to quantization to neutralize quantization-conditioned backdoor attacks, evaluated via the Defense Effectiveness Ratio on multiple models and quantization schemes.

  16. Obliviate: Erasing Concepts from Autoregressive Image Generation Models

    cs.CV 2026-06 unverdicted novelty 7.0

    Obliviate erases targeted concepts from autoregressive image generators via KL supervision on visual tokens over full trajectories, cutting nudity rates sharply on benchmarks while keeping general performance.

  17. Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?

    cs.RO 2026-06 accept novelty 7.0

    VLA language backbones show high redundancy on manipulation benchmarks, with half the LLM blocks removable and even two blocks sufficient to recover baseline performance after fine-tuning, unlike vision and action pathways.

  18. The Topology of Ill-Posed Questions: Persistent Homology for Detection and Steering in LLMs

    cs.AI 2026-06 unverdicted novelty 7.0

    Zero-dimensional persistent homology on transformer layer hidden states yields three descriptors per layer whose concatenation improves ill-posedness classification and enables topology-conditioned activation steering...

  19. RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

    cs.RO 2026-06 unverdicted novelty 7.0

    RoboGaze presents a structured multi-agent VLM pipeline and robotics-specific error taxonomy that improves video evaluation metrics by up to 43 F1 points over zero-shot baselines on a 382-clip dataset.

  20. The Alignment Problem in Constrained Code Generation

    cs.SE 2026-06 unverdicted novelty 7.0

    Incomplete constrainers in constrained decoding push LLMs into low-probability program regions, making unconstrained decoding outperform constrained decoding on functional correctness across seven models and three benchmarks.

  21. CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

    cs.CV 2026-06 unverdicted novelty 7.0

    CheXpercept is a sequential multi-level perception benchmark showing VLMs perform adequately only on coarse lesion detection in chest X-rays while degrading sharply on finer tasks, with medical VLMs offering no advant...

  22. Agentic Discovery of Non-Canonical Antimicrobial Peptides with AMPGAN v3

    q-bio.QM 2026-06 unverdicted novelty 7.0

    AMPGAN v3 generates non-canonical AMPs with D-amino acids and modifications using two discriminators for stability, validated with two active candidates in vitro, alongside the PepCraft multi-agent discovery framework.

  23. UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding

    cs.CL 2026-06 unverdicted novelty 7.0

    UrduMMLU is a new native-source MCQ benchmark for Urdu that reveals top LLMs reach only ~90% accuracy with large gaps on region-specific humanities content.

  24. Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery

    cs.CL 2026-06 unverdicted novelty 7.0

    Structurally distinct circuits for literal sequence copying across token frequency bands implement the same computation, shown by broad transfer of band-specific edges, a shared core recovering 99% performance, and in...

  25. VidMsg: A Benchmark for Implicit Message Inference in Short Videos

    cs.CV 2026-06 unverdicted novelty 7.0

    VidMsg is a new benchmark dataset and QA/retrieval tasks for implicit message inference in short videos, where current models perform poorly.

  26. Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning

    cs.LG 2026-06 unverdicted novelty 7.0

    Analysis of 15 calibration sources shows opposite-sign Spearman correlations between perplexity and retention across General vs. Math/Code dimensions in LLM pruning, and multi-source mixing via IGSP raises total reten...

  27. Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference

    cs.DC 2026-06 unverdicted novelty 7.0

    A new fault-injection framework enables a systematic empirical study that produces 17 takeaways on error propagation in LLM inference and four software-only mitigation directions.

  28. Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs

    cs.CL 2026-05 unverdicted novelty 7.0

    Moral Trolley Arena shows frontier LLMs produce composite moral preferences that are compressed rather than additive functions of calibrated component act strengths across Moral Foundations Theory.

  29. Brain-IT-VQA: From Brain Signals to Answers

    cs.CV 2026-05 unverdicted novelty 7.0

    Brain-IT-VQA decodes visual question answers from fMRI using a transformer to extract language tokens and introduces the NSD-VQA benchmark with 20 controlled questions per image across 20 categories.

  30. SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference

    cs.DC 2026-05 unverdicted novelty 7.0

    SiDP distributes model weights across a DP group with WaS and CaS modes to increase KV cache capacity by up to 1.8x and end-to-end throughput by up to 1.5x over vLLM on H20/H200/B200 GPUs for offline LLM inference.

  31. Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems

    cs.AI 2026-05 unverdicted novelty 7.0

    AgingBench demonstrates multi-dimensional degradation in deployed AI agents through four aging mechanisms diagnosed by temporal graphs and counterfactual probes across hundreds of runs.

  32. Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications

    cs.AI 2026-05 unverdicted novelty 7.0

    POLARIS formalizes policies in FOL, constructs a Semantic Policy Graph to discover compositional violations, and generates natural-language tests, reporting higher coverage and attack success than baselines.

  33. Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0

    Representational convergence across 16 LLMs on 800 reasoning problems is stronger for failed tasks and pre-decision stages but shows minimal causal influence on predictions, pointing to shared processing constraints o...

  34. Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models

    cs.CR 2026-05 conditional novelty 7.0

    ToBAC is the first backdoor attack on unified autoregressive models, using data or model poisoning to make triggers elicit cross-modal malicious behavior in text and image generation.

  35. Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

    cs.LG 2026-05 conditional novelty 7.0

    A framework to identify and convert foldable layer normalizations to RMSNorm for exact equivalence and faster inference in deep neural networks.

  36. BOOKMARKS: Efficient Active Storyline Memory for Role-playing

    cs.CL 2026-05 unverdicted novelty 7.0

    BOOKMARKS introduces searchable bookmarks as reusable answers to storyline questions, enabling active initialization and passive synchronization for more consistent role-playing agent memory than recurrent summarization.

  37. Realtime-VLA FLASH: Speculative Inference Framework for Diffusion-based VLAs

    cs.RO 2026-05 unverdicted novelty 7.0

    A new speculative inference system speeds up diffusion VLAs to 19.1 ms average latency (3.04x faster) on LIBERO by replacing most full 58 ms inferences with 7.8 ms draft rounds while preserving task performance.

  38. TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment

    cs.CL 2026-05 unverdicted novelty 7.0

    TokAlign++ learns token alignments between LLM vocabularies from monolingual representations to enable faster adaptation, better text compression, and effective token-level distillation across 15 languages with minimal steps.

  39. IndicMedDialog: A Parallel Multi-Turn Medical Dialogue Dataset for Accessible Healthcare in Indic Languages

    cs.CL 2026-05 unverdicted novelty 7.0

    A parallel multi-turn medical dialogue dataset spanning English and nine Indic languages is created from synthetic consultations to enable personalized AI healthcare interactions.

  40. CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models

    cs.CV 2026-05 conditional novelty 7.0

    LiteLVLM prunes visual tokens for pixel grounding by reversing CLIP visual-text similarity to retain referent region tokens, outperforming prior methods by over 5% with 22% speedup and 2.3x memory reduction without an...

  41. Controlling Logical Collapse in LLMs via Algebraic Ontology Projection over F2

    cs.LG 2026-05 unverdicted novelty 7.0

    Projecting LLM hidden states onto F2 algebra with 42 pairs yields 93% zero-shot accuracy on logical relations and identifies prompt-preventable late-layer collapse.

  42. Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters

    cs.CV 2026-05 unverdicted novelty 7.0

    Chronicles-OCR is the first benchmark with 2,800 images across the complete evolutionary trajectory of Chinese characters, defining four tasks to evaluate VLLMs' cross-temporal visual perception.

  43. ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    ALAM creates algebraically consistent latent action transitions from videos to act as auxiliary generative targets, raising robot policy success rates from 47.9% to 85.0% on MetaWorld MT50 and 94.1% to 98.1% on LIBERO.

  44. From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Chirality emerges in SMILES translation models through an abrupt encoder-centered reorganization of representations after a long plateau, identified via checkpoint analysis and ablation.

  45. Entropy-informed Decoding: Adaptive Information-Driven Branching

    cs.LG 2026-05 unverdicted novelty 7.0

    EDEN adaptively sets branching factor proportional to next-token entropy, achieving better accuracy per expansion than fixed beam search while providing a proof that monotone entropy-based branching outperforms any fi...

  46. K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

    cs.CL 2026-05 conditional novelty 7.0

    K12-KGraph is a textbook-derived knowledge graph that powers a new benchmark revealing LLMs' poor curriculum cognition and a small training corpus that outperforms general instruction data on educational tasks.

  47. Privacy Without Losing Place: A Paradigm for Private Retrieval in Spatial RAGs

    cs.CR 2026-05 unverdicted novelty 7.0

    PAS encodes locations via relative anchors and bins to deliver roughly 370-400m adversarial error in spatial RAG while retaining over half the baseline retrieval performance and keeping generation quality robust.

  48. Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering

    cs.CR 2026-05 unverdicted novelty 7.0

    Black-box membership inference attacks on retrieval-based in-context learning for document QA succeed via query prefixes, with a novel weighted-averaging method outperforming priors even under paraphrasing.

  49. Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    Decoding-time use of process reward models for bias mitigation raises fairness scores by up to 0.40 on a bilingual benchmark while preserving fluency across four LLMs and extends to open-ended generation with low overhead.

  50. Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidence

    cs.CR 2026-05 unverdicted novelty 7.0

    RAGCharacter localizes poisoned character spans in RAG evidence via prompt-conditioned counterfactual masking and achieves the best accuracy-over-attribution trade-off across tested attacks and models.

  51. The Hidden Cost of Thinking: Energy Use and Environmental Impact of LMs Beyond Pretraining

    cs.CY 2026-05 unverdicted novelty 7.0

    Full development of 7B and 32B Olmo 3 models used 12.3 GWh datacenter energy and emitted 4,251 tCO2eq, with development overheads accounting for 82% of compute and reasoning models costing 17x more to post-train than ...

  52. Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge

    cs.DC 2026-05 unverdicted novelty 7.0

    Tempus delivers 607 GOPS at 10.677 W using fixed 16 AIE cores on Versal AI Edge, with 211.2x better platform-aware utility than spatial SOTA ARIES and zero URAM/DSP utilization.

  53. Attention Is Where You Attack

    cs.CR 2026-04 unverdicted novelty 7.0

    ARA jailbreaks safety-aligned LLMs like LLaMA-3 and Mistral by redirecting attention in safety-heavy heads with as few as 5 tokens, achieving 30-36% attack success while ablating the same heads barely affects refusals.

  54. Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

    cs.LG 2026-04 unverdicted novelty 7.0

    Incompressible Knowledge Probes enable log-linear estimation of LLM parameter counts from factual accuracy on obscure questions, showing continued scaling of knowledge capacity across open and closed models.

  55. Can an MLP Absorb Its Own Skip Connection?

    cs.LG 2026-04 accept novelty 7.0

    Skip-connected MLPs and residual-free MLPs of equal width represent generically disjoint function classes for common activations, with explicit impossibility proofs and a non-generic absorption condition for ReLU and GELU.

  56. Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective

    cs.CL 2026-04 conditional novelty 7.0

    A controlled formal language task reveals fine-tuning outperforms in-context learning on in-distribution generalization but equals it on out-of-distribution, with ICL showing greater sensitivity to model size and toke...

  57. Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective

    cs.CL 2026-04 unverdicted novelty 7.0

    Fine-tuning shows higher proficiency than in-context learning on in-distribution generalization in formal languages, with equal out-of-distribution performance and diverging inductive biases at high proficiency.

  58. ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control

    cs.LG 2026-04 unverdicted novelty 7.0

    ParetoSlider conditions diffusion models on continuous preference weights to approximate the full Pareto front, providing dynamic control over multi-objective rewards at inference time.

  59. RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking

    cs.CV 2026-04 unverdicted novelty 7.0

    RSRCC is a new 126k-question benchmark for fine-grained remote sensing change question-answering, constructed via a hierarchical semi-supervised pipeline with retrieval-augmented Best-of-N ranking.

  60. How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

    cs.CL 2026-04 unverdicted novelty 7.0

    Subword tokenization impairs phonological knowledge encoding in LMs, but an IPA-based fine-tuning method restores it with minimal impact on other capabilities.

Reference graph

Works this paper leans on

91 extracted references · 91 canonical work pages · cited by 291 Pith papers · 24 internal anchors

  1. [1]

    Almazrouei, H

    E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, Étienne Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo. The falcon series of open language models, 2023

    work page 2023

  2. [2]

    Amodei, C

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man \'e . Concrete problems in AI safety. arXiv preprint, 2016

    work page 2016

  3. [5]

    Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. L...

    work page 2022

  4. [6]

    Barham, A

    P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, B. Saeta, P. Schuh, R. Sepassi, L. E. Shafey, C. A. Thekkath, and Y. Wu. Pathways: Asynchronous distributed dataflow for ml, 2022

    work page 2022

  5. [8]

    R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39, 1952

    work page 1952

  6. [11]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H...

    work page 2022

  7. [12]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017

    work page 2017

  8. [14]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    work page 2018

  9. [16]

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. a. Ranzato, A. Senior, P. Tucker, K. Yang, Q. Le, and A. Ng. Large scale distributed deep networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/pap...

    work page 2012

  10. [18]

    Gemini: A family of highly capable multimodal models, 2023

    Gemini Team . Gemini: A family of highly capable multimodal models, 2023

    work page 2023

  11. [20]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    work page 2021

  12. [22]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023

    work page 2023

  13. [24]

    Kavukcuoglu, P

    K. Kavukcuoglu, P. Kohli, L. Ibrahim, D. Bloxwich, and S. Brown. How our principles helped define alphafold’s release, 2022

    work page 2022

  14. [28]

    LeCun, Y

    Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521 0 (7553): 0 436--444, 2015

    work page 2015

  15. [29]

    Efficient Estimation of Word Representations in Vector Space

    T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In Y. Bengio and Y. LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings , 2013. URL http://arxiv.org/abs/1301.3781

    work page internal anchor Pith review arXiv 2013

  16. [31]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 2022

    work page 2022

  17. [32]

    Pacchiardi, A

    L. Pacchiardi, A. J. Chan, S. Mindermann, I. Moscovitz, A. Y. Pan, Y. Gal, O. Evans, and J. Brauner. How to catch an ai liar: Lie detection in black-box llms by asking unrelated questions, 2023

    work page 2023

  18. [35]

    Roberts, H

    A. Roberts, H. W. Chung, A. Levskaya, G. Mishra, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, C. Hawthorne, A. Lewkowycz, A. Salcianu, M. van Zee, J. Austin, S. Goodman, L. B. Soares, H. Hu, S. Tsvyashchenko, A. Chowdhery, J. Bastings, J. Bulian, X. Garcia, J. Ni, A. Chen, K. Kenealy, J. H. Clark, S. Lee, D. Garrette, J. Lee-Thor...

    work page 2022

  19. [36]

    Roberts, H

    A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, et al. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24 0 (377): 0 1--8, 2023

    work page 2023

  20. [41]

    J. M. V. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. In NeurIPS, 2022

    work page 2022

  21. [44]

    Suzgun, N

    M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022

    work page 2022

  22. [45]

    Talmor, J

    A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019

    work page 2019

  23. [46]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023 a

    work page 2023

  24. [47]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev,...

    work page 2023

  25. [51]

    Xla: Optimizing compiler for tensorflow, 2019

    XLA. Xla: Optimizing compiler for tensorflow, 2019. URL https://www.tensorflow.org/xla

    work page 2019

  26. [54]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    work page 2023

  27. [55]

    Zhong, R

    W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023

    work page 2023

  28. [56]

    A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023

    work page 2023

  29. [57]

    Concrete problems in

    Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man. Concrete problems in. arXiv preprint , year =

    work page

  30. [58]

    Quantifying Memorization Across Neural Language Models

    Quantifying memorization across neural language models , author=. arXiv preprint arXiv:2202.07646 , year=

    work page internal anchor Pith review arXiv

  31. [59]

    Scalable Extraction of Training Data from (Production) Language Models

    Scalable extraction of training data from (production) language models , author=. arXiv preprint arXiv:2311.17035 , year=

    work page internal anchor Pith review arXiv

  32. [60]

    30th USENIX Security Symposium (USENIX Security 21) , pages=

    Extracting training data from large language models , author=. 30th USENIX Security Symposium (USENIX Security 21) , pages=

    work page

  33. [61]

    arXiv preprint arXiv:2210.17546 , year=

    Preventing verbatim memorization in language models gives a false sense of privacy , author=. arXiv preprint arXiv:2210.17546 , year=

    work page arXiv

  34. [62]

    Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat

    Madlad-400: A multilingual and document-level large audited dataset , author=. arXiv preprint arXiv:2309.04662 , year=

    work page arXiv

  35. [63]

    NeurIPS , year =

    Defining and Characterizing Reward Gaming , author =. NeurIPS , year =

    work page

  36. [64]

    2023 , eprint=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. 2023 , eprint=

    work page 2023

  37. [65]

    2022 , eprint=

    Scaling Laws for Reward Model Overoptimization , author=. 2022 , eprint=

    work page 2022

  38. [66]

    Mastering the game of

    Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others , journal=. Mastering the game of. 2016 , publisher=

    work page 2016

  39. [67]

    Proceedings of the 50th Annual International Symposium on Computer Architecture , pages=

    Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings , author=. Proceedings of the 50th Annual International Symposium on Computer Architecture , pages=

    work page

  40. [68]

    2023 , eprint=

    Gemini: A Family of Highly Capable Multimodal Models , author=. 2023 , eprint=

    work page 2023

  41. [69]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk and Rowan Zellers and Ronan Le Bras and Jianfeng Gao and Yejin Choi , title =. CoRR , volume =. 2019 , url =. 1911.11641 , timestamp =

    work page internal anchor Pith review arXiv 2019

  42. [70]

    SocialIQA: Commonsense Reasoning about Social Interactions

    Maarten Sap and Hannah Rashkin and Derek Chen and Ronan Le Bras and Yejin Choi , title =. CoRR , volume =. 2019 , url =. 1904.09728 , timestamp =

    work page internal anchor Pith review arXiv 2019

  43. [71]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark and Kenton Lee and Ming. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions , journal =. 2019 , url =. 1905.10044 , timestamp =

    work page internal anchor Pith review arXiv 2019

  44. [72]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew M. and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav. Natura...

    work page doi:10.1162/tacl_a_00276 2019

  45. [73]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt , title =. CoRR , volume =. 2020 , url =. 2009.03300 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  46. [74]

    Program Synthesis with Large Language Models

    Jacob Austin and Augustus Odena and Maxwell I. Nye and Maarten Bosma and Henryk Michalewski and David Dohan and Ellen Jiang and Carrie J. Cai and Michael Terry and Quoc V. Le and Charles Sutton , title =. CoRR , volume =. 2021 , url =. 2108.07732 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  47. [75]

    Evaluating Large Language Models Trained on Code

    Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Pond. Evaluating Large Language Models Trained on Code , journal =. 2021 , url =. 2107.03374 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  48. [76]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Mark Chen and Heewoo Jun and Lukasz Kaiser and Matthias Plappert and Jerry Tworek and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman , title =. CoRR , volume =. 2021 , url =. 2110.14168 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  49. [77]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi , title =. CoRR , volume =. 2019 , url =. 1907.10641 , timestamp =

    work page internal anchor Pith review arXiv 2019

  50. [78]

    Denis Paperno and Germ. The. CoRR , volume =. 2016 , url =. 1606.06031 , timestamp =

    work page Pith review arXiv 2016

  51. [79]

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Mandar Joshi and Eunsol Choi and Daniel S. Weld and Luke Zettlemoyer , title =. CoRR , volume =. 2017 , url =. 1705.03551 , timestamp =

    work page internal anchor Pith review arXiv 2017

  52. [80]

    2023 , eprint=

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author=. 2023 , eprint=

    work page 2023

  53. [81]

    2023 , eprint=

    LLaMA: Open and Efficient Foundation Language Models , author=. 2023 , eprint=

    work page 2023

  54. [82]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    work page 2023

  55. [83]

    2023 , eprint=

    The Falcon Series of Open Language Models , author=. 2023 , eprint=

    work page 2023

  56. [84]

    Sequence to Sequence Learning with Neural Networks

    Ilya Sutskever and Oriol Vinyals and Quoc V. Le , title =. CoRR , volume =. 2014 , url =. 1409.3215 , timestamp =

    work page Pith review arXiv 2014

  57. [85]

    Attention Is All You Need

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , title =. CoRR , volume =. 2017 , url =. 1706.03762 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  58. [86]

    nature , volume=

    Deep learning , author=. nature , volume=. 2015 , publisher=

    work page 2015

  59. [87]

    2022 , eprint=

    Pathways: Asynchronous Distributed Dataflow for ML , author=. 2022 , eprint=

    work page 2022

  60. [88]

    Journal of Machine Learning Research , volume=

    Scaling up models and data with t5x and seqio , author=. Journal of Machine Learning Research , volume=

    work page

  61. [89]

    2019 , url=

    XLA: Optimizing compiler for TensorFlow , author=. 2019 , url=

    work page 2019

  62. [90]

    2022 , publisher=

    How our principles helped define AlphaFold’s release , author=. 2022 , publisher=

    work page 2022

  63. [91]

    Large Scale Distributed Deep Networks , url =

    Dean, Jeffrey and Corrado, Greg and Monga, Rajat and Chen, Kai and Devin, Matthieu and Mao, Mark and Ranzato, Marc aurelio and Senior, Andrew and Tucker, Paul and Yang, Ke and Le, Quoc and Ng, Andrew , booktitle =. Large Scale Distributed Deep Networks , url =

    work page

  64. [92]

    Efficient Estimation of Word Representations in Vector Space , booktitle =

    Tom. Efficient Estimation of Word Representations in Vector Space , booktitle =. 2013 , url =

    work page 2013

  65. [93]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin and Ming. CoRR , volume =. 2018 , url =. 1810.04805 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  66. [94]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. CoRR , volume =. 2019 , url =. 1910.10683 , timestamp =

    work page internal anchor Pith review arXiv 2019

  67. [95]

    Adam Roberts and Hyung Won Chung and Anselm Levskaya and Gaurav Mishra and James Bradbury and Daniel Andor and Sharan Narang and Brian Lester and Colin Gaffney and Afroz Mohiuddin and Curtis Hawthorne and Aitor Lewkowycz and Alex Salcianu and Marc van Zee and Jacob Austin and Sebastian Goodman and Livio Baldini Soares and Haitang Hu and Sasha Tsvyashchenk...

    work page arXiv

  68. [96]

    Fast Transformer Decoding: One Write-Head is All You Need

    Noam Shazeer , title =. CoRR , volume =. 2019 , url =. 1911.02150 , timestamp =

    work page internal anchor Pith review arXiv 2019

  69. [97]

    RoFormer: Enhanced Transformer with Rotary Position Embedding

    Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu , title =. CoRR , volume =. 2021 , url =. 2104.09864 , timestamp =

    work page internal anchor Pith review arXiv 2021

  70. [98]

    GLU Variants Improve Transformer

    Noam Shazeer , title =. CoRR , volume =. 2020 , url =. 2002.05202 , timestamp =

    work page internal anchor Pith review arXiv 2020

  71. [99]

    Root Mean Square Layer Normalization

    Biao Zhang and Rico Sennrich , title =. CoRR , volume =. 2019 , url =. 1910.07467 , timestamp =

    work page internal anchor Pith review arXiv 2019

  72. [100]

    SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

    Kudo, Taku and Richardson, John. S entence P iece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2018. doi:10.18653/v1/D18-2012

    work page internal anchor Pith review doi:10.18653/v1/d18-2012 2018

  73. [101]

    Yuanzhong Xu and HyoukJoong Lee and Dehao Chen and Blake A. Hechtman and Yanping Huang and Rahul Joshi and Maxim Krikun and Dmitry Lepikhin and Andy Ly and Marcello Maggioni and Ruoming Pang and Noam Shazeer and Shibo Wang and Tao Wang and Yonghui Wu and Zhifeng Chen , title =. CoRR , volume =. 2021 , url =. 2105.04663 , timestamp =

    work page internal anchor Pith review arXiv 2021

  74. [102]

    2022 , eprint=

    PaLM: Scaling Language Modeling with Pathways , author=. 2022 , eprint=

    work page 2022

  75. [103]

    PaLM 2 Technical Report

    Palm 2 technical report , author=. arXiv preprint arXiv:2305.10403 , year=

    work page internal anchor Pith review arXiv

  76. [104]

    Advances in Neural Information Processing Systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in Neural Information Processing Systems , volume=

    work page

  77. [105]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

    work page

  78. [106]

    The method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. The method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    work page 1952

  79. [107]

    Machine learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine learning , volume=. 1992 , publisher=

    work page 1992

  80. [108]

    2023 , eprint=

    AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models , author=. 2023 , eprint=

    work page 2023

Showing first 80 references.