REVIEW 2 major objections 60 cited by
Gemma 3 adds vision and 128K context to the Gemma family while a new post-training recipe makes its 4B model competitive with the prior 27B version on math and chat tasks.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-05-22 22:12 UTC
load-bearing objection Gemma 3 is a straightforward model release that adds vision and tweaks the attention ratio for longer context, with the usual self-reported benchmark claims that the released weights will let people check. the 2 major comments →
Gemma 3 Technical Report
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Gemma 3 models achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, the novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks.
What carries the argument
The novel post-training recipe that improves math, chat, instruction-following and multilingual abilities after distillation training.
Load-bearing premise
The chosen benchmarks and evaluation protocols accurately reflect real-world gains in math, chat, and multilingual performance without post-hoc selection or overfitting to the test sets.
What would settle it
New benchmarks or held-out tasks where Gemma3-4B-IT fails to match Gemma2-27B-IT on math and chat metrics would falsify the central performance claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces Gemma 3, a family of multimodal open models (1B–27B parameters) adding vision understanding, broader language coverage, and ≥128K-token context. It modifies the architecture by raising the local-to-global attention layer ratio with short local spans to reduce KV-cache memory, trains via distillation, and applies a novel post-training recipe claimed to improve math, chat, instruction-following, and multilingual performance. This positions Gemma3-4B-IT as competitive with Gemma2-27B-IT and Gemma3-27B-IT as comparable to Gemini-1.5-Pro across benchmarks, with all models released.
Significance. If the empirical results hold under scrutiny, the work is significant for releasing capable open multimodal models that approach closed frontier performance and for practical advances in long-context efficiency via attention architecture. The model release itself and the post-training recipe constitute concrete contributions to the open AI ecosystem.
major comments (2)
- [Abstract] Abstract: the central claim that the novel post-training recipe 'significantly improves' math, chat, instruction-following, and multilingual abilities (making 4B competitive with 27B and 27B comparable to Gemini-1.5-Pro) is stated without any benchmark scores, error bars, dataset splits, or evaluation-protocol details. This directly underpins the headline competitiveness assertions and requires explicit quantitative support.
- [Results / Evaluation] Results / Evaluation sections: absence of variance estimates, exact test-set descriptions, or controls against post-hoc benchmark selection leaves the robustness of the reported gains (and the claim that they reflect genuine capability improvements rather than overfitting) unverified, which is load-bearing for the post-training contribution.
Simulated Author's Rebuttal
We thank the referee for the detailed and constructive review. The comments highlight important areas for strengthening the presentation of our results and claims. We address each major comment below and indicate the revisions we will make.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central claim that the novel post-training recipe 'significantly improves' math, chat, instruction-following, and multilingual abilities (making 4B competitive with 27B and 27B comparable to Gemini-1.5-Pro) is stated without any benchmark scores, error bars, dataset splits, or evaluation-protocol details. This directly underpins the headline competitiveness assertions and requires explicit quantitative support.
Authors: We agree that the abstract would benefit from explicit quantitative support for the central claims. In the revised version we will insert a concise set of representative benchmark scores (e.g., average MMLU, GSM8K, HumanEval, and multilingual metrics) that directly illustrate the competitiveness statements, while keeping the abstract within length limits. This change will make the headline assertions immediately verifiable from the abstract itself. revision: yes
-
Referee: [Results / Evaluation] Results / Evaluation sections: absence of variance estimates, exact test-set descriptions, or controls against post-hoc benchmark selection leaves the robustness of the reported gains (and the claim that they reflect genuine capability improvements rather than overfitting) unverified, which is load-bearing for the post-training contribution.
Authors: The evaluation protocol follows the canonical test splits and official evaluation scripts of each public benchmark; we will expand the relevant section to list the exact dataset versions, prompt formats, and decoding parameters used. Variance estimates from multiple independent training runs are not feasible at this scale due to compute cost, which is standard practice in large-model technical reports. To address concerns about post-hoc selection, we will add a paragraph clarifying that the benchmark suite was fixed prior to final post-training and that the same suite is used for all model variants. These additions will improve transparency without altering the reported numbers. revision: partial
Circularity Check
No significant circularity
full rationale
The paper is a standard technical report on model release. It describes architecture modifications (local/global attention ratio), training with distillation, and a post-training recipe, then reports empirical benchmark scores for math, chat, instruction-following, and multilingual tasks. No equations, first-principles derivations, or 'predictions' are present that reduce by construction to fitted parameters or self-citations. Central claims rest on external benchmark evaluations rather than internal loops. Self-citations to prior Gemma papers are normal and non-load-bearing for the empirical results.
Axiom & Free-Parameter Ledger
read the original abstract
We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.
Forward citations
Cited by 60 Pith papers
-
Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning
A blank-image ablation test reveals that high probe accuracy on VLM spatial reasoning frequently reflects priors or inverted signs rather than image grounding, with horizontal grounded, vertical prior, and depth inverted.
-
Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
Noisy expert imitation learning requires exponential samples for offline methods but polynomial for a variant of on-policy distillation under a noise condition.
-
DataComp-VLM: Improved Open Datasets for Vision-Language Models
DataComp-VLM benchmark shows instruction-heavy data mixing outperforms filtering for VLM training, with DCVLM-Baseline achieving 63.6% on 33 tasks for 8B models (+5.4pp over FineVision).
-
MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents
MedCUA-Bench provides 18 clinical scenarios in 10 domains as a testbed for computer-use agents on medical UIs, with evaluations of 23 agents showing low success rates especially on real systems like OpenEMR.
-
Looped Transformers with Layer Normalization Provably Learn the Power Method
Looped linear transformers with LN provably converge via GD to implement the power method on principal component prediction.
-
Architecture Determines Observability of Transformers
Certain transformer architectures lose internal linear signals for decision quality during training, making observability an architecture-dependent property rather than a universal one.
-
Lost in Translation: Do LVLM Judges Generalize Across Languages?
MM-JudgeBench shows substantial cross-lingual performance variance in 22 LVLM judges, with model size and architecture as poor predictors of multilingual robustness.
-
SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning
SAHM is the first Arabic financial benchmark with seven tasks including AAOIFI standards QA, fatwa reasoning, accounting exams, sentiment analysis, summarization, and event-cause reasoning, showing that Arabic fluency...
-
ArgBench: Benchmarking LLMs on Computational Argumentation Tasks
ArgBench unifies 33 existing datasets into a standardized benchmark for testing LLMs across 46 argumentation tasks and analyzes the impact of prompting techniques and model factors on performance.
-
VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
VAREX benchmark shows structured output compliance limits models under 4B parameters more than extraction ability, with layout-preserving text giving the largest accuracy gains over images.
-
Neural Signals Generate Clinical Notes in the Wild
CELM is the first EEG-to-language foundation model that generates clinical reports from variable-length EEG recordings using a new dataset of 9,922 reports paired with 11,000 hours of data from 9,048 patients.
-
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
Molmo2 delivers state-of-the-art open-weight video VLMs with new grounding datasets and training methods that outperform prior open models and match or exceed some proprietary ones on pointing and tracking tasks.
-
NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity
NEvo performs evolutionary search guided by a dynamic voxel-level encoding model to synthesize videos that maximize predicted activity in target brain ROIs, recovering known selectivities and revealing temporal dynami...
-
EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards
EgoSafetyBench shows VLMs reliably spot hazard-containing videos but miss specific contextual hazards and are degraded by misleading in-scene text.
-
Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA
Self-generated QA supervision for language models is fragile due to non-uniform question selection and instruction compliance during answering, with mitigations that reduce compliance from 88% to 13%.
-
RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization
RCT dataset with sequence-preserving splits demonstrates that tactile-to-text models achieve only 25.1% Recall@1 on held-out materials, exposing generalization as the core challenge.
-
Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?
VSE perturbs images only to probe visual ambiguity in VLMs, clusters outputs into semantic prototypes, and computes mass-weighted dispersion, outperforming prior entropy methods on five VQA benchmarks across five models.
-
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
LaViD distills LLM conceptual knowledge to vision models via LLM-generated MCQ soft labels, outperforming vision-language distillation baselines on fine-grained benchmarks while improving robustness on spurious correl...
-
DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues
DiCoBench is a new high-resolution multi-image benchmark exposing large gaps between top MLLMs and human performance (98.3%) on differential and commonality visual cue perception.
-
Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning
Cliff tokens are single-token failure triggers in LLM mathematical reasoning identified via adaptive statistical threshold; intervening at them recovers performance to 1.0 in resampling and yields up to +6.6 accuracy ...
-
Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning
Cliff tokens are single tokens triggering LLM math reasoning failures, identified via adaptive z-test threshold on token potential; a taxonomy and Cliff-DPO optimization yield up to +6.6 accuracy gains.
-
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
PACE is a clipped per-coordinate controller added to AdamW that improves the limiting error of the returned iterate average in both quadratic analysis and LM experiments.
-
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
PACE is an AdamW wrapper derived from optimal control that improves the limiting error of the returned exponential-moving-average model in both theory and LM experiments.
-
Trustworthy Image Authentication using Forensic Knowledge Graphs
Forensic Knowledge Graphs integrate forensic traces, causal dependencies, and scene links via a new authentication network and Iterative Context Refinement to outperform standard detectors and VLMs on detection, local...
-
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
New benchmark quantifies language- and model-dependent over-alignment in criminal law LLM use and identifies abliteration as an effective mitigation with minimal performance cost.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe decouples access and management of secure resources in TrustZone to enable efficient LLM inference on mobiles, reporting 10.05X TTFT speedup over basic strawman designs and 2.44X over optimized ones.
-
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
FlexServe introduces recallable secure memory and NPU to enable cooperative secure LLM inference on mobile devices, reporting 10.05X TTFT speedup over a basic TrustZone strawman.
-
GitReq: A Gold Standard Dataset for Software Quality Requirements
GitReq is a released dataset of 6,302 expert-labeled GitHub requirements across eight ISO 25010 categories with LLM classification baselines.
-
FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelines
FAPO automates LLM pipeline optimization via iterative diagnosis and prompt-or-structure edits, beating GEPA baseline by +14.1 pp mean across 18 comparisons and +33.8 pp when structural changes occur.
-
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
The normalized inverse-scale direction of LayerNorm's affine parameters is an exact algebraic kernel of the post-final-norm centred activation covariance for any input distribution in LayerNorm transformers.
-
AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model
AMALIA-VL introduces the first open-source instruction-tuned LVLM natively optimized for European Portuguese via vision-language alignment, instruction tuning, preference optimization, and a pt-PT-centric data mix.
-
Image Prompt Reconstruction Attacks on Distributed MLLM Inference Frameworks
First study of image prompt reconstruction attacks on distributed MLLM inference, proposing MPAA for pixel-level and IEDA for semantic reconstruction with 100% embedding extraction accuracy on four model families.
-
TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization
TIGER turns the low-rank attention gradient subspace into a differentiable objective for continuous embedding optimization, improving reconstruction quality and robustness over prior discrete token tests especially un...
-
Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Models
Instruction-based vector steering redirects temporal attention in LALMs to acoustically relevant regions, recovering queried sound event locations with 60.87-68.72% overlap accuracy without training.
-
A History-Aware Visually Grounded Critic for Computer Use Agents
HiViG is a test-time critic that combines macro-action history summarization with visual grounding of execution coordinates to reduce short-sighted and visually erroneous actions in long-horizon GUI agents.
-
Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
Rea2Seg turns image segmentation into candidate mask discovery from MLLM attention followed by MLLM-based comparative scoring and selection, plus a new multi-dimensional reasoning benchmark ReasonSeg-SGDR.
-
When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models
S³E framework finds excess decision-state displacement under semantic stress in multimodal models despite consistent correct forced-choice behavior.
-
When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding
MLLMs fail to detect absent correct answers in video QA tasks across three evaluation settings, defaulting to distractors even with chain-of-thought prompting.
-
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
Sci-Rho is a dynamic multilingual visually-grounded symbolic benchmark for STEM problems that reveals robustness gaps in current VLMs between average and worst-case performance.
-
UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding
UrduMMLU is a new native-source MCQ benchmark for Urdu that reveals top LLMs reach only ~90% accuracy with large gaps on region-specific humanities content.
-
NAVI-Orbital: First In-Orbit Demonstration of a Zero-Shot Vision-Language Model for Autonomous Earth Observation
NAVI-Orbital performed the first claimed in-orbit demonstration of a zero-shot vision-language model for autonomous Earth observation using Gemma 3 on April 16, 2026.
-
Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception
VLMs exhibit anchoring to discrete slant angles rather than graded responses across zero-shot, in-context, and fine-tuned settings, unlike human psychophysical patterns.
-
Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception
VLMs across families and scales show anchoring to discrete slant angles in zero-shot and prompted settings rather than human-like graded texture-based slant perception.
-
MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models
Introduces MMBU benchmark for VLMs in biomedicine and demonstrates that established benchmarks mask perception deficiencies in evaluated models.
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
Text-to-Image Models Need Less from Text Encoders Than You Think
A bag-of-position-tagged-words embedding guides text-to-image diffusion models as effectively as full contextual text embeddings from standard encoders.
-
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
SEA-NLI benchmark shows low performance across 17 LLMs on Southeast Asian cultural NLI, mainly due to missing cultural knowledge, with gains from SEA-adapted models and culture-aware prompting.
-
GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving
GeoDrive-Bench is a new multimodal benchmark and distillation method for testing and improving VLMs on region-specific traffic-rule reasoning in autonomous driving across six countries.
-
Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthromorphism, and Maxims
Presents FRANZ framework and SQUARE corpus for multi-dimensional audit of LLM response framing on subjective cultural queries, applied to three models to reveal differences and couplings.
-
Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging
MERIT enables decentralized instruction tuning via conflict-aware PCA splitting and parameter-space merging, raising average benchmark scores above joint training on multimodal and text mixtures.
-
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Attentive-CoT is an attention-guided fine-tuning objective that improves chain-of-thought performance in multimodal LLMs by delaying answer commitment and increasing sustained visual-token access during rationale generation.
-
OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
OmniOPD replaces token-level logit matching in on-policy distillation with Monte Carlo chunk-level semantic verification and a peak-entropy scheduler.
-
Subliminal Learning Is Steering Vector Distillation
Subliminal learning is steering vector distillation: a student fine-tuned on a steered teacher's outputs learns to imitate the steering vector.
-
How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings
Auto-interpretation labels for SAE features generalize poorly across languages and scripts, missing the same semantic content up to 4x more often in Serbian than English and more in Cyrillic than Latin despite determi...
-
LLMs Need Encoders for Semantic IDs Too
PrefixMem encoder for Semantic IDs improves deepest-level accuracy by up to 46% relative and full-SID retrieval recall by up to 22% relative on Pinterest data across LLM families.
-
FAM-Bench: A Multimodal Benchmark for Condition-Aware Food-as-Medicine Reasoning
FAM-Bench introduces 2500 nutrition-expert-verified multimodal instances across 13 conditions for dish suitability assessment and comparative ranking tasks.
-
The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models
LLM residual streams during addition form an Iso-Raw-Sum Trajectory anchored by digit semantics and modulated by continuous carry signals, with errors arising as geometric slippages across quantization thresholds in a...
-
Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs
Moral Trolley Arena shows frontier LLMs produce composite moral preferences that are compressed rather than additive functions of calibrated component act strengths across Moral Foundations Theory.
-
CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations
CardioLens is a leakage-resistant CMR testbed of 473k slices and 13k QA pairs showing current MLLMs exhibit a large clinical reality gap with category-collapse failures on real workflows.
-
AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
AsymVLM introduces asymmetric token pruning for vision and text in VLMs to deliver up to 54% FLOPs reduction while matching or exceeding prior methods on localized visual tasks.
Reference graph
Works this paper leans on
-
[1]
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyan- skiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245,
work page internal anchor Pith review Pith/arXiv arXiv
- [2]
-
[3]
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lep- ikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report.arXiv preprint arXiv:2305.10403,
work page internal anchor Pith review Pith/arXiv arXiv
- [4]
-
[5]
Program Synthesis with Large Language Models
J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton. Program synthesis with large language models.CoRR, abs/2108.07732,
work page internal anchor Pith review Pith/arXiv arXiv
-
[6]
Longformer: The Long-Document Transformer
I. Beltagy, M. E. Peters, and A. Cohan. Long- former: The long-document transformer.arXiv preprint arXiv:2004.05150,
work page internal anchor Pith review Pith/arXiv arXiv 2004
-
[7]
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. CoRR, abs/1911.11641,
work page internal anchor Pith review Pith/arXiv arXiv 1911
-
[8]
Quantifying Memorization Across Neural Language Models
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memo- rization across neural language models.arXiv preprint arXiv:2202.07646,
work page internal anchor Pith review Pith/arXiv arXiv
-
[9]
Chameleon: Mixed-Modal Early-Fusion Foundation Models
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,
work page internal anchor Pith review Pith/arXiv arXiv
-
[10]
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, 11 Gemma 3 Technical Report G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ry- der, M. Pavlov, A. Power, L. Kaiser, M. Bavar- ian, C. Winter, P. Tillet, F. P. Such, D. Cum- mings, M. Plapper...
work page internal anchor Pith review Pith/arXiv arXiv
-
[11]
S. Chen, S. Wong, L. Chen, and Y. Tian. Extend- ing context window of large language mod- els via positional interpolation.arXiv preprint arXiv:2306.15595,
work page internal anchor Pith review Pith/arXiv arXiv
-
[12]
Microsoft COCO Captions: Data Collection and Evaluation Server
X.Chen, H.Fang, T.-Y.Lin, R.Vedantam, S.Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/1504.00325,
work page internal anchor Pith review Pith/arXiv arXiv
-
[13]
F. Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547,
work page internal anchor Pith review Pith/arXiv arXiv 1911
-
[14]
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Explor- ing the surprising difficulty of natural yes/no questions. CoRR, abs/1905.10044,
work page internal anchor Pith review Pith/arXiv arXiv 1905
-
[15]
Training Verifiers to Solve Math Word Problems
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schul- man. Training verifiers to solve math word problems. CoRR, abs/2110.14168,
work page internal anchor Pith review Pith/arXiv arXiv
-
[16]
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
A. Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,
work page internal anchor Pith review Pith/arXiv arXiv 2010
-
[17]
Test of time: A benchmark for evaluating llms on temporal reasoning
B. Fatemi, M. Kazemi, A. Tsitsulin, K. Malkan, J. Yim, J. Palowitch, S. Seo, J. Halcrow, and B. Perozzi. Test of time: A benchmark for evaluating llms on temporal reasoning.arXiv preprint arXiv:2406.09170,
-
[18]
X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. Blink: Multimodal large language models can see but not perceive.ArXiv, abs/2404.12390,
work page internal anchor Pith review Pith/arXiv arXiv
-
[19]
arXiv preprint arXiv:2410.02089 , year=
12 Gemma 3 Technical Report J. Gehring, K. Zheng, J. Copet, V. Mella, T. Cohen, and G. Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learn- ing. arXiv preprint arXiv:2410.02089,
-
[20]
Gemma 2: Improving Open Language Models at a Practical Size
Gemma Team. Gemma: Open models based on gemini research and technology, 2024a. Gemma Team. Gemma 2: Improving open lan- guage models at a practical size.arXiv preprint arXiv:2408.00118, 2024b. O. Goldman, U. Shaham, D. Malkin, S. Eiger, A. Hassidim, Y. Matias, J. Maynez, A. M. Gi- lady, J.Riesa, S.Rijhwani, L.Rimell, I.Szpektor, R. Tsarfaty, and M. Eyal. ...
work page internal anchor Pith review Pith/arXiv arXiv
-
[21]
Measuring Massive Multitask Language Understanding
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understand- ing. CoRR, abs/2009.03300,
work page internal anchor Pith review Pith/arXiv arXiv 2009
-
[22]
J. Hessel, A. Marasović, J. D. Hwang, L. Lee, J. Da, R. Zellers, R. Mankoff, and Y. Choi. Do an- droids laugh at electric sheep? humor" under- standing"benchmarksfromthenewyorkercap- tion contest. arXiv preprint arXiv:2209.06293,
-
[23]
Distilling the Knowledge in a Neural Network
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,
work page internal anchor Pith review Pith/arXiv arXiv
-
[24]
RULER: What's the Real Context Size of Your Long-Context Language Models?
C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,
work page internal anchor Pith review Pith/arXiv arXiv
-
[25]
arXiv preprint arXiv:2210.17546 , year=
D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini. Preventing verbatim memorization in language models gives a false sense of pri- vacy. arXiv preprint arXiv:2210.17546,
-
[26]
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. CoRR, abs/1705.03551,
work page internal anchor Pith review Pith/arXiv arXiv
-
[27]
M. Kazemi, H. Alvari, A. Anand, J. Wu, X. Chen, and R. Soricut. Geomverse: A systematic eval- uation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241,
-
[28]
M. Kazemi, N. Dikkala, A. Anand, P. Dević, I. Das- gupta, F. Liu, B. Fatemi, P. Awasthi, D. Guo, S. Gollapudi, and A. Qureshi. Remi: A dataset for reasoning with multiple images. ArXiv, abs/2406.09175, 2024a. M. Kazemi, Q. Yuan, D. Bhatia, N. Kim, X. Xu, V. Imbrasaite, and D. Ramachandran. Boardgameqa: A dataset for natural lan- guage reasoning with contr...
-
[29]
A Diagram Is Worth A Dozen Images
A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Ha- jishirzi, and A. Farhadi. A diagram is worth a dozen images. ArXiv, abs/1603.07396,
work page internal anchor Pith review Pith/arXiv arXiv
-
[30]
13 Gemma 3 Technical Report E. Kıcıman, R. Ness, A. Sharma, and C. Tan. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050,
-
[31]
Tulu 3: Pushing Frontiers in Open Language Model Post-Training
N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. T \" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124,
work page internal anchor Pith review Pith/arXiv arXiv
-
[32]
LLaMa Team. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
work page internal anchor Pith review Pith/arXiv arXiv
-
[33]
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. Gsm-symbolic: Understanding the limitations of mathemati- cal reasoning in large language models.arXiv preprint arXiv:2410.05229,
work page internal anchor Pith review Pith/arXiv arXiv
-
[34]
M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scal- able extraction of training data from (pro- duction) language models. arXiv preprint arXiv:2311.17035,
work page internal anchor Pith review Pith/arXiv arXiv
-
[35]
A. Ramé, J. Ferret, N. Vieillard, R. Dadashi, L. Hussenot, P.-L. Cedoz, P. G. Sessa, S. Girgin, A. Douillard, and O. Bachem. WARP: On the benefits of weight averaged rewarded policies, 2024a. A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret. WARM: Onthebenefitsofweightaveragedrewardmod- els. In ICML, 2024b. 14 Gemma 3 T...
work page internal anchor Pith review Pith/arXiv arXiv
-
[36]
How to train data-efficient LLMs
N. Sachdeva, B. Coleman, W.-C. Kang, J. Ni, L. Hong, E. H. Chi, J. Caverlee, J. McAuley, and D. Z. Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.09668,
-
[37]
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641,
work page internal anchor Pith review Pith/arXiv arXiv 1907
-
[38]
E. Sánchez, B. Alastruey, C. Ropers, P. Stenetorp, M. Artetxe, and M. R. Costa-jussà. Linguini: A benchmark for language-agnostic linguistic reasoning. arXiv preprint arXiv:2409.12126 ,
-
[39]
M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y. Choi. Socialiqa: Commonsense reasoning about social interactions. CoRR, abs/1904.09728,
work page internal anchor Pith review Pith/arXiv arXiv 1904
- [40]
-
[41]
H. Singh, N. Gupta, S. Bharadwaj, D. Tewari, and P. Talukdar. Indicgenbench: a multilin- gual benchmark to evaluate generation capabil- ities of llms on indic languages.arXiv preprint arXiv:2404.16816, 2024a. S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Lon...
- [42]
-
[43]
Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M
15 Gemma 3 Technical Report K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640,
-
[44]
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saiful- lah, S. Naidu, et al. Livebench: A challeng- ing, contamination-free llm benchmark.arXiv preprint arXiv:2406.19314,
work page internal anchor Pith review Pith/arXiv arXiv
-
[45]
International Conference on Learning Representations (ICLR) , year=
M. Wortsman, P. J. Liu, L. Xiao, K. Everett, A.Alemi, B.Adlam, J.D.Co-Reyes, I.Gur, A.Ku- mar, R. Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322,
- [46]
- [47]
-
[48]
16 Gemma 3 Technical Report Core contributors Aishwarya Kamath∗ Johan Ferret∗ Shreya Pathak∗ Nino Vieillard∗ Ramona Merhej∗ Sarah Perrin∗ Tatiana Matejovicova∗ Alexandre Ramé∗ Morgane Rivière∗ Louis Rouillard∗ Thomas Mesnard∗ Geoffrey Cideron∗ Jean-bastien Grill∗ Sabela Ramos∗ Edouard Yvinec∗ Michelle Casbon∗ Etienne Pot Ivo Penchev Gaël Liu Francesco Vis...
work page 2019
-
[49]
Evaluation details are described in Table
We consider several standard bench- marks, namely MMLU (Hendrycks et al., 2020), MMLU-Pro (Wang et al., 2024), AGIEval (Zhong et al., 2023), MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), GPQA (Rein et al., 2023), MBPP (Austin et al., 2021), Hu- manEval (Chen et al., 2021). Evaluation details are described in Table
work page 2020
-
[50]
Overall we see a consis- tent improvement over STEM abilities across our Gemma 2 Gemma 3 2B 9B 27B 4B 12B 27B MMLU 52.2 71.2 75.2 59.6 74.5 78.6 MMLUpro 22.2 43.7 49.4 29.2 45.3 52.2 AGIE 31.6 53.1 55.1 42.1 57.4 66.2 MATH 16.4 36.4 42.1 24.2 43.3 50.0 GSM8K 25.0 70.2 74.6 38.4 71.0 82.6 GPQA Diamond 12.5 24.8 26.3 15.0 25.4 24.3 MBPP 31.0 51.2 60.8 46.0 ...
work page 2015
-
[51]
VQAv2 84.8 85.8 85.8 84.1 84.9 85.1 Tally QA 80.6 82.4 82.1 79.0 81.3 81.7 Table 12| Performance of pre-trained checkpoints after fine-tuning on multi-modal benchmarks (without P&S). PaliGemma 2 was transferred at 896x896 resolution for the first four benchmarks, and at 448x448 resolution for the others. Comparison to PaliGemma 2.We fine-tune mul- timodal...
work page 2024
-
[52]
Evaluation details are described in Ta- ble
et al., 2022), XQuAD (Artetxe et al., 2020), ECLeKTic (Goldman et al., 2025), IndicGen- Bench (Singh et al., 2024a), XOR QA (Asai et al., 2020). Evaluation details are described in Ta- ble
work page 2022
-
[53]
benchmarks evaluating at 32K and 128K sequence lengths. 8.1. Performance of IT models We report in Table 18, additional benchmarks on our IT models. Note that N2C refers to Natural2Code, the Gemini 1.0 internal held-out dataset, which uses author-generated sources in- stead of web-based information. BBEH refers to BIG-Bench Extra Hard (Kazemi et al., 2025...
work page 2025
-
[54]
Performance of IT models on video under- standing Additional multimodal evaluations
8.2. Performance of IT models on video under- standing Additional multimodal evaluations. Gemma 3 IT models were evaluated on common vision benchmarks following the evaluation protocol of Gemini 1.5 (Gemini Team, 2024). The results are given in Table 16 when P&S is activated. 4B 12B 27B Perception Test MCVQA 50.6 54.9 58.1 ActivityNet-QA 46.3 50.4 52.8 Ta...
work page 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.