Qwen2 Technical Report
Pith reviewed 2026-05-10 14:02 UTC · model grok-4.3
The pith
Qwen2 releases open models from 0.5B to 72B parameters that outperform most prior open-weight systems on language, coding, math, and reasoning benchmarks while supporting about 30 languages.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The Qwen2 series consists of foundational and instruction-tuned language models from 0.5 to 72 billion parameters, featuring both dense models and a Mixture-of-Experts model. The 72B base model records 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH. The instruction-tuned 72B version scores 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Qwen2 also demonstrates strong capabilities across approximately 30 languages, and all model weights are released openly on Hugging Face and ModelScope along with tools for quantization, fine-tuning, and deployment.
What carries the argument
The Qwen2 model family, a set of scaled dense and Mixture-of-Experts language models whose training yields the reported benchmark gains and multilingual coverage.
If this is right
- Developers can integrate competitive open models into applications for language understanding and generation tasks.
- Researchers obtain new strong baselines for advancing work in multilingual systems, coding assistance, and mathematical reasoning.
- The public release of weights and fine-tuning resources supports customization for domain-specific uses.
- Global users benefit from built-in proficiency across roughly 30 languages for broader accessibility.
Where Pith is reading between the lines
- Widespread use of these models could narrow the practical gap between open and closed AI systems in everyday applications.
- The multilingual reach may accelerate development of tools suited to non-English markets and cross-language tasks.
- Community fine-tuning on the released weights could produce specialized variants that extend performance in targeted areas like coding or reasoning.
Load-bearing premise
The chosen benchmarks accurately and fairly represent the models' overall capabilities without significant evaluation artifacts or selective reporting.
What would settle it
An independent run of the same benchmarks that produces substantially lower scores for Qwen2 models than reported, especially compared to Qwen1.5, would undermine the performance claims.
read the original abstract
This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models. We release a comprehensive suite of foundational and instruction-tuned language models, encompassing a parameter range from 0.5 to 72 billion, featuring dense models and a Mixture-of-Experts model. Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks on language understanding, generation, multilingual proficiency, coding, mathematics, and reasoning. The flagship model, Qwen2-72B, showcases remarkable performance: 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH as a base language model. The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Moreover, Qwen2 demonstrates robust multilingual capabilities, proficient in approximately 30 languages, spanning English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, Vietnamese, and more, underscoring its versatility and global reach. To foster community innovation and accessibility, we have made the Qwen2 model weights openly available on Hugging Face and ModelScope, and the supplementary materials including example code on GitHub. These platforms also include resources for quantization, fine-tuning, and deployment, facilitating a wide range of applications and research endeavors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Qwen2 series of LLMs, including dense models (0.5B to 72B parameters) and a MoE model. It reports that the Qwen2-72B base model achieves 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH. The Qwen2-72B-Instruct scores 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. The models demonstrate strong multilingual capabilities across approximately 30 languages and are released openly along with resources for quantization, fine-tuning, and deployment.
Significance. This technical report contributes significantly to the open-source LLM ecosystem by providing a family of models that achieve competitive or superior performance to prior open models on a wide range of tasks including understanding, coding, math, and reasoning. The public release of the model weights enables direct use and further fine-tuning by the research community, potentially accelerating progress in multilingual and specialized applications.
major comments (1)
- [Evaluation] Evaluation section: The specific scores for benchmarks like GPQA (37.9 for base) and LiveCodeBench (35.7 for instruct) are presented without detailing the prompting method, number of few-shot examples, or decoding parameters used. This omission makes it challenging to independently verify the claims of surpassing other models under identical conditions.
minor comments (3)
- The abstract lists several languages but states 'approximately 30 languages'; a more precise count or complete list in the main text would improve clarity.
- Consider including a comparison table that explicitly lists the scores of Qwen1.5 and other models alongside Qwen2 for direct visual comparison.
- Ensure consistency in reporting whether scores are for base or instruct models across all mentioned benchmarks.
Simulated Author's Rebuttal
We thank the referee for the positive assessment of the Qwen2 Technical Report and the recommendation for minor revision. We address the single major comment below.
read point-by-point responses
-
Referee: [Evaluation] Evaluation section: The specific scores for benchmarks like GPQA (37.9 for base) and LiveCodeBench (35.7 for instruct) are presented without detailing the prompting method, number of few-shot examples, or decoding parameters used. This omission makes it challenging to independently verify the claims of surpassing other models under identical conditions.
Authors: We agree that detailed evaluation protocols are necessary for reproducibility and fair comparison. The current manuscript provides limited information on these aspects for GPQA and LiveCodeBench. In the revised version, we will expand the Evaluation section (and add an appendix if needed) to specify the prompting methods, number of few-shot examples, and decoding parameters (including temperature, top-p, and max new tokens) used for these and other benchmarks. This addition will enable independent verification under identical conditions. revision: yes
Circularity Check
No significant circularity detected
full rationale
The Qwen2 Technical Report is an empirical model release paper whose central claims consist of benchmark scores on external, independently defined datasets (MMLU, GPQA, HumanEval, GSM8K, BBH, MT-Bench, Arena-Hard, LiveCodeBench). No derivations, equations, fitted parameters, or predictions are presented that reduce to self-defined quantities or self-citation chains. Comparisons to prior models and proprietary systems rely on publicly reported numbers rather than internal redefinitions. The results are directly falsifiable by third-party evaluation of the released weights.
Axiom & Free-Parameter Ledger
Forward citations
Cited by 60 Pith papers
-
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
UltraEP is the first exact-load real-time expert balancer for large-EP MoE training and serving on rack-scale nodes, reaching 94.3% of ideal throughput and 1.49x over no-balancing.
-
Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models
Adaptive scheduling of interventions in discrete diffusion language models, timed to attribute-specific commitment schedules discovered with sparse autoencoders, delivers precise multi-attribute steering up to 93% str...
-
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
HeadQ removes 84-94% of excess perplexity from 2-bit key quantization by storing low-rank residuals in a calibration-learned query basis for score-space correction and using A²-weighted distortion for values.
-
AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks
AgentSocialBench demonstrates that privacy preservation is fundamentally harder in human-centered agentic social networks than in single-agent cases due to cross-domain coordination pressures and an abstraction parado...
-
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
Molmo2 delivers state-of-the-art open-weight video VLMs with new grounding datasets and training methods that outperform prior open models and match or exceed some proprietary ones on pointing and tracking tasks.
-
Large Language Diffusion Models
LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.
-
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo VLMs trained on newly collected PixMo open datasets achieve state-of-the-art performance among open-weight models and surpass multiple proprietary VLMs including Claude 3.5 Sonnet and Gemini 1.5 Pro.
-
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
MMMU-Pro is a stricter multimodal benchmark that removes text-only solvable questions, augments options, and requires reading text from images, yielding substantially lower model scores of 16.8-26.9%.
-
COSM: A Cooperative Scheduling Framework for Concurrent PIM and CPU Execution on Mobile Devices
COSM is a cooperative scheduling framework for concurrent PIM and CPU execution on mobile devices that hides PIM latency and overlaps execution with data transfer, achieving up to 2.8x PIM throughput with less than 2%...
-
Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression
DLR creates discrete latent tokens from rendered CoT images via clustering, enabling up to 20x compression and interpretable trajectories that outperform continuous latent baselines on reasoning tasks.
-
The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order
Lie brackets of gradient fields provide a geometric score for optimal transfer order in sequential learning, enabling efficient ranking of many sources via tournament methods.
-
Verifiable Auto-Formalization of Mathematics Using a Relaxed Natural Formal Language
Introduces Relaxed NFL intermediate language for LLM-based auto-formalization, with rule-plus-LLM elaboration to Core NFL and tactic-language discharge of verification conditions.
-
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
RealMath-Eval benchmark shows LLM judges have an evaluation gap, performing worse on diverse real human math reasoning than on synthetic solutions due to greater error diversity and higher surprisal.
-
Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior
INNSteer learns an invertible neural network to map LLM activations into a latent space where linear steering becomes more effective, then applies the inverse map to produce nonlinear interventions in the original space.
-
DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination
DICE formalizes multi-agent LLM coordination as discounted incomplete-information Markov games and introduces Heterogeneous Quantal Response Equilibrium (HQRE) to achieve unique stable equilibria with bounded regret, ...
-
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Fisher-MoE prunes sparse intermediate dimensions in MoE FFNs ranked by Fisher importance, delivering 50% compression that preserves capability while cutting memory ~45% and raising throughput 21%.
-
GeM-NR: Geometry-Aware Multi-View Editing for Nonrigid Scene Changes
GeM-NR performs multi-view consistent nonrigid editing by aligning depth-derived point clouds between edited and unedited scenes then refining projections conditioned on the original query view.
-
Knowledge Index of Noah's Ark
Introduces KINA benchmark with 899 items over 261 disciplines, formal (1-1/e) coverage guarantee and bonus-on-bar tournament theorem, plus evaluations of 42 models with top score 53.17%.
-
UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD
UniCAD supplies a unified multi-modal benchmark and an end-to-end MLLM that performs reconstruction, generation, and QA on CAD data, reporting SOTA results on UniCAD and Fusion360.
-
Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models
Defines representational capacity as the upper bound on distinguishable near-orthogonal directions in transformer latent spaces, derived from embedding similarity distributions and an adjusted Johnson-Lindenstrauss fo...
-
Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
A new fault-injection framework enables a systematic empirical study that produces 17 takeaways on error propagation in LLM inference and four software-only mitigation directions.
-
Task-Induced Representational Invariances Depend on Learning Objective in Deep RL
In navigation tasks, DQN learns MDP-homomorphism-invariant representations while PPO learns action-symmetric ones despite comparable performance, with effects on transfer and in LLMs.
-
Adaptive Order Policies for Masked Diffusion
A policy network learns to choose unmasking order in masked diffusion by reweighting the loss, outperforming random and heuristic baselines on ordering-sensitive tasks.
-
Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs
Moral Trolley Arena shows frontier LLMs produce composite moral preferences that are compressed rather than additive functions of calibrated component act strengths across Moral Foundations Theory.
-
Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning
CSMR is a multimodal reasoning framework that uses a language model to schedule when to acquire visual evidence from an independent perception module, outperforming baselines in zero-shot settings.
-
MolLingo: Molecule-Native Representations for LLM-Powered Scientific Agents
MolLingo introduces a multi-agent framework with BFE molecular representation and docking-grounded reasoning to outperform frontier LLMs on molecular design benchmarks including fourfold docking score gains.
-
Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs
Video-LLMs exhibit directional motion blindness from a direction binding gap; DeltaDirect projector objective lifts synthetic accuracy to 85.4% and real accuracy by 21.9 points while preserving other video capabilities.
-
IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions
IdioLink introduces a benchmark dataset and evaluation showing that strong embedding models struggle to retrieve equivalent meanings across idiomatic and literal forms, relying on shallow cues instead.
-
TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimization
TextReg mitigates prompt distributional overfitting via regularized text-space optimization, reporting up to +16.5% OOD accuracy gains over prior methods on reasoning benchmarks.
-
Learning First Integrals via Backward-Generated Data and Guided Reinforcement Learning
FISolver trains a compact LLM on backward-generated (differential equation, first integral) pairs and uses guided reinforcement learning to outperform larger models and Mathematica on first-integral benchmarks at lower cost.
-
DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Action
DuplexSLA introduces a three-channel full-duplex architecture that synchronizes continuous user audio, discrete assistant audio, and rate-limited textual actions inside a single backbone for native turn-taking and in-...
-
Distribution-Aware Reward: Reinforcement Learning over Predictive Distributions for LLM Regression
Distribution-Aware Reward optimizes LLM regression by treating rollouts as empirical predictive distributions and rewarding marginal improvements in CRPS quality rather than point accuracy alone.
-
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
Chronicle is the first model jointly pretrained from scratch on text and time series in a unified transformer that matches a comparable language model on NLU tasks and sets new bars for time series classification and ...
-
Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
Presents TRUST-Bench benchmark for hidden-trigger tool compromises in LLM agents and VISTA-Guard framework for trajectory-aware risk scoring of final actions under untrusted feedback.
-
Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
Fine-tuning LLMs on an unseen language teaches syntax but fails to transfer semantic competence, leaving Python with up to a 19% performance advantage and no tested intervention closing the gap.
-
Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Space
The paper introduces Manta-LM, which approximates the Hamilton-Jacobi-Bellman optimal policy via Flow Matching in a rectified latent control space to enable high-fidelity parallel language generation.
-
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
KVServe delivers up to 9.13x job completion time speedup and 32.8x time-to-first-token reduction by making KV cache compression service-aware and adaptive in disaggregated LLM serving.
-
Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment
Temperature adjustment on the reference model generalizes inference-time alignment to SLOP ensembles of reward models, with a calibration algorithm that improves robustness to reward hacking while preserving alignment...
-
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
TokAlign++ learns token alignments between LLM vocabularies from monolingual representations to enable faster adaptation, better text compression, and effective token-level distillation across 15 languages with minimal steps.
-
Controlling Logical Collapse in LLMs via Algebraic Ontology Projection over F2
Projecting LLM hidden states onto F2 algebra with 42 pairs yields 93% zero-shot accuracy on logical relations and identifies prompt-preventable late-layer collapse.
-
Towards Automated Air Traffic Safety Assessment Around Non-Towered Airports Using Large Language Models
Large language models achieve macro F1 scores above 0.85 on binary nominal-versus-danger classification from CTAF radio transcripts and METAR weather data using a new synthetic dataset with a 12-category hazard taxonomy.
-
Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
Entropy polarity from a first-order entropy change approximation enables Polarity-Aware Policy Optimization (PAPO) that preserves complementary polarity branches and outperforms baselines on math and agentic RL fine-t...
-
Predictive Maps of Multi-Agent Reasoning: A Successor-Representation Spectrum for LLM Communication Topologies
Successor-representation spectra of row-stochastic communication operators predict perturbation robustness, consensus speed, and error accumulation in multi-agent LLM topologies, with condition number showing perfect ...
-
Privacy-preserving Chunk Scheduling in a BitTorrent Implementation of Federated Learning
FLTorrent achieves within-round source unlinkability in decentralized federated learning via a BitTorrent warm-up with pre-round obfuscation, randomized lags, and coordination-only non-owner-first scheduling, reaching...
-
GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation
GraphInstruct introduces a six-level progressive benchmark with 800 instructions and 1,582 references to diagnose LLM graph generation gaps, plus a verification-guided iterative prompting framework that improves performance.
-
GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation
GraphInstruct is a progressive benchmark with six complexity levels for LLM graph generation that identifies multi-constraint composition as the hardest point and shows a verification-guided iterative framework outper...
-
Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
SeCo performs semantic-driven context compression for LLMs by anchoring on query-relevant semantic centers and applying consistency-weighted token merging, yielding better downstream performance, lower latency, and st...
-
Towards Backdoor-Based Ownership Verification for Vision-Language-Action Models
GuardVLA embeds a stealthy backdoor watermark in VLAs via secret messages in visual data and uses a swap-and-detect mechanism for post-release ownership verification that preserves task performance.
-
Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
Queryable LoRA adds dynamic routing over shared low-rank atoms with attention and language-instruction regularization to make parameter-efficient fine-tuning more adaptive across inputs and layers.
-
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
Chain-based Distillation constructs a sequence of anchor models to enable efficient initialization of variable-sized SLMs through interpolation, with bridge distillation for cross-architecture transfer, yielding bette...
-
Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
The cumulative token IS ratio gives unbiased prefix correction and lower variance than full-sequence ratios for token-level gradients in LLM policy optimization, enabling CTPO to outperform GRPO and GSPO baselines on ...
-
Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions
Performance collapse in layer-pruned LLMs stems from disrupting the Silent Phase of decision-making, which blocks the transition to correct predictions, while the later Decisive Phase is robust to pruning.
-
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
GLoRA replaces raw factor averaging with gauge-aware aggregation in a consensus subspace estimated from client projectors, enabling consistent low-rank federated LoRA under heterogeneity.
-
Logic-Regularized Verifier Elicits Reasoning from LLMs
LOVER creates an unsupervised logic-regularized verifier that reaches 95% of supervised verifier performance on reasoning tasks across 10 datasets.
-
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
A single fused int4 KV cache kernel on Apple Silicon outperforms fp16 in latency with 3x memory compression and near-zero quality loss on tested models.
-
Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation
Decoding-time use of process reward models for bias mitigation raises fairness scores by up to 0.40 on a bilingual benchmark while preserving fluency across four LLMs and extends to open-ended generation with low overhead.
-
QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testing
A multi-agent LLM system cuts false positives in static application security testing by 88.6% on the OWASP Benchmark while dropping recall by only 3.1%.
-
InvEvolve: Evolving White-Box Inventory Policies via Large Language Models with Performance Guarantees
InvEvolve evolves white-box inventory policies from LLMs with statistical safety guarantees and outperforms classical and deep learning methods on synthetic and real retail data.
-
PuzzleMark: Implicit Jigsaw Learning for Robust Code Dataset Watermarking in Neural Code Completion Models
PuzzleMark provides a robust and imperceptible watermarking method for code datasets using adaptive variable name concatenation and statistical verification, achieving perfect detection rates with minimal performance impact.
-
R-CoT: A Reasoning-Layer Watermark via Redundant Chain-of-Thought in Large Language Models
R-CoT embeds watermarks into LLM reasoning paths via redundant CoT and GRPO-based dual optimization, maintaining over 95% true positive rate under fine-tuning and post-training changes.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.