pith. sign in

arxiv: 2405.04434 · v5 · submitted 2024-05-07 · 💻 cs.CL · cs.AI

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI , Aixin Liu , Bei Feng , Bin Wang , Bingxuan Wang , Bo Liu , Chenggang Zhao , Chengqi Dengr
show 149 more authors
Chong Ruan Damai Dai Daya Guo Dejian Yang Deli Chen Dongjie Ji Erhang Li Fangyun Lin Fuli Luo Guangbo Hao Guanting Chen Guowei Li H. Zhang Hanwei Xu Hao Yang Haowei Zhang Honghui Ding Huajian Xin Huazuo Gao Hui Li Hui Qu J.L. Cai Jian Liang Jianzhong Guo Jiaqi Ni Jiashi Li Jin Chen Jingyang Yuan Junjie Qiu Junxiao Song Kai Dong Kaige Gao Kang Guan Lean Wang Lecong Zhang Lei Xu Leyi Xia Liang Zhao Liyue Zhang Meng Li Miaojun Wang Mingchuan Zhang Minghua Zhang Minghui Tang Mingming Li Ning Tian Panpan Huang Peiyi Wang Peng Zhang Qihao Zhu Qinyu Chen Qiushi Du R.J. Chen R.L. Jin Ruiqi Ge Ruizhe Pan Runxin Xu Ruyi Chen S.S. Li Shanghao Lu Shangyan Zhou Shanhuang Chen Shaoqing Wu Shengfeng Ye Shirong Ma Shiyu Wang Shuang Zhou Shuiping Yu Shunfeng Zhou Size Zheng T. Wang Tian Pei Tian Yuan Tianyu Sun W.L. Xiao Wangding Zeng Wei An Wen Liu Wenfeng Liang Wenjun Gao Wentao Zhang X.Q. Li Xiangyue Jin Xianzu Wang Xiao Bi XiaoDong Liu Xiaohan Wang Xiaojin Shen Xiaokang Chen Xiaosha Chen Xiaotao Nie Xiaowen Sun Xiaoxiang Wang Xin Liu Xin Xie Xingkai Yu Xinnan Song Xinyi Zhou Xinyu Yang Xuan Lu Xuecheng Su Y. Wu Y.K. Li Y.X. Wei Y.X. Zhu Yanhong Xu Yanping Huang Yao Li Yao Zhao Yaofeng Sun Yaohui Li Yaohui Wang Yi Zheng Yichao Zhang Yiliang Xiong Yilong Zhao Ying He Ying Tang Yishi Piao Yixin Dong Yixuan Tan Yiyuan Liu Yongji Wang Yongqiang Guo Yuchen Zhu Yuduan Wang Yuheng Zou Yukun Zha Yunxian Ma Yuting Yan Yuxiang You Yuxuan Liu Z.Z. Ren Zehui Ren Zhangli Sha Zhe Fu Zhen Huang Zhen Zhang Zhenda Xie Zhewen Hao Zhihong Shao Zhiniu Wen Zhipeng Xu Zhongyu Zhang Zhuoshu Li Zihan Wang Zihui Gu Zilin Li Ziwei Xie
This is my paper

Pith reviewed 2026-05-11 05:30 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords Mixture-of-ExpertsLanguage ModelEfficient InferenceKV Cache CompressionSparse ComputationLarge Language ModelsParameter EfficiencyMulti-head Latent Attention
0
0 comments X

The pith

DeepSeek-V2 shows a Mixture-of-Experts model with 236 billion total parameters but only 21 billion activated per token can match top open-source language models while lowering training costs and inference demands.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces DeepSeek-V2 as a large language model built with sparse activation to make both training and running the system more practical. It combines a new attention method that shrinks memory use during generation with an expert routing design that limits computation to a small subset of parameters for each token. The model is pretrained on 8.1 trillion tokens and then refined with supervised fine-tuning and reinforcement learning to reach its reported results. A sympathetic reader would care because the approach suggests high-performing language models need not require the full compute budget of dense alternatives, which could widen access to capable systems. The reported outcomes include stronger benchmark scores than the prior 67 billion parameter DeepSeek model along with clear savings in cost, memory, and speed.

Core claim

DeepSeek-V2 is a Mixture-of-Experts model with 236 billion total parameters of which 21 billion activate for each token and a maximum context length of 128 thousand tokens. It incorporates Multi-head Latent Attention to compress the key-value cache into a compact latent vector and DeepSeekMoE to perform sparse computation during training. After pretraining on a high-quality 8.1 trillion token corpus and subsequent supervised fine-tuning plus reinforcement learning, the model surpasses the performance of DeepSeek 67B while cutting training costs by 42.5 percent, shrinking the KV cache by 93.3 percent, and raising maximum generation throughput by a factor of 5.76. The chat versions of DeepSeek

What carries the argument

Multi-head Latent Attention (MLA) and DeepSeekMoE, which together compress the KV cache and restrict computation to a sparse subset of experts so that model capacity grows without proportional increases in active parameters or memory.

If this is right

  • Training budgets for strong language models can be reduced without sacrificing benchmark results.
  • Inference hardware can support higher throughput and longer contexts because the KV cache occupies far less memory.
  • Open-source models become more competitive with closed systems when active parameter counts stay low.
  • Sparse activation patterns allow scaling total model size while keeping per-token compute manageable.
  • Fine-tuning steps such as SFT and RL can further unlock capability after economical pretraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar sparse designs could be adapted to reduce energy use in large-scale AI training across different model families.
  • The efficiency gains may make 128K context practical for more real-time or interactive applications.
  • Future experiments could test whether the same active-parameter ratio holds performance when the model is scaled beyond 236 billion total parameters.
  • The approach connects efficiency improvements directly to accessibility for researchers with modest compute resources.

Load-bearing premise

The reported performance and efficiency advantages arise from the MLA and DeepSeekMoE designs rather than from differences in training data selection or unstated implementation details.

What would settle it

An independent replication that trains the exact architecture on a comparable corpus but fails to match the claimed performance, cost savings, or throughput gains would falsify the central claim.

read the original abstract

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper presents DeepSeek-V2, a 236B total parameter Mixture-of-Experts language model with 21B activated parameters per token and 128K context length. It introduces Multi-head Latent Attention (MLA) for KV cache compression and DeepSeekMoE for sparse computation, pretrained on 8.1T tokens followed by SFT and RL. The central claims are that it significantly outperforms DeepSeek-67B while reducing training costs by 42.5%, KV cache by 93.3%, and increasing generation throughput by 5.76x, achieving top-tier performance among open-source models despite the low activated parameter count.

Significance. If the performance and efficiency results hold under fair, standardized evaluations, this would represent a meaningful advance in economical LLM scaling by showing that targeted architectural innovations in attention and MoE routing can deliver strong results with substantially lower active compute and memory costs. The large-scale pretraining corpus and measured gains provide concrete data points that could inform future work on sparse models.

major comments (2)
  1. Evaluation section: The top-tier performance claim with only 21B activated parameters is load-bearing for the paper's contribution. The manuscript must explicitly document the evaluation protocol (number of shots, prompt templates, decoding strategy, and temperature) applied identically to DeepSeek-V2 and all baselines (including DeepSeek-67B and other open-source models). Without this, it remains possible that reported gains reflect differences in evaluation setup rather than the MLA or DeepSeekMoE innovations.
  2. Training and efficiency claims (abstract and §3): The 42.5% training cost reduction and 93.3% KV cache reduction are presented as direct consequences of the architecture. The paper should provide the precise calculation method (e.g., total FLOPs, wall-clock time on specified hardware, or token throughput) and confirm that the comparison to DeepSeek-67B normalizes for the 8.1T token corpus and any differences in training infrastructure.
minor comments (2)
  1. Abstract: The phrase 'top-tier performance' is used without reference to specific benchmark scores or tables; adding one or two key numbers (e.g., average score on MMLU or GSM8K) would improve clarity for readers.
  2. Notation: The definitions of MLA and DeepSeekMoE are introduced in the abstract and early sections; a brief equation or diagram reference in the main text would help readers quickly locate the formal description.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We have addressed both major comments by expanding the manuscript with explicit documentation of the evaluation protocol and precise methodological details on the efficiency calculations. These revisions strengthen the clarity and reproducibility of our claims without altering the core results.

read point-by-point responses

  1. Referee: Evaluation section: The top-tier performance claim with only 21B activated parameters is load-bearing for the paper's contribution. The manuscript must explicitly document the evaluation protocol (number of shots, prompt templates, decoding strategy, and temperature) applied identically to DeepSeek-V2 and all baselines (including DeepSeek-67B and other open-source models). Without this, it remains possible that reported gains reflect differences in evaluation setup rather than the MLA or DeepSeekMoE innovations.

    Authors: We agree that documenting a uniform evaluation protocol is essential to substantiate the performance claims. In the revised manuscript, we have added a new subsection (Section 4.1) that fully specifies the protocol: all models (DeepSeek-V2, DeepSeek-67B, and other open-source baselines) were evaluated using the LM Evaluation Harness with identical prompt templates, 0-shot prompting for the majority of benchmarks (5-shot only where standard practice requires it, e.g., certain MMLU subsets), greedy decoding (temperature = 0, no top-p or nucleus sampling), and the same maximum generation length. This ensures the reported gains are attributable to the architectural contributions rather than evaluation discrepancies. revision: yes

  2. Referee: Training and efficiency claims (abstract and §3): The 42.5% training cost reduction and 93.3% KV cache reduction are presented as direct consequences of the architecture. The paper should provide the precise calculation method (e.g., total FLOPs, wall-clock time on specified hardware, or token throughput) and confirm that the comparison to DeepSeek-67B normalizes for the 8.1T token corpus and any differences in training infrastructure.

    Authors: We thank the referee for requesting greater precision on these figures. In the revision, we have expanded Section 3 and the abstract with explicit calculation details. The 93.3% KV cache reduction is obtained by comparing the memory footprint of MLA's compressed latent vector (dimension d_c per head) against the full KV cache of standard multi-head attention (2 * num_heads * head_dim per token); the percentage is computed as (1 - (d_c / (2 * num_heads * head_dim))) * 100%. The 42.5% training cost reduction is derived from total FLOPs required to process the identical 8.1T token corpus: DeepSeekMoE activates only 21B parameters per token versus 67B for the dense baseline, yielding lower effective compute per token. Both models were trained on the same 8.1T tokens using the same A100 GPU cluster; costs are normalized by aggregate FLOPs with no differences in infrastructure or data. Wall-clock time and token throughput measurements on identical hardware are also now reported for transparency. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical model presentation with measured results

full rationale

The paper introduces DeepSeek-V2 as an MoE model with MLA and DeepSeekMoE architectures, describes its training on an 8.1T-token corpus followed by SFT/RL, and reports direct empirical measurements of performance, training cost savings (42.5%), KV cache reduction (93.3%), and throughput gains. No mathematical derivation chain, first-principles predictions, or fitted parameters are claimed; results are obtained from actual pretraining and evaluation runs. Comparisons to DeepSeek-67B and other models are presented as measured outcomes rather than outputs derived from the model's own inputs or self-citations. The architecture descriptions and efficiency claims rest on the explicit design choices (latent attention compression, sparse MoE routing) and observed hardware metrics, without reduction to prior fitted constants or self-referential definitions. This is a standard empirical systems paper with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 2 invented entities

The central claim rests on the effectiveness of two newly invented architectural components whose value is demonstrated only through the reported experiments. No independent prior validation or formal proof is supplied.

free parameters (1)
  • Expert count and routing hyperparameters
    Standard MoE design choices that must be tuned to achieve the claimed performance-efficiency trade-off.
axioms (1)
  • domain assumption Transformer attention and feed-forward layers remain effective when sparsified via expert routing
    The model extends the standard transformer and MoE paradigm without re-deriving its foundations.
invented entities (2)
  • Multi-head Latent Attention (MLA) no independent evidence
    purpose: Compress KV cache into a latent vector for efficient inference
    Newly proposed mechanism with no prior independent evidence outside this work.
  • DeepSeekMoE no independent evidence
    purpose: Enable economical training through sparse expert activation
    New MoE variant introduced in this paper.

pith-pipeline@v0.9.0 · 6157 in / 1500 out tokens · 37578 ms · 2026-05-11T05:30:50.710829+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

    cs.CV 2026-05 unverdicted novelty 8.0

    VideoMLA applies multi-head latent attention with 3D-RoPE decoupling to autoregressive video diffusion, delivering 92.7% KV memory reduction while matching short-horizon baselines and leading long-horizon VBench scores.

  2. OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agents

    cs.LG 2026-05 unverdicted novelty 8.0

    OTora provides the first unified framework for reasoning-level denial-of-service attacks on LLM agents, achieving up to 10x more reasoning tokens and order-of-magnitude latency increases while preserving task accuracy...

  3. LiveBench: A Challenging, Contamination-Limited LLM Benchmark

    cs.CL 2024-06 unverdicted novelty 8.0

    LiveBench is a contamination-limited LLM benchmark with auto-scored challenging tasks from recent sources across math, coding, reasoning and more, where top models score below 70%.

  4. CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

    cs.DC 2026-06 unverdicted novelty 7.0

    CrossPool separates weights and KV-cache into distinct GPU pools plus a planner, virtualizer, and layer-wise scheduler to cut P99 time-between-tokens by up to 10.4x versus prior kvcached multi-LLM systems.

  5. MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarios

    cs.LG 2026-06 unverdicted novelty 7.0

    MacroLens is a point-in-time multi-signal benchmark dataset and seven tasks for evaluating contextual financial reasoning models under macroeconomic scenarios.

  6. Dual Dimensionality for Local and Global Attention

    cs.CL 2026-06 unverdicted novelty 7.0

    Distance-Adaptive Representation (DAR) keeps full KV dimensionality inside a local window and reduces it to 1/4 outside, matching full-dimensional baselines on pretraining (70M-410M) and 1B-scale fine-tuning while uni...

  7. Models Take Notes at Prefill: KV Cache Can Be Editable and Composable

    cs.LG 2026-06 unverdicted novelty 7.0

    KV caches function as notebooks of prefilled conclusions, enabling field-level edits that recover decisions (especially with CoT) and position-portable skill composition with near-identical outputs at O(L) cost.

  8. STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

    cs.LG 2026-06 unverdicted novelty 7.0

    STAR-KV applies differentiable soft thresholding for per-head and per-block adaptive low-rank KV cache compression, combined with hybrid decomposition and low-rank-aware quantization, achieving up to 75% compression a...

  9. Depth-Attention: Cross-Layer Value Mixing for Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    Depth-Attention mixes values from earlier layers into the current attention value by having the query attend to previous-layer keys at the same position, yielding lower perplexity and up to 2.3 points higher average a...

  10. LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

    cs.CL 2026-06 unverdicted novelty 7.0

    LazyAttention kernelizes deferred positional encoding to enable zero-copy, position-agnostic KV cache reuse, delivering 1.37× lower TTFT and 1.40× higher throughput than Block-Attention under skewed document distribut...

  11. Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics

    cs.DC 2026-05 unverdicted novelty 7.0

    On a real multi-node H100 cluster the authors show that for MLA, routing the ~1 KB compressed query row is cheaper than moving cache chunks and supply a topology-aware cost model accurate to ~7% on IBGDA fabrics.

  12. Training-Free Looped Transformers

    cs.LG 2026-05 unverdicted novelty 7.0

    Training-free looped transformers retrofit recurrence to frozen models via damped ODE sub-steps on mid-stack blocks, yielding gains such as +2.64 pp on MMLU-Pro for Qwen3-4B.

  13. HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs

    cs.DC 2026-05 unverdicted novelty 7.0

    HyperParallel-MoE achieves up to 1.58x lower Dispatch-to-Combine MoE-FFN latency on Ascend A3 clusters via tile-level heterogeneous scheduling of AIC and AIV resources.

  14. Latent Cache Flow: Model-to-Model Communication Without Text

    cs.LG 2026-05 unverdicted novelty 7.0

    Latent Cache Flow uses a small joint-translation-and-compression adapter to let LLMs with different contexts exchange KV cache summaries, outperforming both larger C2C adapters and text in early experiments.

  15. Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation

    cs.LG 2026-05 unverdicted novelty 7.0

    Text2CAD-Bench supplies 600 dual-prompt examples across four geometric and domain levels to test LLMs on text-to-parametric CAD, finding solid basic performance but sharp drops on complex topology and advanced features.

  16. LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    LLMForge is a NAS framework with Infinite-Head Attention, a Forge-Former surrogate, and Forge-DSE engine that discovers hardware-specific architectures for edge language models, yielding variants with improved accurac...

  17. $\phi$-Balancing for Mixture-of-Experts Training

    cs.LG 2026-05 unverdicted novelty 7.0

    φ-balancing is a convex optimization method for population-level expert balance in MoE training that derives an online EMA adjustment and outperforms heuristic baselines.

  18. GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

    cs.LG 2026-05 unverdicted novelty 7.0

    GQLA exposes dual MQA-absorb and GQA decoding paths from identical parameters to enable hardware-adaptive LLM inference while preserving cache compression on one path and GQA-level traffic on the other.

  19. Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

    stat.ML 2026-05 unverdicted novelty 7.0

    MSD eliminates dequantization from the GEMM path by decomposing BF16 activations into multiple low-precision parts that multiply directly with INT8 or MXFP4 weights, achieving near-16 effective bits for INT8 and 6.6 f...

  20. The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures

    cs.DC 2026-05 unverdicted novelty 7.0

    Power capping is illusory in LLM decode as memory-bound operation leaves power headroom untouched on 700 W GPUs, while SM clock locking saves up to 32% energy and three DVFS classes appear across attention types.

  21. Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference

    cs.DC 2026-05 unverdicted novelty 7.0

    EEP makes wide expert-parallel MoE serving survive single-rank failures with an 11s recovery pause, 8s reintegration pause, and throughput restored to 95% of pre-fault level within 52s while staying within 4.4% of a f...

  22. Simply Stabilizing the Loop via Fully Looped Transformer

    cs.LG 2026-05 unverdicted novelty 7.0

    Fully Looped Transformer stabilizes looped training up to 12 iterations via distributed inter-loop signals and attention injection, improving downstream performance by up to 13.2%.

  23. When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Standard top-k routers in MoE language models often select suboptimal routes for difficult tokens, and updating only the final router layer raises pass@K on AIME and HMMT benchmarks across multiple models.

  24. KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

    cs.LG 2026-05 unverdicted novelty 7.0

    KernelBench-X benchmark shows task category predicts LLM kernel correctness better than method choice, iterative refinement trades performance for higher success rates, and correctness does not ensure efficiency gains...

  25. KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

    cs.LG 2026-05 conditional novelty 7.0

    KernelBenchX benchmark shows task category explains nearly three times more variance in LLM kernel correctness than method choice, iterative refinement boosts correctness but reduces performance, and quantization rema...

  26. Misrouter: Exploiting Routing Mechanisms for Input-Only Attacks on Mixture-of-Experts LLMs

    cs.CR 2026-05 unverdicted novelty 7.0

    Misrouter enables input-only attacks on MoE LLMs by optimizing queries on open-source surrogates to route toward weakly aligned experts and transferring them to public APIs.

  27. When Is the Same Model Not the Same Service? A Measurement Study of Hosted Open-Weight LLM APIs

    cs.PF 2026-05 conditional novelty 7.0

    Hosted open-weight LLMs function as heterogeneous, time-varying services rather than uniform model artifacts, with concentrated demand, decoupled supply and adoption, and measurable gains from task-aware routing.

  28. When Is the Same Model Not the Same Service? A Measurement Study of Hosted Open-Weight LLM APIs

    cs.PF 2026-05 unverdicted novelty 7.0

    Hosted open-weight LLM APIs function as time-varying heterogeneous services rather than fixed model artifacts, with demand concentrated, supply-use mismatches, and task-specific routing yielding major cost and through...

  29. Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

    cs.LG 2026-04 unverdicted novelty 7.0

    Incompressible Knowledge Probes enable log-linear estimation of LLM parameter counts from factual accuracy on obscure questions, showing continued scaling of knowledge capacity across open and closed models.

  30. DPC: A Distributed Page Cache over CXL

    cs.DC 2026-04 conditional novelty 7.0

    DPC maintains exactly one DRAM copy of each file page in a CXL-connected cluster and delivers up to 12.4X speedup (5.6X geometric mean) over replicated caches on data-sharing workloads.

  31. Using large language models for embodied planning introduces systematic safety risks

    cs.AI 2026-04 unverdicted novelty 7.0

    LLM planners for robots often produce dangerous plans even when planning succeeds, with safety awareness staying flat as model scale improves planning ability.

  32. Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

    cs.LG 2026-04 unverdicted novelty 7.0

    Counterfactual Routing awakens dormant experts in MoE models via layer-wise perturbation and a new CEI metric, raising factual accuracy 3.1% on average across TruthfulQA, FACTOR, and TriviaQA without extra inference cost.

  33. The Phase Is the Gradient: Equilibrium Propagation for Frequency Learning in Kuramoto Networks

    cs.LG 2026-04 unverdicted novelty 7.0

    In Kuramoto networks at equilibrium, weak nudging makes phase displacement the exact gradient of loss w.r.t. natural frequencies, enabling frequency learning that beats weight learning and resolves convergence via spe...

  34. Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning

    cs.AI 2026-04 unverdicted novelty 7.0

    COVERT generates verifiable synthetic tool-use environments for RL by validated trajectory synthesis and oracle-preserving augmentations, improving tool-use accuracy on BFCL v3 and ACEBench while remaining complementa...

  35. How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles

    cs.AI 2026-04 unverdicted novelty 7.0

    A new auditing framework reveals widespread behavioral entanglement among LLMs and shows that reweighting ensembles based on measured independence improves verification accuracy by up to 4.5%.

  36. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    cs.LG 2026-03 conditional novelty 7.0

    EvoESAP uses evolutionary search guided by a speculative-decoding-inspired ESAP metric to discover non-uniform layer-wise sparsity allocations for MoE expert pruning, improving generation accuracy up to 19.6% at 50% sparsity.

  37. SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators

    cs.AI 2025-11 unverdicted novelty 7.0

    SnapStream deploys sparse KV attention in a production inference system on dataflow accelerators, delivering 4x on-chip memory savings for DeepSeek-671B at 128k context with up to 1832 tokens/sec and minimal accuracy ...

  38. EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving

    cs.AI 2025-09 unverdicted novelty 7.0

    EngiBench shows LLMs accuracy drops with task complexity, degrades under perturbations, and stays below human performance on open-ended engineering problems.

  39. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

    cs.LG 2024-08 conditional novelty 7.0

    Loss-Free Balancing keeps expert loads balanced in MoE models by dynamically adjusting routing-score biases based on recent usage, avoiding auxiliary-loss interference and yielding better performance.

  40. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

    cs.LG 2024-07 accept novelty 7.0

    FlashAttention-3 achieves 1.5-2x speedup on H100 GPUs for attention, reaching 740 TFLOPs/s (75% utilization) in FP16 and near 1.2 PFLOPs/s in FP8 while cutting numerical error by 2.6x versus baseline FP8 attention.

  41. Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

    cs.DC 2026-07 unverdicted novelty 6.0

    A load-aware prefill deflection scheduler for disaggregated LLM serving reduces P95 TTFT by up to 81% by interleaving chunked prefill on decode nodes and eliminating KV-cache transfers.

  42. MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

    cs.LG 2026-07 unverdicted novelty 6.0

    MosaicKV achieves up to 16x attention speedup, 4.8x lower decode latency, 7.3x higher throughput, and 3x memory reduction with 1.76% accuracy loss via dynamic two-D KV cache compression and management on H800 GPUs.

  43. Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts

    cs.LG 2026-06 unverdicted novelty 6.0

    Geometric and stochastic analysis shows lower-order discontinuities dominate in SMoE; a simple smoothing method enforces continuity with small overhead and empirical gains.

  44. FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

    cs.LG 2026-06 unverdicted novelty 6.0

    FoMoE partitions expert layers across workers in MoE LLMs, skips non-resident experts, and reports up to 1.42x lower communication than baselines plus 1.4x throughput gains while maintaining stable routing.

  45. Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity

    cs.LG 2026-06 unverdicted novelty 6.0

    Rotary positional encodings reduce the symmetry group of functional equivalence in attention compared to sinusoidal encodings, increasing expressivity and altering linear mode connectivity patterns.

  46. RepSelect: Robust LLM Unlearning via Representation Selectivity

    cs.CL 2026-06 unverdicted novelty 6.0

    RepSelect isolates forget-set-specific representations via gradient PCA collapse to achieve 4-50x better post-relearning robustness than baselines across multiple models and forget categories.

  47. Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

    cs.DC 2026-06 unverdicted novelty 6.0

    A CPU-GPU hybrid design with stream-loading prefill, expert parallelism, and disaggregation achieves cloud SLOs for local MoE inference on dual-socket CPUs and consumer GPUs.

  48. End-to-End Context Compression at Scale

    cs.CL 2026-06 unverdicted novelty 6.0

    LCLMs are scaled 0.6B-encoder 4B-decoder compressors pre-trained on over 350B tokens that improve the Pareto frontier for general-task performance, compression speed, and peak memory in long-context language model inference.

  49. STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning

    cs.AI 2026-06 unverdicted novelty 6.0

    STAR rethinks MoE routing as structure-aware subspace learning by adding a GHA-tracked principal subspace to standard routers, yielding more stable specialization and better performance on synthetic, language, and vis...

  50. AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

    cs.LG 2026-06 unverdicted novelty 6.0

    AlphaQ performs calibration-free mixed-precision quantization of MoE models by allocating higher bits to experts whose weight spectra exhibit stronger heavy-tailed structure according to HT-SR theory, outperforming ca...

  51. Value-Aware Stochastic KV Cache Eviction for Reasoning Models

    cs.LG 2026-06 unverdicted novelty 6.0

    VaSE improves KV cache eviction accuracy for reasoning models by over 4% versus prior eviction methods at 4x compression through value-magnitude protection and stochastic diversity.

  52. KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

    cs.LG 2026-06 unverdicted novelty 6.0

    KVarN uses Hadamard rotation plus dual-axis variance normalization on K and V matrices to cut token-scale errors and error accumulation in KV-cache quantization, reaching new SOTA at 2-bit on MATH500, AIME24 and HumanEval.

  53. Do Transformers Need Three Projections? Systematic Study of QKV Variants

    cs.LG 2026-06 conditional novelty 6.0

    Q-K=V projection sharing in transformers matches standard QKV performance with 50% KV cache reduction and combines with GQA/MQA for up to 96.9% reduction across vision and language tasks.

  54. Do Value Vectors in Deep Layers Need Context from the Residual Stream?

    cs.CL 2026-06 unverdicted novelty 6.0

    Deeper transformer layers benefit from context-free token-specific value vectors in a Bank of Values lookup table, improving performance over standard attention with less compute.

  55. Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads

    cs.DC 2026-06 unverdicted novelty 6.0

    Albireo overlaps non-scalable overheads with compute in tensor-parallel LLM inference to raise the empirical optimal TP degree, delivering up to 1.9x throughput and 48% lower latency versus vLLM.

  56. Blurry Window Attention

    cs.LG 2026-05 unverdicted novelty 6.0

    Blurry Window Attention stores a frequency window and reconstructs blurry KV history via Dirichlet kernel interpolation, achieving 8x better state efficiency than sliding window attention on the MQAR synthetic task.

  57. Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    AsyMoE adds hyperbolic geometry for cross-modal hierarchies and evidence-priority experts to address vision-language asymmetry in LVLMs, reporting 1.5% average gains and 25.45% fewer active parameters.

  58. Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models

    cs.CL 2026-05 unverdicted novelty 6.0

    RA-MoE is a three-stage fine-tuning framework that aligns routing in MoE middle layers for multilingual tasks using a four-way example taxonomy and routing alignment loss, outperforming standard SFT across models, tas...

  59. Pruning and Distilling Mixture-of-Experts into Dense Language Models

    cs.CL 2026-05 unverdicted novelty 6.0

    A systematic MoE-to-dense conversion via expert scoring, grouping, and distillation yields +6.3 pp average accuracy over dense-to-dense pruning at matched parameter count on tested models.

  60. MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

    cs.LG 2026-05 unverdicted novelty 6.0

    MONA integrates Nesterov acceleration into Muon's orthogonalization framework, reporting better convergence than Muon and AdamW on MoE models up to 68B parameters trained on 1T tokens and SOTA fine-tuning results.

Reference graph

Works this paper leans on

150 extracted references · 150 canonical work pages · cited by 177 Pith papers · 48 internal anchors

  1. [1]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    work page 2024

  2. [3]

    Introducing Claude , 2023

    Anthropic. Introducing Claude , 2023. URL https://www.anthropic.com/index/introducing-claude

    work page 2023

  3. [7]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herb...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  4. [11]

    D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.06066, 2024. URL https://doi.org/10.48550/arXiv.2401.06066

    work page internal anchor Pith review doi:10.48550/arxiv.2401.06066 2024

  5. [12]

    T. Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning, 2023

    work page 2023

  6. [13]

    DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

    DeepSeek-AI. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401.02954, 2024. URL https://doi.org/10.48550/arXiv.2401.02954

    work page internal anchor Pith review doi:10.48550/arxiv.2401.02954 2024

  7. [16]

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/abs/2101.03961

    work page internal anchor Pith review arXiv 2021

  8. [17]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile : An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  9. [18]

    Introducing gemini: our largest and most capable ai model, 2023

    Google. Introducing gemini: our largest and most capable ai model, 2023. URL https://blog.google/technology/ai/google-gemini-ai/

    work page 2023

  10. [19]

    A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024

    work page 2024

  11. [22]

    Hai-llm: 高效且轻量的大模型训练工具, 2023

    High-flyer. Hai-llm: 高效且轻量的大模型训练工具, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm

    work page 2023

  12. [23]

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y ., and Ginsburg, B

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. CoRR, abs/2401.18079, 2024. URL https://doi.org/10.48550/arXiv.2401.18079

    work page doi:10.48550/arxiv.2401.18079 2024

  13. [25]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.arXiv preprint arXiv:2305.08322, 2023

    Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval : A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023

    work page arXiv 2023

  14. [29]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    work page 2023

  15. [31]

    Lepikhin, H

    D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb

    work page 2021

  16. [32]

    H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU : Measuring massive multitask language understanding in Chinese . arXiv preprint arXiv:2306.09212, 2023

    work page internal anchor Pith review arXiv 2023

  17. [33]

    W. Li, F. Qi, M. Sun, X. Yi, and J. Zhang. Ccpm: A chinese classical poetry matching dataset, 2021

    work page 2021

  18. [36]

    Cheaper, better, faster, stronger: Continuing to push the frontier of ai and making it accessible to all, 2024

    Mistral. Cheaper, better, faster, stronger: Continuing to push the frontier of ai and making it accessible to all, 2024. URL https://mistral.ai/news/mixtral-8x22b

    work page 2024

  19. [37]

    Introducing ChatGPT , 2022

    OpenAI. Introducing ChatGPT , 2022. URL https://openai.com/blog/chatgpt

    work page 2022

  20. [38]

    GPT-4 Technical Report

    OpenAI. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [39]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    work page 2022

  22. [42]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE, 2020

    work page 2020

  23. [43]

    Riquelme, J

    C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 8583--8595, 2021. URL https://proceedings.neurips.cc/paper/202...

    work page 2021

  24. [44]

    Sakaguchi, R

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019

    work page 2019

  25. [46]

    N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  26. [47]

    Shazeer, A

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017 . OpenReview.net, 2017. URL https://openreview.net/forum?id=B1ckMDqlg

    work page 2017

  27. [48]

    J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    work page 2024

  28. [49]

    K. Sun, D. Yu, D. Yu, and C. Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension, 2019

    work page 2019

  29. [51]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, . Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    work page 2017

  30. [53]

    T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023

    work page 2023

  31. [57]

    Y. Zhao, C. Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci. Atom: Low-bit quantization for efficient and accurate LLM serving. CoRR, abs/2310.19102, 2023. URL https://doi.org/10.48550/arXiv.2310.19102

    work page doi:10.48550/arxiv.2310.19102 2023

  32. [59]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    work page 2023

  33. [61]

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  34. [63]

    Zero Bubble Pipeline Parallelism

    Zero Bubble Pipeline Parallelism , author=. arXiv preprint arXiv:2401.10241 , year=

    work page arXiv

  35. [64]

    The Eleventh International Conference on Learning Representations,

    Freda i and Mirac Suzgun and Markus Freitag and Xuezhi Wang and Suraj Srivats and Soroush Vosoughi and Hyung Won Chung and Yi Tay and Sebastian Ruder and Denny Zhou and Dipanjan Das and Jason Wei , title =. The Eleventh International Conference on Learning Representations,. 2023 , url =

    work page 2023

  36. [65]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. arXiv preprint arXiv:2403.07974 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  37. [66]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    work page 2024

  38. [67]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints , author=. arXiv preprint arXiv:2305.13245 , year=

    work page internal anchor Pith review arXiv

  39. [68]

    CoRR , volume =

    Noam Shazeer , title =. CoRR , volume =. 2019 , url =

    work page 2019

  40. [69]

    WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct , author=. arXiv preprint arXiv:2308.09583 , year=

    work page internal anchor Pith review arXiv

  41. [70]

    ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

    Zhibin Gou and Zhihong Shao and Yeyun Gong and Yelong Shen and Yujiu Yang and Minlie Huang and Nan Duan and Weizhu Chen , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2309.17452 , eprinttype =. 2309.17452 , timestamp =

    work page internal anchor Pith review doi:10.48550/arxiv.2309.17452 2023

  42. [71]

    Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

    Wenhu Chen and Xueguang Ma and Xinyi Wang and William W. Cohen , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2211.12588 , eprinttype =. 2211.12588 , timestamp =

    work page internal anchor Pith review doi:10.48550/arxiv.2211.12588 2022

  43. [72]

    International Conference on Machine Learning,

    Luyu Gao and Aman Madaan and Shuyan Zhou and Uri Alon and Pengfei Liu and Yiming Yang and Jamie Callan and Graham Neubig , editor =. International Conference on Machine Learning,. 2023 , url =

    work page 2023

  44. [73]

    Chi and Quoc V

    Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed H. Chi and Quoc V. Le and Denny Zhou , title =. NeurIPS , year =

    work page

  45. [74]

    Mishra, M

    Swaroop Mishra and Matthew Finlayson and Pan Lu and Leonard Tang and Sean Welleck and Chitta Baral and Tanmay Rajpurohit and Oyvind Tafjord and Ashish Sabharwal and Peter Clark and Ashwin Kalyan , editor =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,. 2022 , url =. doi:10.18653/V1/2022.EMNLP-MAIN.392 , timestamp =

    work page doi:10.18653/v1/2022.emnlp-main.392 2022

  46. [75]

    MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

    Xiang Yue and Xingwei Qu and Ge Zhang and Yao Fu and Wenhao Huang and Huan Sun and Yu Su and Wenhu Chen , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2309.05653 , eprinttype =. 2309.05653 , timestamp =

    work page internal anchor Pith review doi:10.48550/arxiv.2309.05653 2023

  47. [76]

    MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

    Longhui Yu and Weisen Jiang and Han Shi and Jincheng Yu and Zhengying Liu and Yu Zhang and James T. Kwok and Zhenguo Li and Adrian Weller and Weiyang Liu , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2309.12284 , eprinttype =. 2309.12284 , timestamp =

    work page internal anchor Pith review doi:10.48550/arxiv.2309.12284 2023

  48. [77]

    Weld and Luke Zettlemoyer , editor =

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel and Zettlemoyer, Luke. T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017. doi:10.18653/v1/P17-1147

    work page doi:10.18653/v1/p17-1147 2017

  49. [78]

    2020 , eprint=

    Language Models are Few-Shot Learners , author=. 2020 , eprint=

    work page 2020

  50. [79]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

    work page

  51. [80]

    Introducing

    OpenAI , url =. Introducing

    work page

  52. [81]

    HAI-LLM: 高效且轻量的大模型训练工具 , author =

    work page

  53. [82]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Megatron-lm: Training multi-billion parameter language models using model parallelism , author=. arXiv preprint arXiv:1909.08053 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  54. [83]

    Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , pages=

    Efficient large-scale language model training on gpu clusters using megatron-lm , author=. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , pages=

    work page

  55. [84]

    Proceedings of Machine Learning and Systems , volume=

    Reducing activation recomputation in large transformer models , author=. Proceedings of Machine Learning and Systems , volume=

    work page

  56. [85]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Flash. Advances in Neural Information Processing Systems , year=

    work page

  57. [86]

    Dao, Tri , year=. Flash

    work page

  58. [87]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    work page

  59. [88]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

    work page

  60. [89]

    SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages=

    Zero: Memory optimizations toward training trillion parameter models , author=. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages=. 2020 , organization=

    work page 2020

  61. [90]

    2021 , eprint=

    CCPM: A Chinese Classical Poetry Matching Dataset , author=. 2021 , eprint=

    work page 2021

  62. [91]

    2018 , eprint=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. 2018 , eprint=

    work page 2018

  63. [92]

    Introducing

    Anthropic , institution =. Introducing

    work page

  64. [93]

    An important next step on our

    Google , url =. An important next step on our

    work page

  65. [94]

    Introducing Gemini: our largest and most capable AI model , author =

    work page

  66. [95]

    2019 , eprint=

    Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension , author=. 2019 , eprint=

    work page 2019

  67. [96]

    A Span-Extraction Dataset for C hinese Machine Reading Comprehension

    Cui, Yiming and Liu, Ting and Che, Wanxiang and Xiao, Li and Chen, Zhipeng and Ma, Wentao and Wang, Shijin and Hu, Guoping. A Span-Extraction Dataset for C hinese Machine Reading Comprehension. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E...

    work page doi:10.18653/v1/d19-1600 2019

  68. [97]

    2019 , eprint=

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. 2019 , eprint=

    work page 2019

  69. [98]

    2023 , eprint=

    CMATH: Can Your Language Model Pass Chinese Elementary School Math Test? , author=. 2023 , eprint=

    work page 2023

  70. [99]

    Measuring Massive Multitask Language Understanding

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  71. [100]

    Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

    Challenging big-bench tasks and whether chain-of-thought can solve them , author=. arXiv preprint arXiv:2210.09261 , year=

    work page internal anchor Pith review arXiv

  72. [101]

    Program Synthesis with Large Language Models

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  73. [102]

    CLUE: A Chinese language understanding evaluation benchmark

    Liang Xu and Hai Hu and Xuanwei Zhang and Lu Li and Chenjie Cao and Yudong Li and Yechen Xu and Kai Sun and Dian Yu and Cong Yu and Yin Tian and Qianqian Dong and Weitang Liu and Bo Shi and Yiming Cui and Junyi Li and Jun Zeng and Rongzhao Wang and Weijian Xie and Yanting Li and Yina Patterson and Zuoyu Tian and Yiwen Zhang and He Zhou and Shaoweihua Liu ...

    work page doi:10.18653/v1/2020.coling-main.419 2020

  74. [103]

    Li, Haonan and Zhang, Yixuan and Koto, Fajri and Yang, Yifei and Zhao, Hai and Gong, Yeyun and Duan, Nan and Baldwin, Timothy , journal=

    work page

  75. [104]

    Chujie Zheng and Minlie Huang and Aixin Sun , editor =. ChID:. Proceedings of the 57th Conference of the Association for Computational Linguistics,. 2019 , url =. doi:10.18653/V1/P19-1075 , timestamp =

    work page doi:10.18653/v1/p19-1075 2019

  76. [105]

    Hovy , editor =

    Guokun Lai and Qizhe Xie and Hanxiao Liu and Yiming Yang and Eduard H. Hovy , editor =. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,. 2017 , url =. doi:10.18653/V1/D17-1082 , timestamp =

    work page doi:10.18653/v1/d17-1082 2017

  77. [106]

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,

    Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner , editor =. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,. 2019 , url =. doi:10.18653/V1/N19-1246 , timestamp =

    work page doi:10.18653/v1/n19-1246 2019

  78. [107]

    Huang, Yuzhen and Bai, Yuzhuo and Zhu, Zhihao and Zhang, Junlei and Zhang, Jinghan and Su, Tangjun and Liu, Junteng and Lv, Chuancheng and Zhang, Yikai and Lei, Jiayi and others , journal=

    work page

  79. [108]

    LLaMA: Open and Efficient Foundation Language Models

    Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth. arXiv preprint arXiv:2302.13971 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  80. [109]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton. Llama 2: Open Foundation and Fine-Tuned Chat Models , journal =. 2023 , url =. doi:10.48550/arXiv.2307.09288 , eprinttype =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2307.09288 2023

Showing first 80 references.