pith. sign in

arxiv: 2401.04088 · v1 · submitted 2024-01-08 · 💻 cs.LG · cs.CL

Mixtral of Experts

Albert Q. Jiang , Alexandre Sablayrolles , Antoine Roux , Arthur Mensch , Blanche Savary , Chris Bamford , Devendra Singh Chaplot , Diego de las Casas
show 18 more authors
Emma Bou Hanna Florian Bressand Gianna Lengyel Guillaume Bour Guillaume Lample L\'elio Renard Lavaud Lucile Saulnier Marie-Anne Lachaux Pierre Stock Sandeep Subramanian Sophia Yang Szymon Antoniak Teven Le Scao Th\'eophile Gervet Thibaut Lavril Thomas Wang Timoth\'ee Lacroix William El Sayed
This is my paper

Pith reviewed 2026-05-24 04:06 UTC · model grok-4.3

classification 💻 cs.LG cs.CL
keywords sparse mixture of expertslanguage modelmodel efficiencybenchmark performanceinstruction tuningopen source modelrouter network
0
0 comments X

The pith

Mixtral 8x7B activates only 13B parameters yet matches or exceeds Llama 2 70B and GPT-3.5 on benchmarks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Mixtral 8x7B as a sparse mixture of experts language model built on the Mistral 7B architecture. Each layer contains eight expert feedforward blocks, and a router network picks two experts to process every token while combining their outputs. This design gives each token access to 47B total parameters but computes only 13B active parameters at inference time. The model is trained with a 32k token context length and is shown to match or surpass Llama 2 70B and GPT-3.5 across evaluated tasks, with especially large gains on mathematics, code generation, and multilingual benchmarks. An instruction-tuned version further outperforms GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B chat on human preference benchmarks.

Core claim

Mixtral 8x7B is a Sparse Mixture of Experts language model that shares the architecture of Mistral 7B except that every layer consists of eight feedforward expert blocks. For each token at each layer a router selects two experts to process the current state and combine their outputs. The result is a model with 47B total parameters that activates only 13B during inference, trained on a 32k token context, and that outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks, particularly on mathematics, code generation, and multilingual tasks, while the instruct version surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B chat on human benchmarks.

What carries the argument

The router network that, at each layer, selects two out of eight expert feedforward blocks to process the current token state and combines their outputs.

If this is right

  • Sparse expert activation allows a higher total parameter count without a proportional increase in inference computation.
  • Different experts can be chosen for different tokens at each timestep, increasing effective model capacity.
  • Strong gains appear on mathematics, code generation, and multilingual tasks relative to dense models of similar active size.
  • Instruction fine-tuning of the base model produces a version that leads on human preference benchmarks.
  • Open release under Apache 2.0 license makes the base and instruct models available for further use and study.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same routing mechanism could be applied at larger scales to keep inference cost manageable while growing total capacity.
  • Expert specialization might be further encouraged by training objectives that reward distinct expert behaviors.
  • Combining sparse activation with other efficiency techniques such as quantization could compound resource savings.
  • Performance on additional domains or longer contexts beyond the reported 32k could be tested to map the limits of the approach.

Load-bearing premise

The reported benchmark scores reflect genuine generalization rather than data contamination, evaluation protocol differences, or undisclosed training choices.

What would settle it

Independent evaluation of both models on a fresh set of benchmarks constructed to have no overlap with possible training data, showing that Mixtral no longer matches or exceeds the compared models.

Figures

Figures reproduced from arXiv: 2401.04088 by Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L\'elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Th\'eophile Gervet, Thibaut Lavril, Thomas Wang, Timoth\'ee Lacroix, William El Sayed.

Figure 1
Figure 1. Figure 1: Mixture of Experts Layer. Each input vector is assigned to 2 of the 8 experts by a router. The layer’s output is the weighted sum of the outputs of the two selected experts. In Mixtral, an expert is a standard feedforward block as in a vanilla transformer architecture. Mixtral demonstrates superior capabilities in mathematics, code generation, and tasks that require multilingual understanding, significantl… view at source ↗
Figure 2
Figure 2. Figure 2: Performance of Mixtral and different Llama models on a wide range of benchmarks [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results on MMLU, commonsense reasoning, world knowledge and reading comprehension, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (Right) shows that the perplexity of Mixtral on a subset of the proof-pile dataset [2] decreases monotonically as the size of the context increases [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Bias Benchmarks. Compared Llama 2 70B, Mixtral presents less bias (higher accuracy on BBQ, lower std on BOLD) and displays more positive sentiment (higher avg on BOLD). To identify possible flaws to be corrected by fine-tuning / preference modeling, we measure the base model performance on Bias Benchmark for QA (BBQ) [24] and Bias in Open-Ended Language Generation Dataset (BOLD) [10]. BBQ is a dataset of h… view at source ↗
Figure 6
Figure 6. Figure 6: LMSys Leaderboard. (Screenshot from Dec 22, 2023) Mixtral 8x7B Instruct v0.1 achieves an Arena Elo rating of 1121 outperforming Claude-2.1 (1117), all versions of GPT-3.5-Turbo (1117 best), Gemini Pro (1111), and Llama-2-70b-chat (1077). Mixtral is currently the best open-weights model by a large margin. 3 https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard 6 [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 7
Figure 7. Figure 7: Proportion of tokens assigned to each expert on different domains from The Pile dataset for [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Text samples where each token is colored with the first expert choice. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Proportion of tokens assigned to each expert on different subsets from The Pile dataset, separated [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Repeated consecutive assignments per MoE layer. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
read the original abstract

We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces Mixtral 8x7B, a sparse mixture-of-experts (SMoE) language model with the same architecture as Mistral 7B but with 8 feedforward experts per layer; a router selects 2 experts per token, yielding 47B total parameters but only 13B active during inference. The base model is claimed to match or outperform Llama 2 70B and GPT-3.5 across benchmarks (with particularly large gains on mathematics, code generation, and multilingual tasks), while the instruction-tuned variant surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B chat on human preference benchmarks. Both models are released under Apache 2.0.

Significance. If the reported benchmark results prove robust, the work provides concrete evidence that sparse MoE designs can deliver competitive performance at substantially lower inference cost than dense models of comparable total size. The open-source release of both base and instruct checkpoints is a clear strength that enables independent verification and downstream research.

major comments (2)
  1. [Abstract] Abstract: The central claims that Mixtral 'outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks' and 'vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks' are stated without any description of the training corpus, decontamination statistics, or overlap analysis with the reported benchmarks (MMLU, HumanEval, MATH, etc.). Because these empirical head-to-head comparisons constitute the paper's primary contribution, the absence of this information renders the headline performance deltas impossible to evaluate for contamination or protocol artifacts.
  2. [Abstract] Abstract and evaluation sections: No information is supplied on exact prompting templates, few-shot counts, decoding parameters, or statistical significance testing for any benchmark score. The lack of error bars, multiple-run statistics, or protocol disclosure means the reported deltas cannot be assessed for reliability or reproducibility, directly undermining the strength of the outperformance assertions.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below, focusing on the substance of the concerns regarding transparency in training and evaluation details.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claims that Mixtral 'outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks' and 'vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks' are stated without any description of the training corpus, decontamination statistics, or overlap analysis with the reported benchmarks (MMLU, HumanEval, MATH, etc.). Because these empirical head-to-head comparisons constitute the paper's primary contribution, the absence of this information renders the headline performance deltas impossible to evaluate for contamination or protocol artifacts.

    Authors: We acknowledge the absence of training corpus details and decontamination statistics in the manuscript. As a commercial lab, we cannot disclose proprietary training data composition or perform public overlap analysis. Standard decontamination practices were followed during training, and the open release of model weights enables independent verification of the reported results by the community. The primary contribution remains the architectural demonstration and efficiency gains rather than exhaustive data provenance, which is consistent with prior open model releases. revision: no

  2. Referee: [Abstract] Abstract and evaluation sections: No information is supplied on exact prompting templates, few-shot counts, decoding parameters, or statistical significance testing for any benchmark score. The lack of error bars, multiple-run statistics, or protocol disclosure means the reported deltas cannot be assessed for reliability or reproducibility, directly undermining the strength of the outperformance assertions.

    Authors: We agree that additional protocol details would strengthen reproducibility. In the revised version we will add an evaluation appendix specifying the exact few-shot counts, prompting templates (following the original benchmark papers), decoding parameters, and noting that scores reflect single-run evaluations using standard setups, as is conventional for large-model reports. This addresses the request without altering the core claims. revision: yes

standing simulated objections not resolved
  • Full disclosure of the training corpus composition, decontamination statistics, and overlap analysis, which are proprietary.

Circularity Check

0 steps flagged

No circularity: empirical benchmark claims are external comparisons

full rationale

The paper introduces an SMoE architecture and reports direct head-to-head benchmark scores against independently developed external models (Llama 2 70B, GPT-3.5, etc.). No derivation chain, equations, or fitted parameters are presented as predictions; the central claims rest on external evaluation protocols rather than self-referential reductions or self-citation load-bearing steps. This is a standard empirical model release paper whose performance assertions are falsifiable against the cited baselines.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

Performance claims rest on the effectiveness of the learned router, the choice of exactly 8 experts and top-2 routing, and the training data and procedure, none of which receive independent justification in the abstract.

free parameters (2)
  • number of experts = 8
    Architectural hyperparameter set to 8 per layer to control total capacity versus active compute.
  • experts activated per token = 2
    Routing hyperparameter set to 2 to achieve the stated active-parameter budget.
axioms (1)
  • domain assumption A learned router can reliably select task-relevant experts without degrading overall model quality.
    Invoked by the SMoE design when claiming that activating only two experts preserves or improves performance relative to dense baselines.

pith-pipeline@v0.9.0 · 5864 in / 1478 out tokens · 69153 ms · 2026-05-24T04:06:44.736429+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

    cs.DC 2026-06 unverdicted novelty 8.0

    UltraEP is the first exact-load real-time expert balancer for large-EP MoE training and serving on rack-scale nodes, reaching 94.3% of ideal throughput and 1.49x over no-balancing.

  2. HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts

    cs.LG 2026-05 unverdicted novelty 8.0

    HodgeCover isolates the harmonic kernel of a simplicial Laplacian on an expert 2-complex to identify irreducible merge cycles and selects experts for aggressive compression, matching or exceeding baselines on open-wei...

  3. Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Models

    cs.AR 2026-05 conditional novelty 8.0

    Sieve dynamically schedules MoE experts across GPU and PIM hardware to handle bimodal token distributions, achieving 1.3x to 1.6x gains in throughput and interactivity over static prior PIM systems on three large models.

  4. ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning

    cs.LG 2026-05 conditional novelty 8.0

    ReLibra uses pre-known token-to-expert routing from RL rollouts to perform inter-batch expert reordering and intra-batch replication, delivering up to 1.6x higher throughput than Megatron-LM and 1.2x over oracle-equip...

  5. Efficient Training on Multiple Consumer GPUs with RoundPipe

    cs.DC 2026-04 conditional novelty 8.0

    RoundPipe achieves near-zero-bubble pipeline parallelism for LLM training on consumer GPUs by dynamically dispatching computation stages round-robin, yielding 1.48-2.16x speedups and enabling 235B model fine-tuning on...

  6. When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models

    cs.LG 2026-03 conditional novelty 8.0

    Content-based routing succeeds only when models provide bidirectional context and perform pairwise comparisons, with bidirectional Mamba plus rank-1 projection reaching 99.7% precision at linear inference cost.

  7. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    cs.AI 2024-08 unverdicted novelty 8.0

    The AI Scientist framework enables LLMs to independently conduct the full scientific process from idea generation to paper writing and review, demonstrated across three ML subfields with papers costing under $15 each.

  8. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

    cs.HC 2024-05 conditional novelty 8.0

    AgentClinic is a multimodal agent benchmark demonstrating that LLM diagnostic accuracy on MedQA drops to below one-tenth in sequential clinical simulations, with Claude-3.5 leading and large tool-use differences acros...

  9. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

    cs.AI 2024-04 accept novelty 8.0

    OSWorld provides the first unified real-computer benchmark for open-ended multimodal agent tasks, exposing large performance gaps between humans and state-of-the-art LLM/VLM agents.

  10. RULER: What's the Real Context Size of Your Long-Context Language Models?

    cs.CL 2024-04 accept novelty 8.0

    RULER shows most long-context LMs drop sharply in performance on complex tasks as length and difficulty increase, with only half maintaining results at 32K tokens.

  11. GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0

    GeMoE adaptively sets the number of experts per token via gating entropy, retaining 99.5% of static-routing performance while raising average sparsity by 36.5%.

  12. CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

    cs.DC 2026-06 unverdicted novelty 7.0

    CrossPool separates weights and KV-cache into distinct GPU pools plus a planner, virtualizer, and layer-wise scheduler to cut P99 time-between-tokens by up to 10.4x versus prior kvcached multi-LLM systems.

  13. Toward Calibrated Mixture-of-Experts Under Distribution Shift

    cs.AI 2026-06 unverdicted novelty 7.0

    Expert calibration suffices for MoE calibration under distribution shifts in hard-routed models but not soft-routed ones; adversarial reweighting improves the accuracy-calibration tradeoff across models and shifts.

  14. Comparing Linear Probes with Mahalanobis Cosine Similarity

    cs.LG 2026-06 unverdicted novelty 7.0

    For balanced Gaussian class projections, OOD AUROC is a linear function of MCS to the reference probe because both are sigmoid-shaped functions of the probe SNR on test data.

  15. Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks

    cs.CV 2026-06 unverdicted novelty 7.0

    The paper constructs the VIBE benchmark and evaluates six visual in-context learning models on 14 datasets, 12 tasks, and 106 combinations under a unified one-shot protocol, revealing limitations and failure modes.

  16. DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination

    cs.LG 2026-06 unverdicted novelty 7.0

    DICE formalizes multi-agent LLM coordination as discounted incomplete-information Markov games and introduces Heterogeneous Quantal Response Equilibrium (HQRE) to achieve unique stable equilibria with bounded regret, ...

  17. MOSAIC: A Workload-Driven Simulation and Design-Space Exploration Framework for Heterogeneous NPUs

    cs.AR 2026-06 unverdicted novelty 7.0

    MOSAIC is a simulation and DSE framework for heterogeneous NPUs that finds designs achieving 46.91% mean iso-area energy savings over homogeneous baselines on 20 workloads.

  18. Knowledge Index of Noah's Ark

    cs.AI 2026-06 unverdicted novelty 7.0

    Introduces KINA benchmark with 899 items over 261 disciplines, formal (1-1/e) coverage guarantee and bonus-on-bar tournament theorem, plus evaluations of 42 models with top score 53.17%.

  19. LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling

    cs.LG 2026-06 unverdicted novelty 7.0

    LoopMoE is a looped MoE language model that outperforms matched vanilla MoE on 8 of 9 downstream benchmarks at 3B scale and continues to outperform at 9B scale under strictly controlled budgets.

  20. ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

    cs.AI 2026-06 unverdicted novelty 7.0

    ClinicalMC is a benchmark of 1,275 Chinese and 5,804 English multi-course clinical samples across four stages, evaluated via a multi-agent framework on closed-source, open-source, and medical LLMs in static and dynami...

  21. ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving

    cs.DC 2026-05 unverdicted novelty 7.0

    ViBE co-optimizes expert placement with measured GPU performance variability in MoE inference to cut execution-time imbalance, delivering 14% better SLO attainment and up to 45% lower P90 TTFT.

  22. EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

    cs.LG 2026-05 unverdicted novelty 7.0

    EST-PRM stress-tests five PRM models on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench using three label-preserving transformations and reports model-specific vulnerability patterns.

  23. Next-Billion AI Index: The compass for AI utility and adoption in the global majority

    cs.CY 2026-05 unverdicted novelty 7.0

    Introduces nexbax, a diagnostic framework with three themes and 10 dimensions for evaluating AI economic viability, operational practicality, and societal integrity in next-billion-user contexts.

  24. Model-Native Computing Architecture: Envisioning Future System Architecture Through the Lens of Computer Architecture

    cs.AI 2026-05 unverdicted novelty 7.0

    Proposes the Intelligent Computing Architecture (ICA) as a six-layer framework with dual probabilistic-deterministic planes and three Amdahl-style heuristics to unify design of LLM-based systems.

  25. Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs

    cs.CL 2026-05 unverdicted novelty 7.0

    Moral Trolley Arena shows frontier LLMs produce composite moral preferences that are compressed rather than additive functions of calibrated component act strengths across Moral Foundations Theory.

  26. Latent Performance Profiling of Large Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Introduces Latent Performance Profiling (LPP) as a task-agnostic framework deriving scalar metrics from LLM latent representations and dynamics to complement benchmark evaluations.

  27. Large Language Model Selection with Limited Annotations

    cs.CL 2026-05 unverdicted novelty 7.0

    SELECT-LLM is the first active model selection framework for LLMs that uses expected information gain from pairwise output similarities to minimize required annotations, reporting up to 84.78% cost reduction across 23...

  28. HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs

    cs.DC 2026-05 unverdicted novelty 7.0

    HyperParallel-MoE achieves up to 1.58x lower Dispatch-to-Combine MoE-FFN latency on Ascend A3 clusters via tile-level heterogeneous scheduling of AIC and AIV resources.

  29. Fine-grained Claim-level RAG Benchmark for Law

    cs.CL 2026-05 unverdicted novelty 7.0

    ClaimRAG-LAW provides a claim-level bilingual benchmark and evaluation framework for legal RAG systems that reveals limitations in current retrieval and generation performance.

  30. Fine-grained Claim-level RAG Benchmark for Law

    cs.CL 2026-05 unverdicted novelty 7.0

    ClaimRAG-LAW is a French-English legal RAG benchmark with claim-level granularity for experts and non-experts that reveals limitations in current retrieval and generation performance.

  31. Fine-grained Claim-level RAG Benchmark for Law

    cs.CL 2026-05 unverdicted novelty 7.0

    ClaimRAG-LAW is a new English-French dataset with diverse question types and a claim-level evaluation framework that reveals limitations in current legal RAG systems for retrieval, generation, and analysis.

  32. CaMo: Camera Motion Grounded Evaluation and Training for Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Proposes Spatial Narrative Score (SNS) evaluation for VLMs' camera motion understanding and introduces CaMo model achieving consistent performance on SNS and direct QA.

  33. Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

    math.OC 2026-05 conditional novelty 7.0

    Proposes equivariant optimizers matched to the symmetry groups of embeddings, SwiGLU projections and MoE routers, with experiments showing consistent gains over AdamW on language model pre-training.

  34. From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

    cs.CL 2026-05 unverdicted novelty 7.0

    A dataset-agnostic framework converts text tool-calling benchmarks to paired audio versions via TTS and noise, showing model-dependent performance with small text-to-voice gaps of 1.8-4.8 points on Confetti and When2Call.

  35. Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts

    cs.LG 2026-05 unverdicted novelty 7.0

    Routers in SMoE models form geometric alignments with their experts through shared gradient directions, enabling effective specialization that auxiliary load-balancing losses tend to disrupt.

  36. HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

    cs.CL 2026-05 unverdicted novelty 7.0

    Hebatron is the first open-weight Hebrew MoE LLM adapted from Nemotron-3, reaching 73.8% on Hebrew reasoning benchmarks while activating only 3B parameters per pass and supporting 65k-token context.

  37. Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

    cs.DC 2026-05 unverdicted novelty 7.0

    Dooly reduces LLM inference profiling costs by 56.4% via configuration-agnostic taint-based labeling and selective database reuse, delivering simulation accuracy within 5% MAPE for TTFT and 8% for TPOT across 12 models.

  38. Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts

    cs.LG 2026-05 unverdicted novelty 7.0

    Evaluation artifacts substantially inflate the measured unsolvability ceiling in multi-LLM routing, leading to distorted router training and overstated headroom.

  39. MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference

    cs.LG 2026-05 conditional novelty 7.0

    MISA routes to a small subset of indexer heads via block statistics, matching full DSA performance on LongBench with 4-8x fewer heads and 3.82x speedup while recovering over 92% of selected tokens.

  40. When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Standard top-k routers in MoE language models often select suboptimal routes for difficult tokens, and updating only the final router layer raises pass@K on AIME and HMMT benchmarks across multiple models.

  41. Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity

    stat.ML 2026-05 unverdicted novelty 7.0

    Every fixed finite feedforward neural network definable in an o-minimal structure has finite sample complexity in the agnostic PAC setting.

  42. Geometry-Aware State Space Model: A New Paradigm for Whole-Slide Image Representation

    cs.CV 2026-05 unverdicted novelty 7.0

    BatMIL uses hybrid hyperbolic-Euclidean geometry, an S4 state-space backbone, and chunk-level mixture-of-experts to outperform prior multiple-instance learning methods on seven whole-slide image datasets across six cancers.

  43. Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

    cs.DC 2026-05 unverdicted novelty 7.0

    Coral cuts multi-LLM serving costs by up to 2.79x and raises goodput by up to 2.39x on heterogeneous GPUs through adaptive joint optimization and a lossless two-stage decomposition that solves quickly.

  44. MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

    cs.LG 2026-05 unverdicted novelty 7.0

    MoE-Prefill achieves 1.35-1.59x higher throughput for prefill-only MoE serving by using asynchronous expert parallelism to overlap weight AllGather with computation and prefix-aware routing with true-FLOPs tracking.

  45. DARE: Diffusion Language Model Activation Reuse for Efficient Inference

    cs.LG 2026-05 unverdicted novelty 7.0

    DARE reuses up to 87% of attention activations in diffusion LLMs through KV caching and output reuse, delivering 1.2x per-layer latency gains with average performance drops of 1.2-2.0%.

  46. Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts

    cs.LG 2026-05 conditional novelty 7.0

    Adding temporal memory via LIF, precision-weighted gating, and anticipatory prediction to MoE routers recovers effective expert selection at distribution transitions, with ablation confirming a super-additive beta-ant...

  47. RouteProfile: Graph-Based Profiling for Cold-Start LLM Routing

    cs.NI 2026-04 unverdicted novelty 7.0

    RouteProfile organizes LLM profile design into organizational form, representation type, aggregation depth, and learning configuration, with evaluations showing structured profiles outperform flat ones and aid general...

  48. MASCing: Configurable Mixture-of-Experts Behavior via Activation Steering Masks

    cs.CR 2026-04 unverdicted novelty 7.0

    MASCing uses an LSTM surrogate and optimized steering masks to enable flexible, inference-time control over MoE expert routing for safety objectives, improving jailbreak defense and content generation success rates su...

  49. Machine Collective Intelligence for Explainable Scientific Discovery

    cs.AI 2026-04 unverdicted novelty 7.0

    Machine collective intelligence uses coordinated AI agents to evolve symbolic hypotheses and recover governing equations from observations in deterministic, stochastic, and uncharacterized systems, achieving up to six...

  50. Authority Inversion in LLM-Mediated Ubiquitous Systems: When Models Trust Users Over Sensors

    cs.AI 2026-04 unverdicted novelty 7.0

    LLMs exhibit authority inversion by prioritizing natural-language user claims over numerical sensor data in conflicts, diagnosed with new geometric metrics and mitigated via layer-level calibration.

  51. Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations

    cs.RO 2026-04 unverdicted novelty 7.0

    ACO-MoE employs agent-centric mixture-of-experts to decouple task-relevant features from dynamic visual perturbations in RL, recovering 95.3% of clean performance on the new VDCS benchmark.

  52. Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations

    cs.RO 2026-04 unverdicted novelty 7.0

    ACO-MoE recovers 95.3% of clean-input performance in visual control tasks under Markov-switching corruptions by routing restoration experts and anchoring representations to clean foreground masks.

  53. Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

    cs.LG 2026-04 unverdicted novelty 7.0

    Incompressible Knowledge Probes enable log-linear estimation of LLM parameter counts from factual accuracy on obscure questions, showing continued scaling of knowledge capacity across open and closed models.

  54. DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models

    cs.LG 2026-04 unverdicted novelty 7.0

    DPRM introduces a Doob h-transform process reward module as a plug-in for token ordering in diffusion language models, with convergence proofs and empirical gains over confidence baselines especially on hard reasoning...

  55. On Bayesian Softmax-Gated Mixture-of-Experts Models

    stat.ML 2026-04 unverdicted novelty 7.0

    Bayesian softmax-gated mixture-of-experts models achieve posterior contraction for density estimation and parameter recovery using Voronoi losses, plus two strategies for choosing the number of experts.

  56. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 7.0

    Expert upcycling duplicates experts in an existing MoE checkpoint and continues pre-training to match fixed-size baseline performance with 32% less compute.

  57. Multi-Domain Learning with Global Expert Mapping

    cs.CV 2026-04 unverdicted novelty 7.0

    GEM replaces learned routers in MoE models with a global scheduler based on linear programming relaxation and hierarchical rounding, achieving SOTA on the UODB multi-domain benchmark with gains on rare domains.

  58. GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

    cs.CL 2026-04 conditional novelty 7.0

    GTA-2 benchmark shows frontier models achieve below 50% on atomic tool tasks and only 14.39% success on realistic long-horizon workflows, with execution harnesses like Manus providing substantial gains.

  59. SecureRouter: Encrypted Routing for Efficient Secure Inference

    cs.CR 2026-04 unverdicted novelty 7.0

    SecureRouter accelerates secure transformer inference by 1.95x via an encrypted router that selects input-adaptive models from an MPC-optimized pool with negligible accuracy loss.

  60. Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap

    cs.SE 2026-04 unverdicted novelty 7.0

    Atropos uses GCN on inference graphs for early failure prediction and hotswaps to larger LLMs, achieving 74% of large-model performance at 24% cost.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages · cited by 335 Pith papers · 23 internal anchors

  1. [1]

    Program Synthesis with Large Language Models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  2. [2]

    Llemma: An Open Language Model For Mathematics

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432–7439, 2020

    work page 2020

  4. [4]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [5]

    QuAC : Question Answering in Context

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. arXiv preprint arXiv:1808.07036, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  6. [6]

    Unified scaling laws for routed language models

    Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pages 4057–4086. PMLR, 2022

    work page 2022

  7. [7]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  8. [8]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  9. [9]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [10]

    Bold: Dataset and metrics for measuring biases in open-ended language generation

    Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862–872, 2021

    work page 2021

  11. [11]

    arXiv preprint arXiv:2312.17238 (2023)

    Artyom Eliseev and Denis Mazur. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238, 2023

    work page arXiv 2023

  12. [12]

    A review of sparse expert mod- els in deep learning,

    William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667, 2022

    work page arXiv 2022

  13. [13]

    Megablocks: Efficient sparse training with mixture-of-experts,

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. arXiv preprint arXiv:2211.15841, 2022

    work page arXiv 2022

  14. [14]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  15. [15]

    Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning

    Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335–29347, 2021. 9

    work page 2021

  16. [16]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  17. [17]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  18. [18]

    Mistral 7B

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  20. [20]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, pages 453–466, 2019

    work page 2019

  21. [21]

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  22. [22]

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  23. [23]

    arXiv preprint arXiv:2305.16300 , year=

    Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023

    work page arXiv 2023

  24. [24]

    BBQ: A Hand-Built Bias Benchmark for Question Answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  25. [25]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, pages 99–106, 2021

    work page 2021

  27. [27]

    SocialIQA: Commonsense Reasoning about Social Interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  28. [28]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  29. [29]

    Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  30. [30]

    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  31. [31]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    work page 2017

  32. [32]

    HellaSwag: Can a Machine Really Finish Your Sentence?

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  33. [33]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. 10

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  35. [35]

    Either choice

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022. 11 0 0.1 0.2 0.3 Layer 0 -- Either choice 0 0.1 0.2 0.3 Layer 0 -- First choice 0 0.1 0.2 0.3 Layer 0 -- Second choice 0 0...

    work page 2022