pith. sign in

arxiv: 2312.00752 · v2 · submitted 2023-12-01 · 💻 cs.LG · cs.AI

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Albert Gu , Tri Dao This is my paper

Pith reviewed 2026-05-10 11:48 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords state space modelssequence modelingselective SSMlinear-time architectureslanguage modelingTransformerslong sequences
0
0 comments X

The pith

Selective SSMs let Mamba model sequences linearly while matching larger Transformers on language tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Mamba, a sequence model built on selective state space models. Prior subquadratic models could not perform content-based reasoning because their parameters stayed fixed across the sequence. The fix is to compute the SSM parameters as functions of each input token so the model can decide what information to keep or discard. A hardware-aware parallel algorithm makes the computation efficient despite the loss of convolution structure. The resulting architecture uses no attention or MLP blocks, scales linearly with length, runs with five times the inference throughput of Transformers, and reaches strong results on language, audio, and genomics data.

Core claim

By allowing the state transition parameters of a state space model to depend on the current input, the model gains the ability to selectively propagate or forget information along the sequence. When these selective SSMs are stacked into a simplified end-to-end network without attention or MLP blocks, the architecture achieves linear scaling in sequence length, five times higher inference throughput than Transformers, and state-of-the-art performance across modalities. On language modeling a 3B-parameter Mamba model outperforms Transformers of the same size and matches Transformers twice its size in both pretraining and downstream evaluation.

What carries the argument

Selective SSMs, in which the state transition and output parameters are computed from the input at each step to enable content-dependent propagation or forgetting of information.

If this is right

  • Performance on real data improves as sequence length grows to a million tokens.
  • A 3B Mamba model outperforms same-size Transformers and matches twice-as-large Transformers on language pretraining and downstream tasks.
  • Inference throughput reaches five times that of comparable Transformers while maintaining linear scaling.
  • State-of-the-art results appear on language, audio, and genomics without attention or MLP blocks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same input-dependent selectivity pattern could be added to other linear recurrent architectures to improve their handling of long-range dependencies.
  • Hardware-aware parallel scans for selective recurrence may become a standard optimization for any model that trades attention for linear time.
  • If the pattern generalizes, smaller models built this way could replace larger attention-based models in applications that need long context windows.

Load-bearing premise

Making SSM parameters depend on the input is enough to overcome the content-based reasoning weakness of earlier subquadratic models, and the resulting selective SSMs can be trained stably at scale in a simplified architecture without attention or MLP blocks.

What would settle it

Training Mamba models on long language sequences and observing that they underperform same-size Transformers on standard benchmarks, or that wall-clock inference time grows faster than linearly with sequence length, would falsify the central claims.

read the original abstract

Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5$\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces Mamba, a simplified sequence model architecture built from selective state space models (SSMs). It identifies the lack of content-based reasoning in prior subquadratic models (linear attention, gated convolutions, standard SSMs) as their key limitation on discrete modalities like language. The core technical contribution is making the SSM parameters Δ, B, and C input-dependent, which enables selective propagation or forgetting of information along the sequence. A hardware-aware parallel scan algorithm is derived to enable efficient training despite the loss of convolution structure. The resulting Mamba block stack contains no attention or MLP layers. Empirical claims include linear scaling to million-length sequences, 5× higher inference throughput than Transformers, and state-of-the-art results across language, audio, and genomics; specifically, a 3B-parameter Mamba model outperforms same-size Transformers and matches twice-as-large Transformers on both pretraining perplexity and downstream tasks.

Significance. If the empirical results and the attribution to selectivity are reproducible, the work is significant. It supplies a concrete, scalable mechanism that converts a long-standing weakness of SSMs into a strength while preserving linear complexity and fast inference. The combination of a parameter-efficient selective recurrence with a custom parallel algorithm offers a plausible path toward replacing attention-based backbones on long-context tasks. The paper also ships the implementation details and scaling curves needed for follow-up work.

major comments (2)
  1. [§5.2] §5.2 (Language Modeling Results) and Table 2: the headline claim that Mamba-3B matches Transformers twice its size is load-bearing for the architectural conclusion. However, the manuscript provides no ablation that holds model size, training tokens, optimizer, and residual structure fixed while disabling input-dependence of Δ, B, C (i.e., reverting to a standard SSM). Without this control, it remains possible that the observed gains arise from the particular projection dimensions, the simplified block design, or hyper-parameter differences rather than selectivity itself.
  2. [§3.3] §3.3 (Hardware-Aware Algorithm) and Algorithm 1: the parallel scan is presented as numerically stable and hardware-efficient, yet the paper does not report the condition number of the discretized state transition matrix or any ablation on floating-point precision (FP16 vs. BF16) across sequence lengths up to 1M. Because selectivity makes the recurrence input-dependent, small numerical errors could accumulate differently than in time-invariant SSMs; this should be quantified to support the “stable training at scale” claim.
minor comments (2)
  1. [Figure 2] Figure 2 (scaling curves) uses log-log axes but does not label the exact sequence lengths or batch sizes used for the throughput measurements; this makes direct comparison with the Transformer baselines harder.
  2. [§3.1] Notation: the symbol Δ is overloaded between the continuous-time step size and the input-dependent discretization parameter; a brief clarification in §3.1 would avoid confusion for readers familiar with the original S4 formulation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below and have incorporated revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [§5.2] §5.2 (Language Modeling Results) and Table 2: the headline claim that Mamba-3B matches Transformers twice its size is load-bearing for the architectural conclusion. However, the manuscript provides no ablation that holds model size, training tokens, optimizer, and residual structure fixed while disabling input-dependence of Δ, B, C (i.e., reverting to a standard SSM). Without this control, it remains possible that the observed gains arise from the particular projection dimensions, the simplified block design, or hyper-parameter differences rather than selectivity itself.

    Authors: We agree that a tightly controlled ablation isolating input-dependent selectivity (while fixing model size, data, optimizer, and block structure) would provide stronger evidence for the architectural conclusion. Prior comparisons in the manuscript were to external models such as S4 rather than an internal non-selective control. We have added this ablation in the revised Section 5.2 and appendix: a non-selective Mamba-3B variant (time-invariant Δ, B, C) trained under identical conditions shows a clear performance degradation relative to the selective version, supporting that selectivity drives the gains rather than other design choices. revision: yes

  2. Referee: [§3.3] §3.3 (Hardware-Aware Algorithm) and Algorithm 1: the parallel scan is presented as numerically stable and hardware-efficient, yet the paper does not report the condition number of the discretized state transition matrix or any ablation on floating-point precision (FP16 vs. BF16) across sequence lengths up to 1M. Because selectivity makes the recurrence input-dependent, small numerical errors could accumulate differently than in time-invariant SSMs; this should be quantified to support the “stable training at scale” claim.

    Authors: We acknowledge that explicit quantification of numerical properties under input-dependent selectivity strengthens the stability claim. The parallel scan uses standard associative operations, and we observed no instability during training. In the revision we have added (i) condition-number statistics for the discretized state matrices across sequence lengths, showing they remain well-bounded, and (ii) FP16/BF16 precision ablations up to 1M tokens demonstrating equivalent convergence and no differential error accumulation. These results are reported in the updated §3.3 and appendix. revision: yes

Circularity Check

0 steps flagged

No circularity: claims rest on empirical validation of input-dependent SSMs

full rationale

The paper motivates selective SSMs by identifying a content-based reasoning weakness in prior subquadratic models, proposes making parameters (Δ, B, C) input-dependent as a direct fix, and validates the resulting Mamba architecture through large-scale training and benchmarking on language, audio, and genomics tasks. No derivation step reduces a claimed prediction to a fitted parameter by construction, invokes a self-citation as an unverified uniqueness theorem, or renames an empirical pattern as a first-principles result. The hardware-aware algorithm and end-to-end architecture are presented as engineering choices whose performance is measured externally, leaving the central claims independently falsifiable.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

The central performance claims rest on the empirical success of the selective SSM design and the assumption that input-dependent parameters suffice for content-based reasoning; no external benchmarks or formal derivations are cited in the abstract.

free parameters (1)
  • model size
    Performance is reported specifically for a 3B parameter model; the scaling behavior depends on this choice.
axioms (1)
  • domain assumption Input-dependent SSM parameters enable content-based reasoning
    Invoked to justify the selective mechanism as addressing the weakness of prior subquadratic models.
invented entities (1)
  • selective SSM no independent evidence
    purpose: To allow the model to selectively propagate or forget information along the sequence based on current token content
    New concept introduced to overcome the content-based reasoning limitation of standard SSMs.

pith-pipeline@v0.9.0 · 5553 in / 1385 out tokens · 27894 ms · 2026-05-10T11:48:06.209749+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Preisach Attention: A Hysteretic Model of Sequential Memory

    cs.LG 2026-05 unverdicted novelty 8.0

    PAL uses the classical Preisach hysteresis operator with learned thresholds and an extrema stack to model sequences, proving O(1)-depth Turing completeness via two-stack PDA simulation and incomparability with standar...

  2. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0

    WriteSAE is the first sparse autoencoder that factors decoder atoms into the native d_k x d_v cache write shape of recurrent models and supplies a closed-form per-token logit shift for atom substitution.

  3. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0

    WriteSAE introduces sparse autoencoders with rank-1 matrix atoms for recurrent state updates, allowing replacement tests that outperform deletion on 92.4% of positions and a formula predicting logit changes with R²=0.98.

  4. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0

    WriteSAE decomposes recurrent model cache writes into substitutable atoms with a closed-form logit shift, achieving high substitution success and targeted behavioral installs on models like Qwen3.5 and Mamba-2.

  5. Convergent Stochastic Training of Attention and Understanding LoRA

    cs.LG 2026-05 unverdicted novelty 8.0

    Attention and LoRA regression losses induce Poincaré inequalities under mild regularization, so SGD-mimicking SDEs converge to minimizers with no assumptions on data or model size.

  6. Learning the Signature of Memorization in Autoregressive Language Models

    cs.CL 2026-04 accept novelty 8.0

    A classifier trained only on transformer fine-tuning data detects an invariant memorization signature that transfers to Mamba, RWKV-4, and RecurrentGemma with AUCs of 0.963, 0.972, and 0.936.

  7. The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

    cs.LG 2026-04 unverdicted novelty 8.0

    Transformer weight spectra exhibit transient compression waves that propagate layer-wise, persistent non-monotonic depth gradients in power-law exponents, and Q/K-V asymmetry, with the spectral exponent alpha predicti...

  8. When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models

    cs.LG 2026-03 conditional novelty 8.0

    Content-based routing succeeds only when models provide bidirectional context and perform pairwise comparisons, with bidirectional Mamba plus rank-1 projection reaching 99.7% precision at linear inference cost.

  9. Sparse Attention as Compact Kernel Regression

    cs.LG 2026-01 unverdicted novelty 8.0

    Sparse attention arises from compact kernel regression, with Epanechnikov and similar kernels mapping to normalized ReLU, sparsemax, and alpha-entmax attention.

  10. Learning to (Learn at Test Time): RNNs with Expressive Hidden States

    cs.LG 2024-07 conditional novelty 8.0

    TTT layers treat the hidden state as a trainable model updated at test time, allowing linear-complexity sequence models to scale perplexity reduction with context length unlike Mamba.

  11. RULER: What's the Real Context Size of Your Long-Context Language Models?

    cs.CL 2024-04 accept novelty 8.0

    RULER shows most long-context LMs drop sharply in performance on complex tasks as length and difficulty increase, with only half maintaining results at 32K tokens.

  12. VMamba: Visual State Space Model

    cs.CV 2024-01 conditional novelty 8.0

    VMamba introduces a state-space vision backbone using 2D selective scanning across four routes to achieve linear complexity and strong performance on image tasks.

  13. Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding

    cs.LG 2026-07 unverdicted novelty 7.0

    Set diffusion factorizes likelihood over arbitrary token sets and uses a set-causal diffusion architecture to support KV caching and any-order decoding, yielding improved speed-quality tradeoffs versus prior diffusion LMs.

  14. Multi-Head Recurrent Memory Agents

    cs.LG 2026-07 unverdicted novelty 7.0

    The paper proposes Multi-Head Recurrent Memory (MHM) with a select-then-update strategy to improve memory retention in long-context recurrent agents.

  15. Agentic generation of verifiable rules for deterministic, self-expanding reaction classification

    cs.AI 2026-07 unverdicted novelty 7.0

    Multi-agent LLMs generate and verify 14,073 deterministic reaction rules from 665,901 patents, enabling 97.7% classification of unseen reactions with finer resolution than fixed proprietary systems.

  16. SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference

    cs.CV 2026-07 conditional novelty 7.0

    SpiralFovea is a parameter-free input-adaptive tokenization method that replaces fixed ViT grids with entropy-driven multi-scale spiral patches, delivering 1.7-2.1 pp accuracy gains and 60% fewer tokens on fine-graine...

  17. SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference

    cs.CV 2026-07 unverdicted novelty 7.0

    SpiralFovea is a parameter-free input-adaptive foveated tokeniser that uses local visual entropy to generate <=78 spiral-ring patches, replacing fixed ViT grids and delivering accuracy gains plus large reductions in t...

  18. RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception

    cs.CV 2026-06 accept novelty 7.0

    RESOLVE provides a controlled multi-resolution LiDAR and camera benchmark for evaluating 3D detection and tracking under point sparsity variations in roadside cooperative perception.

  19. Morphing into Hybrid Attention Models

    cs.CL 2026-06 unverdicted novelty 7.0

    FlashMorph formulates hybrid layer selection as budget-constrained optimization, trains per-layer gates on synthetic retrieval data with linearization regularization, then discretizes and distills to produce efficient...

  20. Interventional Flow Matching: Prospective Dose-Response Forecasting with Velocity-Field Jacobian Regularization

    cs.LG 2026-06 unverdicted novelty 7.0

    IFM conditions flow-matching velocity fields on patient history and planned treatments, using velocity-field Jacobian regularization to enforce signed, dose-bounded insulin-lowering and carbohydrate-raising effects on...

  21. CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

    cs.CL 2026-06 conditional novelty 7.0

    CARVE introduces key-axis content-aware gating and value-efficient scalar writes in recurrent linear attention, outperforming GDN-2 on perplexity and retrieval tasks while cutting parameters and memory.

  22. CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

    cs.CL 2026-06 unverdicted novelty 7.0

    CARVE fixes memory-blind gating in recurrent models by erasing only on the key axis using the recurrent output as content signal and a scalar per head, yielding better perplexity and benchmarks than GDN-2 at 1.3B scal...

  23. Structure Before Collapse: Transient semantic geometry in next-token prediction

    cs.LG 2026-06 unverdicted novelty 7.0

    Semantic geometry emerges transiently early in next-token prediction training before collapsing to Neural Collapse symmetry in synthetic settings with latent semantic factors.

  24. SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

    cs.LG 2026-06 unverdicted novelty 7.0

    HRM adapters via Hankel reduced-order modeling outperform LoRA on long-context tasks in Mistral-7B when used as SSM residual modules with FFT-based parallel scan.

  25. FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation

    cs.CV 2026-06 unverdicted novelty 7.0

    FLAT maps compressed video diffusion latents to explicit triangle splats via ray-centered rotation parameterization and a product window function, reporting better geometric accuracy than 3D Gaussian baselines under i...

  26. Frequency Domain Reservoir Computing

    cs.LG 2026-06 unverdicted novelty 7.0

    FRESCO is a frequency-domain Echo State Network using zero-padding embeddings, packed readout, and native frequency non-linearity to achieve O(N) complexity while matching SOTA on memory and forecasting benchmarks.

  27. MeshFlow: Mesh Generation with Equivariant Flow Matching

    cs.GR 2026-06 unverdicted novelty 7.0

    MeshFlow applies equivariant optimal-transport flow matching to generate triangle meshes as soups, matching autoregressive quality with an 18x inference speedup.

  28. Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime

    cs.LG 2026-06 unverdicted novelty 7.0

    Proves GD convergence to stationary point neighborhoods for general NN architectures beyond NTK via block-level analysis, analyticity, and local smoothness conditions.

  29. Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

    cs.LG 2026-06 unverdicted novelty 7.0

    Execution-state capsules enable graph-bound full-state checkpointing and sub-millisecond restore for LLMs including KV and recurrent states, yielding 3.9x-27x TTFT speedups in on-device physical-AI serving.

  30. Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision

    cs.CV 2026-06 unverdicted novelty 7.0

    Neural events compress event camera streams into fewer informative tokens via discrete asynchronous autoencoders, achieving on-par or better performance on detection and classification with 2x lower event rate.

  31. ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

    cs.AI 2026-06 unverdicted novelty 7.0

    ITNet unifies convolution, self-attention, and autoregressive recurrence as special cases of a single learnable integral transform implemented via an MLP kernel and matches specialized models on ImageNet, GLUE, ModelN...

  32. MIDS: Detecting Stealthy Masquerade and Tampering Attacks on CAN Bus via Bidirectional Mamba

    cs.CR 2026-06 unverdicted novelty 7.0

    MIDS uses bidirectional Mamba for CAN bus masquerade and tampering detection, achieving F1 of 96.94% on a new Tesla dataset and 93.70-99.61% on public benchmarks while outperforming baselines.

  33. Topical Phase Transitions in Artificial Intelligence Research: Large-Scale Evidence and an Early-Warning Signature for Emerging Topics

    cs.AI 2026-06 unverdicted novelty 7.0

    AI research topics exhibit abrupt phase transitions detectable via an early-warning signature with 27% precision and 63% recall on held-out data.

  34. Between Amnesia and Chaos: A Memory Stability Expressivity Trilemma for Trainable Dissipative Oscillator Networks

    cs.LG 2026-06 unverdicted novelty 7.0

    Trainable dissipative oscillator networks exhibit a trilemma in which damping governs memory horizon, gradient stability, and Lyapunov exponent, with learned substrates outperforming frozen ones only at short horizons...

  35. MOSAIC: A Workload-Driven Simulation and Design-Space Exploration Framework for Heterogeneous NPUs

    cs.AR 2026-06 unverdicted novelty 7.0

    MOSAIC is a simulation and DSE framework for heterogeneous NPUs that finds designs achieving 46.91% mean iso-area energy savings over homogeneous baselines on 20 workloads.

  36. Learning Long Range Spatio-Temporal Representations over Continuous Time Dynamic Graphs with State Space Models

    cs.LG 2026-06 unverdicted novelty 7.0

    CTDG-SSM introduces CTT-HiPPO, a Laplacian-polynomial projection of HiPPO, to create a parameter-efficient state-space formulation for continuous-time dynamic graphs that captures long-range spatio-temporal patterns.

  37. MARS: Multi-rate Aggregation of Recency Signals for Sequential Recommendation across Sparse and Dense Regimes

    cs.IR 2026-06 conditional novelty 7.0

    MARS is a new encoder-agnostic operator that produces multiple recency-scale summaries from timestamps and fuses them adaptively, achieving top HR@10 on five benchmarks with gains up to 36% on sparse data and efficien...

  38. AURA: Action-Gated Memory for Robot Policies at Constant VRAM

    cs.AI 2026-06 unverdicted novelty 7.0

    AURA-Mem uses an action-gated recurrent memory trained on closed-loop action error to deliver constant 4,224-byte state and 5-9x fewer writes than baselines while matching base policy success on LIBERO-Long.

  39. Forget Attention: Importance-Aware Attention Is All You Need

    cs.AI 2026-06 unverdicted novelty 7.0

    SISA adds an SSM importance term inside the attention score and runs the full operation as one SDPA call on augmented Q/K vectors, reporting better LAMBADA and perfect NIAH at small scale.

  40. ScanWeaver: Compiler-Driven Parallelization of Affine Recurrences via Associative Scan Lowering

    cs.PL 2026-05 unverdicted novelty 7.0

    ScanWeaver lowers affine recurrences to compiler-generated Blelloch scans in MLIR, producing executable GPU code validated on selective-scan workloads against PyTorch, CUDA, and Mamba baselines.

  41. Model-Native Computing Architecture: Envisioning Future System Architecture Through the Lens of Computer Architecture

    cs.AI 2026-05 unverdicted novelty 7.0

    Proposes the Intelligent Computing Architecture (ICA) as a six-layer framework with dual probabilistic-deterministic planes and three Amdahl-style heuristics to unify design of LLM-based systems.

  42. Ego-METAS: Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark

    cs.CV 2026-05 unverdicted novelty 7.0

    Ego-METAS is a new benchmark providing unified egocentric video data, splits, features and baselines for online multimodal temporal action segmentation under hardware-representative energy constraints.

  43. AdaState: Self-Evolving Anchors for Streaming Video Generation

    cs.CV 2026-05 unverdicted novelty 7.0

    AdaState replaces the static first-frame KV anchor with an evolving hidden latent that the model denoises alongside content, treating time as relative to enable recurrence and richer dynamics in streaming video generation.

  44. From XXLTraffic to EvoXXLTraffic: Scaling Traffic Forecasting to Sensor-Evolving Networks

    cs.AI 2026-05 unverdicted novelty 7.0

    Introduces evolutionary traffic datasets and a yearly streaming protocol, finding that many SOTA methods fail when sensor networks grow over decades.

  45. CaMBRAIN: Real-time, Continuous EEG Inference with Causal State Space Models

    cs.AI 2026-05 unverdicted novelty 7.0

    CaMBRAIN introduces a causal Mamba-based SSM with a multi-stage self-supervised training pipeline that achieves SOTA results on three EEG datasets while enabling linear-time long-range inference.

  46. Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference

    cs.LG 2026-05 unverdicted novelty 7.0

    Meta-Attention introduces per-token Bayesian routing among attention mechanisms via amortised variational inference with a Dirichlet prior, yielding lower projected FLOP cost than prior-free routing on a Tiny LM benchmark.

  47. Prompting Is All You Need: Multi-view Prompting Large Language Models for Aspect-Based Sentiment Analysis

    cs.CL 2026-05 unverdicted novelty 7.0

    LLM-MvP adapts multi-view prompting to LLMs via constrained decoding to close the gap between few-shot prompting and fine-tuned models on five ABSA benchmarks while cutting inference overhead.

  48. From Item-Only to Query-Item: Query-Conditioned Generative Search with QGS in Quark

    cs.IR 2026-05 unverdicted novelty 7.0

    QGS introduces query-item pair encoding and query-conditioned prediction with a linear HSTU encoder and HFG-Attention to reduce noise from query switches in generative search ranking, reporting online gains in a comme...

  49. UWM-JEPA: Predictive World Models That Imagine in Belief Space

    cs.LG 2026-05 unverdicted novelty 7.0

    UWM-JEPA uses a density-matrix latent and unitary predictor in JEPA to preserve joint-state spectrum during blind rollouts, achieving 0.77 accuracy on a five-step hidden-velocity task versus 0.53 for an LSTM baseline.

  50. SliceWorld: A Predictive and Controllable World-State Model for CT Report Generation

    cs.CV 2026-05 unverdicted novelty 7.0

    SliceWorld introduces a world-state model for CT report generation that uses predictive and factor-aware objectives on axial slice sequences.

  51. Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference

    cs.LG 2026-05 unverdicted novelty 7.0

    AVMP separates KV and SSM cache pools behind unified virtual addressing with failure-triggered migration, cutting OOM events 7.6% and raising throughput 1.83-13.3x on synthetic loads and 2.36x on ShareGPT traces.

  52. Tensor Cache: Eviction-conditioned Associative Memory for Transformers

    cs.LG 2026-05 unverdicted novelty 7.0

    Tensor Cache augments sliding-window attention with an eviction-fed outer-product associative memory and a training correction to improve long-context performance under bounded memory.

  53. Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU

    cs.DC 2026-05 conditional novelty 7.0

    LlamaWeb is a WebGPU backend for llama.cpp that uses static memory planning, tunable kernels, and templated multi-precision support to cut memory use by 29-33% and raise decode throughput by 45-69% versus prior browse...

  54. KVBuffer: IO-aware Serving for Linear Attention

    cs.LG 2026-05 unverdicted novelty 7.0

    KVBuffer reduces linear attention decoding latency by up to 45% and increases speculative decoding throughput 5x by buffering keys/values for flexible chunked and parallel computation.

  55. Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation

    cs.CV 2026-05 unverdicted novelty 7.0

    Patch-MoE Mamba introduces patch-ordered hierarchical scanning and an MoE-based directional fusion module to improve Mamba segmentation models on polyp and skin lesion datasets.

  56. Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory

    cs.CV 2026-05 unverdicted novelty 7.0

    Mamba-VGGT introduces a Sliding Window Mamba memory module and Zero-Init Spatial Memory Injector to enable persistent long-range geometric reasoning in VGGT for extended video sequences.

  57. Identify Then Project: Contrastive Learning of Latent Dynamics from Partial Observations with Port-Hamiltonian Structure

    cs.LG 2026-05 unverdicted novelty 7.0

    A two-stage contrastive teacher-student framework learns and then projects latent dynamics onto port-Hamiltonian submanifolds from partial observations.

  58. Public-Decay Homomorphic State Space Models for Private Sequence Inference

    cs.CR 2026-05 unverdicted novelty 7.0

    Public-decay HSSMs achieve exact plaintext-matching accuracy (0.7505/0.7420) on Rotten Tomatoes and SST-2 while running ~5x faster than polynomial attention under FHE constraints.

  59. Transformer-like Inference from Optimal Control

    cs.LG 2026-05 unverdicted novelty 7.0

    Derives transformer-like dual-filter inference layers from first-principles optimal control on nonlinear discrete and linear Gaussian sequence models.

  60. Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Social-Mamba introduces a Cycle Mamba block and social triplet factorization to achieve state-of-the-art trajectory forecasting accuracy with linear-time social interaction modeling on five benchmarks.

Reference graph

Works this paper leans on

122 extracted references · 122 canonical work pages · cited by 503 Pith papers · 13 internal anchors

  1. [1]

    Unitary Evolution Recurrent Neural Networks

    Martin Arjovsky, Amar Shah, and Yoshua Bengio. “Unitary Evolution Recurrent Neural Networks”. In:The Interna- tional Conference on Machine Learning (ICML) . 2016, pp. 1120–1128. 17

    work page 2016

  2. [2]

    Effective Gene Expression Prediction from Sequence by Integrating Long-range Interactions

    Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. “Effective Gene Expression Prediction from Sequence by Integrating Long-range Interactions”. In: Nature Methods 18.10 (2021), pp. 1196–1203

    work page 2021

  3. [3]

    Using Fast Weights to Attend to the Recent Past

    Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. “Using Fast Weights to Attend to the Recent Past”. In: Advances in Neural Information Processing Systems (NeurIPS) 29 (2016)

    work page 2016

  4. [4]

    Layer Normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. “Layer Normalization”. In:arXiv preprint arXiv:1607.06450 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  5. [5]

    Neural Machine Translation by Jointly Learning to Align and Translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural Machine Translation by Jointly Learning to Align and Translate”. In: The International Conference on Learning Representations (ICLR) . 2015

    work page 2015

  6. [6]

    Strongly-typed Recurrent Neural Networks

    David Balduzzi and Muhammad Ghifary. “Strongly-typed Recurrent Neural Networks”. In:International Conference on Machine Learning. PMLR. 2016, pp. 1292–1300

    work page 2016

  7. [7]

    Pythia: A Suite for Analyzing Large Language Models across Training and Scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. “Pythia: A Suite for Analyzing Large Language Models across Training and Scaling”. In: The International Conference on Machine Learning (ICML) . PMLR. 2023, pp. 2397–2430

    work page 2023

  8. [8]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. “PIQA: Reasoning about Physical Commonsense in Natural Language”. In: Proceedings of the AAAI conference on Artificial Intelligence . Vol. 34. 2020

    work page 2020

  9. [9]

    GPT-NeoX-20B: An Open-Source Autoregressive Language Model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. “Gpt-NeoX-20B: An Open-source Autoregressive Language Model”. In:arXiv preprint arXiv:2204.06745 (2022)

    work page Pith review arXiv 2022

  10. [10]

    Prefix Sums and Their Applications

    Guy E Blelloch. “Prefix Sums and Their Applications”. In: (1990)

    work page 1990

  11. [11]

    Quasi-Recurrent Neural Networks

    James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. “Quasi-recurrent Neural Networks”. In: arXiv preprint arXiv:1611.01576 (2016)

    work page Pith review arXiv 2016

  12. [12]

    Language Models are Few-shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. “Language Models are Few-shot Learners”. In: Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), pp. 1877–1901

    work page 2020

  13. [13]

    arXiv preprint arXiv:2304.11062 , year=

    Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. “Scaling Transformer to 1M tokens and Beyond with RMT”. In: arXiv preprint arXiv:2304.11062 (2023)

    work page arXiv 2023

  14. [14]

    Generating Long Sequences with Sparse Transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. “Generating Long Sequences with Sparse Transformers”. In: arXiv preprint arXiv:1904.10509 (2019)

    work page internal anchor Pith review arXiv 1904

  15. [15]

    Rethinking Attention with Performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. “Rethinking Attention with Performers”. In: The International Conference on Learning Representations (ICLR) . 2021

    work page 2021

  16. [16]

    PaLM: Scaling Language Modeling with Pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. “PaLM: Scaling Language Modeling with Pathways”. In: Journal of Machine Learning Research 24.240 (2023), pp. 1–113. url: http://jmlr.org/papers/v24/22- 1144.html

    work page 2023

  17. [17]

    Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling”. In: arXiv preprint arXiv:1412.3555 (2014)

    work page internal anchor Pith review arXiv 2014

  18. [18]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge”. In:arXiv preprint arXiv:1803.05457 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  19. [19]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    Tri Dao. “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning”. In:The International Conference on Learning Representations (ICLR) . 2024

    work page 2024

  20. [20]

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”. In:Advances in Neural Information Processing Systems (NeurIPS) . 2022

    work page 2022

  21. [21]

    Hungry Hungry Hippos: Towards Language Modeling with State Space Models

    Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. “Hungry Hungry Hippos: Towards Language Modeling with State Space Models”. In:The International Conference on Learning Representations (ICLR). 2023

    work page 2023

  22. [22]

    Language Modeling with Gated Convolutional Networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. “Language Modeling with Gated Convolutional Networks”. In: The International Conference on Machine Learning (ICML) . PMLR. 2017, pp. 933–941

    work page 2017

  23. [23]

    SampleRNN

    DeepSound. SampleRNN. https://github.com/deepsound-project/samplernn-pytorch. 2017

    work page 2017

  24. [24]

    arXiv preprint arXiv:2307.02486 , year =

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. “LongNet: Scaling Transformers to 1,000,000,000 Tokens”. In:arXiv preprint arXiv:2307.02486 (2023). 18

    work page arXiv 2023

  25. [25]

    Adversarial Audio Synthesis

    Chris Donahue, Julian McAuley, and Miller Puckette. “Adversarial Audio Synthesis”. In:The International Conference on Learning Representations (ICLR) . 2019

    work page 2019

  26. [26]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”. In: The International Conference on Learning Representations (ICLR) . 2020

    work page 2020

  27. [27]

    A Mathematical Framework for Transformer Circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. “...

    work page 2021

  28. [28]

    Block-State Transformer

    Mahan Fathi, Jonathan Pilault, Pierre-Luc Bacon, Christopher Pal, Orhan Firat, and Ross Goroshin. “Block-State Transformer”. In: arXiv preprint arXiv:2306.09539 (2023)

    work page arXiv 2023

  29. [29]

    Multi-Head State Space Model for Speech Recognition

    Yassir Fathullah, Chunyang Wu, Yuan Shangguan, Junteng Jia, Wenhan Xiong, Jay Mahadeokar, Chunxi Liu, Yangyang Shi, Ozlem Kalinli, Mike Seltzer, and Mark J. F. Gales. “Multi-Head State Space Model for Speech Recognition”. In: Proc. INTERSPEECH 2023 . 2023, pp. 241–245. doi: 10.21437/Interspeech.2023-1036

    work page doi:10.21437/interspeech.2023-1036 2023

  30. [30]

    Dynamic Causal Modelling

    Karl J Friston, Lee Harrison, and Will Penny. “Dynamic Causal Modelling”. In:Neuroimage 19.4 (2003), pp. 1273– 1302

    work page 2003

  31. [31]

    Simple Hardware-efficient Long Convolutions for Sequence Modeling

    Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. “Simple Hardware-efficient Long Convolutions for Sequence Modeling”. In:The International Conference on Machine Learning (ICML) (2023)

    work page 2023

  32. [32]

    Approximation of Dynamical Systems by Continuous Time Recurrent Neural Networks

    Ken-ichi Funahashi and Yuichi Nakamura. “Approximation of Dynamical Systems by Continuous Time Recurrent Neural Networks”. In: Neural Networks 6.6 (1993), pp. 801–806

    work page 1993

  33. [33]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling”. In: arXiv preprint arXiv:2101.00027 (2020)

    work page internal anchor Pith review arXiv 2020

  34. [34]

    2021 , publisher =

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A Framework for Few-shot Language Model Evaluation . Version v0.0.1. Sept. 2021. doi: 10.5281/zenodo.5371628. url: http...

    work page doi:10.5281/zenodo.5371628 2021

  35. [35]

    It’s Raw! Audio Generation with State-Space Models

    Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. “It’s Raw! Audio Generation with State-Space Models”. In: The International Conference on Machine Learning (ICML) . 2022

    work page 2022

  36. [36]

    HIPPO: Recurrent Memory with Optimal Polynomial Projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. “HIPPO: Recurrent Memory with Optimal Polynomial Projections”. In: Advances in Neural Information Processing Systems (NeurIPS) . 2020

    work page 2020

  37. [37]

    Efficiently Modeling Long Sequences with Structured State Spaces

    Albert Gu, Karan Goel, and Christopher Ré. “Efficiently Modeling Long Sequences with Structured State Spaces”. In: The International Conference on Learning Representations (ICLR) . 2022

    work page 2022

  38. [38]

    Improving the Gating Mechanism of Recurrent Neural Networks

    Albert Gu, Caglar Gulcehre, Tom Le Paine, Matt Hoffman, and Razvan Pascanu. “Improving the Gating Mechanism of Recurrent Neural Networks”. In: The International Conference on Machine Learning (ICML) . 2020

    work page 2020

  39. [39]

    On the Parameterization and Initialization of Diagonal State Space Models

    Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. “On the Parameterization and Initialization of Diagonal State Space Models”. In: Advances in Neural Information Processing Systems (NeurIPS) . 2022

    work page 2022

  40. [40]

    Combining Recurrent, Convolutional, and Continuous-time Models with the Linear State Space Layer

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. “Combining Recurrent, Convolutional, and Continuous-time Models with the Linear State Space Layer”. In:Advances in Neural Information Processing Systems (NeurIPS). 2021

    work page 2021

  41. [41]

    How to Train Your HIPPO: State Space Models with Generalized Basis Projections

    Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. “How to Train Your HIPPO: State Space Models with Generalized Basis Projections”. In: The International Conference on Learning Representations (ICLR) . 2023

    work page 2023

  42. [42]

    Diagonal State Spaces are as Effective as Structured State Spaces

    Ankit Gupta, Albert Gu, and Jonathan Berant. “Diagonal State Spaces are as Effective as Structured State Spaces”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 22982–22994

    work page 2022

  43. [43]

    arXiv preprint arXiv:2212.00768 , year=

    Ankit Gupta, Harsh Mehta, and Jonathan Berant. “Simplifying and Understanding State Space Models with Diagonal Linear RNNs”. In: arXiv preprint arXiv:2212.00768 (2022)

    work page arXiv 2022

  44. [44]

    HyperNetworks

    David Ha, Andrew Dai, and Quoc V. Le. “HyperNetworks”. In:The International Conference on Learning Representa- tions (ICLR). 2017

    work page 2017

  45. [45]

    Dream to Control: Learning Behaviors by Latent Imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. “Dream to Control: Learning Behaviors by Latent Imagination”. In: The International Conference on Learning Representations (ICLR) . 2020. 19

    work page 2020

  46. [46]

    Liquid Structural State-Space Models

    Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. “Liquid Structural State-Space Models”. In: The International Conference on Learning Representations (ICLR) . 2023

    work page 2023

  47. [47]

    Recurrent Orthogonal Networks and Long-Memory Tasks

    Mikael Henaff, Arthur Szlam, and Yann LeCun. “Recurrent Orthogonal Networks and Long-Memory Tasks”. In: The International Conference on Machine Learning (ICML) . 2016

    work page 2016

  48. [48]

    Gaussian Error Linear Units (GELUs)

    Dan Hendrycks and Kevin Gimpel. “Gaussian Error Linear Units (GELUs)”. In: arXiv preprint arXiv:1606.08415 (2016)

    work page Pith review arXiv 2016

  49. [49]

    Untersuchungen zu dynamischen neuronalen Netzen

    Sepp Hochreiter. “Untersuchungen zu dynamischen neuronalen Netzen”. In: Diploma, Technische Universität München 91.1 (1991), p. 31

    work page 1991

  50. [50]

    Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-term Dependencies

    Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-term Dependencies. 2001

    work page 2001

  51. [51]

    Long Short-Term Memory

    Sepp Hochreiter and Jürgen Schmidhuber. “Long Short-Term Memory”. In:Neural Computation 9.8 (1997), pp. 1735– 1780

    work page 1997

  52. [52]

    An Empirical Analysis of Compute- Optimal Large Language Model Training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. “An Empirical Analysis of Compute- Optimal Large Language Model Training”. In:Advances in Neural Information Processing Systems (NeurIPS) 35 (2022), pp. 30016–30030

    work page 2022

  53. [53]

    Transformer Quality in Linear Time

    Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. “Transformer Quality in Linear Time”. In:The International Conference on Machine Learning (ICML) . PMLR. 2022, pp. 9099–9117

    work page 2022

  54. [54]

    Deep Learning for Time Series Classification: A Review

    Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. “Deep Learning for Time Series Classification: A Review”. In: Data Mining and Knowledge Discovery 33.4 (2019), pp. 917– 963

    work page 2019

  55. [55]

    Data Movement is All You Need: A Case Study on Optimizing Transformers

    Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. “Data Movement is All You Need: A Case Study on Optimizing Transformers”. In: Proceedings of Machine Learning and Systems 3 (2021), pp. 711–732

    work page 2021

  56. [56]

    Gated Orthogonal Recurrent Units: On Learning to Forget

    Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. “Gated Orthogonal Recurrent Units: On Learning to Forget”. In: Neural Computation 31.4 (2019), pp. 765–783

    work page 2019

  57. [57]

    A New Approach to Linear Filtering and Prediction Problems

    Rudolph Emil Kalman. “A New Approach to Linear Filtering and Prediction Problems”. In: (1960)

    work page 1960

  58. [58]

    Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”. In:International Conference on Machine Learning . PMLR. 2020, pp. 5156–5165

    work page 2020

  59. [59]

    Linear Dynamical Systems as a Core Computational Primitive

    Shiva Kaul. “Linear Dynamical Systems as a Core Computational Primitive”. In:Advances in Neural Information Processing Systems 33 (2020), pp. 16808–16820

    work page 2020

  60. [60]

    DiffWave: A Versatile Diffusion Model for Audio Synthesis

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. “DiffWave: A Versatile Diffusion Model for Audio Synthesis”. In:International Conference on Learning Representations . 2021

    work page 2021

  61. [61]

    Time-Parameterized Convolutional Neural Networks for Irregularly Sampled Time Series

    Chrysoula Kosma, Giannis Nikolentzos, and Michalis Vazirgiannis. “Time-Parameterized Convolutional Neural Networks for Irregularly Sampled Time Series”. In: arXiv preprint arXiv:2308.03210 (2023)

    work page arXiv 2023

  62. [62]

    ImageNet Classification with Deep Convolutional Neural Networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks”. In: Advances in Neural Information Processing Systems (NeurIPS) 25 (2012)

    work page 2012

  63. [63]

    When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute

    Tao Lei. “When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute”. In:Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . 2021, pp. 7633–7648

    work page 2021

  64. [64]

    Simple Recurrent Units for Highly Parallelizable Recur- rence

    Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi. “Simple Recurrent Units for Highly Parallelizable Recurrence”. In: arXiv preprint arXiv:1709.02755 (2017)

    work page internal anchor Pith review arXiv 2017

  65. [65]

    Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

    Mario Lezcano-Casado and David Martínez-Rubio. “Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group”. In: The International Conference on Machine Learning (ICML). 2019

    work page 2019

  66. [66]

    What Makes Convolutional Models Great on Long Sequence Modeling?

    Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. “What Makes Convolutional Models Great on Long Sequence Modeling?” In: The International Conference on Learning Representations (ICLR) . 2023

    work page 2023

  67. [67]

    Time-aware Large Kernel Convolutions

    Vasileios Lioutas and Yuhong Guo. “Time-aware Large Kernel Convolutions”. In:The International Conference on Machine Learning (ICML). PMLR. 2020, pp. 6172–6183

    work page 2020

  68. [68]

    Structured State Space Models for In-Context Reinforcement Learning

    Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. “Structured State Space Models for In-Context Reinforcement Learning”. In:Advances in Neural Information Processing Systems (NeurIPS). 2023

    work page 2023

  69. [69]

    Focus Your Attention (with Adaptive IIR Filters)

    Shahar Lutati, Itamar Zimerman, and Lior Wolf. “Focus Your Attention (with Adaptive IIR Filters)”. In:arXiv preprint arXiv:2305.14952 (2023). 20

    work page arXiv 2023

  70. [70]

    Mega: Moving Average Equipped Gated Attention

    Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. “Mega: Moving Average Equipped Gated Attention”. In:The International Conference on Learning Representations (ICLR). 2023

    work page 2023

  71. [71]

    Parallelizing Linear Recurrent Neural Nets Over Sequence Length

    Eric Martin and Chris Cundy. “Parallelizing Linear Recurrent Neural Nets Over Sequence Length”. In:The Interna- tional Conference on Learning Representations (ICLR) . 2018

    work page 2018

  72. [72]

    SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

    Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. “SampleRNN: An Unconditional End-to-End Neural Audio Generation Model”. In:The International Conference on Learning Representations (ICLR) . 2017

    work page 2017

  73. [73]

    Long Range Language Modeling via Gated State Spaces

    Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. “Long Range Language Modeling via Gated State Spaces”. In: The International Conference on Learning Representations (ICLR) . 2023

    work page 2023

  74. [74]

    Efficient Orthogonal Parametrisation of Recurrent Neural Networks using Householder Reflections

    Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. “Efficient Orthogonal Parametrisation of Recurrent Neural Networks using Householder Reflections”. In:International Conference on Machine Learning . PMLR. 2017, pp. 2401–2409

    work page 2017

  75. [75]

    S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces

    Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. “S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces”. In:Advances in Neural Information Processing Systems (NeurIPS) . 2022

    work page 2022

  76. [76]

    HyenaDNA: Long-range Genomic Sequence Modeling at Single Nucleotide Resolution

    Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, et al. “HyenaDNA: Long-range Genomic Sequence Modeling at Single Nucleotide Resolution”. In: Advances in Neural Information Processing Systems (NeurIPS) . 2023

    work page 2023

  77. [77]

    In-context Learning and Induction Heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, a...

    work page 2022

  78. [78]

    WaveNet: A Generative Model for Raw Audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. “WaveNet: A Generative Model for Raw Audio”. In: arXiv preprint arXiv:1609.03499 (2016)

    work page internal anchor Pith review arXiv 2016

  79. [79]

    Resurrecting Recurrent Neural Networks for Long Sequences

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. “Resurrecting Recurrent Neural Networks for Long Sequences”. In:The International Conference on Machine Learning (ICML). 2023

    work page 2023

  80. [80]

    The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. “The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context”. In:Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics . 2016, pp. 1525–1534

    work page 2016

Showing first 80 references.