pith. sign in

arxiv: 2111.00396 · v3 · pith:WJCRQIERnew · submitted 2021-10-31 · 💻 cs.LG

Efficiently Modeling Long Sequences with Structured State Spaces

Albert Gu , Karan Goel , Christopher R\'e This is my paper

Pith reviewed 2026-05-11 10:37 UTC · model grok-4.3

classification 💻 cs.LG
keywords structured state spacesstate space modelslong-range dependenciessequence modelingS4Long Range ArenaCauchy kernel
0
0 comments X

The pith

S4 uses a low-rank correction to the state matrix A so state space models can be computed efficiently via Cauchy kernels while retaining long-range power.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks a single sequence model that works across modalities and scales to sequences of 10,000 or more steps. Prior state space models could theoretically capture arbitrary long dependencies but were too slow and memory-heavy for practical use. By conditioning the state matrix with a low-rank correction, S4 stabilizes diagonalization and reduces the core computation to a well-studied Cauchy kernel. This yields a model that matches or beats Transformers on image and language tasks, runs generation 60 times faster, and reaches state-of-the-art accuracy on every Long Range Arena task, including the previously unsolved 16k-length Path-X problem.

Core claim

Conditioning the state matrix A of the continuous-time SSM with a low-rank correction permits stable diagonalization; the resulting system can be evaluated exactly by computing a Cauchy kernel, preserving the theoretical long-range modeling capacity of the underlying SSM at far lower time and memory cost.

What carries the argument

Low-rank correction to the state matrix A, which enables stable diagonalization of the SSM and reduces its evaluation to Cauchy kernel computation.

If this is right

  • A single architecture can now address sequential CIFAR-10 at 91% accuracy without augmentation or auxiliary losses, on par with a larger 2-D ResNet.
  • S4 closes most of the gap to Transformers on image and language modeling while generating sequences 60 times faster.
  • Every Long Range Arena task, including the 16k-step Path-X problem that defeated all earlier methods, is solved at state-of-the-art accuracy with the same efficiency as competing models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the Cauchy-kernel reduction preserves the SSM's theoretical properties across domains, the same parameterization may be applied directly to other continuous-time dynamical systems that currently rely on attention or recurrence.
  • The efficiency gain could allow scaling the same model to sequences of hundreds of thousands of steps in domains such as genomics or long video without changing the core mathematics.
  • Because S4 avoids the quadratic cost of attention yet matches its long-range performance, it supplies a concrete alternative architecture whose scaling behavior can be compared directly against Transformer variants on the same long-context benchmarks.

Load-bearing premise

The low-rank correction to the state matrix A permits stable diagonalization and the resulting Cauchy kernel computation fully preserves the theoretical long-range modeling strengths of the underlying SSM without introducing approximation errors that degrade performance on real data.

What would settle it

An experiment showing that S4 either loses accuracy relative to the uncorrected SSM on a long-range task such as Path-X or requires asymptotically more than linear time for sequences of length 16k.

read the original abstract

A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly on long-range dependencies. Although conventional models including RNNs, CNNs, and Transformers have specialized variants for capturing long dependencies, they still struggle to scale to very long sequences of $10000$ or more steps. A promising recent approach proposed modeling sequences by simulating the fundamental state space model (SSM) \( x'(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t) \), and showed that for appropriate choices of the state matrix \( A \), this system could handle long-range dependencies mathematically and empirically. However, this method has prohibitive computation and memory requirements, rendering it infeasible as a general sequence modeling solution. We propose the Structured State Space sequence model (S4) based on a new parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches while preserving their theoretical strengths. Our technique involves conditioning \( A \) with a low-rank correction, allowing it to be diagonalized stably and reducing the SSM to the well-studied computation of a Cauchy kernel. S4 achieves strong empirical results across a diverse range of established benchmarks, including (i) 91\% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet, (ii) substantially closing the gap to Transformers on image and language modeling tasks, while performing generation $60\times$ faster (iii) SoTA on every task from the Long Range Arena benchmark, including solving the challenging Path-X task of length 16k that all prior work fails on, while being as efficient as all competitors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper proposes the Structured State Space (S4) sequence model, a new parameterization of the continuous-time state space model (SSM) x'(t) = A x(t) + B u(t), y(t) = C x(t) + D u(t). By applying a low-rank correction to the state matrix A, S4 enables stable diagonalization and reduces the SSM to an exact Cauchy kernel computation. This yields an efficient model that preserves the theoretical long-range dependency modeling strengths of SSMs. Empirically, S4 reports strong results including 91% accuracy on sequential CIFAR-10, closing the gap to Transformers on image/language tasks with 60x faster generation, and state-of-the-art performance on all Long Range Arena tasks including solving the 16k-length Path-X task where prior methods fail, while matching competitor efficiency.

Significance. If the results hold, this work is significant as it delivers a principled, scalable alternative to Transformers and other sequence models for long-range dependencies across modalities. It combines the mathematical advantages of SSMs with practical efficiency via the structured Cauchy kernel, addressing a key limitation of prior SSM approaches. Credit is due for consistent empirical validation on established benchmarks like LRA and CIFAR-10, and for the explicit structural parameterization that avoids hidden approximations in the kernel computation.

major comments (1)
  1. [Abstract, §3] Abstract and §3 (parameterization): the central claim that the low-rank correction to A 'permits stable diagonalization' and 'reduces the SSM to the well-studied computation of a Cauchy kernel' while exactly preserving long-range modeling power lacks an ablation isolating the correction's contribution. Without this, it is difficult to confirm that performance on Path-X (length 16k) and other LRA tasks stems from the claimed structural properties rather than hyperparameter tuning or other implementation details.
minor comments (2)
  1. [Experiments section] The manuscript would benefit from expanded details on training procedures, hyperparameter sensitivity, and full experimental setup (e.g., optimizer, learning rate schedules, and data preprocessing) to support reproducibility of the reported SoTA numbers.
  2. [§3] Notation for the low-rank correction parameters and the resulting diagonalized form could be clarified with an explicit equation showing how the correction is applied to A before diagonalization.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment and recommendation for minor revision. We address the major comment below.

read point-by-point responses

  1. Referee: [Abstract, §3] Abstract and §3 (parameterization): the central claim that the low-rank correction to A 'permits stable diagonalization' and 'reduces the SSM to the well-studied computation of a Cauchy kernel' while exactly preserving long-range modeling power lacks an ablation isolating the correction's contribution. Without this, it is difficult to confirm that performance on Path-X (length 16k) and other LRA tasks stems from the claimed structural properties rather than hyperparameter tuning or other implementation details.

    Authors: We thank the referee for this constructive comment. Section 3 derives that the low-rank correction to A is what permits stable diagonalization (avoiding the numerical instability of the HiPPO matrix) and reduces the SSM kernel computation exactly to a Cauchy matrix, thereby preserving the theoretical long-range modeling properties without approximation. While this is a mathematical property rather than an empirical one, we agree that an explicit ablation would strengthen the presentation. We will add such an ablation to the revised manuscript, comparing the full S4 model against an SSM variant without the low-rank correction on the LRA benchmark (including Path-X) to isolate its contribution. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper starts from the standard continuous-time SSM equations x'(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t) and introduces a new parameterization via low-rank correction to A. This choice is explicitly structural, enabling stable diagonalization and exact reduction to Cauchy kernel computation as a direct mathematical consequence rather than a redefinition or fit. Reported results consist of empirical performance on external benchmarks (sequential CIFAR-10, LRA tasks including Path-X of length 16k) that are not quantities predicted or fitted by construction from the model's own inputs. No load-bearing self-citations, uniqueness theorems, or ansatzes imported from prior author work are invoked to force the central claims; the efficiency and long-range modeling preservation follow from the closed-form kernel structure and are validated externally. The derivation remains self-contained against independent benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the SSM framework from prior work plus the new low-rank correction enabling efficient exact computation; no new physical entities are postulated.

free parameters (1)
  • low-rank correction parameters
    The rank and values of the correction term added to A are part of the model and learned from data.
axioms (1)
  • domain assumption For appropriate choices of A the continuous SSM can capture long-range dependencies mathematically
    Invoked in the abstract as the foundation that prior SSM work established but could not compute efficiently.

pith-pipeline@v0.9.0 · 5616 in / 1357 out tokens · 46526 ms · 2026-05-11T10:37:10.486355+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rotation Equivariant Mamba for Vision Tasks

    cs.CV 2026-03 unverdicted novelty 8.0

    EQ-VMamba adds rotation-equivariant cross-scan and group Mamba blocks to enforce end-to-end rotation equivariance, yielding better rotation robustness, competitive accuracy, and roughly 50% fewer parameters than non-e...

  2. Test-Time Training with KV Binding Is Secretly Linear Attention

    cs.LG 2026-02 conditional novelty 8.0

    Test-time training with KV binding reduces to learned linear attention.

  3. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

    cs.CL 2023-09 unverdicted novelty 8.0

    Promptbreeder evolves both task prompts and the mutation prompts that improve them using LLMs, outperforming Chain-of-Thought and Plan-and-Solve on arithmetic and commonsense reasoning benchmarks.

  4. Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding

    cs.LG 2026-07 unverdicted novelty 7.0

    Set diffusion factorizes likelihood over arbitrary token sets and uses a set-causal diffusion architecture to support KV caching and any-order decoding, yielding improved speed-quality tradeoffs versus prior diffusion LMs.

  5. MASS: Motion-Aligned Selective Scan for Refinement in Flow-Based Video Frame Interpolation

    cs.CV 2026-06 unverdicted novelty 7.0

    MASS reformulates SSM-based feature scanning in flow-based VFI to follow dynamic motion trajectories via learnable path integration and velocity-aware sampling, claiming SOTA on challenging large-displacement cases.

  6. SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

    cs.LG 2026-06 unverdicted novelty 7.0

    HRM adapters via Hankel reduced-order modeling outperform LoRA on long-context tasks in Mistral-7B when used as SSM residual modules with FFT-based parallel scan.

  7. Frequency Domain Reservoir Computing

    cs.LG 2026-06 unverdicted novelty 7.0

    FRESCO is a frequency-domain Echo State Network using zero-padding embeddings, packed readout, and native frequency non-linearity to achieve O(N) complexity while matching SOTA on memory and forecasting benchmarks.

  8. ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

    cs.AI 2026-06 unverdicted novelty 7.0

    ITNet unifies convolution, self-attention, and autoregressive recurrence as special cases of a single learnable integral transform implemented via an MLP kernel and matches specialized models on ImageNet, GLUE, ModelN...

  9. LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    LongSpike integrates fractional-order state-space modeling into spiking neural networks, enabling better long-sequence performance than prior SNNs on LRA, WikiText-103, and Speech Commands benchmarks while retaining s...

  10. Between Amnesia and Chaos: A Memory Stability Expressivity Trilemma for Trainable Dissipative Oscillator Networks

    cs.LG 2026-06 unverdicted novelty 7.0

    Trainable dissipative oscillator networks exhibit a trilemma in which damping governs memory horizon, gradient stability, and Lyapunov exponent, with learned substrates outperforming frozen ones only at short horizons...

  11. MOSAIC: A Workload-Driven Simulation and Design-Space Exploration Framework for Heterogeneous NPUs

    cs.AR 2026-06 unverdicted novelty 7.0

    MOSAIC is a simulation and DSE framework for heterogeneous NPUs that finds designs achieving 46.91% mean iso-area energy savings over homogeneous baselines on 20 workloads.

  12. Learning Long Range Spatio-Temporal Representations over Continuous Time Dynamic Graphs with State Space Models

    cs.LG 2026-06 unverdicted novelty 7.0

    CTDG-SSM introduces CTT-HiPPO, a Laplacian-polynomial projection of HiPPO, to create a parameter-efficient state-space formulation for continuous-time dynamic graphs that captures long-range spatio-temporal patterns.

  13. AURA: Action-Gated Memory for Robot Policies at Constant VRAM

    cs.AI 2026-06 unverdicted novelty 7.0

    AURA-Mem uses an action-gated recurrent memory trained on closed-loop action error to deliver constant 4,224-byte state and 5-9x fewer writes than baselines while matching base policy success on LIBERO-Long.

  14. Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing

    cs.LG 2026-05 unverdicted novelty 7.0

    Presents a structured generalized linear token mixing framework that extends recurrence equations to multiple past states, enabling new patterns with provable complexity-expressivity trade-offs for causal generation.

  15. UWM-JEPA: Predictive World Models That Imagine in Belief Space

    cs.LG 2026-05 unverdicted novelty 7.0

    UWM-JEPA uses a density-matrix latent and unitary predictor in JEPA to preserve joint-state spectrum during blind rollouts, achieving 0.77 accuracy on a five-step hidden-velocity task versus 0.53 for an LSTM baseline.

  16. Multi-view Consistent 3D Gaussian Head Avatars 'without' Multi-view Generation

    cs.CV 2026-05 unverdicted novelty 7.0

    MVCHead uses a hierarchical state space model with bi-directional scans and an SE(3) critic to enforce 3D consistency in Gaussian avatars trained only on 2D images.

  17. Exact expression for maximum Lyapunov exponent during transients in computationally powerful dynamical networks

    nlin.CD 2026-05 unverdicted novelty 7.0

    Exact analytical expression for the time-dependent maximum Lyapunov exponent during transients in a network supporting dynamics-based computation.

  18. Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Social-Mamba introduces a Cycle Mamba block and social triplet factorization to achieve state-of-the-art trajectory forecasting accuracy with linear-time social interaction modeling on five benchmarks.

  19. A Novel Schur-Decomposition-Based Weight Projection Method for Stable State-Space Neural-Network Architectures

    cs.LG 2026-05 unverdicted novelty 7.0

    A real Schur decomposition projection maps the state matrix of discrete-time state-space layers onto its nearest stable counterpart, delivering accuracy comparable to prior stable identification methods with fewer weights.

  20. QLAM: A Quantum Long-Attention Memory Approach to Long-Sequence Token Modeling

    cs.LG 2026-05 unverdicted novelty 7.0

    QLAM extends state-space models with quantum superposition in the hidden state for linear-time long-sequence modeling and reports consistent gains over RNN and transformer baselines on sequential image tasks.

  21. Parallel Scan Recurrent Neural Quantum States for Scalable Variational Monte Carlo

    cond-mat.str-el 2026-05 conditional novelty 7.0

    PSR-NQS makes recurrent neural quantum states scalable for variational Monte Carlo by using parallel scan recurrence, reaching accurate results on 52x52 two-dimensional lattices.

  22. Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

    cs.CV 2026-05 unverdicted novelty 7.0

    Radar-Modulated Selection perturbs only the step size Δ and readout C parameters inside Mamba's selective scan with radar data while keeping other components image-only, yielding state-of-the-art depth estimation on n...

  23. TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles

    cs.CV 2026-05 unverdicted novelty 7.0

    TCP-SSM conditions stable poles on visual tokens to explicitly control memory decay and oscillation in SSMs, cutting computation up to 44% while matching or exceeding accuracy on classification, segmentation, and detection.

  24. TIDES: Implicit Time-Awareness in Selective State Space Models

    cs.LG 2026-05 unverdicted novelty 7.0

    TIDES reconciles selective SSM expressivity with continuous-time physical discretization by moving input dependence onto the state matrix, enabling native irregular time series handling and achieving SOTA on UEA and P...

  25. PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

    cs.LG 2026-05 unverdicted novelty 7.0

    PairAlign learns compact audio token sequences via self-alignment of paired content views using an autoregressive decoder, achieving strong cross-view consistency and edit-distance preservation while reducing token co...

  26. PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

    cs.LG 2026-05 unverdicted novelty 7.0

    PairAlign learns compact variable-length token sequences for audio via self-alignment on paired content-preserving views, achieving 55% fewer archive tokens than VQ while preserving edit-distance retrieval at 12.71 tokens/s.

  27. Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement

    cs.CV 2026-05 unverdicted novelty 7.0

    NOVA represents world states as INR weights for decoder-free rendering, compactness, and unsupervised disentanglement of background, foreground, and motion in video world models.

  28. How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

    cs.LG 2026-05 unverdicted novelty 7.0

    In linear recurrent models, infinite-width signal propagation remains accurate only for depths t much smaller than sqrt(width n), with a critical regime at t ~ c sqrt(n) where finite-width effects emerge and dominate ...

  29. The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence

    cs.LG 2026-05 unverdicted novelty 7.0

    Predictive representation learning structurally favors encoding slower or less noisy environment modes over causal system modes, as shown by an impossibility theorem for linear-Gaussian dynamics and large-scale neural...

  30. FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    FLUID is a continuous-time transformer using Liquid Attention Networks to model attention as stable ODE solutions that interpolate between discrete SDPA and CT-RNNs, with an explicit sink gate and liquid hyper-connect...

  31. Rethink MAE with Linear Time-Invariant Dynamics

    cs.CV 2026-04 unverdicted novelty 7.0

    Token order in frozen visual representations is exploitable via SSM-based LTI probes, revealing pre-training-dependent heterogeneity that fixed pooling misses.

  32. Mamba Sequence Modeling meets Model Predictive Control

    math.OC 2026-04 unverdicted novelty 7.0

    Mamba-MPC stabilizes and tracks references on SISO and MIMO systems in simulation and hardware while outperforming LSTM-MPC with faster computation.

  33. RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

    cs.CV 2026-04 unverdicted novelty 7.0

    RSGMamba introduces a reliability-aware self-gated Mamba block for dynamic cross-modal feature selection in semantic segmentation, delivering state-of-the-art mIoU on RGB-D and RGB-T benchmarks with 48.6M parameters.

  34. Is Flow Matching Just Trajectory Replay for Sequential Data?

    stat.ML 2026-02 unverdicted novelty 7.0

    Flow matching on time series targets a closed-form nonparametric velocity field that is a similarity-weighted mixture of observed transition velocities, making neural models approximations to an ideal memory-augmented...

  35. Hidden State Poisoning Attacks against Mamba-based Language Models

    cs.CL 2026-01 unverdicted novelty 7.0

    Short input phrases can irreversibly overwrite hidden states in Mamba models, impairing information retrieval on a new benchmark while leaving pure Transformer models unaffected.

  36. Kinetic-Mamba: Mamba-Assisted Predictions of Stiff Chemical Kinetics

    cs.LG 2025-12 unverdicted novelty 7.0

    Mamba-based neural operators predict stiff chemical kinetics evolution with high fidelity from initial states on Syngas and GRI-Mech 3.0 mechanisms.

  37. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

  38. STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction

    cs.LG 2025-08 unverdicted novelty 7.0

    STM3 is a spatio-temporal mixture-of-multiscale-Mamba model that uses disentangled experts, stable routing, and causal contrastive learning to achieve state-of-the-art long-term forecasting on ten real-world benchmarks.

  39. L2RU: a Structured State Space Model with prescribed L2-bound

    eess.SY 2025-03 unverdicted novelty 7.0

    L2RU parametrizes SSMs to enforce a prescribed L2-gain bound for guaranteed input-output stability and robustness in all parameter regimes.

  40. Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Space

    cs.LG 2025-01 unverdicted novelty 7.0

    MbaGCN combines message aggregation, selective state space transitions, and node state prediction to create a more scalable deep graph convolutional network.

  41. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

    cs.LG 2024-02 unverdicted novelty 7.0

    Griffin hybrid model matches Llama-2 performance while trained on over 6 times fewer tokens and offers lower inference latency with higher throughput.

  42. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

    cs.CV 2024-01 conditional novelty 7.0

    Vim is a bidirectional Mamba vision backbone that outperforms DeiT in accuracy on standard tasks while being substantially faster and more memory-efficient for high-resolution images.

  43. Liquid Fusion of Heterogeneous Representations Towards General Salient Object Detection

    cs.CV 2026-06 unverdicted novelty 6.0

    LFNet fuses complementary representations from state space and convolutional backbones using liquid fusion and saliency-guided upsampling to reach state-of-the-art performance on five salient object detection tasks.

  44. MambaRaw: Selective State Space Modeling for Efficient 4K Raw Image Reconstruction

    cs.CV 2026-06 unverdicted novelty 6.0

    MambaRaw uses SSM-based context modeling with TileMambaBlock and EAR modules for efficient JPEG-guided 4K raw reconstruction, reporting 1.2-1.4 dB PSNR gains and 9% lower latency over baselines on Sony, Olympus, and S...

  45. Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling

    cs.LG 2026-06 unverdicted novelty 6.0

    TND is a neuron-wise dynamical system on a directed graph that achieves 17.47 consecutive catches in Pong behavior cloning, more than three times the strongest baseline.

  46. Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling

    cs.LG 2026-06 unverdicted novelty 6.0

    TND shifts to neuron-wise dynamics on explicit graph topology and achieves 17.47 mean consecutive catches in Pong behavior cloning, more than 3x the strongest baseline.

  47. Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling

    cs.LG 2026-06 unverdicted novelty 6.0

    TND models sequences via independent neuron dynamics on a directed graph and reports over three times more consecutive catches than strong baselines on a Pong behavior-cloning task.

  48. Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling

    cs.LG 2026-06 unverdicted novelty 6.0

    TND models sequences via independent neuron evolution on a directed graph and outperforms RNN, LSTM, CfC, and Transformer baselines on Pong behavior cloning with over 3x more consecutive catches.

  49. Spatial-Aware Reduction Framework: Towards Efficient and Faithful Visual State Space Models

    cs.CV 2026-06 unverdicted novelty 6.0

    STORM is a training-free spatial-aware token reduction framework that reformulates compression on spatial units to preserve grid topology and neighborhood coherence in visual state space models.

  50. SL-S4Wave: Self-Supervised Learning of Physiological Waveforms with Structured State Space Models

    cs.LG 2026-06 unverdicted novelty 6.0

    SL-S4Wave combines contrastive learning with a multiscale S4 encoder to outperform baselines in arrhythmia detection and EEG tasks while showing strong label efficiency and robustness on long sequences.

  51. ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

    cs.AI 2026-06 unverdicted novelty 6.0

    ITNet frames convolution, attention, and recurrence as special cases of one learnable integral transform with an MLP kernel and shows a single shared operator plus modality encoders matches specialized models on Image...

  52. NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama

    cs.CL 2026-06 unverdicted novelty 6.0

    NarrativeWorldBench evaluates 21 LLMs on nine narrative metrics across horizons to 200 episodes and introduces N-VSSM, a 256-dimensional variational state-space model that achieves plot-beat F1 >=0.84 with 4x lower co...

  53. Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

    cs.AI 2026-06 unverdicted novelty 6.0

    Reinforcement learning after SFT conversion narrows the performance gap between sliding-window attention and full self-attention on math reasoning benchmarks while preserving linear complexity.

  54. Free Parametrization of L_2-Bounded Structured State-Space Controllers for Nonlinear Control with Stability Guarantees

    eess.SY 2026-06 unverdicted novelty 6.0

    A new free parametrization of L2-bounded LTI systems creates L2RU SSM layers that enforce stability by design, allowing unconstrained nonlinear controller optimization with guarantees via small-gain theorem.

  55. End-to-End Context Compression at Scale

    cs.CL 2026-06 unverdicted novelty 6.0

    LCLMs are scaled 0.6B-encoder 4B-decoder compressors pre-trained on over 350B tokens that improve the Pareto frontier for general-task performance, compression speed, and peak memory in long-context language model inference.

  56. Vision-Language Guided Hyperspectral Object Tracking via Semantics Fusion and Contextual Template Updating

    cs.CV 2026-06 unverdicted novelty 6.0

    VLHTrack integrates LLM-guided band selection and Mamba-based dynamic template updating to outperform prior methods on HOT2023 and HOT2024 hyperspectral tracking benchmarks.

  57. Chiaroscuro Attention: Spending Compute in the Dark

    cs.CL 2026-06 unverdicted novelty 6.0

    CHIAR-Former routes tokens via spectral entropy to DCT mixing or attention, yielding 35-40% FLOP savings at 400M parameters with modest perplexity increase on WikiText-103.

  58. Pretraining Recurrent Networks without Recurrence

    cs.LG 2026-06 unverdicted novelty 6.0

    SMT reduces RNN training to supervised learning on memory transitions (m_t, x_{t+1}) to m_{t+1} obtained from a Transformer encoder, enabling time-parallel training with O(1) gradient paths.

  59. Mamba-Assisted Non-Markovian Closure for Reduced-Order Modeling

    cs.LG 2026-06 unverdicted novelty 6.0

    Mamba-Assisted Closure (MAC) trains a Mamba sequence model on resolved trajectories to predict non-Markovian closures and couples it with reduced-order equations, outperforming Markovian, GRU, and Wilks baselines on B...

  60. Physically-Constrained Mamba-SDE for Remaining Useful Life Prediction under Irregular Observations

    cs.AI 2026-06 unverdicted novelty 6.0

    PC-MambaSDE combines Mamba with physics-constrained SDE for RUL prediction under irregular observations, with theoretical stability guarantees and empirical outperformance on benchmarks.

Reference graph

Works this paper leans on

52 extracted references · 52 canonical work pages · cited by 156 Pith papers · 5 internal anchors

  1. [1]

    Unitary evolution recurrent neural networks

    Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In The International Conference on Machine Learning (ICML) , pages 1120–1128, 2016

    work page 2016

  2. [2]

    Adaptive Input Representations for Neural Language Modeling

    Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018

    work page Pith review arXiv 2018

  3. [3]

    An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018

    work page internal anchor Pith review arXiv 2018

  4. [4]

    Trellis networks for sequence modeling

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. In The International Conference on Learning Representations (ICLR) , 2019

    work page 2019

  5. [5]

    Dilated recurrent neural networks

    Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2017

    work page 2017

  6. [6]

    Generating Long Sequences with Sparse Transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  7. [7]

    Parallelizing legendre memory unit training

    Narsimha Chilkuri and Chris Eliasmith. Parallelizing legendre memory unit training. The International Conference on Machine Learning (ICML) , 2021

    work page 2021

  8. [8]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In The International Conference on Learning Representations (ICLR) , 2020

    work page 2020

  9. [9]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning , pages 933–941. PMLR, 2017

    work page 2017

  10. [10]

    Gru-ode-bayes: Continuous modeling of sporadically-observed time series

    Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. In Advances in Neural Information Processing Systems (NeurIPS) , 2019

    work page 2019

  11. [11]

    Adversarial audio synthesis

    Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In ICLR, 2019

    work page 2019

  12. [12]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  13. [13]

    Lipschitz recurrent neural networks

    N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Representations, 2021

    work page 2021

  14. [14]

    It’s raw! audio generation with state-space models

    Karan Goel, Albert Gu, Chris Donahue, and Christopher R´ e. It’s raw! audio generation with state-space models. arXiv preprint arXiv:2202.09729 , 2022

    work page arXiv 2022

  15. [15]

    Matrix computations, volume 3

    Gene H Golub and Charles F Van Loan. Matrix computations, volume 3. JHU press, 2013

    work page 2013

  16. [16]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R´ e. Hippo: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems (NeurIPS) , 2020

    work page 2020

  17. [17]

    Improving the gating mechanism of recurrent neural networks

    Albert Gu, Caglar Gulcehre, Tom Le Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In The International Conference on Machine Learning (ICML) , 2020. 13

    work page 2020

  18. [18]

    Combining recurrent, convolutional, and continuous-time models with the structured learnable linear state space layer

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R´ e. Combining recurrent, convolutional, and continuous-time models with the structured learnable linear state space layer. In Advances in Neural Information Processing Systems (NeurIPS) , 2021

    work page 2021

  19. [19]

    On the Parameterization and Initialization of Diagonal State Space Models,

    Albert Gu, Ankit Gupta, Karan Goel, and Christopher R´ e. On the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206.11893 , 2022

    work page arXiv 2022

  20. [20]

    How to train your hippo: State space models with generalized orthogonal basis projections

    Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher R´ e. How to train your hippo: State space models with generalized basis projections. arXiv preprint arXiv:2206.12037 , 2022

    work page arXiv 2022

  21. [21]

    Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    work page 1997

  22. [22]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸ cois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020

    work page 2020

  23. [23]

    URL:https://arxiv.org/ abs/2005.08926.arXiv:2005.08926

    Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint arXiv:2005.08926 , 2020

    work page arXiv 2005

  24. [24]

    Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group

    Mario Lezcano-Casado and David Mart´ ınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In The International Conference on Machine Learning (ICML), 2019

    work page 2019

  25. [25]

    Independently recurrent neural network (IndRNN): Building a longer and deeper RNN

    Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (IndRNN): Building a longer and deeper RNN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5457–5466, 2018

    work page 2018

  26. [26]

    Time-aware large kernel convolutions

    Vasileios Lioutas and Yuhong Guo. Time-aware large kernel convolutions. In International Conference on Machine Learning, pages 6172–6183. PMLR, 2020

    work page 2020

  27. [27]

    Scalable language modeling: Wikitext-103 on a single gpu in 12 hours

    Stephen Merity, Nitish Shirish Keskar, James Bradbury, and Richard Socher. Scalable language modeling: Wikitext-103 on a single gpu in 12 hours. SysML, 2018

    work page 2018

  28. [28]

    WaveNet: A Generative Model for Raw Audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 2016

    work page internal anchor Pith review arXiv 2016

  29. [29]

    Structured matrices and polynomials: unified superfast algorithms

    Victor Pan. Structured matrices and polynomials: unified superfast algorithms . Springer Science & Business Media, 2001

    work page 2001

  30. [30]

    Fast approximate computations with cauchy matrices and polynomials

    Victor Pan. Fast approximate computations with cauchy matrices and polynomials. Mathematics of Computation, 86(308):2799–2826, 2017

    work page 2017

  31. [31]

    Transformations of matrix structures work again

    Victor Y Pan. Transformations of matrix structures work again. Linear Algebra and Its Applications , 465:107–138, 2015

    work page 2015

  32. [32]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310–1318, 2013

    work page 2013

  33. [33]

    Fast parametric learning with activation memorization

    Jack Rae, Chris Dyer, Peter Dayan, and Timothy Lillicrap. Fast parametric learning with activation memorization. The International Conference on Machine Learning (ICML) , 2018

    work page 2018

  34. [34]

    Fast generation for convolutional autoregressive models

    Prajit Ramachandran, Tom Le Paine, Pooya Khorrami, Mohammad Babaeizadeh, Shiyu Chang, Yang Zhang, Mark A Hasegawa-Johnson, Roy H Campbell, and Thomas S Huang. Fast generation for convolutional autoregressive models. arXiv preprint arXiv:1704.06001 , 2017

    work page internal anchor Pith review arXiv 2017

  35. [35]

    arXiv preprint arXiv:2102.02611 , year=

    David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611 , 2021. 14

    work page arXiv 2021

  36. [36]

    Flexconv: Continuous kernel convolutions with differentiable kernel sizes

    David W Romero, Robert-Jan Bruintjes, Jakub M Tomczak, Erik J Bekkers, Mark Hoogendoorn, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In The International Conference on Learning Representations (ICLR) , 2022

    work page 2022

  37. [37]

    Latent ordinary differential equations for irregularly-sampled time series

    Yulia Rubanova, Tian Qi Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. In Advances in Neural Information Processing Systems, pages 5321–5331, 2019

    work page 2019

  38. [38]

    Unicornn: A recurrent model for learning very long time dependencies

    T Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. The International Conference on Machine Learning (ICML) , 2021

    work page 2021

  39. [39]

    PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017

    work page Pith review arXiv 2017

  40. [40]

    Long range arena : A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=qVyeW-grC2k

    work page 2021

  41. [41]

    Mlp-mixer: An all-mlp architecture for vision, 2021

    Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601 , 2021

    work page arXiv 2021

  42. [42]

    Learning longer-term dependencies in RNNs with auxiliary losses

    Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoc V Le. Learning longer-term dependencies in RNNs with auxiliary losses. In The International Conference on Machine Learning (ICML) , 2018

    work page 2018

  43. [43]

    A method of analysing the behaviour of linear systems in terms of time series

    Arnold Tustin. A method of analysing the behaviour of linear systems in terms of time series. Journal of the Institution of Electrical Engineers-Part IIA: Automatic Regulators and Servo Mechanisms , 94(1): 130–142, 1947

    work page 1947

  44. [44]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    work page 2017

  45. [45]

    Legendre memory units: Continuous-time representation in recurrent neural networks

    Aaron Voelker, Ivana Kaji´ c, and Chris Eliasmith. Legendre memory units: Continuous-time representation in recurrent neural networks. In Advances in Neural Information Processing Systems, pages 15544–15553, 2019

    work page 2019

  46. [46]

    Dynamical systems in spiking neuromorphic hardware

    Aaron Russell Voelker. Dynamical systems in spiking neuromorphic hardware . PhD thesis, University of Waterloo, 2019

    work page 2019

  47. [47]

    Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

    Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. ArXiv, abs/1804.03209, 2018

    work page Pith review arXiv 2018

  48. [48]

    Inverting modified matrices

    Max A Woodbury. Inverting modified matrices. Memorandum report, 42:106, 1950

    work page 1950

  49. [49]

    Pay less attention with lightweight and dynamic convolutions

    Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In The International Conference on Learning Representations (ICLR), 2019

    work page 2019

  50. [50]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference , volume 35, pages 11106– 11115. AAAI Press, 2021. 15 A Discussion Related Work. Our work...

    work page 2021

  51. [51]

    Gu et al

    derived a non-trainable SSM motivated from approximating a neuromorphic spiking model, and Chilkuri and Eliasmith [7] showed that it could be sped up at train time with a convolutional view. Gu et al. [16] extended this special case to a general continuous-time function approximation framework with several more special cases of A matrices designed for lon...

    work page

  52. [52]

    Our S4 model uses the same Transformer backbone as in [ 2]

    and many more. Our S4 model uses the same Transformer backbone as in [ 2]. The model consists of 16 blocks of S4 layers alternated with position-wise feedforward layers, with a feature dimension of 1024. Because our S4 layer has around 1/4 the number of parameters as a self-attention layer with the same dimension, we made two modifications to match the par...

    work page 2040