pith. sign in

arxiv: 2605.22337 · v2 · pith:RF5AMAPLnew · submitted 2026-05-21 · 💻 cs.AI

Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression

Wei Luo , Yi Huang , Songchen Ma , Huanyu Qu , Jiang Cai , Mingkun Xu This is my paper

Pith reviewed 2026-06-30 17:25 UTC · model grok-4.3

classification 💻 cs.AI
keywords KV cache compressionsoft tokensmeta-libraryattention flowcontext preservationGumbel-SoftmaxLLM inferenceeviction methods
0
0 comments X

The pith

Meta-Soft dynamically synthesizes adaptive soft tokens from a meta-library to compress KV cache while redistributing evicted semantics via attention flow.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models face memory growth and efficiency loss from expanding KV caches during long-context processing. Fixed soft-token eviction methods rely on static queries that fail to adapt to varying prompts and permanently discard information. The paper introduces a meta-library built on a learnable orthogonal basis matrix together with a Gumbel-Softmax selector network that produces sparse weights to form targeted soft tokens on the fly. These tokens are appended to probe the sequence, after which an attention-flow mechanism redistributes the semantic content of removed KV pairs into retained ones. Experiments across datasets are reported to show gains over prior eviction techniques.

Core claim

The paper claims that a probe-driven compression framework, built around a meta-library whose learnable orthogonal basis matrix supplies composable components and a selector network that uses Gumbel-Softmax to produce differentiable sparse weights for synthesizing the most relevant k soft tokens from prompt features, when combined with an attention-flow integration step that redistributes semantics of dropped tokens, enables dynamic, context-preserving KV cache compression that avoids irreversible loss and adapts to each input.

What carries the argument

Meta-library with learnable orthogonal basis matrix L plus Gumbel-Softmax selector network that produces sparse combination weights to synthesize dynamic soft tokens, followed by attention-flow integration for semantic redistribution.

If this is right

  • KV cache memory footprint shrinks while task performance on long inputs is maintained.
  • Evicted token information is redistributed rather than discarded, reducing context breaks.
  • Compression becomes input-dependent instead of relying on fixed static queries.
  • A new eviction strategy is available that can be applied at inference time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could be tested on decoder-only models beyond those in the experiments to check generality.
  • Combining the meta-token synthesis with quantization might yield multiplicative memory savings.
  • If the selector network generalizes, similar composable tokens might compress other attention stores such as in vision transformers.
  • One could measure whether the synthesized tokens themselves become useful for downstream analysis of prompt relevance.

Load-bearing premise

Dynamically synthesized soft tokens can precisely capture complex and changing task relevance, and attention-flow redistribution preserves all semantic information without irreversible loss.

What would settle it

Measure accuracy or factual recall on long-context tasks using the compressed cache versus the full KV cache; a consistent drop when critical details are evicted would falsify the preservation claim.

Figures

Figures reproduced from arXiv: 2605.22337 by Huanyu Qu, Jiang Cai, Mingkun Xu, Songchen Ma, Wei Luo, Yi Huang.

Figure 1
Figure 1. Figure 1: Motivation and overview of Meta-Soft. Left: Existing KV-cache compression often relies on static queries for eviction, which fail to adapt across diverse tasks and may cause cross-task mismatch; moreover, hard eviction permanently deletes KV entries, leading to irreversible information loss and broken context. Right: Meta-Soft uses input-dependent dynamic soft tokens synthesized from a meta-library to prob… view at source ↗
Figure 2
Figure 2. Figure 2: Meta-Soft framework overview. Meta-Soft trains a Meta-Library and selector offline with Ground-Truth Attention supervision and compresses the KV cache online by generating prompt-conditioned soft tokens to probe, partition, and consolidate context for decoding. Cache Partitioning Based on Asof t and budget B, we par￾tition the cache: Ikeep = TopK(Asof t, B), Idrop = {1, . . . , L} \ Ikeep (6) This yields t… view at source ↗
read the original abstract

The KV cache used in large language models has linearly growing time complexity, so LLMs face memory blow-up and reduced decoding efficiency when they process long contexts. Current KV Cache eviction has become an important research direction; however, existing methods based on fixed Soft Tokens (e.g., Judge Q) rely on a static parameter set as the query to evaluate the importance of KV pairs, so they cannot adapt dynamically to different input prompts, and they cannot precisely capture complex and changing task relevance. Also, evicted KV pairs are discarded permanently, so this causes irreversible information loss and context breaks. To address this problem, we propose Meta-Soft, a dynamic compression framework based on probe-driven context integration. Specifically, we build a meta-library with a learnable orthogonal basis matrix $\mathcal{L}$, and we use a selector network with Gumbel-Softmax to produce differentiable sparse combination weights, so we dynamically synthesize the most targeted $k$ Soft Tokens from the input prompt features. We append these Soft Tokens to the end of the input sequence to probe key information. We also introduce an attention-flow based integration mechanism, which redistributes the semantic information of removed tokens into retained tokens, and this keeps the dropped context information effectively. Experiments on multiple datasets show that our method outperforms existing state-of-the-art eviction methods and provides a new solution for KV Cache compression.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes Meta-Soft, a KV cache compression framework for LLMs that constructs a meta-library from a learnable orthogonal basis matrix L, employs a selector network with Gumbel-Softmax to dynamically synthesize k adaptive soft tokens from input prompt features, appends them to probe key information, and applies an attention-flow integration mechanism to redistribute semantic content from evicted tokens into retained ones, claiming to avoid irreversible loss and outperform prior static soft-token eviction methods on multiple datasets.

Significance. If the performance claims hold with rigorous validation, the method could advance context-preserving compression by replacing fixed-parameter queries with prompt-adaptive meta-tokens and by attempting to conserve evicted information via attention redistribution, offering a potential improvement over methods that discard KV pairs permanently.

major comments (2)
  1. [Abstract] Abstract: the central claim that 'experiments on multiple datasets show that our method outperforms existing state-of-the-art eviction methods' is unsupported because the abstract (and the manuscript as presented) supplies no experimental setup, baselines, metrics, error bars, ablation studies, or results tables; without these the outperformance assertion cannot be evaluated and appears unsubstantiated.
  2. [Abstract] Abstract: the assumption that the selector network and attention-flow mechanism 'precisely capture complex and changing task relevance' and 'redistribute the semantic information of removed tokens without irreversible loss' is stated but receives no derivation, proof, or empirical test in the provided text, leaving the core technical contribution unverified.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract. We address each point below and will revise the manuscript to strengthen the presentation of our claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'experiments on multiple datasets show that our method outperforms existing state-of-the-art eviction methods' is unsupported because the abstract (and the manuscript as presented) supplies no experimental setup, baselines, metrics, error bars, ablation studies, or results tables; without these the outperformance assertion cannot be evaluated and appears unsubstantiated.

    Authors: The referee is correct that the provided manuscript text is limited to the abstract and does not include experimental details, setups, baselines, metrics, or results tables. We will revise the manuscript by expanding it to include dedicated experimental sections with full setup descriptions, baseline comparisons (e.g., against static soft-token methods), evaluation metrics, and results tables to substantiate the outperformance claim. revision: yes

  2. Referee: [Abstract] Abstract: the assumption that the selector network and attention-flow mechanism 'precisely capture complex and changing task relevance' and 'redistribute the semantic information of removed tokens without irreversible loss' is stated but receives no derivation, proof, or empirical test in the provided text, leaving the core technical contribution unverified.

    Authors: We acknowledge that the abstract states these properties without accompanying derivation, proof, or empirical tests in the presented text. In revision, we will add explicit mathematical derivations for the Gumbel-Softmax selector and attention-flow integration, along with targeted ablation studies and empirical validations to test these aspects of the core contribution. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The abstract and framework description introduce a new construction (meta-library with learnable orthogonal basis matrix L, Gumbel-Softmax selector, synthesized soft tokens, and attention-flow integration) without any equations, derivations, or self-citations that reduce the claimed mechanisms or performance to inputs by construction. No fitted-input-called-prediction, self-definitional, or uniqueness-imported patterns appear. Experimental outperformance is asserted as external validation rather than an internal reduction. This is the common honest finding for papers whose central claims rest on empirical results rather than definitional equivalences.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 1 invented entities

Only the abstract is available; no equations, training details, or explicit assumptions are provided, preventing identification of specific free parameters, axioms, or invented entities beyond the high-level components named in the abstract.

free parameters (1)
  • learnable orthogonal basis matrix L
    Described as learnable; its values are presumably fitted during training but no details given.
invented entities (1)
  • meta-tokens no independent evidence
    purpose: Dynamically synthesized soft tokens to probe key information from the input prompt
    Introduced as core of the framework; no independent evidence outside the method itself is mentioned.

pith-pipeline@v0.9.1-grok · 5785 in / 1224 out tokens · 55165 ms · 2026-06-30T17:25:26.681154+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 21 canonical work pages · 11 internal anchors

  1. [1]

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

    [Baiet al., 2023 ] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding.arXiv preprint arXiv:2308.14508,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Token Merging: Your ViT But Faster

    [Bolyaet al., 2022 ] Daniel Bolya, Cheng-Yang Fu, Xiao- liang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461,

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    Lococo: Dropping in convolutions for long context compression.arXiv preprint arXiv:2406.05317,

    Accepted at ICLR 2023 (Oral), per arXiv. [Caiet al., 2024a ] Ruisi Cai, Yuandong Tian, Zhangyang Wang, and Beidi Chen. Lococo: Dropping in convo- lutions for long context compression.arXiv preprint arXiv:2406.05317,

    work page arXiv 2023

  4. [4]

    PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

    [Caiet al., 2024b ] Zefan Cai, Yichi Zhang, Bofei Gao, Yu- liang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal infor- mation funneling.arXiv preprint arXiv:2406.02069,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2305.14788 , year=

    [Chevalieret al., 2023 ] Alexis Chevalier, Alexander Wet- tig, Anirudh Ajith, and Danqi Chen. Adapting lan- guage models to compress contexts.arXiv preprint arXiv:2305.14788,

    work page arXiv 2023

  6. [6]

    Gonzalez, Ion Stoica, and Eric P

    [Chianget al., 2023 ] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chat- bot impressing gpt-4 with 90%* chatgpt quality, March

    2023

  7. [7]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e

    [Daoet al., 2022 ] Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness.Ad- vances in Neural Information Processing Systems,

    2022

  8. [8]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    [Gaoet al., 2020 ] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  9. [9]

    Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

    [Geet al., 2023 ] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    Accurate kv cache eviction via anchor direction projection for efficient llm inference

    [Genget al., 2025 ] Zijie Geng, Jie Wang, Ziqi Liu, Feng Ju, Yiming Li, Xing Li, Mingxuan Yuan, Jianye Hao, Defu Lian, Enhong Chen, and Feng Wu. Accurate kv cache eviction via anchor direction projection for efficient llm inference. InAdvances in Neural Information Processing Systems,

    2025

  11. [11]

    The Llama 3 Herd of Models

    NeurIPS 2025 (OpenReview). [Grattafioriet al., 2024 ] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Zipcache: Accurate and efficient kv cache quantization with salient token iden- tification,

    [Heet al., 2024 ] Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accu- rate and efficient kv cache quantization with salient token identification.arXiv preprint arXiv:2405.14256,

    work page arXiv 2024

  13. [13]

    RULER: What's the Real Context Size of Your Long-Context Language Models?

    [Hsiehet al., 2024 ] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    [Jianget al., 2023 ] Albert Q

    COLM 2024 (per arXiv). [Jianget al., 2023 ] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth ´ee Lacroix, and Willi...

    2024

  15. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    [Kwonet al., 2023 ] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Effi- cient memory management for large language model serv- ing with pagedattention. InProceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23),

    2023

  16. [16]

    SnapKV: LLM Knows What You are Looking for Before Generation

    [Liet al., 2024 ] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.arXiv preprint arXiv:2404.14469,

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time.arXiv preprint arXiv:2305.17118, 2023

    [Liuet al., 2023 ] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anasta- sios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time.arXiv preprint arXiv:2305.17118,

    work page arXiv 2023

  18. [18]

    arXiv preprint arXiv:2509.10798 , year=

    [Liuet al., 2025b ] Yijun Liu, Yixuan Wang, Yuzhuang Xu, Shiyu Ji, Yang Xu, Qingfu Zhu, and Wanxiang Che. Judge q: Trainable queries for optimized information retention in kv cache eviction.arXiv preprint arXiv:2509.10798,

    work page arXiv

  19. [19]

    Learning to Compress Prompts with Gist Tokens, 2023

    [Muet al., 2023 ] Jesse Mu, Xiang Lisa Li, and Noah Good- man. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304.08467,

    work page arXiv 2023

  20. [20]

    [Popeet al., 2023 ] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean

    NeurIPS 2023 (per arXiv). [Popeet al., 2023 ] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, volume 5,

    2023

  21. [21]

    Compressive Transformers for Long-Range Sequence Modelling

    [Raeet al., 2019 ] Jack W. Rae, Anna Potapenko, Sid- dhant M. Jayakumar, and Timothy P. Lillicrap. Com- pressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507,

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  22. [22]

    [Ren and Zhu, 2024] Siyu Ren and Kenny Q. Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference.arXiv preprint arXiv:2402.06262,

    work page arXiv 2024

  23. [23]

    Slimpajama-dc: Understanding data combinations for llm training,

    [Shenet al., 2024 ] Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-dc: Understanding data combinations for llm training,

    2024

  24. [24]

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang

    [Wanet al., 2024 ] Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. D2o: Dy- namic discriminative operations for efficient long-context inference of large language models.arXiv preprint arXiv:2406.13035,

    work page arXiv 2024

  25. [25]

    [Wanget al., 2025 ] Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, and Wanxiang Che

    ICLR 2025 (per arXiv). [Wanget al., 2025 ] Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, and Wanxiang Che. Lookahead q-cache: Achieving more consistent kv cache eviction via pseudo query.arXiv preprint arXiv:2505.20334,

    work page arXiv 2025

  26. [26]

    Efficient Streaming Language Models with Attention Sinks

    Accepted by EMNLP 2025 Main (per arXiv). [Xiaoet al., 2023 ] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks.arXiv preprint arXiv:2309.17453,

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    [Yanget al., 2024 ] June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left be- hind: Reliable kv cache compression via importance- aware mixed precision quantization.arXiv preprint arXiv:2402.18096,

    work page arXiv 2024

  28. [28]

    H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

    [Zhanget al., 2023 ] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R ´e, Clark Barrett, Zhangyang Wang, and Beidi Chen. H 2O: Heavy-hitter oracle for efficient generative inference of large language models.arXiv preprint arXiv:2306.14048,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    CaM: Cache merging for memory-efficient LLMs inference

    [Zhanget al., 2024 ] Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Ron- grong Ji. CaM: Cache merging for memory-efficient LLMs inference. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proceedings of the 41st International Conference on M...

    2024