REVIEW 2 major objections 2 minor 3 references
AdapShot uses output entropy from short probes to pick per-query shot counts and reuses KV caches to raise many-shot ICL accuracy while cutting inference cost.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-07-01 00:16 UTC pith:A4NJWH45
load-bearing objection The entropy probe's link to actual accuracy is the untested core assumption here. the 2 major comments →
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
AdapShot dynamically optimizes shot counts for each query by selecting the number that produces the lowest output entropy in a probe evaluation, combined with a semantics-aware KV cache reuse strategy that decouples and re-encodes cached pairs to handle positional incompatibilities, enabling efficient many-shot in-context learning without fixed shot limits.
What carries the argument
Probe-based evaluation mechanism that uses output entropy to select the optimal shot count, together with semantics-aware KV cache reuse that decouples content from position and re-encodes for flexible ordering.
Load-bearing premise
The shot count that minimizes probe entropy will also produce the highest final answer accuracy on the actual query.
What would settle it
On a test set where every query is exhaustively evaluated at every feasible shot count, measure whether the entropy-chosen count matches the true highest-accuracy count in most cases; the claim fails if average accuracy falls below the best fixed-shot baseline.
If this is right
- Many-shot ICL can be applied to queries of unknown difficulty without manual tuning or excessive compute.
- Redundant prefilling work is eliminated across both the probing stage and the final generation stage.
- Positional encoding mismatches no longer prevent reuse of cached examples from different shot counts.
- Inference runs 4.64 times faster on average while accuracy rises roughly 10 percent over prior adaptive methods.
Where Pith is reading between the lines
- The same entropy signal could be checked for guiding other choices such as example ordering or temperature within a single ICL call.
- The decoupling technique might transfer to other KV-cache-heavy workloads like long-context retrieval or multi-turn chat.
- A direct comparison of entropy minima against human preference for shot count would test whether the metric aligns with user judgment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AdapShot for many-shot in-context learning in LLMs. It introduces a probe-based mechanism that measures output entropy to dynamically select the per-query optimal shot count k, combined with a semantics-aware KV-cache reuse strategy that includes a decoupling and re-encoding step to resolve positional-encoding incompatibilities when reordering cached pairs. Experiments are reported to show an average ~10% performance gain and 4.64x speedup relative to the state-of-the-art DBSA baseline.
Significance. If the entropy probe is shown to reliably select shot counts that improve downstream accuracy (rather than merely reducing model uncertainty) and the cache-reuse overhead remains negligible, the approach would address a practical bottleneck in many-shot ICL by adapting context length to query difficulty while preserving efficiency.
major comments (2)
- [Abstract, §3] Abstract and §3 (probe mechanism): the claim that output entropy identifies the query-specific optimal shot count is load-bearing for both the 10% gain and the net 4.64x speedup, yet no validation is supplied that lower entropy predicts higher task accuracy rather than simply lower model uncertainty; without this correlation or an ablation showing selected k values outperform fixed-k or oracle-k baselines, the performance and speedup numbers cannot be substantiated.
- [§4] §4 (experimental results): the reported 10% average gain and 4.64x speedup are presented without per-task variance, number of runs, or ablation on probe overhead (number of forward passes, tokens whose entropy is measured); if the probe cost is non-negligible or frequently selects suboptimal k, both headline numbers are at risk.
minor comments (2)
- [§3.1] Notation for the entropy probe (which tokens, which layer, temperature) should be formalized with an equation rather than left descriptive.
- [§3.2] The decoupling/re-encoding procedure for KV pairs would benefit from a small diagram or pseudocode to clarify how positional encodings are restored after reordering.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the probe validation and experimental details. We address each major comment below and will revise the manuscript to incorporate additional analysis and reporting.
read point-by-point responses
-
Referee: [Abstract, §3] Abstract and §3 (probe mechanism): the claim that output entropy identifies the query-specific optimal shot count is load-bearing for both the 10% gain and the net 4.64x speedup, yet no validation is supplied that lower entropy predicts higher task accuracy rather than simply lower model uncertainty; without this correlation or an ablation showing selected k values outperform fixed-k or oracle-k baselines, the performance and speedup numbers cannot be substantiated.
Authors: We agree that a direct correlation analysis between entropy and accuracy would strengthen the justification for the probe. The manuscript reports end-to-end gains from probe-selected k versus the DBSA baseline (which uses fixed or heuristic k), but does not include an explicit entropy-accuracy plot or comparison to an oracle-k. In revision we will add a dedicated ablation subsection that (i) reports Pearson/Spearman correlation between probe entropy and downstream accuracy across queries, (ii) compares probe-selected k against fixed-k, random-k, and length-based heuristics on the same tasks, and (iii) discusses why an exhaustive oracle-k is infeasible at scale while showing that AdapShot still outperforms all static baselines. This will substantiate that the entropy signal is not merely reducing uncertainty but selecting practically better k. revision: yes
-
Referee: [§4] §4 (experimental results): the reported 10% average gain and 4.64x speedup are presented without per-task variance, number of runs, or ablation on probe overhead (number of forward passes, tokens whose entropy is measured); if the probe cost is non-negligible or frequently selects suboptimal k, both headline numbers are at risk.
Authors: We accept that variance, run counts, and probe-cost ablations are necessary for credibility. The current manuscript reports only aggregate numbers. In the revised version we will (i) add per-task tables with mean and standard deviation over at least three random seeds, (ii) state the exact number of runs, and (iii) include a new overhead analysis quantifying the probe’s forward-pass count, tokens evaluated for entropy, and wall-clock contribution relative to the KV-cache savings. These additions will allow readers to verify that probe overhead remains negligible and that the reported 4.64× net speedup is robust. revision: yes
Circularity Check
No circularity; method is a procedural proposal validated by experiments
full rationale
The paper introduces AdapShot as an engineering combination of entropy-based probing for dynamic shot count and a semantics-aware KV-cache reuse technique with decoupling/re-encoding. No equations, fitted parameters, or self-citations are presented in the provided text that would make any claimed performance gain or speedup equivalent to the inputs by construction. The central claims rest on external experimental comparisons rather than a closed derivation loop. This is the normal case of a non-circular systems paper.
Axiom & Free-Parameter Ledger
read the original abstract
Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs). However, existing methods typically rely on a predetermined, fixed number of shots. This static approach often fails to adapt to the varying difficulty of different queries, leading to either insufficient context or interference from noise. Furthermore, the prohibitive computational and memory costs of long contexts severely limit Many-Shot's feasibility. To address the above limitations, we propose AdapShot, which dynamically optimizes shot counts and leverages KV cache reuse for efficient inference. Specifically, we design a probe-based evaluation mechanism that utilizes output entropy to determine the optimal number of shots. To bypass the redundant prefilling computation during both the probing and inference phases, we incorporate a semantics-aware KV cache reuse strategy. Within this reuse strategy, to address positional encoding incompatibilities, we introduce a decoupling and re-encoding method that enables the flexible reordering of cached key-value pairs. Extensive experiments demonstrate that AdapShot achieves an average performance gain of around 10% and a 4.64x speedup compared to state-of-the-art DBSA.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2501.15030 (2025)
Many-shot in-context learning.Advances in Neural Information Processing Systems, 37:76930– 76966. Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. 2025. In-context learning with long-context models: An in-depth exploration. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of...
-
[2]
Llama 2: Open Foundation and Fine-Tuned Chat Models
Parallel context windows for large language models. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pages 6383–6402. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2...
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[3]
Joint lossless compression and steganography for medical images via large language models.arXiv preprint arXiv:2508.01782. Pengcheng Zheng, Chaoning Zhang, Jiarong Mo, Guo- Hui Li, Jiaquan Zhang, Jiahao Zhang, Sihan Cao, Sheng Zheng, Caiyan Qin, Guoqing Wang, and 1 others. 2026. Llava-fa: Learning fourier approxima- tion for compressing large multimodal m...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.