Pith. sign in

REVIEW 2 major objections 2 minor 24 cited by

Existing benchmarks fall short for testing LLM memory and continual learning from user feedback.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-18 06:16 UTC pith:WIYSYBKL

load-bearing objection MemoryBench introduces a simulation-based benchmark for continual learning from user feedback in LLMs, but the headline result on poor baseline performance rests on an unvalidated simulator. the 2 major comments →

arxiv 2510.17281 v7 pith:WIYSYBKL submitted 2025-10-20 cs.LG cs.AIcs.IR

MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems

Qingyao Ai , Yichen Tang , Changyue Wang , Jianming Long , Weihang Su , Yiqun Liu This is my paper
classification cs.LG cs.AIcs.IR
keywords LLM memorycontinual learningbenchmarkuser feedback simulationLLM systemsevaluation frameworkmemory optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM systems have largely improved through scaling data, parameters, and computation, but these approaches are reaching their limits. The paper introduces MemoryBench, which uses a user feedback simulation to test how well LLM systems can learn continually from interactions in various domains and languages. Experiments demonstrate that current state-of-the-art methods are neither effective nor efficient enough. This matters because it points to a new direction for advancing AI by mimicking how humans learn from experience rather than just getting bigger.

Core claim

We propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLM systems. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying.

What carries the argument

User feedback simulation framework that generates realistic interactions to build the MemoryBench for assessing continual learning in LLM systems.

Load-bearing premise

The user feedback simulation framework accurately represents real user behavior in deployed LLM services.

What would settle it

Running a controlled experiment where an LLM system is deployed with real users and comparing its learning performance over time to the benchmark predictions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLM systems should incorporate better mechanisms to learn from accumulated user feedback over time.
  • Evaluation of memory capabilities needs to shift from long reading comprehension to diverse, interactive tasks.
  • Future optimization algorithms for LLMs may focus on continual learning to overcome scaling limits.
  • New methods are needed to improve both the effectiveness and efficiency of memory in LLM services.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark results hold, real-world LLM deployments could benefit from feedback-driven updates to reduce errors over time.
  • This approach connects to broader efforts in lifelong learning for AI systems beyond LLMs.
  • Extending the benchmark to include more complex user behaviors could reveal additional weaknesses in current systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces MemoryBench, a benchmark for memory and continual learning in LLM systems. It argues that prior benchmarks emphasize homogeneous reading-comprehension tasks with long inputs, whereas MemoryBench employs a user-feedback simulation framework to evaluate learning from accumulated service-time interactions across multiple domains, languages, and task types. Experiments on state-of-the-art baselines are reported to show unsatisfactory effectiveness and efficiency.

Significance. If the simulation framework produces interactions representative of real deployed LLM usage, the benchmark could usefully expose gaps in current memory mechanisms and motivate more practical continual-learning algorithms. The work supplies a new evaluation resource in an area where existing tests are acknowledged to be limited.

major comments (2)
  1. [§4] §4 (User Feedback Simulation Framework): No quantitative validation is provided for the simulation (e.g., statistical comparison to production logs, human-rater studies of simulated vs. real sessions, or ablation on simulation parameters). This is load-bearing for the central claim that baselines perform poorly on service-time learning, because the observed results could be artifacts of how queries, feedback signals, and task distributions are generated.
  2. [Experiments section] Experiments section and associated tables: Results are presented without error bars, details on data-selection criteria, or explicit description of how the benchmark instances were constructed and filtered. These omissions make it difficult to judge the reliability and generalizability of the headline finding that SOTA methods are “far from satisfying.”
minor comments (2)
  1. [Abstract] The abstract and introduction could more explicitly state the number of domains, languages, and task types covered by the benchmark to help readers assess its breadth.
  2. [§3] Notation for feedback signals and memory-update rules should be defined once in a dedicated subsection rather than introduced piecemeal.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the thoughtful and constructive comments on our manuscript. We address each major comment below in detail and indicate where revisions will be made to strengthen the paper.

read point-by-point responses

  1. Referee: [§4] §4 (User Feedback Simulation Framework): No quantitative validation is provided for the simulation (e.g., statistical comparison to production logs, human-rater studies of simulated vs. real sessions, or ablation on simulation parameters). This is load-bearing for the central claim that baselines perform poorly on service-time learning, because the observed results could be artifacts of how queries, feedback signals, and task distributions are generated.

    Authors: We agree that quantitative validation of the simulation is important for supporting claims about real-world applicability. We do not have access to proprietary production logs and thus cannot perform statistical comparisons against them. However, we will add human-rater evaluation studies comparing simulated sessions to real user interactions and include ablations on simulation parameters (such as query generation and feedback signal distributions) in the revised manuscript to demonstrate representativeness. revision: partial

  2. Referee: [Experiments section] Experiments section and associated tables: Results are presented without error bars, details on data-selection criteria, or explicit description of how the benchmark instances were constructed and filtered. These omissions make it difficult to judge the reliability and generalizability of the headline finding that SOTA methods are “far from satisfying.”

    Authors: We acknowledge that the current presentation lacks these details, which limits assessment of reliability. In the revised version, we will add error bars computed over multiple random seeds or runs, provide explicit data-selection criteria, and include a detailed description of benchmark instance construction, filtering steps, and task distribution generation. revision: yes

standing simulated objections not resolved
  • Statistical comparison to production logs, as we lack access to proprietary real-world deployment data.

Circularity Check

0 steps flagged

No significant circularity in benchmark proposal or experimental claims

full rationale

The paper proposes a user feedback simulation framework and a new benchmark covering multiple domains and tasks to evaluate continual learning in LLM systems. It reports direct experimental results showing that state-of-the-art baselines perform poorly in effectiveness and efficiency. This does not involve any self-definitional reduction, fitted inputs renamed as predictions, or load-bearing self-citations that collapse the central claim to prior unverified work by the same authors. The benchmark construction and evaluation results are independent of the reported outcomes; the simulation defines the test environment rather than deriving the performance numbers by construction. The paper is self-contained against its own defined tasks with no evident circular derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that simulated user feedback can stand in for real interactions and that the chosen baselines are representative of current best practice.

axioms (1)
  • domain assumption Simulated user feedback accurately reflects real-world continual learning scenarios for LLM systems.
    Invoked to justify the benchmark design and experimental conclusions.

pith-pipeline@v0.9.0 · 5716 in / 1147 out tokens · 38822 ms · 2026-05-18T06:16:57.254358+00:00 · methodology

0 comments
read the original abstract

Scaling up data, parameters, and test-time computation has been the mainstream methods to improve LLM systems (LLMsys), but their upper bounds are almost reached due to the gradual depletion of high-quality data and marginal gains obtained from larger computational resource consumption. Inspired by the abilities of human and traditional AI systems in learning from practice, constructing memory and continual learning frameworks for LLMsys has become an important and popular research direction in recent literature. Yet, existing benchmarks for LLM memory often focus on evaluating the system on homogeneous reading comprehension tasks with long-form inputs rather than testing their abilities to learn from accumulated user feedback in service time. Therefore, we propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLMsys. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying, and we hope this benchmark could pave the way for future studies on LLM memory and optimization algorithms. Website: https://memorybench.thuir.cn Code: https://github.com/THUIR/MemoryBench Data: https://huggingface.co/datasets/THUIR/MemoryBench Data-Full: https://huggingface.co/datasets/THUIR/MemoryBench-Full

Figures

Figures reproduced from arXiv: 2510.17281 by Changyue Wang, Jianming Long, Qingyao Ai, Weihang Su, Yichen Tang, Yiqun Liu.

Figure 1
Figure 1. Figure 1: An illustrative example of different types of memory and user feedback. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The framework of MemoryBench includes three major modules: (1) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The off-policy experimental results of base [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Time consumption of different LLMsys in off-policy experiments on four task-format [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: On-policy performance of some LLMsys on the Open domain. The x-axis denotes training [PITH_FULL_IMAGE:figures/full_fig_p041_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: On-policy performance of some LLMsys on the Academic domain. The x-axis denotes [PITH_FULL_IMAGE:figures/full_fig_p041_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: On-policy performance of some LLMsys on the Legal domain. The x-axis denotes training [PITH_FULL_IMAGE:figures/full_fig_p042_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Stepwise off-policy performance of some LLMsys on the Open domain. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p042_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Stepwise off-policy performance of some LLMsys on the Academic domain. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p043_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Stepwise off-policy performance of some LLMsys on the Legal domain. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p043_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Mem0 exhibits severe slowdown during memorization across different tasks. (a) For [PITH_FULL_IMAGE:figures/full_fig_p045_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Probability of giving like feedback at different satisfaction scores under two noise levels. [PITH_FULL_IMAGE:figures/full_fig_p048_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance of different LLMsys under noisy like feedback with varying [PITH_FULL_IMAGE:figures/full_fig_p048_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Performance of different LLMsys under noisy like feedback with varying [PITH_FULL_IMAGE:figures/full_fig_p049_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

    cs.AI 2026-06 unverdicted novelty 8.0

    CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.

  2. GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

    cs.CL 2026-05 conditional novelty 8.0

    GroupMemBench shows leading LLM memory systems reach only 46% average accuracy on multi-party tasks, with a simple BM25 baseline matching or beating most of them.

  3. MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

    cs.IR 2026-07 unverdicted novelty 7.0

    MemSyco-Bench is a new benchmark with five tasks to assess memory-induced sycophancy in LLM agent systems.

  4. MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

    cs.IR 2026-07 unverdicted novelty 7.0

    MemSyco-Bench is a benchmark covering five tasks to evaluate memory-induced sycophancy in LLM agents, testing rejection of invalid memory, scope respect, conflict resolution, update tracking, and valid personalization.

  5. When Classic Cache Policies Fail: Learning-Augmented Replacement for Semantic Retrieval Buffers

    cs.DB 2026-07 unverdicted novelty 7.0

    SOLAR is a learning-augmented policy for semantic cache replacement that achieves constant competitive ratio 3 and 5-75% gains over FIFO on retrieval workloads.

  6. Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

    cs.CV 2026-06 unverdicted novelty 7.0

    RNG-Bench evaluates MLLMs on hidden-observation reconstruction in non-Markov games, finds forgetting as the dominant error source, and shows fine-tuning on optimal rollouts improves performance with transfer to other ...

  7. MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

    cs.AI 2026-06 unverdicted novelty 7.0

    MemTrace shows that evidence utilization, not retrieval, is the dominant failure mode in LLM long-term memory systems across tested configurations.

  8. MemGym: a Long-Horizon Memory Environment for LLM Agents

    cs.CL 2026-05 unverdicted novelty 7.0

    MemGym unifies agent gyms into a memory benchmark with isolated scoring across tool-use, research, coding, and computer-use regimes plus a lightweight reward model for tractable coding evaluation.

  9. EXG: Self-Evolving Agents with Experience Graphs

    cs.AI 2026-05 unverdicted novelty 7.0

    EXG is an experience graph framework for self-evolving LLM agents that supports online real-time growth and offline reuse to enhance solution quality and efficiency on code generation and reasoning benchmarks.

  10. GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

    cs.CL 2026-05 unverdicted novelty 7.0

    GroupMemBench is a new benchmark exposing that LLM agent memory systems fail on group conversation properties like speaker-grounded tracking and audience-adapted responses, with top systems at 46% accuracy.

  11. Skill Retrieval Augmentation for Agentic AI

    cs.CL 2026-04 unverdicted novelty 7.0

    Introduces the SRA paradigm and SRA-Bench benchmark showing retrieval-based skill augmentation improves agent performance but skill incorporation remains a bottleneck regardless of retrieval quality.

  12. Skill Retrieval Augmentation for Agentic AI

    cs.CL 2026-04 unverdicted novelty 7.0

    Agents improve when they retrieve skills on demand from large corpora, yet current models cannot selectively decide when to load or ignore a retrieved skill.

  13. PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

    cs.AI 2026-03 unverdicted novelty 7.0

    PERMA is a new benchmark using temporally ordered events, text variability, and linguistic alignment to evaluate LLM memory agents on persona consistency beyond simple retrieval.

  14. MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation

    cs.CL 2026-06 unverdicted novelty 6.0

    MemDelta shows agent memory evaluations are confounded by LLM family and embedding model, with RAG often matching full context and self-memory underperforming basic retrieval.

  15. Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

    cs.CL 2026-05 unverdicted novelty 6.0

    RefMem-Bench benchmarks reflective memory in dialogue with 26K instances across eight dimensions, and REMIND improves model accuracy via hierarchical evidence retrieval, grounding, and abstraction.

  16. EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

    cs.CL 2026-05 unverdicted novelty 6.0

    EvoMemBench evaluates 15 memory methods for LLM agents and finds long-context baselines competitive with no single memory approach working consistently across settings.

  17. State Contamination in Memory-Augmented LLM Agents

    cs.AI 2026-05 unverdicted novelty 6.0

    Toxic context can be laundered into memory summaries that stay below toxicity thresholds while still driving higher downstream toxicity in LLM agents compared to neutral baselines.

  18. Skill Retrieval Augmentation for Agentic AI

    cs.CL 2026-04 unverdicted novelty 6.0

    Introduces SRA paradigm and SRA-Bench benchmark (5,400 tasks, 26,262 skills) showing retrieval improves performance but LLMs fail to selectively incorporate retrieved skills.

  19. Trust Your Memory: Verifiable Control of Smart Homes through Reinforcement Learning with Multi-dimensional Rewards

    cs.AI 2026-04 unverdicted novelty 6.0

    Introduces MemHome benchmark and RL with multi-dimensional rewards for memory-driven smart home device control.

  20. ATANT: An Evaluation Framework for AI Continuity

    cs.AI 2026-04 unverdicted novelty 6.0

    ATANT defines AI continuity via seven properties and offers a 10-checkpoint, LLM-free test using 250 stories to check if systems retrieve correct facts without cross-contamination.

  21. Exploring Cross-Scenario Generality of Agentic Memory Systems: Diagnostics and a Strong Baseline

    cs.AI 2026-06 unverdicted novelty 5.0

    An agentic harness letting the LLM self-manage flat text-file storage via tool calls outperforms eight prior memory systems on cross-scenario generality across QA, chat, trajectory, stress-test, and long-horizon tasks.

  22. AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

    cs.AI 2026-06 unverdicted novelty 5.0

    AgentCL constructs controlled task streams with intentional reusability and introduces MemProbe to evaluate non-parametric memory designs for continual learning in language agents across coding, research, and reasoning tasks.

  23. Improve Large Language Model Systems with User Logs

    cs.CL 2026-02 unverdicted novelty 5.0

    UNO distills user logs into semi-structured rules and preferences, applies query-and-feedback clustering to handle heterogeneity, quantifies cognitive gaps to filter noise, and builds primary and reflective modules th...

  24. ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks

    cs.AI 2026-04 unverdicted novelty 4.0

    Existing memory benchmarks cover at most two of the seven continuity properties from ATANT v1.0, with a median of one and none covering more than two.