ScrapMem: A Bio-inspired Framework for On-device Personalized Agent Memory via Optical Forgetting
Pith reviewed 2026-07-01 00:06 UTC · model grok-4.3
The pith
ScrapMem applies optical forgetting to lower the resolution of older memories inside a causal-temporal graph, making long-term multimodal agent memory feasible on edge devices.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
ScrapMem integrates multimodal data into Scrapbook Pages and introduces Optical Forgetting, which progressively reduces the resolution of older memories to lower storage cost while suppressing low-value details, and constructs an Episodic Memory Graph to organize key events into a causal-temporal structure that maintains semantic consistency.
What carries the argument
Optical Forgetting, the mechanism that progressively reduces resolution of older memories, paired with the Episodic Memory Graph that structures events causally and temporally.
If this is right
- Storage requirements for long-term agent memory drop substantially while joint task accuracy reaches new benchmark levels.
- Recall on memory-dependent queries improves through the structured aggregation provided by the graph.
- Multimodal data can be retained on-device for extended periods without requiring constant external resources.
- The combination of compression and graph organization supports efficient handling of complex personal histories.
Where Pith is reading between the lines
- The same resolution-reduction approach could be tested on sequential personal data streams such as sensor logs or interaction histories outside agent settings.
- If the graph maintains consistency after compression, the method might scale to shared memories across multiple agents.
- Hardware implementations that mimic the progressive detail loss could be explored for even lower power memory modules.
Load-bearing premise
Progressively lowering the resolution of older memories removes low-value details without damaging the semantic consistency that the episodic graph is meant to preserve.
What would settle it
A controlled test that measures semantic accuracy or task performance on recall queries before and after applying resolution reduction to the same set of older memories, where accuracy falls below baseline levels.
Figures
read the original abstract
Long-term personalized memory for LLM agents is challenging on resource-limited edge devices due to high storage costs and multimodal complexity. To address this, we propose ScrapMem, a framework that integrates multimodal data into "Scrapbook Page." ScrapMem introduces Optical Forgetting, an optical compression mechanism that progressively reduces the resolution of older memories, lowering storage cost while suppressing low-value details. To maintain semantic consistency, we construct an Episodic Memory Graph (EM-Graph) that organizes key events into a causal-temporal structure. Extensive experiments on the multimodal ATM-Bench showcase that ScrapMem provides three main benefits: (1) strong performance, achieving a new state-of-the-art with a 51.0% Joint@10 score; (2) high storage efficiency, reducing memory usage by up to 93% via optical forgetting; and (3) improved recall, increasing Recall@10 to 70.3% through structured aggregation. ScrapMem offers an effective and storage-efficient solution for on-device long-term memory in multimodal LLM agents.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ScrapMem, a framework for long-term personalized memory in multimodal LLM agents on edge devices. It integrates data into 'Scrapbook Page' structures, applies 'Optical Forgetting' to progressively lower resolution of older memories for up to 93% storage reduction while suppressing low-value details, and uses an Episodic Memory Graph (EM-Graph) to organize events in a causal-temporal structure for semantic consistency. On the ATM-Bench dataset, it claims new SOTA results of 51.0% Joint@10, 70.3% Recall@10 via structured aggregation, and high storage efficiency.
Significance. If the performance, efficiency, and consistency claims hold with proper validation, the work could address a practically important challenge in on-device agent memory by providing a bio-inspired compression approach that trades resolution for storage without (claimed) loss of utility. The combination of optical-style forgetting with graph-structured episodic memory is a novel angle worth exploring for resource-constrained multimodal agents.
major comments (3)
- [Abstract] Abstract: The headline claims (51.0% Joint@10 SOTA, 93% memory reduction, 70.3% Recall@10) are presented with no experimental setup, baselines, error bars, dataset splits, or implementation description of Optical Forgetting or EM-Graph construction. This makes it impossible to assess whether the numbers can be attributed to the proposed mechanisms rather than unstated factors.
- [Abstract] Abstract: The central premise that Optical Forgetting 'suppresses low-value details' while the EM-Graph 'maintains semantic consistency' is stated but unsupported by any ablation, consistency metric, or comparison of full-resolution vs. forgotten memory on identical queries. Without such evidence the efficiency and recall claims rest on an untested assumption.
- [Abstract] Abstract: No equations, algorithms, or pseudocode are supplied for Optical Forgetting (resolution reduction schedule, optical model) or EM-Graph (node/edge definitions, causal-temporal construction, aggregation method), despite these being load-bearing for all three claimed benefits.
Simulated Author's Rebuttal
We thank the referee for the detailed feedback on the abstract. While abstracts are necessarily concise, we address each comment by clarifying where the supporting details appear in the full manuscript and indicate our willingness to revise the abstract for improved transparency.
read point-by-point responses
-
Referee: [Abstract] Abstract: The headline claims (51.0% Joint@10 SOTA, 93% memory reduction, 70.3% Recall@10) are presented with no experimental setup, baselines, error bars, dataset splits, or implementation description of Optical Forgetting or EM-Graph construction. This makes it impossible to assess whether the numbers can be attributed to the proposed mechanisms rather than unstated factors.
Authors: The abstract is a high-level summary; the full experimental setup, baselines, error bars, dataset splits on ATM-Bench, and implementation details for Optical Forgetting and EM-Graph are provided in Sections 4 and 5. We will revise the abstract to include a brief reference to the evaluation protocol and main baselines. revision: partial
-
Referee: [Abstract] Abstract: The central premise that Optical Forgetting 'suppresses low-value details' while the EM-Graph 'maintains semantic consistency' is stated but unsupported by any ablation, consistency metric, or comparison of full-resolution vs. forgotten memory on identical queries. Without such evidence the efficiency and recall claims rest on an untested assumption.
Authors: Section 5.3 presents ablation studies comparing full-resolution vs. forgotten memory on identical queries, along with consistency metrics and the contribution of the EM-Graph. We will revise the abstract to reference these supporting experiments. revision: partial
-
Referee: [Abstract] Abstract: No equations, algorithms, or pseudocode are supplied for Optical Forgetting (resolution reduction schedule, optical model) or EM-Graph (node/edge definitions, causal-temporal construction, aggregation method), despite these being load-bearing for all three claimed benefits.
Authors: Section 3 provides the equations for the Optical Forgetting resolution schedule and optical model, plus formal definitions and construction details for the EM-Graph; pseudocode appears in the appendix. We will revise the abstract to briefly note the core mechanisms. revision: partial
Circularity Check
No circularity; empirical claims rest on reported experiments without derivations or self-referential reductions.
full rationale
The provided abstract and structure contain no equations, parameter fittings, uniqueness theorems, or self-citations that could create circularity. Performance, efficiency, and recall figures are presented as experimental outcomes on ATM-Bench rather than derived by construction from inputs or prior author work. The framework description (Optical Forgetting, EM-Graph) is introduced as a proposal without load-bearing self-referential steps, making the derivation chain self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Multimodal data can be integrated into Scrapbook Pages without loss of essential structure
- ad hoc to paper Optical forgetting can reduce resolution while the EM-Graph maintains semantic consistency
invented entities (3)
-
Optical Forgetting
no independent evidence
-
Episodic Memory Graph (EM-Graph)
no independent evidence
-
Scrapbook Page
no independent evidence
Reference graph
Works this paper leans on
-
[1]
Sina Abdollahi, Mohammad M Maheri, Javad Forough, Amir Al Sadi, Josh Millar, David Kotz, Marios Kogias, and Hamed Haddadi. 2026. Agentee: Confidential llm agent execution on edge devices. In Proceedings of the Sixth European Workshop on Machine Learning and Systems, pages 473--480
2026
-
[2]
Shuai Bai, Yuqi Cai, Ruoyi Chen, Kai Chen, Xu Chen, Zhihao Cheng, and 1 others. 2025 a . Qwen3-vl technical report. arXiv preprint arXiv:2511.21631
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[3]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2025 b . Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[4]
TING BAI, JIAYANG FAN, XIAOSHUAI WEN, JIAZHENG KANG, HENGZHI LAN, RUOCHEN ZHAO, PINGZHENG WU, ZEPENG ZHANG, YUTIAN ZHONG, GEZI LI, and 1 others. 2025. Survey on ai memory: Theories, taxonomies, evaluations, and emerging trends
2025
-
[5]
Ting Bai, Le Huang, Yue Yu, Cheng Yang, Cheng Hou, Zhe Zhao, and Chuan Shi. 2025. https://doi.org/10.1145/3736403 Efficient multi-task prompt tuning for recommendation . ACM Transactions on Information Systems, 43(4)
-
[6]
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, and 1 others. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206--2240. PMLR
2022
-
[7]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877--1901
2020
- [8]
-
[9]
Jialin Chen, Ziyu Zhao, Gaukhar Nurbek, Aosong Feng, Ali Maatouk, Leandros Tassiulas, Yifeng Gao, and Rex Ying. 2026. Trace: Grounding time series in context for multimodal embedding and retrieval. Advances in Neural Information Processing Systems, 38:2440--2472
2026
-
[10]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[11]
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[12]
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130
work page internal anchor Pith review Pith/arXiv arXiv 2024
- [13]
-
[14]
Yicheng Fu, Raviteja Anantha, and Jianpeng Cheng. 2025. https://doi.org/10.18653/v1/2025.realm-1.3 Camphor: Collaborative agents for multi-input planning and high-order reasoning on device . In Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025), pages 33--51, Vienna, Austria. Association for Computational Linguistics
-
[15]
Bernal J Guti \'e rrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems, 37:59532--59569
2024
-
[16]
Bernal Jim \'e nez Guti \'e rrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From rag to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[17]
Canny, and Ian Fischer
Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John F. Canny, and Ian Fischer. 2024. https://proceedings.mlr.press/v235/lee24c.html A human-inspired reading agent with gist memory of very long contexts . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 22928--22951. PMLR
2024
-
[18]
u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459--9474
2020
-
[19]
Jinze Li, Yang Zhang, Xin Yang, Jiayi Qu, Jinfeng Xu, Shuo Yang, Junhua Ding, and Edith Cheuk-Han Ngai. 2026. Ocr-memory: Optical context retrieval for long-horizon agent memory. arXiv preprint arXiv:2604.26622
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[20]
Junming Liu, Yifei Sun, Weihua Cheng, Haodong Lei, Yirong Chen, Licheng Wen, Xuemeng Yang, Daocheng Fu, Pinlong Cai, Nianchen Deng, Yi Yu, Shuyue Hu, Botian Shi, and Ding Wang. 2025. https://api.semanticscholar.org/CorpusID:283466545 Memverse: Multimodal memory for lifelong learning agents . ArXiv, abs/2512.03627
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[21]
Miao Lu, Weiwei Sun, Weihua Du, Zhan Ling, Xuesong Yao, Kang Liu, and Jiecao Chen. 2025. https://doi.org/10.48550/arXiv.2510.06727 Scaling llm multi-turn rl with end-to-end summarization-based context management . arXiv preprint arXiv:2510.06727
-
[22]
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing
2025
-
[23]
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. https://doi.org/10.18653/v1/2024.acl-long.747 Evaluating very long-term conversational memory of llm agents . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851--13870
-
[24]
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419
1995
- [25]
-
[26]
Charles Packer, Vivian Fang, Shishir\_G Patil, Kevin Lin, Sarah Wooders, and Joseph\_E Gonzalez. 2023. Memgpt: towards llms as operating systems
2023
-
[27]
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. https://doi.org/10.1145/3586183.3606763 Generative agents: Interactive simulacra of human behavior . In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pages 1--22
-
[28]
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982--3992
2019
-
[29]
Yaorui Shi, Shugui Liu, Yu Yang, Wenyu Mao, Yuxin Chen, Qi Gu, Hui Su, Xunliang Cai, Xiang Wang, and An Zhang. 2026. Memocr: Layout-aware visual memory for efficient long-horizon reasoning. arXiv preprint arXiv:2601.21468
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[30]
Weinan Sun, Madhu Advani, Nelson Spruston, Andrew Saxe, and James E Fitzgerald. 2023. Organizing memories for generalization in complementary learning systems. Nature neuroscience, 26(8):1438--1448
2023
- [31]
-
[32]
Mamatha Thota, Dewei Yi, and Georgios Leontidis. 2023. Lleda—lifelong self-supervised domain adaptation. Knowledge-Based Systems, 279:110959
2023
- [33]
-
[34]
Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. Deepseek-ocr: Contexts optical compression. arXiv preprint arXiv:2510.18234
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[35]
Biao Wu, Yanda Li, Zhiwei Zhang, Yunchao Wei, Meng Fang, and Ling Chen. 2024. https://doi.org/10.48550/arXiv.2411.02006 Foundations and recent trends in multimodal mobile agents: A survey . arXiv preprint arXiv:2411.02006
- [36]
-
[37]
Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2026. A-mem: Agentic memory for llm agents. Advances in Neural Information Processing Systems, 38:17577--17604
2026
- [38]
-
[39]
Zhongkai Yu, Shengwen Liang, Tianyun Ma, Yunke Cai, Ziyuan Nan, Di Huang, Xinkai Song, Yifan Hao, Jie Zhang, Tian Zhi, Yongwei Zhao, Zidong Du, Xing Hu, Qi Guo, and Tianshi Chen. 2024. https://doi.org/10.1109/MICRO61859.2024.00108 Cambricon-llm: A chiplet-based hybrid architecture for on-device inference of 70b llm . In 2024 57th IEEE/ACM International Sy...
-
[40]
Delong Zeng, Yuexiang Xie, Yaliang Li, and Ying Shen. 2025. Enhancing multimodal retrieval via complementary information extraction and alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22092--22105
2025
- [41]
-
[42]
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. https://doi.org/10.18653/v1/2024.acl-demos.38 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400--410
-
[43]
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 19724--19731
2024
-
[44]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[45]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.