RAG over Thinking Traces Can Improve Reasoning Tasks

Matei Zaharia; Negar Arabzadeh; Sewon Min; Wenjie Ma

arxiv: 2605.03344 · v2 · pith:BNMDKKCEnew · submitted 2026-05-05 · 💻 cs.IR · cs.AI· cs.CL

RAG over Thinking Traces Can Improve Reasoning Tasks

Negar Arabzadeh , Wenjie Ma , Sewon Min , Matei Zaharia This is my paper

Pith reviewed 2026-07-01 00:31 UTC · model grok-4.3

classification 💻 cs.IR cs.AIcs.CL

keywords retrieval-augmented generationthinking tracesreasoning tasksmath benchmarkscode generationRAG corpus

0 comments

The pith

Retrieving thinking traces as the RAG corpus improves reasoning performance on math and code benchmarks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper challenges the view that retrieval-augmented generation adds little value to reasoning tasks. It shows the issue is the corpus choice, and proposes using thinking traces—intermediate trajectories from problem-solving attempts—instead of web documents. A retrieve-then-generate pipeline with these traces raises accuracy across models and benchmarks such as AIME 2025-2026, LiveCodeBench, and GPQA-Diamond. Gains appear even when newer models retrieve traces generated by prior models. An offline method called T3 that turns traces into structured forms further increases the benefit.

Core claim

Thinking traces serve as an effective retrieval corpus for reasoning-intensive tasks. A retrieve-then-generate pipeline using these traces outperforms both non-RAG baselines and retrieval over standard web corpora on benchmarks including AIME 2025-2026, LiveCodeBench, and GPQA-Diamond. The approach produces relative gains such as +56.3% on AIME 2025-2026 for Gemini-2.5-Flash when traces come from Gemini-2-thinking. Transforming traces offline into structured, compact, or diagnostic representations unlocks additional improvements.

What carries the argument

Thinking traces, defined as intermediate thinking trajectories generated during problem-solving attempts, used as the retrieval corpus in a retrieve-then-generate pipeline, with an optional offline T3 transformation to create structured representations.

If this is right

Reasoning accuracy rises on math and code benchmarks when the retrieval corpus consists of thinking traces rather than documents.
The same pipeline outperforms retrieval over standard web corpora on the tested benchmarks.
Gains hold for multiple strong models, including cases where the target model is newer than the one that generated the traces.
Applying the T3 offline transformation to structure the traces produces larger improvements than raw traces alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Systems could maintain growing collections of thinking traces to support repeated retrieval for similar problem types.
The transfer of traces across models suggests that reasoning patterns captured in traces may be reusable even as base models advance.
Similar trace-based retrieval might apply to other structured reasoning domains if comparable intermediate trajectories can be collected.

Load-bearing premise

That thinking traces generated by one group of models form a sufficiently general corpus that transfers to improve performance of other models on new benchmarks without harmful distribution shift.

What would settle it

A controlled run of the retrieve-then-generate pipeline on AIME 2025-2026 or GPQA-Diamond that shows no accuracy increase or a decrease relative to the non-retrieval baseline would falsify the claim.

Figures

Figures reproduced from arXiv: 2605.03344 by Matei Zaharia, Negar Arabzadeh, Sewon Min, Wenjie Ma.

**Figure 1.** Figure 1: Overview of T 3 . Offline, a large reasoning model (e.g., Gemini-2-thinking) solves a set of problems and produces raw thinking traces. A smaller model (e.g., Gemini-2-FlashLite) then rewrites them into structured representations, forming a retrieval-friendly corpus. At inference time, a previously unseen query, which is not part of initial problem set, is retrieved against this corpus, and the retrieved … view at source ↗

**Figure 2.** Figure 2: A case study of T 3 - Reflect . Without retrieval, Gemini-2.5-Flash fails to reach a correct answer in 8 attempts. Retrieval over full traces is also insufficient and does not lead to a correct solution. In contrast, retrieval over T 3 provides targeted reasoning guidance that enables the model to solve 7 out of 8 attempts correctly. Retrieved examples and solutions are shortened for brevity. Our comments … view at source ↗

**Figure 3.** Figure 3: Average cost–accuracy trade-off over three reasoning benchmarks (AIME 2025- view at source ↗

**Figure 4.** Figure 4: , 5 and 6, respectively. Additionally, we provide our simple RAG inference prompt in view at source ↗

**Figure 5.** Figure 5: Prompt for Semantic transformation. T 3 - Reflection Instruction. Extract failure patterns and negative knowledge from the reasoning trace. Guidelines. • Focus on common mistakes and misleading reasoning paths. • Explain why these mistakes are tempting. • Highlight how to detect and avoid them. • Provide contrast with the correct approach. • Do not reproduce the full solution. Output format. Problem: ... C… view at source ↗

**Figure 6.** Figure 6: Prompt for Reflect transformation. RAG Inference Instruction. Solve the main problem by using useful hints and strategies from the retrieved examples. Example 1: ... Example 2: ... Example 3: ... Main problem: view at source ↗

**Figure 7.** Figure 7: Prompt for RAG inference using retrieved examples. view at source ↗

**Figure 8.** Figure 8: Corpus statistics for thinking traces. (Left) Domain distribution of the two view at source ↗

**Figure 9.** Figure 9: Impact of the number of retrieved documents ( view at source ↗

**Figure 10.** Figure 10: A single reasoning trace transformed by each strategy, with token counts. All view at source ↗

**Figure 11.** Figure 11: Example of generated and transformed thinking traces from a physics problem. view at source ↗

**Figure 12.** Figure 12: Example of generated and transformed thinking traces from a coding / optimiza view at source ↗

read the original abstract

Retrieval-augmented generation (RAG) has proven effective for knowledge-intensive tasks, but is widely believed to offer limited benefit for reasoning-intensive problems such as math and code generation. We challenge this assumption by showing that the limitation lies not in RAG itself, but in the choice of corpus. Instead of retrieving documents, we propose retrieving thinking traces, i.e., intermediate thinking trajectories generated during problem solving attempts. We show that thinking traces are already a strong retrieval source, and further introduce T3, an offline method that transforms them into structured, retrieval-friendly representations, to improve usability. Using these traces as a corpus, a simple retrieve-then-generate pipeline consistently improves reasoning performance across strong models and benchmarks such as AIME 2025--2026, LiveCodeBench, and GPQA-Diamond, outperforming both non-RAG baselines and retrieval over standard web corpora. For instance, on AIME 2025-2026, RAG with traces generated by Gemini-2-thinking achieves relative gains of +56.3%, +8.6%, and +7.6% for Gemini-2.5-Flash, GPT-OSS-120B, and GPT-5, respectively, even though these are more recent models. Overall, our results suggest that thinking traces are an effective retrieval corpus for reasoning tasks, and transforming them into structured, compact, or diagnostic representations unlocks even stronger gains. Code available at https://github.com/Narabzad/t3.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Retrieving thinking traces beats document RAG for reasoning benchmarks, but the gains rest on unexamined details of how the trace corpus was built and filtered.

read the letter

The main point is that this paper shows retrieval over model-generated thinking traces improves performance on math and code reasoning tasks more than either no retrieval or standard web document retrieval, and their T3 method helps by turning raw traces into structured forms.

The new element is the corpus choice itself. Prior RAG work for reasoning mostly pulls external facts or passages; here the authors treat intermediate trajectories from problem-solving attempts as the primary source. They report consistent lifts across AIME 2025-2026, LiveCodeBench, and GPQA-Diamond, with the largest relative gain (+56.3%) coming when Gemini-2.5-Flash retrieves from Gemini-2-thinking traces. The fact that it helps newer models and beats web corpora is the concrete result.

The experiments are direct comparisons, and releasing code is helpful for reproduction. That said, the trace collection protocol is the soft spot. The abstract says traces come from "problem solving attempts" and T3 makes them retrieval-friendly, but gives little on whether only successful traces were kept, whether sampling favored easier problems, or how coverage was ensured across difficulty levels. If the corpus ends up skewed toward traces that already solve problems similar to the test set, the retrieve-then-generate step can look stronger than it is for general reasoning reuse. The transfer to newer models is interesting, yet without more controls on distribution shift or selection, the headline numbers are harder to interpret.

This is for researchers working on inference-time augmentation or scaling reasoning data. Anyone trying to build a reusable trace store would get practical value from the T3 description and the benchmark numbers. It deserves peer review because the core claim is testable, the comparisons are clear, and the direction is distinct enough to warrant discussion even if the corpus details need tightening.

Referee Report

2 major / 2 minor

Summary. The paper claims that standard RAG is limited for reasoning tasks not because of the technique itself but due to the choice of corpus; instead, retrieving over 'thinking traces' (intermediate trajectories from problem-solving attempts) yields consistent gains on math/code benchmarks (AIME 2025-2026, LiveCodeBench, GPQA-Diamond). It introduces T3, an offline method to convert raw traces into structured, retrieval-friendly representations, and reports that a retrieve-then-generate pipeline using Gemini-2-thinking traces outperforms both non-RAG baselines and web-corpus RAG, with large relative gains even when applied to newer models (e.g., +56.3% on AIME for Gemini-2.5-Flash).

Significance. If the empirical results are robust, the work would establish thinking traces as a viable and often superior retrieval corpus for reasoning augmentation, opening a path to self-referential RAG that does not rely on external web data. The availability of code is a positive factor for reproducibility.

major comments (2)

[Experimental Setup / Trace Generation] The central claim that trace-based RAG transfers to newer models without distribution shift or selection artifacts (e.g., the +56.3% relative gain on AIME 2025-2026 for Gemini-2.5-Flash using Gemini-2-thinking traces) is load-bearing, yet the manuscript provides no protocol details on trace collection, success filtering, or problem sampling. This leaves open the possibility that reported gains arise from retrieving near-solutions rather than general reasoning support.
[Results / Baselines] The comparison to 'retrieval over standard web corpora' is presented as decisive, but without explicit controls for corpus size, indexing method, or retrieval hyperparameters matched across conditions, it is unclear whether the advantage is due to the nature of thinking traces or to differences in corpus quality and coverage.

minor comments (2)

[§3 (T3 Method)] The abstract states that T3 produces 'structured, retrieval-friendly representations' but does not define the precise transformation steps or output format; a short example in §3 would clarify usability.
[Results] Table or figure reporting absolute accuracies (not only relative gains) alongside the non-RAG and web-RAG baselines would make the magnitude of improvement easier to interpret.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and will make the corresponding revisions to strengthen the manuscript.

read point-by-point responses

Referee: [Experimental Setup / Trace Generation] The central claim that trace-based RAG transfers to newer models without distribution shift or selection artifacts (e.g., the +56.3% relative gain on AIME 2025-2026 for Gemini-2.5-Flash using Gemini-2-thinking traces) is load-bearing, yet the manuscript provides no protocol details on trace collection, success filtering, or problem sampling. This leaves open the possibility that reported gains arise from retrieving near-solutions rather than general reasoning support.

Authors: We agree that additional protocol details are required to support the central claim and rule out selection artifacts. In the revised manuscript we will add a dedicated subsection in the Experimental Setup that specifies the trace generation procedure (models, prompts, temperature), success filtering criteria (e.g., traces retained only when the final answer matches the ground truth or reaches a defined partial-progress threshold), and the exact problem-sampling strategy used for the corpus. We will also include a quantitative check (cosine similarity of retrieved traces to test problems) demonstrating that performance gains persist even after excluding near-duplicate traces. revision: yes
Referee: [Results / Baselines] The comparison to 'retrieval over standard web corpora' is presented as decisive, but without explicit controls for corpus size, indexing method, or retrieval hyperparameters matched across conditions, it is unclear whether the advantage is due to the nature of thinking traces or to differences in corpus quality and coverage.

Authors: We acknowledge that the current baseline comparison lacks explicit matching of corpus size, indexing pipeline, and hyperparameter tuning. In the revision we will report the exact token counts of both corpora, use the identical embedding model and vector index for both conditions, and separately optimize the retrieval hyper-parameters (k, similarity threshold) for each corpus before reporting results. These controls will be added to the Results section and the associated tables. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical performance comparisons

full rationale

The paper reports direct experimental results on retrieve-then-generate pipelines using thinking traces as corpus, with gains measured against non-RAG baselines and web corpora on held-out benchmarks. No equations, parameter fits, or derivations are present that could reduce reported improvements to quantities defined by the paper's own inputs. No self-citation chains, uniqueness theorems, or ansatzes are invoked to justify the central claim. The work is self-contained against external benchmarks via explicit model and dataset comparisons.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only; no explicit free parameters, axioms, or invented entities are stated. T3 is presented as a method rather than a new entity with independent evidence.

pith-pipeline@v0.9.1-grok · 5804 in / 1109 out tokens · 23742 ms · 2026-07-01T00:31:16.552889+00:00 · methodology

Review history (3 revisions) →

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Natural Language Query to Configuration for Retrieval Agents
cs.AI 2026-05 unverdicted novelty 6.0

BRANE maps queries to optimal retrieval pipeline configurations using LLM-derived features and per-configuration correctness predictors, improving the cost-quality Pareto frontier on three benchmarks.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages · cited by 1 Pith paper · 1 internal anchor

[1]

doi: 10.18653/v1/2024.findings-emnlp

Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp

work page doi:10.18653/v1/2024.findings-emnlp 2024
[2]

URLhttps://aclanthology.org/2024.findings-emnlp.496/. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish ...

work page arXiv 2024
[3]

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

URLhttps://arxiv.org/abs/2503.14476. A Prompts Here, we present the prompts used for transforming thinking traces and for RAG inference. The prompt used for transformation strategies introduced in Section 3.2, namely Structural Normalization (Struct), Semantic Distillation (Semantic), and Reflection (Reflect) are shown in Figure 4, 5 and 6, respectively. ...

work page internal anchor Pith review Pith/arXiv arXiv 2000

[1] [1]

doi: 10.18653/v1/2024.findings-emnlp

Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp

work page doi:10.18653/v1/2024.findings-emnlp 2024

[2] [2]

URLhttps://aclanthology.org/2024.findings-emnlp.496/. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish ...

work page arXiv 2024

[3] [3]

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

URLhttps://arxiv.org/abs/2503.14476. A Prompts Here, we present the prompts used for transforming thinking traces and for RAG inference. The prompt used for transformation strategies introduced in Section 3.2, namely Structural Normalization (Struct), Semantic Distillation (Semantic), and Reflection (Reflect) are shown in Figure 4, 5 and 6, respectively. ...

work page internal anchor Pith review Pith/arXiv arXiv 2000