pith. sign in

arxiv: 2511.09149 · v4 · submitted 2025-11-12 · 💻 cs.LG · cs.AI· cs.MA

Enabling Agents to Communicate Entirely in Latent Space

Zhuoyun Du , Runze Wang , Huiyu Bai , Zouying Cao , Xiaoyong Zhu , Yu Cheng , Bo Zheng , Wei Chen
show 1 more author
Haochao Ying
This is my paper

Pith reviewed 2026-05-17 23:38 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.MA
keywords latent space communicationmulti-agent systemsLLM agentshidden statescompressionchain-of-thoughtinter-agent communicationcollaborative agents
0
0 comments X

The pith

Agents can communicate directly via compressed LLM hidden states instead of natural language tokens.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Natural language forces agents to downsample rich internal representations into discrete tokens, limiting the nuance available for collaboration. The paper introduces Interlat to bypass this by letting agents transmit the continuous last hidden states of their LLMs as direct representations of thought. A learned compression process then enables further reasoning inside the latent space while cutting communication cost. Experiments show the method beats both fine-tuned chain-of-thought prompting and single-agent baselines, even when models differ, and yields up to 24 times faster inference. The work frames this as a feasibility demonstration that latent-space exchange can preserve and use information that language discards.

Core claim

Interlat lets LLM-based agents share compressed last hidden states as thought representations, allowing direct latent communication that avoids tokenization losses and supports more exploratory joint problem-solving with substantially lower inference time.

What carries the argument

Interlat, which transmits compressed versions of an LLM's last hidden states as continuous thought vectors for inter-agent exchange.

If this is right

  • Collaborative tasks can access richer internal information than language permits.
  • Heterogeneous models can still exchange useful signals without shared vocabulary or fine-tuning.
  • Inference accelerates by up to 24 times while performance stays competitive.
  • Agents display more exploratory behavior because they are not forced to produce language outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same latent channel might extend to non-text modalities where internal states are already continuous.
  • Coordination among larger numbers of agents could expose new emergent patterns once language bottlenecks are removed.
  • Security and alignment questions arise because latent messages are harder to audit than readable text.

Load-bearing premise

The last hidden states of an LLM contain transferable, task-useful information that other agents can interpret and act on without language as an intermediary.

What would settle it

An experiment in which agents using Interlat show no performance gain or a clear drop relative to language-based baselines on the same collaborative tasks.

Figures

Figures reproduced from arXiv: 2511.09149 by Bo Zheng, Haochao Ying, Huiyu Bai, Runze Wang, Wei Chen, Xiaoyong Zhu, Yu Cheng, Zhuoyun Du, Zouying Cao.

Figure 1
Figure 1. Figure 1: A comparison of Interlat with language-space communication. In language space, an agent transmits a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training the reasoning model, with frozen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of the separation loss: an [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: PCA visualization of latent communications, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task-averaged relative Cross-Entropy (∆CE) increase vs. communication rate. Learned compression consistently yields lower CE increase than training-free truncation, demonstrating that our method better pre￾serves predictive confidence and task-critical informa￾tion under reduced communication bandwidth. 5.2 Compression Analysis Compression Performance. Theoretically, due to their substantially higher expre… view at source ↗
Figure 6
Figure 6. Figure 6: Analysis of parallelism in latent communication across the first six steps. The [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics of the cross-entropy loss [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Result of compression on seen and unseen tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Parallelism in latent communication over the first six steps. Red indicates latents from the trained model, [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Extended analysis (32 steps). Same construction as [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: PCA visualization of latent communications grouped by ALFWorld task templates ( [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Joint PCA visualization of latent communications grouped by ALFWorld task templates. Each point [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
read the original abstract

While natural language is the de facto communication medium for LLM-based agents, it presents a fundamental constraint. The process of downsampling rich, internal latent states into discrete tokens inherently limits the depth and nuance of information that can be transmitted, thereby hindering collaborative problem-solving. Inspired by telepathy, which bypasses symbolic language in communication, we propose Interlat (Inter-agent Latent Space Communication), a paradigm that leverages the continuous last hidden states of an LLM as a representation of its thought for direct communication (termed latent communication). An additional learned compression process further compresses latent communication via latent space reasoning. Experiments demonstrate that Interlat outperforms both fine-tuned chain-of-thought (CoT) prompting and single-agent baselines, even across heterogeneous models, promoting more exploratory behavior and enabling genuine utilization of latent information. Further compression not only substantially accelerates inference by up to 24 times but also maintains competitive performance through an efficient information-preserving mechanism. We position this work as a feasibility study of entirely latent space inter-agent communication, and our results highlight its potential, offering valuable insights for future research. Our code is available at https://github.com/XiaoDu-flying/Interlat.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes Interlat, a paradigm for inter-agent communication among LLM-based agents that replaces natural language with direct transmission of (compressed) last hidden states as latent representations of thought. A learned compression network enables further latent-space reasoning; experiments are claimed to show outperformance versus fine-tuned CoT prompting and single-agent baselines, including across heterogeneous models, together with up to 24× inference speedup while preserving competitive task performance.

Significance. If the central experimental claims hold, the work constitutes a concrete feasibility demonstration that latent-space channels can support richer, faster inter-agent collaboration than token-level language. The public release of code is a clear strength that enables direct verification and extension.

major comments (2)
  1. [§4 and §5] §4 (Experimental Setup) and §5 (Results): the headline claim that Interlat outperforms baselines “even across heterogeneous models” is load-bearing for the feasibility argument, yet the manuscript provides no explicit description of the dimension-matching or alignment mechanism (projection, adapter, or shared space) used when hidden-state dimensionalities differ, nor an ablation that isolates performance when all agents use identical models. Without these controls it remains unclear whether observed gains arise from genuine latent transfer or from residual language leakage or compressor artifacts.
  2. [§5.2] §5.2 (Compression and Speedup): the reported 24× inference acceleration is presented as maintaining competitive performance via an “efficient information-preserving mechanism,” but no quantitative information-loss metric (e.g., mutual information, reconstruction error on task-relevant features, or comparison against language-token compression) is supplied to substantiate that the compressor retains task-useful content rather than simply discarding it.
minor comments (2)
  1. [Abstract and §3] The abstract and §3 refer to “last hidden states” without specifying which layer or token position is used; an equation or diagram would remove ambiguity.
  2. [Figures and Tables] Table captions and axis labels in the results figures should explicitly state the number of runs, random seeds, and statistical tests used to support the reported outperformance.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our work. We address each of the major comments below and outline the revisions we will make to improve the clarity and rigor of the manuscript.

read point-by-point responses

  1. Referee: [§4 and §5] §4 (Experimental Setup) and §5 (Results): the headline claim that Interlat outperforms baselines “even across heterogeneous models” is load-bearing for the feasibility argument, yet the manuscript provides no explicit description of the dimension-matching or alignment mechanism (projection, adapter, or shared space) used when hidden-state dimensionalities differ, nor an ablation that isolates performance when all agents use identical models. Without these controls it remains unclear whether observed gains arise from genuine latent transfer or from residual language leakage or compressor artifacts.

    Authors: We agree that an explicit description of the dimension alignment mechanism is necessary for reproducibility and to substantiate the cross-model claims. In the revised manuscript, we will add a detailed description in §4 of the linear projection layers used to map hidden states between models with differing dimensionalities. These projections are trained jointly with the compression network. We also acknowledge the value of an ablation study using identical models for all agents. While our current experiments focus on heterogeneous setups to demonstrate broader applicability, we will include results from same-model configurations in the revision to isolate the effects of latent transfer. Regarding potential language leakage, we note that communication occurs solely via the compressed latent states without any token generation or decoding during inter-agent exchange, which we will emphasize more clearly. revision: yes

  2. Referee: [§5.2] §5.2 (Compression and Speedup): the reported 24× inference acceleration is presented as maintaining competitive performance via an “efficient information-preserving mechanism,” but no quantitative information-loss metric (e.g., mutual information, reconstruction error on task-relevant features, or comparison against language-token compression) is supplied to substantiate that the compressor retains task-useful content rather than simply discarding it.

    Authors: We concur that providing quantitative measures of information preservation would better support the claims about the compressor's efficiency. Although task performance is preserved under compression, this indirect evidence may not fully address concerns about what information is retained. In the revised version, we will include additional metrics such as reconstruction error on the hidden states and a comparison of mutual information estimates between original and compressed representations. We will also add a baseline comparison against compressing language tokens to highlight the advantages of latent-space compression. revision: yes

Circularity Check

0 steps flagged

No significant circularity; claims rest on experimental comparisons

full rationale

The paper presents Interlat as a feasibility study for latent-space agent communication, with central claims supported by experiments showing outperformance over fine-tuned CoT prompting and single-agent baselines (including heterogeneous models). No equations, derivations, or self-citations appear that reduce performance metrics to fitted parameters or inputs by construction. The learned compression process and latent communication paradigm are introduced as proposals and validated empirically rather than through self-definitional steps or load-bearing self-citations. The work is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 1 invented entities

The central claim rests on the unstated premise that LLM hidden states are sufficiently rich and interpretable across agents for direct use, plus the existence of a trainable compression function that preserves performance-critical information. No explicit free parameters or invented physical entities are named in the abstract.

free parameters (1)
  • compression network parameters
    The learned compression process necessarily introduces trainable weights that are fitted to data to achieve the reported speedup and performance retention.
invented entities (1)
  • latent communication channel no independent evidence
    purpose: Direct exchange of hidden-state representations between agents without tokenization
    This is the core proposed mechanism whose effectiveness is asserted by the experiments but lacks independent verification outside the paper.

pith-pipeline@v0.9.0 · 5528 in / 1262 out tokens · 34657 ms · 2026-05-17T23:38:10.646978+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Latent Agents Lie: KV-Cache Integrity in Multi-Agent LLM Collaboration

    cs.MA 2026-06 conditional novelty 7.0

    KV-cache sharing boosts multi-agent QA performance but enables undetectable tampering; HMAC manifests binding agent, session, and payload reliably detect changes.

  2. See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

    cs.MA 2026-06 unverdicted novelty 7.0

    Heterogeneous agents achieve dense latent KV-cache communication via lightweight cross-model transformation and two-phase training, outperforming text at lower compute in context-aware settings and enabling context-un...

  3. Good Agentic Friends Do Not Just Give Verbal Advice: They Can Update Your Weights

    cs.CL 2026-05 unverdicted novelty 7.0

    TFlow enables multi-agent LLMs to collaborate via transient low-rank LoRA perturbations derived from sender activations, yielding up to 8.5 accuracy gains and 83% token reduction versus text-based baselines on Qwen3-4...

  4. Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier

    cs.AI 2026-06 unverdicted novelty 6.0

    Agent systems lose uncertainty at decision handoffs, causing downstream over-trust; the paper proposes latent uncertainty as a carrier to preserve pre-commitment fragility across interfaces.

  5. HyLaT: Efficient Multi-Agent Communication via Hybrid Latent-Text Protocol

    cs.CL 2026-05 unverdicted novelty 6.0

    HyLaT proposes a hybrid latent-text communication protocol with two-stage training that reduces overhead while maintaining performance in multi-agent LLM systems.

  6. LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems

    cs.AI 2026-05 unverdicted novelty 6.0

    LCGuard applies adversarial training to transform KV cache artifacts in multi-agent LLMs, reducing reconstructable sensitive information while preserving task performance.

  7. LACO: Adaptive Latent Communication for Collaborative Driving

    cs.AI 2026-05 unverdicted novelty 6.0

    LACO introduces Iterative Latent Deliberation, Cross-Horizon Saliency Attribution, and Structured Semantic Knowledge Distillation to enable low-latency latent communication in collaborative driving while preserving pe...

  8. Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus

    cs.LG 2026-04 conditional novelty 6.0

    LLM agent committees exhibit representational collapse with mean cosine similarity of 0.888, and diversity-aware consensus reaches 87% accuracy on GSM8K versus 84% for self-consistency at lower cost.

  9. A Token/KV-Cache Communication Media Selection and Resource Allocation Strategy for Multi-Agent Collaboration

    eess.SP 2026-05 unverdicted novelty 5.0

    A joint media selection and resource allocation algorithm (JMSRA) adaptively chooses token or KV-cache transmission and bandwidth allocation to reduce E2E latency compared to fixed baselines in wireless multi-agent systems.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 9 Pith papers · 1 internal anchor

  1. [1]

    Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

    Scaling up test-time compute with latent rea- soning: A recurrent depth approach.arXiv preprint arXiv:2502.05171. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Kr- ishna Menon, Sanjiv Kumar, and Vaishnavh Nagara- jan. 2023. Think before you speak: Training lan- guage models with pause tokens.arXiv preprint arXiv:2310.02226. Shibo Hao, Sainbayar Sukhba...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Let’s think dot by dot: Hidden computation in transformer language models.arXiv preprint arXiv:2404.15758, 2024

    Let’s think dot by dot: Hidden computa- tion in transformer language models.arXiv preprint arXiv:2404.15758. Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plummer, Zhaoran Wang, and Hongxia Yang. 2023. Let models speak ci- phers: Multiagent debate through embeddings.arXiv preprint arXiv:2310.06272. Samyam Rajbhandari,...

    work page arXiv 2023

  3. [3]

    arXiv preprint arXiv:2501.14082 , year=

    IEEE. Vignav Ramesh and Kenneth Li. 2025. Communicating activations between language model agents.arXiv preprint arXiv:2501.14082. Alireza Salemi, Mihir Parmar, Palash Goyal, Yiwen Song, Jinsung Yoon, Hamed Zamani, Hamid Palangi, and Tomas Pfister. 2025. Llm-based multi-agent blackboard system for information discovery in data science.arXiv preprint arXiv...

    work page arXiv 2025

  4. [4]

    In inference, the model receives a complete CoT plan before generating answers

    CoT (full).We use complete Chain-of-Thought (CoT) traces produced by a related instruction- tuned model (Qwen2.5-7B-Instruct, Qwen2.5-0.5B-Instruct, and LLaMA3.1-8B-Instruct) to perform full-parameter supervised fine-tuning. In inference, the model receives a complete CoT plan before generating answers. Rationale.This baseline serves as a strong upper bou...

    work page

  5. [5]

    Rationale.This baseline isolates the contribution of any communication signal

    No-CoT.The language model is trained to produce the final answer directly, without receiving any plan from other agents. Rationale.This baseline isolates the contribution of any communication signal. It tests whether inter-agent exchange, latent or linguistic, is necessary for solving multi-step tasks. In addition, we evaluate controlledvariantsof Interla...

    work page

  6. [6]

    15 Rationale.This variant keeps the interaction protocol unchanged while varying only the communica- tion channel

    Text.Instead of latent communication, we feed the corresponding CoT plan (in language space) to the actor. 15 Rationale.This variant keeps the interaction protocol unchanged while varying only the communica- tion channel. It enables a direct comparison between language-space and latent-space communication under matched training conditions, disentangling a...

    work page

  7. [7]

    No-Comm.We remove any communication from the actor’s input. This variant quantifies the intrinsic benefit of communication in our framework and verifies that performance improvements do not arise solely from modifications to the underlying model parameters

    work page

  8. [8]

    Rationale.This variant examines whether the actor is genuinely interpreting task-specific latent content

    CrossTask.We replace the current task’s latent communication with one sampled from a different task. Rationale.This variant examines whether the actor is genuinely interpreting task-specific latent content. A substantial degradation indicates reliance on meaningful information encoded in the latents, rather than superficial distributional shortcuts

    work page

  9. [9]

    Rationale.Noise-based perturbations interrogate the robustness and locality of latent-space semantics

    Noised.We add perturbations to the latent communication H: (a)CovNoise-0.5 ×/1.0×: covariance- shaped noise εt ∼ N(0, ˆΣ) with optional strength λ∈ {0.5,1.0} , where ˆΣ is the sample covariance of the originalH; (b)WhiteNoise: a control drawn fromN(0, I)with the same length. Rationale.Noise-based perturbations interrogate the robustness and locality of la...

    work page

  10. [10]

    These preserve first-second order moments while removing higher-order structure and temporal alignment

    CovGauss.We replace the entire H withi.i.d.samples Ht ∼ N(0, ˆΣ) (0µ) and report a robustness check with N(ˆµ,ˆΣ) (µ). These preserve first-second order moments while removing higher-order structure and temporal alignment. Rationale.This variant preserves the mean and covariance of the original latent distribution while discarding all higher-order statist...

    work page

  11. [11]

    Rationale.This preserves the mean/covariance exactly while disrupting higher-order structure

    RandomRot.We apply a structure-preserving but information-scrambling transform H ′ = ˆµ+ (H−ˆµ) ˆΣ−1/2 Q ˆΣ1/2, whereQis a Haar-random orthogonal matrix (Mezzadri, 2006). Rationale.This preserves the mean/covariance exactly while disrupting higher-order structure. Random rotation strictly preserves the first two moments of the latent distribution while sc...

    work page 2006

  12. [12]

    Ours Full

    Cross-Family.We evaluate Interlat under a cross-model-family setting, where the sender and actor belong to different pretrained model families. Specifically, latent communications are generated by a sender model from one family (In this work, we use Qwen2.5-7B-Instruct) and consumed by an actor model from another family (LLaMA3.1-8B-Base), without sharing...

    work page 2020

  13. [13]

    clean obj with recep

    work page

  14. [14]

    Nothing happened

    cool obj with recep where obj and recep correspond to objects and receptacles. After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. if the envrionment output "Nothing happened", that means the previous action is invalid and you should try more options. Your response should use the following fo...

    work page