pith. sign in

arxiv: 2606.09331 · v1 · pith:GXU3XXDEnew · submitted 2026-06-08 · 💻 cs.MM · cs.AI· cs.LG

Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

Shiyu Li , Zhiyuan Hu , Yifan Wang , Peiming Li , Zheng Wei , Yang Tang This is my paper

Pith reviewed 2026-06-27 14:14 UTC · model grok-4.3

classification 💻 cs.MM cs.AIcs.LG
keywords omni-modal retrievaltask vector fusionprojector driftmodality specialistsembedding modelsmultimodal fusionretrieval systems
0
0 comments X

The pith

A decouple-fuse-recover process merges modality specialists into one backbone for omni-modal retrieval.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that training independent modality specialists and fusing their task vectors into a shared dense backbone can compose retrieval capabilities across text, image, video, and document inputs. This approach matters to a sympathetic reader because separate models for each modality have long been necessary due to mismatched data distributions and optimization needs, and a single embedding space would simplify omni-modal systems. The fusion succeeds for most modalities but creates projector drift when an external audio encoder attaches via a projector, leaving the projector miscalibrated to the fused backbone. The authors correct this with full-parameter projector fine-tuning while the backbone stays frozen, plus balanced multi-modal rehearsal, to restore audio retrieval without harming other modalities.

Core claim

Decoupled Specialist Fusion trains modality specialists independently then fuses their task vectors into one backbone, composing visual, video, and document retrieval capabilities; however, attaching audio through an external encoder and projector produces Projector Drift that regresses audio performance even when audio modules are copied unchanged. Projector Recovery repairs the drift by full-parameter fine-tuning of the projector with the backbone frozen, followed by balanced multi-modal rehearsal, so the final model supports all retrieval pathways in one backbone.

What carries the argument

Decoupled Specialist Fusion (independent specialist training followed by task-vector fusion into a shared backbone) together with Projector Recovery (full-parameter projector fine-tuning under frozen backbone plus balanced rehearsal) to correct calibration drift.

If this is right

  • Fusion successfully composes visual, video, and document retrieval capabilities in the shared backbone.
  • Projector-attached modalities experience regression from projector drift after fusion.
  • Full-parameter projector fine-tuning with frozen backbone followed by balanced rehearsal restores audio capability.
  • The recovered model supports multiple retrieval pathways within a single backbone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same fusion-plus-recovery pattern may extend to other projector-based modalities such as 3D or sensor inputs.
  • Maintaining separate specialist training before fusion could prove more stable than end-to-end joint training for additional modalities.
  • Projector drift is likely to appear in any architecture that attaches external encoders via projectors, suggesting recovery steps could become standard.

Load-bearing premise

Task vectors from modality specialists can be fused into one backbone without destroying performance except for projector-attached modalities, and projector recovery plus rehearsal can restore the lost modality without degrading the others.

What would settle it

An experiment that measures audio retrieval scores immediately after fusion and again after Projector Recovery to test whether the regression reverses without losses in other modalities.

Figures

Figures reproduced from arXiv: 2606.09331 by Peiming Li, Shiyu Li, Yang Tang, Yifan Wang, Zheng Wei, Zhiyuan Hu.

Figure 1
Figure 1. Figure 1: Projector Drift across Transformer depth. The semicircles show audio-token directions at layers 0, 17, and 35, measured relative to the audio specialist (blue). Direct fusion (red) drifts away in deeper layers, while Projector Recovery (green) stays closer to the specialist and restores AudioCaps performance. space for text, image, video, visual-document, and audio retrieval (Zhan et al., 2024; Lin et al.,… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Conan-embedding-v3. Stage 1 trains modality specialists from a shared initialization θ0 using modality-specific LoRA, where the audio specialist additionally grafts an audio encoder and projector. These specialists are then combined in Stage 2 by fusing their shared-backbone task vectors while copying audio modules directly, which induces Projector Drift. Finally, Stage 3 applies Projector Reco… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation on fusion weights before recov￾ery. Increasing the audio task-vector coefficient αA improves audio capability but degrades visual retrieval performance, illustrating the inherent conflict. 4.4 Qualitative Experiments 4.4.1 Task-Vector Geometry To understand why direct fusion disproportionately affects the grafted audio pathway, we analyze the model-space task vectors of the specialists in Fig￾ure … view at source ↗
Figure 4
Figure 4. Figure 4: Geometry of the four modality task vectors. (a) Pairwise cosine similarity indicates audio is nearly orthogonal to visual updates (cos ≤ 0.001). (b) Audio has the largest global update norm (≈ 42.3). (c) Audio updates remain large across the backbone and peak in deeper layers. Specialist (R@1 = 92%) Direct merge (R@1 = 56%) Recovered (R@1 = 91%) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Output-space view of Projector Drift on diagnostic audio–text retrieval pairs. The audio specialist (left) aligns paired embeddings (R@1 = 92%). Direct fusion (middle) separates the audio and text manifolds (R@1 = 56%). Projector Recovery (right) reduces this separation and restores retrieval performance (R@1 = 91%). sual updates provide little direct directional support for audio during fusion. Second, Fi… view at source ↗
read the original abstract

Omni-modal retrieval promises a single embedding space for text, image, video, document, and audio inputs, but building such a unified retriever is difficult since these modalities differ in data distribution, architecture, and optimization dynamics. In this work, we present Conan-embedding-v3, a decouple--fuse--recover framework for omni-modal retrieval. Conan-embedding-v3 first trains modality specialists independently and fuses their task vectors into a single dense backbone, a strategy we call Decoupled Specialist Fusion. We show that this fusion composes visual, video, and document retrieval capabilities, but also exposes a failure mode for projector-based modalities: when audio is attached through an external encoder and projector, fusing the backbone leaves the projector calibrated to the audio-specialist backbone, causing a large audio retrieval regression despite copying all audio-specific modules unchanged. We call this failure Projector Drift. To repair it, Conan-embedding-v3 applies Projector Recovery (i.e., full-parameter fine-tuning of the projector while keeping the backbone frozen) followed by balanced multi-modal rehearsal. The resulting model supports these retrieval pathways in one backbone, achieving 74.9 scores on MMEB while obtaining 55.61 on the 30-task MAEB audio suite.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper presents Conan-embedding-v3, a decouple-fuse-recover framework for omni-modal retrieval. Modality specialists are trained independently; their task vectors are fused into one backbone via Decoupled Specialist Fusion. This composes visual/video/document capabilities but induces Projector Drift for audio (attached via external encoder and projector). The drift is repaired by Projector Recovery (full-parameter projector fine-tuning with backbone frozen) plus balanced multi-modal rehearsal. The final model reports 74.9 on MMEB and 55.61 on the 30-task MAEB audio suite.

Significance. If the recovery step demonstrably restores audio performance without side-effects on the already-fused modalities, the work would offer a concrete, modular route to omni-modal embeddings that avoids full joint training. The explicit identification of Projector Drift as a failure mode and the proposed fix constitute a useful diagnostic contribution for projector-based modality attachment.

major comments (1)
  1. [Abstract] Abstract: The central claim that Decoupled Specialist Fusion followed by Projector Recovery + balanced rehearsal 'restores audio capability without degrading' the fused visual/video/document modalities is load-bearing for the paper's contribution, yet only the final 74.9 MMEB / 55.61 MAEB numbers are supplied. No intermediate MMEB scores (post-fusion, post-recovery, post-rehearsal) or comparisons against the individual modality specialists are reported, making it impossible to verify that the recovery step satisfies the 'without destroying performance' condition.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful reading and for highlighting the need for clearer verification of the central claim. We address the major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that Decoupled Specialist Fusion followed by Projector Recovery + balanced rehearsal 'restores audio capability without degrading' the fused visual/video/document modalities is load-bearing for the paper's contribution, yet only the final 74.9 MMEB / 55.61 MAEB numbers are supplied. No intermediate MMEB scores (post-fusion, post-recovery, post-rehearsal) or comparisons against the individual modality specialists are reported, making it impossible to verify that the recovery step satisfies the 'without destroying performance' condition.

    Authors: We agree that the absence of intermediate MMEB scores makes it difficult for readers to directly verify that Projector Recovery restores audio performance without side-effects on the already-fused modalities. In the revised version we will add an explicit ablation table (new Table X) that reports MMEB after (i) Decoupled Specialist Fusion, (ii) Projector Recovery with backbone frozen, and (iii) balanced multi-modal rehearsal. The same table will also include the MMEB scores of the individual modality specialists prior to fusion for direct comparison. These additions will be referenced from both the abstract and the experimental section so that the load-bearing claim can be evaluated quantitatively. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical framework with no derivations or self-referential claims

full rationale

The manuscript presents a sequence of training steps (independent specialist training, task-vector fusion, projector recovery, and rehearsal) and reports final benchmark numbers (74.9 MMEB, 55.61 MAEB). No equations, uniqueness theorems, or fitted-parameter predictions appear; the central claims are empirical outcomes of the described procedure rather than quantities defined from or forced by the inputs themselves. Absence of any load-bearing self-citation chain or ansatz smuggling keeps the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no equations, no fitted constants, and no explicit axioms; all technical content is at the level of high-level method description.

pith-pipeline@v0.9.1-grok · 5771 in / 1022 out tokens · 19890 ms · 2026-06-27T14:14:37.207567+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    2025 , eprint=

    VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents , author=. 2025 , eprint=

    2025

  2. [2]

    2025 , eprint=

    Qwen2.5-Omni Technical Report , author=. 2025 , eprint=

    2025

  3. [3]

    2026 , eprint=

    MAEB: Massive Audio Embedding Benchmark , author=. 2026 , eprint=

    2026

  4. [4]

    Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , month=

    Muennighoff, Niklas and Tazi, Nouamane and Magne, Loic and Reimers, Nils. MTEB : Massive Text Embedding Benchmark. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023. doi:10.18653/v1/2023.eacl-main.148

    work page doi:10.18653/v1/2023.eacl-main.148 2023

  5. [5]

    Audiocaps: Generating captions for audios in the wild,

    Kim, Chris Dongjoo and Kim, Byeongchang and Lee, Hyunmin and Kim, Gunhee. A udio C aps: Generating Captions for Audios in The Wild. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019. doi:10.18653/v1/N19-1011

    work page doi:10.18653/v1/n19-1011 2019

  6. [6]

    and Ellis, Daniel P

    Gemmeke, Jort F. and Ellis, Daniel P. W. and Freedman, Dylan and Jansen, Aren and Lawrence, Wade and Moore, R. Channing and Plakal, Manoj and Ritter, Marvin , booktitle=. Audio Set: An ontology and human-labeled dataset for audio events , year=

  7. [7]

    The Benefit of Temporally-Strong Labels in Audio Event Classification , year=

    Hershey, Shawn and Ellis, Daniel P W and Fonseca, Eduardo and Jansen, Aren and Liu, Caroline and Channing Moore, R and Plakal, Manoj , booktitle=. The Benefit of Temporally-Strong Labels in Audio Event Classification , year=

  8. [8]

    Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle =

    Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle =. 2022 , url =

    2022

  9. [9]

    2024 , eprint=

    OLIVE: Object Level In-Context Visual Embeddings , author=. 2024 , eprint=

    2024

  10. [10]

    2025 , howpublished =

    2025

  11. [11]

    2026 , eprint=

    Think Then Embed: Generative Context Improves Multimodal Embedding , author=. 2026 , eprint=

    2026

  12. [12]

    2025 , eprint=

    RzenEmbed: Towards Comprehensive Multimodal Retrieval , author=. 2025 , eprint=

    2025

  13. [13]

    2026 , eprint=

    e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings , author=. 2026 , eprint=

    2026

  14. [14]

    2026 , eprint=

    BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs , author=. 2026 , eprint=

    2026

  15. [15]

    2024 , eprint=

    Qwen2-Audio Technical Report , author=. 2024 , eprint=

    2024

  16. [16]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  17. [17]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Learning Transferable Visual Models From Natural Language Supervision , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    2021

  18. [18]

    2024 , eprint=

    E5-V: Universal Embeddings with Multimodal Large Language Models , author=. 2024 , eprint=

    2024

  19. [19]

    The Thirteenth International Conference on Learning Representations (ICLR) , year=

    VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks , author=. The Thirteenth International Conference on Learning Representations (ICLR) , year=

  20. [20]

    2026 , eprint=

    Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking , author=. 2026 , eprint=

    2026

  21. [21]

    CLAP Learning Audio Concepts from Natural Language Supervision , year=

    Elizalde, Benjamin and Deshmukh, Soham and Ismail, Mahmoud Al and Wang, Huaming , booktitle=. CLAP Learning Audio Concepts from Natural Language Supervision , year=

  22. [22]

    Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation , year=

    Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo , booktitle=. Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation , year=

  23. [23]

    2025 , eprint=

    Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality , author=. 2025 , eprint=

    2025

  24. [24]

    2024 , eprint=

    ColPali: Efficient Document Retrieval with Vision Language Models , author=. 2024 , eprint=

    2024

  25. [25]

    2025 , eprint=

    Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video , author=. 2025 , eprint=

    2025

  26. [26]

    Scaling Language-centric Omnimodal Representation Learning , url =

    Xiao, Chenghao and Chan, Hou Pong (Ken) and Zhang, Hao and Xu, Weiwen and Aljunied, Mahani and Rong, Yu , booktitle =. Scaling Language-centric Omnimodal Representation Learning , url =

  27. [27]

    2026 , eprint=

    jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers , author=. 2026 , eprint=

    2026

  28. [28]

    Editing Models with Task Arithmetic

    Editing Models with Task Arithmetic , author=. International Conference on Learning Representations (ICLR) , year=. 2212.04089 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    TIES-Merging: Resolving Interference When Merging Models , url =

    Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit , booktitle =. TIES-Merging: Resolving Interference When Merging Models , url =

  30. [30]

    2020 , eprint=

    Gradient Surgery for Multi-Task Learning , author=. 2020 , eprint=

    2020

  31. [31]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    2024

  32. [32]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  33. [33]

    Merging Models with Fisher-Weighted Averaging , url =

    Matena, Michael S and Raffel, Colin , booktitle =. Merging Models with Fisher-Weighted Averaging , url =

  34. [34]

    Gradient Surgery for Multi-Task Learning , url =

    Yu, Tianhe and Kumar, Saurabh and Gupta, Abhishek and Levine, Sergey and Hausman, Karol and Finn, Chelsea , booktitle =. Gradient Surgery for Multi-Task Learning , url =

  35. [35]

    Conflict-Averse Gradient Descent for Multi-task learning , url =

    Liu, Bo and Liu, Xingchao and Jin, Xiaojie and Stone, Peter and Liu, Qiang , booktitle =. Conflict-Averse Gradient Descent for Multi-task learning , url =

  36. [36]

    2025 , eprint=

    Qwen3-Omni Technical Report , author=. 2025 , eprint=

    2025

  37. [37]

    2025 , eprint=

    Scaling Language-Centric Omnimodal Representation Learning , author=. 2025 , eprint=

    2025

  38. [38]

    2014 , eprint=

    Microsoft COCO: Common Objects in Context , author=. 2014 , eprint=

    2014

  39. [39]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , year=

    Visual News: Benchmark and Challenges in News Image Captioning , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , year=

    2021

  40. [40]

    2024 , eprint=

    Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward , author=. 2024 , eprint=

    2024

  41. [41]

    2024 , eprint=

    VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents , author=. 2024 , eprint=

    2024

  42. [42]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Girdhar, Rohit and El-Nouby, Alaaeldin and Liu, Zhuang and Singh, Mannat and Alwala, Kalyan Vasudev and Joulin, Armand and Misra, Ishan , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2023 , pages =

    2023

  43. [43]

    OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces , url =

    wang, zehan and Zhang, Ziang and Hong, Minjie and Zhang, Hang and Liu, Luping and Huang, Rongjie and Cheng, Xize and Ji, Shengpeng and Jin, Tao and Zhao, Hengshuang and Zhao, Zhou , booktitle =. OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces , url =

  44. [44]

    LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment , url =

    Zhu, Bin and Lin, Bin and Ning, Munan and YAN, YANG and Cui, Jiaxi and HongFa, WANG and Pang, Yatian and Jiang, Wenhao and Zhang, Junwu and Li, Zongwei and Zhang, Cai and Li, Zhifeng and Liu, Wei and Li, Yuan , booktitle =. LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment , url =

  45. [45]

    AnyGPT: Unified multimodal LLM with discrete sequence modeling

    Zhan, Jun and Dai, Junqi and Ye, Jiasheng and Zhou, Yunhua and Zhang, Dong and Liu, Zhigeng and Zhang, Xin and Yuan, Ruibin and Zhang, Ge and Li, Linyang and Yan, Hang and Fu, Jie and Gui, Tao and Sun, Tianxiang and Jiang, Yu-Gang and Qiu, Xipeng. A ny GPT : Unified Multimodal LLM with Discrete Sequence Modeling. Proceedings of the 62nd Annual Meeting of ...

    work page doi:10.18653/v1/2024.acl-long.521 2024

  46. [46]

    2025 , eprint=

    SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model , author=. 2025 , eprint=

    2025

  47. [47]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Which Tasks Should Be Learned Together in Multi-task Learning? , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    2020

  48. [48]

    Multi-Task Learning as Multi-Objective Optimization , url =

    Sener, Ozan and Koltun, Vladlen , booktitle =. Multi-Task Learning as Multi-Objective Optimization , url =

  49. [49]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =

    Chen, Hao Mark and Hu, Shell Xu and Luk, Wayne and Hospedales, Timothy and Fan, Hongxiang , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =. 2025 , pages =

    2025

  50. [50]

    2025 , eprint=

    No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces , author=. 2025 , eprint=

    2025

  51. [51]

    Visual Instruction Tuning , url =

    Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae , booktitle =. Visual Instruction Tuning , url =

  52. [52]

    SALMONN: Towards Generic Hearing Abilities for Large Language Models , url =

    Tang, Changli and Yu, Wenyi and Sun, Guangzhi and Chen, Xianzhao and Tan, Tian and Li, Wei and Lu, Lu and MA, Zejun and Zhang, Chao , booktitle =. SALMONN: Towards Generic Hearing Abilities for Large Language Models , url =