Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Animesh Garg; Jeannette Bohg; Krishnan Srinivasan; Li Fei-Fei; Michelle A. Lee; Parth Shah; Silvio Savarese; Yuke Zhu

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1810.10191 v2 pith:Y7A353KS submitted 2018-10-24 cs.RO cs.AIcs.LG

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Michelle A. Lee , Yuke Zhu , Krishnan Srinivasan , Parth Shah , Silvio Savarese , Li Fei-Fei , Animesh Garg , Jeannette Bohg This is my paper

classification cs.RO cs.AIcs.LG

keywords learningcontact-richdifferentinputsmultimodalrealrobotsample

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.

discussion (0)

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

TactX: Learning Shared Tactile Representations Across Diverse Sensors
cs.RO 2026-06 unverdicted novelty 6.0

TactX learns a shared latent representation across three tactile sensor modalities via joint training on paired contacts, enabling zero-shot policy transfer and higher success on pick-and-place, insertion, wiping, and...
Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation
cs.RO 2026-06 unverdicted novelty 6.0

MiTaS fuses multi-resolution tactile data from GelSight and Evetac sensors with vision using modality-specific stems and transformer fusion to condition flow-matching policies, reporting 80% average success on five co...
Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
cs.RO 2025-11 unverdicted novelty 6.0

MSDP pretrains a transformer encoder via masked multisensory reconstruction and feeds the embeddings into an asymmetric actor-critic RL setup, yielding faster learning and high real-robot success rates with only 6,000...
Grasping Using Tactile Sensing and Deep Calibration
cs.RO 2019-07 unverdicted novelty 3.0

A tactile feedback approach for robot grasping evaluated on a real robot, using deep learning to eliminate bias in force-torque sensor data.