pith. sign in

arxiv: 2403.12945 · v2 · submitted 2024-03-19 · 💻 cs.RO

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky , Karl Pertsch , Suraj Nair , Ashwin Balakrishna , Sudeep Dasari , Siddharth Karamcheti , Soroush Nasiriany , Mohan Kumar Srirama
show 93 more authors
Lawrence Yunliang Chen Kirsty Ellis Peter David Fagan Joey Hejna Masha Itkina Marion Lepert Yecheng Jason Ma Patrick Tree Miller Jimmy Wu Suneel Belkhale Shivin Dass Huy Ha Arhan Jain Abraham Lee Youngwoon Lee Marius Memmel Sungjae Park Ilija Radosavovic Kaiyuan Wang Albert Zhan Kevin Black Cheng Chi Kyle Beltran Hatch Shan Lin Jingpei Lu Jean Mercat Abdul Rehman Pannag R Sanketi Archit Sharma Cody Simpson Quan Vuong Homer Rich Walke Blake Wulfe Ted Xiao Jonathan Heewon Yang Arefeh Yavary Tony Z. Zhao Christopher Agia Rohan Baijal Mateo Guaman Castro Daphne Chen Qiuyu Chen Trinity Chung Jaimyn Drake Ethan Paul Foster Jensen Gao Vitor Guizilini David Antonio Herrera Minho Heo Kyle Hsu Jiaheng Hu Muhammad Zubair Irshad Donovon Jackson Charlotte Le Yunshuang Li Kevin Lin Roy Lin Zehan Ma Abhiram Maddukuri Suvir Mirchandani Daniel Morton Tony Nguyen Abigail O'Neill Rosario Scalise Derick Seale Victor Son Stephen Tian Emi Tran Andrew E. Wang Yilin Wu Annie Xie Jingyun Yang Patrick Yin Yunchu Zhang Osbert Bastani Glen Berseth Jeannette Bohg Ken Goldberg Abhinav Gupta Abhishek Gupta Dinesh Jayaraman Joseph J Lim Jitendra Malik Roberto Mart\'in-Mart\'in Subramanian Ramamoorthy Dorsa Sadigh Shuran Song Jiajun Wu Michael C. Yip Yuke Zhu Thomas Kollar Sergey Levine Chelsea Finn
This is my paper

Pith reviewed 2026-05-11 05:47 UTC · model grok-4.3

classification 💻 cs.RO
keywords robot manipulationimitation learningdatasetgeneralizationdemonstration learningin-the-wild roboticspolicy trainingreal-world data
0
0 comments X

The pith

Training on the DROID dataset of 350 hours of real-world robot manipulation data produces policies with higher performance and stronger generalization.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a very large robot manipulation dataset collected across many different real-world scenes and tasks can be used to train policies that succeed more often and transfer better to new situations than policies trained on smaller, more controlled datasets. Current robot learning is held back by the difficulty and cost of gathering diverse data outside the lab, so most policies remain narrow. By releasing 76,000 trajectories gathered by non-experts over 564 scenes and 84 tasks, the work tests whether scale and variety in ordinary environments can overcome those limits. The reported result is that models trained with this data outperform those trained without it on both seen and unseen tasks.

Core claim

Policies trained with the DROID dataset achieve higher success rates and improved generalization compared with policies trained only on smaller laboratory datasets. The dataset supplies 76k demonstration trajectories totaling 350 hours, gathered across 564 distinct scenes and 84 tasks by 50 non-expert collectors in homes and offices in three continents over twelve months. The authors open-source the trajectories, the policy-training code, and instructions for reproducing the robot hardware.

What carries the argument

The DROID dataset itself: a collection of 76k real-world demonstration trajectories spanning 564 scenes and 84 tasks, gathered by non-experts in varied everyday environments.

If this is right

  • Policies trained on DROID data reach higher success rates on the same tasks used during collection.
  • The same policies transfer more successfully to rooms and object arrangements never seen in training.
  • Robot learning pipelines can now incorporate data gathered by non-experts without specialized facilities.
  • Open release of the full trajectories and training code allows other groups to reproduce and extend the results directly.
  • Future dataset efforts can prioritize geographic and scene diversity over perfect control of every variable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the generalization gains hold, robot training may shift from controlled lab collection toward distributed, low-cost gathering in homes and workplaces.
  • The result implies that scene diversity can substitute for some amount of expert supervision and hardware precision.
  • A natural next test would be whether the same dataset also improves performance when used for imitation learning from video only, without robot proprioception.
  • The collection protocol could be applied to other robot platforms to test whether the benefits are hardware-specific.

Load-bearing premise

The data collected by non-experts in ordinary, uncontrolled settings is high enough in quality and coverage to produce measurable gains in policy performance and generalization beyond what smaller lab datasets already provide.

What would settle it

Train identical policy architectures on DROID versus on a comparable number of trajectories from existing lab datasets, then measure success rates on a held-out set of tasks performed in new rooms; if the DROID-trained policies show no advantage or perform worse, the central claim is falsified.

read the original abstract

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper introduces DROID, a large-scale in-the-wild robot manipulation dataset with 76k demonstration trajectories (350 hours) collected across 564 scenes and 84 tasks by 50 non-expert collectors in North America, Asia, and Europe. It claims that policies trained on DROID achieve higher performance and improved generalization ability relative to smaller lab-collected datasets, and releases the full dataset, policy learning code, and hardware reproduction guide.

Significance. If the empirical results hold, DROID would be a valuable resource for robot learning by substantially increasing the scale and diversity of available real-world manipulation data, addressing a core limitation of current policies trained in limited environments. The open-sourcing of the dataset, code, and detailed hardware guide is a clear strength that supports reproducibility and community use.

major comments (1)
  1. [Section 5] Section 5 (Experiments): The manuscript reports that training with DROID yields higher performance and better generalization, but does not present volume-controlled ablations (e.g., comparing full DROID against an equal-sized subset collected in a single lab environment). Without such controls it is difficult to isolate whether gains arise from the in-the-wild diversity, geographic spread, and non-expert collection rather than from increased total interaction time, which directly affects the central attribution in the abstract and title.
minor comments (1)
  1. [Abstract] Abstract: The claim of improved performance and generalization is stated without any quantitative metrics, baselines, or effect sizes; including one or two key numbers would make the summary more informative.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed review and constructive feedback on our manuscript. The primary concern raised is the lack of volume-controlled ablations in Section 5 to better isolate the contributions of in-the-wild diversity versus raw data volume. We address this point directly below and commit to revisions that strengthen the empirical claims.

read point-by-point responses

  1. Referee: [Section 5] Section 5 (Experiments): The manuscript reports that training with DROID yields higher performance and better generalization, but does not present volume-controlled ablations (e.g., comparing full DROID against an equal-sized subset collected in a single lab environment). Without such controls it is difficult to isolate whether gains arise from the in-the-wild diversity, geographic spread, and non-expert collection rather than from increased total interaction time, which directly affects the central attribution in the abstract and title.

    Authors: We agree that volume-controlled ablations would provide stronger evidence for attributing performance gains specifically to the in-the-wild aspects of DROID rather than scale alone. Our current experiments in Section 5 compare policies trained on DROID against those trained on smaller existing lab-collected datasets (e.g., Bridge, RT-1), which implicitly vary both volume and diversity. However, we did not include explicit controls matching DROID's full volume against an equivalently sized single-lab subset, as our data collection prioritized geographic and scene diversity across 564 environments rather than concentrating volume in one setting. In the revised manuscript, we will add volume-controlled experiments by (1) subsampling DROID to match the sizes of the baseline datasets used in our comparisons and (2) constructing a volume-matched subset drawn from the largest individual scenes or geographic clusters in DROID where sufficient data exists. We will also include a discussion of the practical challenges in collecting large-scale single-lab data at DROID's volume. These additions will clarify the relative contributions of scale and diversity. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical dataset release with direct experimental validation

full rationale

The paper's core contribution is the collection and public release of the DROID dataset (76k trajectories across 564 scenes) followed by empirical policy training results showing performance gains. No derivation chain, equations, fitted parameters, or predictions exist that reduce to inputs by construction. Claims rest on direct experimentation and comparisons to baselines rather than self-definitional loops, self-citation load-bearing premises, or renamed known results. The work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper rests on the domain assumption that in-the-wild data collection yields higher-quality training signals for generalization than lab data; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Diverse real-world robot interaction data improves policy generalization over limited lab data
    Invoked when claiming higher performance and generalization from training on DROID.

pith-pipeline@v0.9.0 · 5912 in / 1221 out tokens · 54192 ms · 2026-05-11T05:47:41.116365+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

    cs.RO 2026-06 unverdicted novelty 8.0

    HABIT is a large-scale robot demonstration dataset for human-present environments that elicits spatiotemporal synchronization, yielding, and gesture grounding behaviors absent from robot-only training data.

  2. SpatialBench: Is Your Spatial Foundation Model an All-Round Player?

    cs.CV 2026-05 unverdicted novelty 8.0

    SpatialBench evaluates 41 spatial foundation models across 6 paradigms and 5 task suites, finds they are not all-round players, and introduces the DA-Next-5M dataset plus DA-Next baseline model.

  3. Membership Inference Attacks on Vision-Language-Action Models

    cs.CR 2026-05 unverdicted novelty 8.0

    Vision-language-action models are highly vulnerable to membership inference attacks, including practical black-box versions that exploit generated actions and motion trajectories.

  4. Adapting Generalist Robot Policies with Semantic Reinforcement Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    SARL optimizes language prompt inputs to generalist vision-language-action policies through online RL to solve complex long-horizon tasks by composing existing skills.

  5. RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

    cs.RO 2026-06 unverdicted novelty 7.0

    RoboGaze presents a structured multi-agent VLM pipeline and robotics-specific error taxonomy that improves video evaluation metrics by up to 43 F1 points over zero-shot baselines on a 382-clip dataset.

  6. Geometric Entropy: When Trajectory Diversity Helps and Hurts in Imitation Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    Geometric diversity of demonstration trajectories exhibits an inverted-U effect on imitation learning success, with the peak shifting lower as mastery increases via more data, easier tasks, or stronger priors.

  7. HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

    cs.CV 2026-06 unverdicted novelty 7.0

    Processed egocentric human video outperforms teleoperated real-robot trajectories as pretraining data for embodied foundation models, delivering 24% lower validation loss and 52.5-90% higher task success rates under m...

  8. EgoCS-400K: An Egocentric Gameplay Dataset for World Models

    cs.CV 2026-06 unverdicted novelty 7.0

    EgoCS-400K is a new 400K-video egocentric CS dataset with action-state-event alignment from public match demos for world model training.

  9. FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation

    cs.RO 2026-06 unverdicted novelty 7.0

    FTP-1 is the first foundation tactile policy pretrained on ~3000 hours of data from 26 sources across 21 sensors that improves performance on seen setups by 17.2% and transfers to unseen sensors with 31% success rate gain.

  10. M*: A Modular, Extensible, Serving System for Multimodal Models

    cs.LG 2026-06 unverdicted novelty 7.0

    M* introduces the Walk Graph abstraction to serve arbitrary compositions of multimodal model components and reports latency and throughput gains over vLLM-Omni and other baselines on text-to-image, text-to-speech, and...

  11. EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations

    cs.RO 2026-06 unverdicted novelty 7.0

    EgoEngine transforms egocentric human videos into high-fidelity robot data enabling zero-shot visuomotor dexterous policy learning without real-robot demonstrations.

  12. UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data

    cs.RO 2026-06 unverdicted novelty 7.0

    UMI-Bench 1.0 is presented as the first open benchmark dedicated to reproducible real-world evaluation of Universal Manipulation Interface policies.

  13. Targeting World Models to Compromise Robot Learning Pipelines

    cs.RO 2026-06 unverdicted novelty 7.0

    World models introduce a stealthy poisoning vector into robot learning pipelines where malicious prompts or dynamics in teleoperated data activate only during synthetic trajectory generation, enabling backdoors in dow...

  14. ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    ActProbe is an action-space detector that uses temporal consistency error and action chunk magnitude from policy outputs, mapped via LSTM-MLP, to predict failures earlier than baselines across policies and real-robot tasks.

  15. HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    HapTile introduces a visuotactile dataset with haptic-informed teleoperation for language-conditioned contact-rich manipulation tasks and provides baseline policy benchmarks.

  16. Same Weights, Different Robot: A Deployment Safety View of VLA Policies

    cs.CR 2026-06 unverdicted novelty 7.0

    The paper identifies a deployment safety gap in VLA policies where identical checkpoints can be executable-inequivalent due to action metadata mismatches, supported by a derived closed-form transform and empirical dri...

  17. X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding

    cs.CV 2026-06 unverdicted novelty 7.0

    X-Stream benchmark shows SOTA MLLMs score ~50% on concurrent multi-stream tasks and lack proactive ability, using a dual-verification pipeline to avoid single-stream bias.

  18. X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding

    cs.CV 2026-06 unverdicted novelty 7.0

    X-Stream benchmark shows state-of-the-art MLLMs achieve only about 50% on multi-stream video tasks and exhibit poor proactive ability.

  19. MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models

    cs.AI 2026-05 unverdicted novelty 7.0

    MiraBench defines action-conditioned reliability via three levels (physics adherence, action-following fidelity, optimism bias detection) and applies it to 12 model configurations using a 16,000-judgment human corpus,...

  20. What-If World: A Causal Benchmark for General World Models in Embodied Scenarios

    cs.CV 2026-05 unverdicted novelty 7.0

    What-If World is a new paired-prompt benchmark showing that nine state-of-the-art video generation models achieve at most 52% on causal intervention tests and cluster near 28% for open-source systems.

  21. Point Tracking Improves World Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    JOPAT jointly models pixels, point tracks, and actions in a diffusion transformer and reports gains over pixel-only baselines on long-horizon robot tasks with occlusion and off-screen motion.

  22. RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents

    cs.CR 2026-05 unverdicted novelty 7.0

    RoboJailBench creates a taxonomy-based benchmark, intent-contrast datasets, and evaluation framework for jailbreak attacks and defenses in embodied robotic AI systems.

  23. Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR

    cs.LG 2026-05 conditional novelty 7.0

    Pion modifies Muon's Newton-Schulz iterations into a controllable high-pass filter that anchors dominant singular values at 1 while suppressing noisy tails, outperforming Muon and AdamW in VLA and RLVR regimes.

  24. RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0

    RoboFlow4D is an end-to-end lightweight flow world model that predicts multi-frame 3D flows from visual observations and textual instructions to provide explicit planning for real-time robotic manipulation.

  25. From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0

    MoLA infers a mixture of latent actions from generated future videos via modality-aware inverse dynamics models to improve robot manipulation policies.

  26. Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation

    cs.LG 2026-05 unverdicted novelty 7.0

    MA-BC partitions divergent expert data while pooling non-conflicting pairs in MOMDPs, converging faster to Pareto-optimal policies than independent learners and matching a new minimax lower bound.

  27. Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation

    cs.LG 2026-05 unverdicted novelty 7.0

    MA-BC partitions divergent expert data and pools non-conflicting pairs to achieve faster convergence to Pareto-optimal policies in MOMDPs, with a matching minimax lower bound.

  28. Coordinated Diffusion: Generating Multi-Agent Behavior Without Multi-Agent Demonstrations

    cs.RO 2026-05 unverdicted novelty 7.0

    CoDi decomposes the multi-agent diffusion score into pre-trained single-agent policies plus a gradient-free cost guidance term to generate coordinated behavior from single-agent data alone.

  29. Offline Policy Evaluation for Manipulation Policies via Discounted Liveness Formulation

    cs.RO 2026-05 conditional novelty 7.0

    A liveness-based Bellman operator enables conservative offline policy evaluation for manipulation tasks by encoding task progression and reducing truncation bias from finite horizons.

  30. CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Capability vectors extracted from parameter differences between standard and auxiliary-finetuned VLA models can be merged into pretrained weights to match auxiliary-training performance while reducing computational ov...

  31. ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

    cs.CV 2026-05 unverdicted novelty 7.0

    ACWM-Phys is a controllable simulator benchmark with in- and out-of-distribution protocols for evaluating action-conditioned world models across rigid, kinematic, deformable, and particle dynamics.

  32. OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0

    OA-WAM uses persistent address vectors and dynamic content vectors in object slots to enable addressable world-action prediction, improving robustness on manipulation benchmarks under scene changes.

  33. OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction

    cs.RO 2026-04 unverdicted novelty 7.0

    A 48-camera residential platform delivers real-time occlusion-robust 3D perception and coordinated actuation for multi-human multi-robot interaction in a shared home workspace.

  34. Atomic-Probe Governance for Skill Updates in Compositional Robot Policies

    cs.RO 2026-04 unverdicted novelty 7.0

    A cross-version swap protocol reveals dominant skills that swing composition success by up to 50 percentage points, and an atomic probe with selective revalidation governs updates at lower cost than always re-testing ...

  35. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 accept novelty 7.0

    3D generation for embodied AI is shifting from visual realism toward interaction readiness, organized into data generation, simulation environments, and sim-to-real bridging roles.

  36. Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Models

    cs.RO 2026-04 unverdicted novelty 7.0

    MoSS augments VLAs with decoupled modality streams for multiple physical signals, achieving synergistic gains in real-world robot tasks via joint attention and auxiliary future-signal prediction.

  37. CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models

    cs.RO 2026-04 unverdicted novelty 7.0

    CodeGraphVLP uses a semantic-graph state and executable code planner to enable reliable long-horizon non-Markovian robot manipulation, improving task success and lowering latency over standard VLA baselines.

  38. Action Images: End-to-End Policy Learning via Multiview Video Generation

    cs.CV 2026-04 unverdicted novelty 7.0

    Action Images turn robot arm motions into interpretable multiview pixel videos, letting video backbones serve as zero-shot policies for end-to-end robot learning.

  39. Tune to Learn: How Controller Gains Shape Robot Policy Learning

    cs.RO 2026-04 conditional novelty 7.0

    Controller gains affect learnability differently for behavior cloning, RL from scratch, and sim-to-real transfer, so optimal gains depend on the learning paradigm rather than desired task behavior.

  40. VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

    cs.RO 2026-03 unverdicted novelty 7.0

    VP-VLA decouples high-level reasoning from low-level control in VLA models by rendering spatial anchors as visual prompts directly in the RGB observation space, outperforming end-to-end baselines.

  41. UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

    cs.RO 2026-02 unverdicted novelty 7.0

    UniLACT improves VLA models by adding depth-aware unified latent action pretraining that outperforms RGB-only baselines on seen and unseen manipulation tasks.

  42. DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

    cs.RO 2026-02 unverdicted novelty 7.0

    DreamDojo is a foundation world model pretrained on the largest human video dataset to date that uses continuous latent actions to transfer interaction knowledge and achieves controllable physics simulation after robo...

  43. Large Video Planner Enables Generalizable Robot Control

    cs.RO 2025-12 conditional novelty 7.0

    A video foundation model trained on human demonstrations generates zero-shot plans that convert to executable robot actions on novel scenes and tasks.

  44. RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation

    cs.RO 2025-11 accept novelty 7.0

    RoboCOIN is a large multi-embodiment bimanual manipulation dataset with hierarchical annotations and an open processing pipeline that improves model performance across robotic platforms.

  45. USIM and U0: A Vision-Language-Action Dataset and Model for General Underwater Robots

    cs.RO 2025-10 unverdicted novelty 7.0

    Introduces USIM simulation dataset and U0 VLA model with CAP module for general underwater robot tasks, reporting 0.0359 offline error and 43.1% online success rate.

  46. DreamGen: Unlocking Generalization in Robot Learning through Video World Models

    cs.RO 2025-05 unverdicted novelty 7.0

    DreamGen trains robot policies on synthetic trajectories from adapted video world models, enabling a humanoid robot to perform 22 new behaviors in seen and unseen environments from a single pick-and-place teleoperatio...

  47. Kaiwu: A Multimodal Manipulation Dataset and Framework for Robot Learning and Human-Robot Interaction

    cs.RO 2025-03 unverdicted novelty 7.0

    Introduces the Kaiwu multimodal dataset and framework with 11,664 synchronized assembling demonstrations including hand motions, pressures, sounds, multi-view videos, motion capture, eye gaze, and EMG signals with tim...

  48. Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

    cs.RO 2026-07 unverdicted novelty 6.0

    TAP uses two-stage pretraining on unlabeled data to learn physical competence before language grounding, matching 1M-expert models with far less labeled data and showing robustness on real robots.

  49. RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

    cs.RO 2026-07 unverdicted novelty 6.0

    RoboWorld introduces an automated pipeline using autoregressive video world models and task-progress VLM scoring, plus Step Forcing for long-horizon stability, to achieve high correlation with real robot policy evaluation.

  50. ABot-M0.5: Unified Mobility-and-Manipulation World Action Model

    cs.CV 2026-07 unverdicted novelty 6.0

    ABot-M0.5 proposes a unified mobility-and-manipulation world action model using three alignment strategies that achieves state-of-the-art performance on mobile and fine-grained manipulation benchmarks.

  51. Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments

    cs.RO 2026-06 unverdicted novelty 6.0

    Human-as-Humanoid converts ego-exo human videos into executable 60-DoF humanoid actions through embodiment alignment and retargeting, enabling zero-shot real-robot policy deployment without target-task teleoperation data.

  52. 3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

    cs.RO 2026-06 unverdicted novelty 6.0

    3D HAMSTER augments a VLM with depth encoding and reconstruction to predict 3D waypoints that directly guide pointcloud policies, outperforming 2D baselines especially under distribution shifts.

  53. 3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

    cs.RO 2026-06 unverdicted novelty 6.0

    3D HAMSTER adds depth encoding and reconstruction to VLMs to produce 3D waypoint sequences that feed directly into pointcloud policies, claiming better generalization than 2D baselines under shifts.

  54. Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

    cs.RO 2026-06 unverdicted novelty 6.0

    CI-MSE improves Spearman's rank correlation between offline validation error and real rollout performance from -0.61 (raw MSE) to -0.87 across policy checkpoints in simulation and real-world robot manipulation experiments.

  55. SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

    cs.RO 2026-06 unverdicted novelty 6.0

    SimFoundry automates zero-shot real-to-sim scene generation from video, producing digital twins and cousins that enable policy training with 0.911 mean Pearson correlation to real-world results and 17-40% success gain...

  56. Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience

    cs.RO 2026-06 unverdicted novelty 6.0

    SCORE constrains sim RL to the support of a real-data policy via flow steering, raising average success on eight dexterous tasks from 37.8% to 89.9%.

  57. Scalable Behavior Cloning with Open Data, Training, and Evaluation

    cs.RO 2026-06 unverdicted novelty 6.0

    Releases the largest open teleoperation dataset for robot manipulation together with hardware, simulation, and training infrastructure to support scalable behavior cloning.

  58. LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

    cs.RO 2026-06 unverdicted novelty 6.0

    LA4VLA creates a 33K language-action dataset from existing demos and shows that pretraining on language-action pairs before or alongside vision-language-action training boosts success rates in sim and real robot tasks.

  59. LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

    cs.RO 2026-06 unverdicted novelty 6.0

    LA4VLA pretrains on language-action pairs from decomposed demonstrations to create reusable action priors, yielding up to 45 percentage point gains in real-world VLA success rates when mixed with standard training.

  60. WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation

    cs.RO 2026-06 unverdicted novelty 6.0

    WatchAct is a new benchmark of 3000 instances across 14 tasks in four cognitive domains for evaluating video-grounded robot manipulation, with current systems achieving at most 16.3% success.

Reference graph

Works this paper leans on

79 extracted references · 79 canonical work pages · cited by 209 Pith papers · 15 internal anchors

  1. [1]

    Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023.https://arxiv.org/abs/2309.01918

    Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918, 2023

    work page arXiv 2023

  2. [2]

    RT-1: Robotics Transformer for Real-World Control at Scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    Scaling data-driven robotics with reward sketching and batch reinforcement learning

    Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. RSS, 2019

    work page 2019

  4. [4]

    nuScenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. preprint arXiv:1903.11027, 2019

    work page internal anchor Pith review arXiv 1903

  5. [5]

    ShapeNet: An Information-Rich 3D Model Repository

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015

    work page internal anchor Pith review arXiv 2015

  6. [6]

    Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning

    Lawrence Yunliang Chen, Chenfeng Xu, Karthik Dhar- marajan, Muhammad Zubair Irshad, Richard Cheng, Kurt Keutzer, Masayoshi Tomizuka, Quan Vuong, and Ken Goldberg. Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning. In Conference on Robot Learning (CoRL) , Munich, Germany, 2024

    work page 2024

  7. [7]

    Dif- fusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- fusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS) , 2023

    work page 2023

  8. [8]

    Robonet: Large-scale multi-robot learning

    Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. CoRL, 2019

    work page 2019

  9. [9]

    Objaverse-XL: A Universe of 10M+ 3D Objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram V oleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.05663, 2023

    work page internal anchor Pith review arXiv 2023

  10. [10]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023

    work page 2023

  11. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009

    work page 2009

  12. [12]

    Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv:1812.00568, 2018

    work page Pith review arXiv 2018

  13. [13]

    Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

    Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396 , 2021

    work page internal anchor Pith review arXiv 2021

  14. [14]

    Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot

    Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 3:5, 2023

    work page 2023

  15. [15]

    Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography

    Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM , 24(6):381–395, 1981

    work page 1981

  16. [16]

    Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation

    Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024

    work page internal anchor Pith review arXiv 2024

  17. [17]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  18. [18]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012

    work page 2012

  19. [19]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995– 19012, 2022

    work page 2022

  20. [20]

    Robot learning in homes: Improving generalization and reducing dataset bias

    Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashc- hand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems , 31, 2018

    work page 2018

  21. [21]

    Scaling up and distilling down: Language-guided robot skill acquisition

    Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning , pages 3766–3777. PMLR, 2023

    work page 2023

  22. [22]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840–6851, 2020

    work page 2020

  23. [23]

    spaCy 2: Natural language understanding with Bloom embeddings, con- volutional neural networks and incremental parsing

    Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, con- volutional neural networks and incremental parsing. To appear, 2017

    work page 2017

  24. [24]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning , pages 991–1002. PMLR, 2022

    work page 2022

  25. [25]

    VIMA: Robot ma- nipulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: Robot ma- nipulation with multimodal prompts. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on...

    work page 2023

  26. [26]

    QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018

    work page Pith review arXiv 2018

  27. [27]

    Mt-opt: Continuous multi- task robotic reinforcement learning at scale

    Dmitry Kalashnkov, Jake Varley, Yevgen Chebotar, Ben Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi- task robotic reinforcement learning at scale. arXiv, 2021

    work page 2021

  28. [28]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation

    Haresh Karnan, Anirudh Nair, Xuesu Xiao, Garrett War- nell, Sören Pirk, Alexander Toshev, Justin Hart, Joydeep Biswas, and Peter Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation Letters, 7(4):11807–11814, 2022

    work page 2022

  29. [29]

    Segment Anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. URL https: //arxiv.org/abs/2304.02643

    work page internal anchor Pith review arXiv 2023

  30. [30]

    Berg, Wan-Yen Lo, et al

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment Anything, April 2023

    work page 2023

  31. [31]

    Ep n p: An accurate o (n) solution to the p n p problem

    Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Ep n p: An accurate o (n) solution to the p n p problem. International journal of computer vision , 81: 155–166, 2009

    work page 2009

  32. [32]

    Learning hand-eye coordination for robotic grasping with large-scale data collection

    Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with large-scale data collection. In International Symposium on Experimental Robotics . Springer, 2016

    work page 2016

  33. [33]

    Wang, Giovanni Sutanto, Ak- shara Rai, and Franziska Meier

    Yixin Lin, Austin S. Wang, Giovanni Sutanto, Ak- shara Rai, and Franziska Meier. Polymetis. https: //facebookresearch.github.io/fairo/polymetis/, 2021

    work page 2021

  34. [34]

    Ctrnet-x: Camera-to-robot pose estimation in real-world conditions using a single camera

    Jingpei Lu, Zekai Liang, Tristin Xie, Florian Ritcher, Shan Lin, Sainan Liu, and Michael C Yip. Ctrnet-x: Camera-to-robot pose estimation in real-world conditions using a single camera. arXiv preprint arXiv:2409.10441 , 2024

    work page arXiv 2024

  35. [35]

    Interactive language: Talking to robots in real time

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters , 2023

    work page 2023

  36. [36]

    Roboturk: A crowdsourcing platform for robotic skill learning through imitation

    Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning , pages 879–

    work page

  37. [37]

    What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298 , 2021

    work page internal anchor Pith review arXiv 2021

  38. [38]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. https://octo-models.github.io, 2023

    work page 2023

  39. [39]

    Open X-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Her- zog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bern- hard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Hua...

    work page internal anchor Pith review arXiv 2023

  40. [40]

    Supersizing self- supervision: Learning to grasp from 50k tries and 700 robot hours

    Lerrel Pinto and Abhinav Gupta. Supersizing self- supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA) , pages 3406–3413. IEEE, 2016

    work page 2016

  41. [41]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  42. [42]

    Robot learning with sensorimotor pre-training

    Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning, 2023

    work page 2023

  43. [43]

    Common crawl – building an open web-scale crawl using hadoop, 2010

    Ahad Rana. Common crawl – building an open web-scale crawl using hadoop, 2010. URL https://www.slideshare. net/hadoopusergroup/common-crawlpresentation

    work page 2010

  44. [44]

    Accelerating 3D Deep Learning with PyTorch3D

    Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.08501, 2020

    work page internal anchor Pith review arXiv 2007

  45. [45]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108,

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  46. [46]

    URL https://api.semanticscholar.org/CorpusID: 203626972

    work page

  47. [47]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278–25294, 2022

    work page 2022

  48. [48]

    On bringing robots home, 2023

    Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home, 2023

    work page 2023

  49. [49]

    Gnm: A general navigation model to drive any robot

    Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, and Sergey Levine. Gnm: A general navigation model to drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 7226–7233. IEEE, 2023

    work page 2023

  50. [50]

    ViNT: A foundation model for visual navigation,

    Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachow- icz, Kevin Black, Noriaki Hirose, and Sergey Levine. ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning , 2023. URL https://arxiv.org/abs/2306.14846

    work page arXiv 2023

  51. [51]

    Multiple interactions made easy (mime): Large scale demonstrations data for imitation

    Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhi- nav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning , pages 906–915. PMLR, 2018

    work page 2018

  52. [52]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022

    work page 2022

  53. [53]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning , pages 785–799. PMLR, 2023

    work page 2023

  54. [54]

    Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations

    Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters , 5(3):4978–4985, 2020

    work page 2020

  55. [55]

    Sridhar, D

    Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navi- gation and exploration. arXiv preprint arXiv:2310.07896 , 2023

    work page arXiv 2023

  56. [56]

    Scalability in perception for autonomous driving: Waymo open dataset, 2019

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception...

    work page 2019

  57. [57]

    View-invariant policy learning via zero-shot novel view synthesis

    Stephen Tian, Blake Wulfe, Kyle Sargent, Katherine Liu, Sergey Zakharov, Vitor Guizilini, and Jiajun Wu. View-invariant policy learning via zero-shot novel view synthesis. arXiv, 2024

    work page 2024

  58. [58]

    Tartandrive: A large-scale dataset for learning off-road dynamics models

    Samuel Triest, Matthew Sivaprakasam, Sean J Wang, Wenshan Wang, Aaron M Johnson, and Sebastian Scherer. Tartandrive: A large-scale dataset for learning off-road dynamics models. In 2022 International Conference on Robotics and Automation (ICRA), pages 2546–2552. IEEE, 2022

    work page 2022

  59. [59]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998–6008, 2017

    work page 2017

  60. [60]

    Bridgedata v2: A dataset for robot learning at scale, 2023

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen- Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale, 2023

    work page 2023

  61. [61]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    work page 2025

  62. [62]

    Dust3r: Geometric 3d vision made easy, 2024

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy, 2024. URL https://arxiv.org/abs/2312. 14132

    work page 2024

  63. [63]

    Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli

    Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2025

    work page 2025

  64. [64]

    Visual imitation made easy, 2020

    Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy, 2020

    work page 2020

  65. [65]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020

    work page 2020

  66. [66]

    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  67. [67]

    3D-VLA: A 3D Vision-Language-Action Generative World Model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d- vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631 , 2024

    work page internal anchor Pith review arXiv 2024

  68. [68]

    in-the-wild

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In 7th Annual Conference on Robot Learning , 2023. APPENDIX A CONTRIBUTIONS Project Leads: Alexander Khazatsky, Karl Pertsch Research Leads (contri...

    work page 2023

  69. [69]

    Industrial office: industrial office tables and chairs, conference rooms, conference TVs

    work page

  70. [70]

    Industrial kitchen: industrial refrigerator, sink, coffee maker

    work page

  71. [71]

    Industrial dining room: industrial setting with dining tables

    work page

  72. [72]

    Home office: desk or desk chairs in a home setting

    work page

  73. [73]

    Home kitchen: refrigerator, kitchen sink, kitchen tabletop in a home setting

    work page

  74. [74]

    Home dining room: dining table, dining chairs, in a home setting

    work page

  75. [75]

    Bedroom: room with a bed

    work page

  76. [76]

    Bathroom: Showers, baths, toilets, bathroom sinks

    work page

  77. [77]

    Living room: places with couches, armchairs, coffee tables, tvs in a home setting

    work page

  78. [78]

    Hallway / closet: areas between rooms, situations where the robot is interacting with a door or objects in a closet

    work page

  79. [79]

    Other: any other location that does not fit into those categories 12: Unknown: a scene that’s too hard to classify because the image is dark or too close up Listing C.1: The prompt provided to GPT4V in order to classify scene types. A. Diffusion Policy Architecture and Hyperparameters We build our diffusion policy [ 7] training pipeline on the Robomimic c...

    work page