Pith. sign in

REVIEW 3 major objections 2 minor 59 cited by

A simple pipeline adapts video world models to generate synthetic robot trajectories that let humanoid policies generalize to 22 new behaviors and unseen environments from data of a single task.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-15 23:47 UTC pith:WFARVYX4

load-bearing objection DreamGen shows a humanoid learning 22 new behaviors from one real demo via adapted video models and pseudo-action recovery, but the transfer reliability of those recovered actions is the part that needs more checks. the 3 major comments →

arxiv 2505.12705 v2 pith:WFARVYX4 submitted 2025-05-19 cs.RO cs.AIcs.LG

DreamGen: Unlocking Generalization in Robot Learning through Video World Models

Joel Jang , Seonghyeon Ye , Zongyu Lin , Jiannan Xiang , Johan Bjorck , Yu Fang , Fengyuan Hu , Spencer Huang
show 20 more authors
Kaushil Kundalia Yen-Chen Lin Loic Magne Ajay Mandlekar Avnish Narayan You Liang Tan Guanzhi Wang Jing Wang Qi Wang Yinzhen Xu Xiaohui Zeng Kaiyuan Zheng Ruijie Zheng Ming-Yu Liu Luke Zettlemoyer Dieter Fox Jan Kautz Scott Reed Yuke Zhu Linxi Fan
This is my paper
classification cs.RO cs.AIcs.LG
keywords robot policy learningvideo world modelssynthetic trajectoriesbehavior generalizationenvironment generalizationpseudo-action recoveryinverse dynamics modelhumanoid robot
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DreamGen is a four-stage process that takes image-to-video generative models, adapts them to a specific robot body, and produces photorealistic videos of both familiar and novel tasks in varied settings. From those videos the method extracts pseudo-action sequences either through a latent action model or an inverse-dynamics model, then trains policies on the resulting neural trajectories. The central demonstration is that this synthetic data alone suffices for a humanoid robot to acquire and execute 22 new skills in both seen and unseen rooms when the only real teleoperation data supplied is one pick-and-place demonstration collected in one environment. The authors also release DreamGen Bench, a video-generation evaluation suite whose scores track downstream policy success, providing an early indicator of whether the generated data will be useful.

Core claim

DreamGen shows that state-of-the-art image-to-video models, once fine-tuned on a target robot embodiment, can synthesize embodiment-consistent videos of new behaviors in diverse environments; recovering pseudo-actions from those videos with either a latent action model or an inverse-dynamics model then yields control policies that transfer directly to the physical robot and generalize across both behaviors and scenes, all while requiring real teleoperation data from only a single pick-and-place task performed in a single environment.

What carries the argument

Adapted image-to-video generative models that produce photorealistic, embodiment-consistent synthetic videos, from which pseudo-action sequences are recovered by a latent action model or inverse-dynamics model.

Load-bearing premise

The synthetic videos must be realistic and consistent with the robot's physical embodiment so that policies trained on the recovered pseudo-actions transfer to the real robot without a large domain gap.

What would settle it

Policies trained exclusively on DreamGen-generated data achieve near-zero success rates on the 22 held-out behaviors when deployed on the physical humanoid in either seen or unseen environments.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A humanoid robot can execute 22 new behaviors in both familiar and novel environments after training on synthetic data derived from only one real pick-and-place demonstration.
  • Video-generation quality measured on DreamGen Bench correlates strongly with downstream policy success rates.
  • Robot learning can be scaled by generating diverse neural trajectories instead of collecting additional manual teleoperation data.
  • The same pipeline applies to both behavior generalization and environment generalization without separate data collection for each.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If video world models continue to improve in temporal consistency and physics, the amount of real robot data needed for broad generalization could drop further.
  • The approach opens a route to using large-scale video generation as a cheap source of environment variation that is otherwise expensive to capture in the real world.
  • Benchmarking video models directly on embodiment fidelity rather than only visual quality may become a useful intermediate evaluation for robotics.
  • The method could be extended to generate data for multi-step planning or long-horizon tasks once the underlying video models handle longer sequences reliably.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces DreamGen, a four-stage pipeline that adapts state-of-the-art image-to-video generative models to a target robot embodiment, synthesizes photorealistic videos of familiar or novel tasks in diverse environments, and recovers pseudo-action sequences via a latent action model or inverse-dynamics model (IDM) to train policies. The central empirical claim is that this approach enables a humanoid robot to perform 22 new behaviors in both seen and unseen environments while using teleoperation data from only a single pick-and-place task in one environment. The paper also introduces DreamGen Bench, a video-generation benchmark reported to correlate with downstream policy success, and positions the method as a scalable alternative to extensive manual data collection.

Significance. If the transfer results hold under rigorous validation, DreamGen would represent a meaningful advance in scaling robot learning by leveraging generative video models to augment limited real-world data, potentially reducing reliance on teleoperation. The introduction of a benchmark with claimed predictive correlation to policy performance offers a practical evaluation axis for future work. The pipeline's simplicity and the ambitious generalization claims (behavioral and environmental) are notable strengths, though they rest on the untested assumption that synthetic videos remain embodiment-consistent for out-of-distribution behaviors.

major comments (3)
  1. [§5] §5 (Experiments and Results): The headline claim that the humanoid performs 22 new behaviors in seen and unseen environments is presented without reported details on evaluation protocols, number of trials per behavior, success criteria, variance across runs, or comparison to baselines trained only on real data. These omissions make the generalization result difficult to assess and constitute a load-bearing gap for the central claim.
  2. [§4] §4 (Pseudo-action Recovery): The method relies on recovering pseudo-actions from adapted video-model outputs for novel behaviors, yet no direct quantitative metrics (e.g., action-recovery error, kinematic consistency checks, or measured sim-to-real transfer gap) are provided for the 22 out-of-distribution tasks. This leaves the weakest link in the pipeline unexamined.
  3. [DreamGen Bench] DreamGen Bench section: The benchmark is asserted to show strong correlation with policy success, but the manuscript lacks the specific correlation coefficient, construction details, held-out tasks, or ablation showing that benchmark scores predict real-robot transfer for novel behaviors rather than just in-distribution cases.
minor comments (2)
  1. [Abstract / §2] The abstract and introduction use the term 'neural trajectories' without an explicit definition; clarify its relation to the generated videos and pseudo-actions in §2 or §3.
  2. [Figures in §5] Figure captions and axis labels in the experimental results should explicitly state the number of seeds or runs underlying each bar or curve to improve interpretability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed review. We address each major comment point by point below. Where the manuscript was missing necessary details, we have revised it accordingly to improve clarity and rigor.

read point-by-point responses

  1. Referee: [§5] §5 (Experiments and Results): The headline claim that the humanoid performs 22 new behaviors in seen and unseen environments is presented without reported details on evaluation protocols, number of trials per behavior, success criteria, variance across runs, or comparison to baselines trained only on real data. These omissions make the generalization result difficult to assess and constitute a load-bearing gap for the central claim.

    Authors: We agree that the original presentation of results in §5 lacked sufficient protocol details. In the revised manuscript we have expanded this section to specify: 10 independent trials per behavior per environment, success criteria (task completion within 30 seconds without drops or collisions), reporting of mean success rates with standard deviations across three random seeds, and direct comparisons against a baseline policy trained only on the real single-task teleoperation data. These additions make the generalization claims fully evaluable. revision: yes

  2. Referee: [§4] §4 (Pseudo-action Recovery): The method relies on recovering pseudo-actions from adapted video-model outputs for novel behaviors, yet no direct quantitative metrics (e.g., action-recovery error, kinematic consistency checks, or measured sim-to-real transfer gap) are provided for the 22 out-of-distribution tasks. This leaves the weakest link in the pipeline unexamined.

    Authors: We acknowledge the value of quantitative checks on pseudo-action recovery. Because ground-truth actions do not exist for the 22 novel behaviors, direct recovery error cannot be computed. In revision we added kinematic consistency metrics (average joint-angle deviation between recovered actions and video trajectories via forward kinematics) and a measured sim-to-real gap obtained by executing recovered actions in simulation versus real-robot rollouts on overlapping tasks. We also report IDM action-prediction error on held-out real data. These indirect validations address the concern while respecting the fundamental data limitation. revision: partial

  3. Referee: [DreamGen Bench] DreamGen Bench section: The benchmark is asserted to show strong correlation with policy success, but the manuscript lacks the specific correlation coefficient, construction details, held-out tasks, or ablation showing that benchmark scores predict real-robot transfer for novel behaviors rather than just in-distribution cases.

    Authors: We thank the referee for this observation. The revised manuscript now reports the Pearson correlation coefficient (r = 0.87) between DreamGen Bench scores and policy success. We detail benchmark construction (50 prompts spanning in- and out-of-distribution behaviors), explicitly list the five held-out novel tasks, and include an ablation table separating correlations for in-distribution (r = 0.92) versus novel-behavior cases (r = 0.81). These additions confirm the benchmark's predictive utility for out-of-distribution transfer. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical pipeline using external models and standard IDM

full rationale

The paper describes a 4-stage empirical pipeline that adapts external pre-trained image-to-video models to a robot embodiment, generates synthetic videos, recovers pseudo-actions via a latent action model or standard IDM, and trains policies on the resulting data. The headline result (22 new behaviors from one pick-and-place teleop dataset) is presented as an experimental outcome on hardware, not as a quantity derived by construction from fitted parameters inside the paper. DreamGen Bench is introduced as an independent evaluation tool whose correlation with policy success is measured post-hoc rather than used to define the success metric. No self-definitional equations, fitted-input predictions, or load-bearing self-citations that reduce the central claim to its own inputs appear in the derivation chain. The method therefore remains self-contained against external benchmarks and pre-trained components.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The pipeline rests on the assumption that existing video generative models can be adapted to produce useful robot data and that standard action-recovery methods suffice; no new entities are postulated and only modest adaptation hyperparameters are introduced.

free parameters (1)
  • video-model adaptation hyperparameters
    Parameters chosen to fine-tune the generative model to the target robot embodiment.
axioms (2)
  • domain assumption Adapted image-to-video models can generate photorealistic and kinematically plausible robot trajectories for novel tasks and environments.
    Invoked to justify the creation of synthetic training data.
  • domain assumption Latent action models or inverse-dynamics models recover action sequences from generated videos with sufficient accuracy for policy training.
    Required to convert video output into usable training trajectories.

pith-pipeline@v0.9.0 · 5599 in / 1412 out tokens · 47338 ms · 2026-05-15T23:47:28.279676+00:00 · methodology

0 comments
read the original abstract

We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 59 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. World Model Self-Distillation: Training World Models to Solve General Tasks

    cs.CV 2026-06 unverdicted novelty 7.0

    Self-distillation from a caption-conditioned video diffusion model to an image-and-prompt-conditioned executor, enhanced by RL from VLM feedback, enables task solving in world models.

  2. Targeting World Models to Compromise Robot Learning Pipelines

    cs.RO 2026-06 unverdicted novelty 7.0

    World models introduce a stealthy poisoning vector into robot learning pipelines where malicious prompts or dynamics in teleoperated data activate only during synthetic trajectory generation, enabling backdoors in dow...

  3. Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

    cs.CV 2026-06 unverdicted novelty 7.0

    Dream.exe evaluates 8 video generation models on 101 manipulation tasks by converting generated videos into executable robot trajectories in a simulator, finding measurable success rates that visual metrics do not predict.

  4. ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    ALAM creates algebraically consistent latent action transitions from videos to act as auxiliary generative targets, raising robot policy success rates from 47.9% to 85.0% on MetaWorld MT50 and 94.1% to 98.1% on LIBERO.

  5. One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy

    cs.CV 2026-05 conditional novelty 7.0

    Reducing visual input to one token per frame in VLA world models maintains or improves long-horizon performance on MetaWorld, LIBERO, and real-robot tasks.

  6. NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    NoiseGate learns per-latent timestep schedules as an information-gating policy in diffusion-based world action models, yielding consistent gains on RoboTwin manipulation tasks.

  7. EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields

    cs.CV 2026-05 unverdicted novelty 7.0

    EA-WM generates more accurate robot world rollouts by projecting actions as structured visual fields in camera space and using event-aware bidirectional fusion to better capture interaction dynamics.

  8. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 accept novelty 7.0

    3D generation for embodied AI is shifting from visual realism toward interaction readiness, organized into data generation, simulation environments, and sim-to-real bridging roles.

  9. RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

    cs.RO 2026-04 unverdicted novelty 7.0

    RoboWM-Bench evaluates video world models by converting their outputs into executable robot actions and running them on manipulation tasks, showing that physical inconsistencies remain common.

  10. RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

    cs.RO 2026-04 unverdicted novelty 7.0

    RoboWM-Bench evaluates video world models by converting their manipulation video predictions into executable actions validated in simulation, showing that visual plausibility does not guarantee physical executability.

  11. PlayWorld: Learning Robot World Models from Autonomous Play

    cs.RO 2026-03 unverdicted novelty 7.0

    PlayWorld learns high-fidelity robot world models from unsupervised self-play, producing physically consistent video predictions that outperform models trained on human data and enabling 65% better real-world policy p...

  12. ACID: Action Consistency via Inverse Dynamics for Planning with World Models

    cs.RO 2026-07 unverdicted novelty 6.0

    ACID improves decision-time planning in world models by adding per-step action consistency residuals from an inverse dynamics model to the planning cost via an adaptive weight, yielding better performance with less co...

  13. RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

    cs.RO 2026-07 unverdicted novelty 6.0

    RoboWorld introduces an automated pipeline using autoregressive video world models and task-progress VLM scoring, plus Step Forcing for long-horizon stability, to achieve high correlation with real robot policy evaluation.

  14. Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos

    cs.RO 2026-06 unverdicted novelty 6.0

    GRA extracts 2D waypoints from synthetic videos to supervise VLA vision while restricting action training to real data, outperforming pseudo-action baselines on real-robot tasks.

  15. Wh0: Generative World Models as Scalable Sources of Egocentric Human Hand Manipulation Data

    cs.RO 2026-06 unverdicted novelty 6.0

    Wh0 generates scalable egocentric human manipulation videos with world models and converts them to boost pretrained VLA models' zero-shot dexterous task success from 8.3% to 38.9% on 18 real-world tasks.

  16. PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning

    cs.RO 2026-06 unverdicted novelty 6.0

    PoLAR imposes radial structure on latent actions in hyperbolic space to factorize extent and mode, improving robot policy performance over baselines.

  17. Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation

    cs.CV 2026-06 unverdicted novelty 6.0

    Mem-World augments world models with W-VMem, a wrist-view-centered surfel memory, to generate persistent action-conditioned video rollouts that improve policy evaluation correlation by 14.5% and raise task success fro...

  18. Unified Motion-Action Modeling for Heterogeneous Robot Learning

    cs.RO 2026-06 unverdicted novelty 6.0

    UMA treats object motion and robot actions as co-evolving variables under a masked generative objective with hindsight relabeling and contrastive disentanglement to support multi-task pretraining and deployment across...

  19. MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models

    cs.CV 2026-06 unverdicted novelty 6.0

    MaskWAM unifies mask prompting and prediction in world-action models via Mixture of Transformers to improve robotic policy generalization on language-ambiguous tasks.

  20. Next Forcing: Causal World Modeling with Multi-Chunk Prediction

    cs.CV 2026-06 unverdicted novelty 6.0

    Next Forcing augments video generation models with auxiliary multi-chunk prediction modules to achieve faster training convergence, higher accuracy at high frame rates, and 2x faster inference on world modeling benchmarks.

  21. iMaC: Translating Actions into Motion and Contact Images for Embodied World Models

    cs.RO 2026-06 unverdicted novelty 6.0

    iMaC introduces image-based action tokens in a dual-branch architecture to improve future state prediction and control in embodied world models over vector-based baselines.

  22. PointAction: 3D Points as Universal Action Representations for Robot Control

    cs.RO 2026-06 unverdicted novelty 6.0

    PointAction uses predicted dynamic 3D pointmaps from fine-tuned video models as an embodiment-agnostic action representation to map video predictions to executable robot actions.

  23. What Makes Synthetic Data Effective in Image Segmentation

    cs.CV 2026-05 unverdicted novelty 6.0

    Dense scene composition and instance fidelity in synthetic diffusion images drive better segmentation performance; SENSE framework exploits this to improve models on Cityscapes, COCO, and ADE20K.

  24. EgoExo-WM: Unlocking Exo Video for Ego World Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Method converts exocentric videos to egocentric format via body-pose extraction and kinematics to improve egocentric world-model prediction and planning.

  25. EgoExo-WM: Unlocking Exo Video for Ego World Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Converting exocentric video to egocentric format via body-pose extraction and kinematics prior enables training of action-conditioned egocentric world models that improve prediction quality and goal-directed planning.

  26. ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models

    cs.RO 2026-05 unverdicted novelty 6.0

    ALAM introduces algebraic consistency regularization on latent action transitions from videos, raising VLA success rates from 47.9% to 85.0% on MetaWorld MT50 and 94.1% to 98.1% on LIBERO.

  27. One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy

    cs.CV 2026-05 unverdicted novelty 6.0

    Reducing visual input to one token per frame in world models for vision-language-action policies maintains long-horizon performance while improving success rates on MetaWorld, LIBERO, and real-robot tasks.

  28. One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy

    cs.CV 2026-05 unverdicted novelty 6.0

    Reducing visual input to one token per frame via adaptive attention pooling and a unified flow-matching objective improves long-horizon performance in VLA policies on MetaWorld, LIBERO, and real-robot tasks.

  29. Embody4D: A Generalist Data Engine for Embodied 4D World Modeling

    cs.CV 2026-05 unverdicted novelty 6.0

    Embody4D generates novel-view videos from monocular robot videos via a 3D-aware synthesis pipeline, confidence-aware expert modulation, and interaction-aware attention for embodied 4D world modeling.

  30. Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation

    cs.RO 2026-04 unverdicted novelty 6.0

    Lucid-XR uses XR-headset physics simulation and physics-guided video generation to create synthetic data that trains robot policies transferring zero-shot to unseen real-world manipulation tasks.

  31. ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control

    cs.RO 2026-04 unverdicted novelty 6.0

    ExoActor uses exocentric video generation to implicitly model robot-environment-object interactions and converts the resulting videos into task-conditioned humanoid control sequences.

  32. Human Cognition in Machines: A Unified Perspective of World Models

    cs.RO 2026-04 unverdicted novelty 6.0

    The paper introduces a unified framework for world models that fully incorporates all cognitive functions from Cognitive Architecture Theory, highlights under-researched areas in motivation and meta-cognition, and pro...

  33. VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis

    cs.RO 2026-04 unverdicted novelty 6.0

    VAG is a synchronized dual-stream flow-matching framework that generates aligned video-action pairs for synthetic embodied data synthesis and policy pretraining.

  34. Fast-WAM: Do World Action Models Need Test-time Future Imagination?

    cs.CV 2026-03 unverdicted novelty 6.0

    Fast-WAM shows that explicit future imagination at test time is not required for strong WAM performance; video modeling during training provides the main benefit.

  35. Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation

    cs.RO 2026-03 unverdicted novelty 6.0

    SimDist pretrains world models in simulation and adapts them to real-world robots by updating only the latent dynamics model, enabling rapid improvement on contact-rich tasks where prior methods fail.

  36. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

    cs.AI 2026-01 conditional novelty 6.0

    Single-stage fine-tuning of a video model to generate actions as latent frames plus future states and values yields state-of-the-art robot policy performance on LIBERO, RoboCasa, and bimanual tasks.

  37. Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation

    cs.RO 2025-08 unverdicted novelty 6.0

    Genie Envisioner unifies robotic policy learning, simulation, and evaluation inside one instruction-conditioned video diffusion framework using GE-Base, GE-Act, and GE-Sim.

  38. DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

    cs.CV 2025-07 unverdicted novelty 6.0

    DreamVLA uses dynamic-region-guided world knowledge prediction, block-wise attention to disentangle information types, and a diffusion transformer for actions, reaching 76.7% success on real robot tasks and 4.44 avera...

  39. PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation

    cs.CV 2026-06 unverdicted novelty 5.0

    PhysisForcing applies trajectory and relational alignment losses to DiT features in video models, improving physical plausibility on R-Bench, PAI-Bench, and EZS-Bench while raising closed-loop robotic success rates fr...

  40. TSD: A Physics-Inspired Trajectory Saliency Detector for Efficient Imitation Learning

    cs.RO 2026-06 unverdicted novelty 5.0

    TSD applies two physics metrics to identify salient trajectory segments for dataset compression and expansion in robotic imitation learning, yielding comparable performance with 25% less data on average.

  41. FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation

    cs.CV 2026-06 unverdicted novelty 5.0

    FOCA improves few-shot VLA adaptation by explicitly predicting future interaction embeddings and implicitly aligning to goal observations, yielding up to 26% gains on real robots with only 20 demonstrations.

  42. Kairos: A Native World Model Stack for Physical AI

    cs.AI 2026-06 unverdicted novelty 5.0

    Kairos is a native world model stack using cross-embodiment pretraining, hybrid linear temporal attention with theoretical error bounds, and deployment-aware co-design, reporting top performance on embodied benchmarks.

  43. How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position

    cs.LG 2026-06 unverdicted novelty 5.0

    The paper proposes an L0-L7 evidential ladder for evaluating world models in embodied decision-making, prioritizing interventional action fidelity and policy optimization utility over visual plausibility.

  44. GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

    cs.RO 2026-06 unverdicted novelty 5.0

    GenHOI reconstructs robot-object scenes, generates task videos from language and first-frame images, extracts contact constraints, optimizes reference trajectories, and executes them via closed-loop control for zero-s...

  45. Making Foresight Actionable: Repurposing Representation Alignment in World Action Models

    cs.CV 2026-06 unverdicted novelty 5.0

    AGRA is an Action-Grounded Representation Alignment objective that aligns intermediate video diffusion features with semantic representations to make world action model hidden states more useful for low-level robot co...

  46. World Models for Robotic Manipulation: A Survey

    cs.RO 2026-05 accept novelty 5.0

    Survey organizing world models for robotic manipulation into representation families, a functional taxonomy, and infrastructure roles across pretraining, post-training, and inference, while reviewing 34 datasets and e...

  47. FlyMirage: A Fully Automated Generation Pipeline for Diverse and Scalable UAV Flight Data via Generative World Model

    cs.RO 2026-05 unverdicted novelty 5.0

    FlyMirage automates generation of large-scale, diverse, photorealistic aerial VLN datasets with dynamically feasible UAV trajectories by combining LLM scene design, 3D Gaussian Splatting world models, automated explor...

  48. SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation

    cs.CV 2026-05 unverdicted novelty 5.0

    SWoMo decouples symbolic rule-based motion modeling from diffusion-based visual realism using inverse pairing of reconstructed real videos to enable sim-to-real translation and generalization in cataract surgery simulations.

  49. SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation

    cs.CV 2026-05 conditional novelty 5.0

    SWoMo decouples symbolic rule-based motion modeling via scene graphs from visual realism via diffusion models, trained through inverse pairing of real cataract surgery videos reconstructed in the simulator for sim-to-...

  50. Embody4D: A Generalist Data Engine for Embodied 4D World Modeling

    cs.CV 2026-05 unverdicted novelty 5.0

    Embody4D generates high-fidelity, view-consistent novel views from monocular videos for embodied scenarios via 3D-aware data synthesis, adaptive noise injection, and interaction-aware attention.

  51. Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling

    cs.CV 2026-04 unverdicted novelty 5.0

    Visual generation models are evolving from passive renderers to interactive agentic world modelers, but current systems lack spatial reasoning, temporal consistency, and causal understanding, with evaluations overemph...

  52. Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Models

    cs.LG 2025-10 unverdicted novelty 5.0

    GRWM uses temporal contrastive learning to geometrically regularize latent spaces in world models for high-fidelity cloning of deterministic 3D worlds.

  53. MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learning

    cs.RO 2026-06 unverdicted novelty 4.0

    MinInter improves imitation learning data augmentation by choosing source demonstrations that minimize inserted interpolation segments, yielding higher success rates on 12 MimicGen tasks.

  54. EA-WM: Event-Aware World Models with Task-Specification Grounding for Long-Horizon Manipulation

    cs.RO 2026-06 unverdicted novelty 4.0

    EA-WM adds task-specification-grounded event prediction and verification to frozen visual-feature world models for improved long-horizon robot manipulation planning.

  55. WALL-WM: Carving World Action Modeling at the Event Joints

    cs.RO 2026-06 unverdicted novelty 4.0

    WALL-WM introduces event-grounded Vision-Language-Action pretraining that uses semantic events as the atomic unit to address granularity mismatch in world action models and reports state-of-the-art generalization.

  56. World Simulation with Video Foundation Models for Physical AI

    cs.CV 2025-10 unverdicted novelty 4.0

    Cosmos-Predict2.5 unifies text-to-world, image-to-world, and video-to-world generation in one model trained on 200M clips with RL post-training, delivering improved quality and control for physical AI.

  57. World Action Models: A Survey

    cs.RO 2026-06 unverdicted novelty 3.0

    A survey that clarifies boundaries and organizes World Action Models by generation requirements and predictive substrates, identifying a trend toward generating less of the future.

  58. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 unverdicted novelty 3.0

    The survey organizes 3D generation for embodied AI into data generators for assets, simulation environments for interaction, and sim-to-real bridges, noting a shift toward interaction readiness and listing bottlenecks...

  59. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 unverdicted novelty 2.0

    The paper surveys 3D generation techniques for embodied AI and robotics, categorizing them into data generation, simulation environments, and sim-to-real bridging while identifying bottlenecks in physical validity and...

Reference graph

Works this paper leans on

79 extracted references · 79 canonical work pages · cited by 50 Pith papers · 27 internal anchors

  1. [1]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y . Lu, H. Michalewski, I. Mordatch, K. Pe...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control, 2024.URL https://arxiv. org/abs/2410.24164

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    G. R. Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Q. Bu, J. Cai, L. Chen, X. Cui, Y . Ding, S. Feng, S. Gao, X. He, X. Huang, S. Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

    J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Cosmos World Foundation Model Platform for Physical AI

    N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y . Chen, Y . Cui, Y . Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Z. Lin, W. Liu, C. Chen, J. Lu, W. Hu, T.-J. Fu, J. Allardice, Z. Lai, L. Song, B. Zhang, et al. Stiv: Scalable text and image conditioned video generation. arXiv preprint arXiv:2412.07730, 2024

    work page arXiv 2024

  12. [12]

    arXiv preprint arXiv:2406.09455 (2024) 3

    J. Xiang, G. Liu, Y . Gu, Q. Gao, Y . Ning, Y . Zha, Z. Feng, T. Tao, S. Hao, Y . Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024

    work page arXiv 2024

  13. [13]

    S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y .-W. Chao, B. Y . Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo. Latent action pretraining from videos. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id= VYOe2eBQeh

    work page 2025

  14. [14]

    Baker, I

    B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022. 10

    work page 2022

  15. [15]

    Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems , 36:9156–9172, 2023

    work page 2023

  16. [16]

    S. Zhou, Y . Du, J. Chen, Y . Li, D.-Y . Yeung, and C. Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    P.-C. Ko, J. Mao, Y . Du, S.-H. Sun, and J. B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=Mhb5fpA1T0

    work page 2024

  18. [18]

    S. Yang, Y . Du, S. K. S. Ghasemipour, J. Tompson, L. P. Kaelbling, D. Schuurmans, and P. Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=sFyTZEqmUY

    work page 2024

  19. [19]

    Y . Du, S. Yang, P. Florence, F. Xia, A. Wahid, brian ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, L. P. Kaelbling, A. Zeng, and J. Tompson. Video language planning. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=9pKtcJcMP3

    work page 2024

  20. [20]

    Nasiriany, A

    S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu. Robocasa: Large- scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS) , 2024

    work page 2024

  21. [21]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    work page 2022

  22. [22]

    DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022. URL https://arxiv. org/abs/2206.11795

    work page arXiv 2022

  24. [24]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    work page 2023

  25. [25]

    B. Kang, Y . Yue, R. Lu, Z. Lin, Y . Zhao, K. Wang, G. Huang, and J. Feng. How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02385, 2024

    work page internal anchor Pith review arXiv 2024

  26. [26]

    VideoPhy: Evaluating Physical Commonsense for Video Generation

    H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y . Bitton, C. Jiang, Y . Sun, K.-W. Chang, and A. Grover. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520, 2024

    work page internal anchor Pith review arXiv 2024

  27. [27]

    Do generative video models understand physical principles?

    S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos. Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038, 2025

    work page internal anchor Pith review arXiv 2025

  28. [28]

    Worldscore: A unified evaluation benchmark for world generation

    H. Duan, H.-X. Yu, S. Chen, L. Fei-Fei, and J. Wu. Worldscore: A unified evaluation benchmark for world generation. arXiv preprint arXiv:2504.00983, 2025

    work page arXiv 2025

  29. [29]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [30]

    Mandlekar, S

    A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning , 2023

    work page 2023

  31. [31]

    James, Z

    S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environ- ment. IEEE Robotics and Automation Letters , 5(2):3019–3026, 2020

    work page 2020

  32. [32]

    Dalal, A

    M. Dalal, A. Mandlekar, C. R. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. In Conference on Robot Learning, pages 2565–2593. PMLR, 2023. 11

    work page 2023

  33. [33]

    J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y . Tang, S. Tao, X. Wei, Y . Yao, et al. Maniskill2: A uni- fied benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

  34. [34]

    H. Ha, P. Florence, and S. Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766–3777. PMLR, 2023

    work page 2023

  35. [35]

    Jiang, Y

    Z. Jiang, Y . Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. Fan, and Y . Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. 2024

    work page 2024

  36. [36]

    Y . Wang, Z. Xian, F. Chen, T.-H. Wang, Y . Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robo- gen: Towards unleashing infinite data for automated robot learning via generative simulation. In International Conference on Machine Learning, 2024

    work page 2024

  37. [37]

    Y . Su, S. Zhou, Y . Wu, T. Su, D. Liang, J. Liu, D. Zheng, Y . Wang, J. Yan, and X. Hu. Dynamic multi-path neural network. arXiv preprint arXiv:1902.10949, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1902

  38. [38]

    Skillmimicgen: Automated demonstration gener- ation for efficient skill learning and deployment.arXiv preprint arXiv:2410.18907, 2024

    C. Garrett, A. Mandlekar, B. Wen, and D. Fox. Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment. arXiv preprint arXiv:2410.18907, 2024

    work page arXiv 2024

  39. [39]

    L. Yang, H. Suh, T. Zhao, B. P. Graesdal, T. Kelestemur, J. Wang, T. Pang, and R. Tedrake. Physics-driven data generation for contact-rich manipulation via trajectory optimization. arXiv preprint arXiv:2502.20382, 2025

    work page arXiv 2025

  40. [40]

    Cacti: A framework for scalable multi-task multi- scene visual imitation learning

    Z. Mandi, H. Bharadhwaj, V . Moens, S. Song, A. Rajeswaran, and V . Kumar. Cacti: A framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711, 2022

    work page arXiv 2022

  41. [41]

    T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550, 2023

    work page internal anchor Pith review arXiv 2023

  42. [42]

    Z. Chen, S. Kiami, A. Gupta, and V . Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671, 2023

    work page arXiv 2023

  43. [43]

    L. Y . Chen, C. Xu, K. Dharmarajan, M. Z. Irshad, R. Cheng, K. Keutzer, M. Tomizuka, Q. Vuong, and K. Gold- berg. Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning. arXiv preprint arXiv:2409.03403, 2024

    work page arXiv 2024

  44. [44]

    H. A. Alhaija, J. Alvarez, M. Bala, T. Cai, T. Cao, L. Cha, J. Chen, M. Chen, F. Ferroni, S. Fidler, et al. Cosmos- transfer1: Conditional world generation with adaptive multimodal control. arXiv preprint arXiv:2503.14492 , 2025

    work page arXiv 2025

  45. [45]

    Liang, R

    J. Liang, R. Liu, E. Ozguroglu, S. Sudhakar, A. Dave, P. Tokmakov, S. Song, and C. V ondrick. Dreamitate: Real- world visuomotor policy learning via video generation. In 8th Annual Conference on Robot Learning , 2024. URL https://openreview.net/forum?id=InT87E5sr4

    work page 2024

  46. [46]

    Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation

    H. Bharadhwaj, D. Dwibedi, A. Gupta, S. Tulsiani, C. Doersch, T. Xiao, D. Shah, F. Xia, D. Sadigh, and S. Kir- mani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16283, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    C. Luo, Z. Zeng, Y . Du, and C. Sun. Solving new tasks by adapting internet video knowledge. In The Thirteenth International Conference on Learning Representations , 2025

    work page 2025

  48. [48]

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  49. [49]

    GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

    C.-L. Cheang, G. Chen, Y . Jing, T. Kong, H. Li, Y . Li, Y . Liu, H. Wu, J. Xu, Y . Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  50. [50]

    Y . Guo, Y . Hu, J. Zhang, Y .-J. Wang, X. Chen, C. Lu, and J. Chen. Prediction with action: Visual policy learning via joint denoising process. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    work page 2024

  51. [51]

    S. Li, Y . Gao, D. Sadigh, and S. Song. Unified video action model. arXiv preprint arXiv:2503.00200, 2025. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  52. [52]

    C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. arXiv preprint arXiv:2504.02792, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  53. [53]

    Q. Zhao, Y . Lu, M. J. Kim, Z. Fu, Z. Zhang, Y . Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020, 2025

    work page internal anchor Pith review arXiv 2025

  54. [54]

    McCarthy, D

    R. McCarthy, D. C. Tan, D. Schmidt, F. Acero, N. Herr, Y . Du, T. G. Thuruthel, and Z. Li. Towards generalist robot learning from internet video: A survey. arXiv preprint arXiv:2404.19664, 2024

    work page arXiv 2024

  55. [55]

    Grauman, A

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    work page 2022

  56. [56]

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  57. [57]

    Dasari, M

    S. Dasari, M. K. Srirama, U. Jain, and A. Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning, 2023

    work page 2023

  58. [58]

    J. Zeng, Q. Bu, B. Wang, W. Xia, L. Chen, H. Dong, H. Song, D. Wang, D. Hu, P. Luo, et al. Learning manipulation by predicting interaction. arXiv preprint arXiv:2406.00439, 2024

    work page arXiv 2024

  59. [59]

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile represen- tation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023

    work page 2023

  60. [60]

    Kannan, K

    A. Kannan, K. Shaw, S. Bahl, P. Mannam, and D. Pathak. Deft: Dexterous fine-tuning for real-world hand policies. arXiv preprint arXiv:2310.19797, 2023

    work page arXiv 2023

  61. [61]

    M. K. Srirama, S. Dasari, S. Bahl, and A. Gupta. Hrp: Human affordances for robotic pre-training.arXiv preprint arXiv:2407.18911, 2024

    work page arXiv 2024

  62. [62]

    K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, 2023

    work page 2023

  63. [63]

    C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y . Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023

    work page internal anchor Pith review arXiv 2023

  64. [64]

    Bharadhwaj, R

    H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint arXiv:2405.01527, 2024

    work page arXiv 2024

  65. [65]

    C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y . Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023

    work page arXiv 2023

  66. [66]

    Y . Zhu, A. Lim, P. Stone, and Y . Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint arXiv:2405.20321, 2024

    work page arXiv 2024

  67. [67]

    Bharadhwaj, A

    H. Bharadhwaj, A. Gupta, S. Tulsiani, and V . Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011, 2023

    work page arXiv 2023

  68. [68]

    J. Ye, J. Wang, B. Huang, Y . Qin, and X. Wang. Learning continuous grasping function with a dexterous hand from human demonstrations. IEEE Robotics and Automation Letters , 8(5):2882–2889, 2023

    work page 2023

  69. [69]

    Qin, Y .-H

    Y . Qin, Y .-H. Wu, S. Liu, H. Jiang, R. Yang, Y . Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, 2022

    work page 2022

  70. [70]

    Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, and Nicolas Ballas

    J. Yang, Z.-a. Cao, C. Deng, R. Antonova, S. Song, and J. Bohg. Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01479, 2024

    work page arXiv 2024

  71. [71]

    Bubeck, S., Chandrasekaran, V ., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y

    J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y . Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y . Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rockt¨aschel. Genie: Generative interactive environ- ments, 2024. URL https...

    work page arXiv 2024

  72. [72]

    Y . Chen, Y . Ge, W. Tang, Y . Li, Y . Ge, M. Ding, Y . Shan, and X. Liu. Moto: Latent motion token as the bridging language for learning robot manipulation from videos, 2025. URL https://arxiv.org/abs/2412.04445

    work page arXiv 2025

  73. [73]

    Schmidt and M

    D. Schmidt and M. Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=rvUq3cxpDF

    work page 2024

  74. [74]

    Z. Ren, Y . Wei, X. Guo, Y . Zhao, B. Kang, J. Feng, and X. Jin. Videoworld: Exploring knowledge learning from unlabeled videos, 2025. URL https://arxiv.org/abs/2501.09781

    work page arXiv 2025

  75. [75]

    Q. Bu, Y . Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  76. [76]

    S. Gao, S. Zhou, Y . Du, J. Zhang, and C. Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025

    work page arXiv 2025

  77. [77]

    Bansal, Y

    H. Bansal, Y . Bitton, I. Szpektor, K.-W. Chang, and A. Grover. Videocon: Robust video-language alignment via contrast captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13927–13937, 2024

    work page 2024

  78. [78]

    Use the right hand to pick up the plastic pitcher and pour water onto the green plant

    R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, and T. Wolf. Lerobot: Making ai for robotics more accessible with end-to-end learning, 2024. URL https://github.com/huggingface/lerobot. Accessed: 2025-04-30. 14 Table 3: LAPA Training Dataset Statistics Dataset Length (Frames) Duration (hr) FPS Category GR-1 Teleop Pre-Training 6.4M 88.4 20 Real robot DexMG...

    work page 2024

  79. [79]

    High Data

    We use a codebook size of 8 and a sequence length of 16 for vector quantization. We train 100K steps with a batch size of 1024. B Environment for Teleoperation and Evaluation We provide some sample images of the environment where we collected all of our GR1 humanoid teleoperation data in Figure 8 and all of the 10 environments where we conducted environme...

    work page