pith. sign in

arxiv: 2406.09246 · v3 · submitted 2024-06-13 · 💻 cs.RO · cs.LG

OpenVLA: An Open-Source Vision-Language-Action Model

Moo Jin Kim , Karl Pertsch , Siddharth Karamcheti , Ted Xiao , Ashwin Balakrishna , Suraj Nair , Rafael Rafailov , Ethan Foster
show 10 more authors
Grace Lam Pannag Sanketi Quan Vuong Thomas Kollar Benjamin Burchfiel Russ Tedrake Dorsa Sadigh Sergey Levine Percy Liang Chelsea Finn
This is my paper

Pith reviewed 2026-05-10 14:41 UTC · model grok-4.3

classification 💻 cs.RO cs.LG
keywords vision-language-action modelsrobot learningimitation learningopen-source roboticsgeneralist policiesfine-tuningmanipulation tasks
0
0 comments X

The pith

A 7B open-source vision-language-action model beats a 55B closed model by 16.5% on robot manipulation tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents OpenVLA, an open model that integrates language understanding, vision processing, and action generation for robot control. It is built by combining a Llama 2 language model with visual features from DINOv2 and SigLIP, then trained on 970k real-world robot demonstrations. This setup produces higher success rates than larger closed models across 29 tasks on multiple robot types. The model also supports efficient fine-tuning for new multi-object and language-based scenarios, outperforming from-scratch methods. The core goal is to make generalist robot policies publicly available and practical to adapt without proprietary resources.

Core claim

OpenVLA is a 7B-parameter vision-language-action model trained on a diverse set of 970k real robot demonstrations that outperforms closed models such as RT-2-X (55B) by 16.5% absolute task success rate on 29 tasks across embodiments while using 7x fewer parameters, and that can be fine-tuned effectively for new multi-task, multi-object settings with strong language grounding.

What carries the argument

A Llama 2 language model fused with pretrained DINOv2 and SigLIP visual encoders, trained end-to-end on diverse robot demonstration data to map visual-language inputs to actions.

If this is right

  • Generalist robot policies can be obtained by fine-tuning a single open model instead of training separate behaviors from scratch for each new task.
  • Researchers without access to closed models gain a practical starting point for visuomotor control that already handles diverse objects and language instructions.
  • Low-rank adaptation and quantization allow the model to run and adapt on consumer-grade GPUs without loss of downstream performance.
  • Strong results in multi-object and language-grounding settings suggest the approach scales to more complex instruction-following scenarios.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Widespread release of the model and training code could let the community extend the approach to additional robot hardware and longer-horizon tasks.
  • If the performance advantage holds on new domains, open VLAs may reduce dependence on proprietary training pipelines for practical robot deployment.
  • A natural next test would be whether the same architecture maintains its edge when applied to tasks requiring extended sequences of actions or novel object categories.

Load-bearing premise

The 29-task benchmark and reported fine-tuning results on multi-object and multi-language cases represent broader real-world performance without major distribution shifts or evaluation bias.

What would settle it

A new evaluation set of tasks or robot embodiments where OpenVLA's success rate falls below that of the larger closed models after the same fine-tuning protocol.

read the original abstract

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces OpenVLA, a 7B-parameter open-source vision-language-action model trained on 970k real-world robot demonstrations. It combines a Llama 2 language model with a visual encoder fusing DINOv2 and SigLIP features. The central claims are that OpenVLA outperforms the closed RT-2-X (55B) model by 16.5% absolute success rate across 29 tasks and multiple embodiments with 7x fewer parameters, that it can be effectively fine-tuned for new multi-object/multi-language settings while outperforming Diffusion Policy by 20.4%, and that it supports efficient consumer-GPU fine-tuning via LoRA and quantized inference. The authors release model checkpoints, fine-tuning notebooks, and PyTorch code with Open X-Embodiment support.

Significance. If the performance margins hold under matched conditions, the work is significant for providing the first publicly accessible large-scale VLA, directly addressing the closed nature of prior models like RT-2-X. The open release of code, checkpoints, and scalable training support on Open X-Embodiment datasets is a concrete strength that could enable broader community experimentation. The demonstration of practical fine-tuning and serving efficiency further supports potential impact on generalist robot policy research.

major comments (2)
  1. [Abstract and Evaluation section] Abstract and Evaluation section: The headline claim of a 16.5% absolute success-rate gain over RT-2-X (55B) across 29 tasks is load-bearing for the assertion of superior generalist manipulation. The manuscript does not include an explicit side-by-side task list, confirmation of identical success criteria, or verification that RT-2-X was re-run on the same protocol and embodiments rather than citing prior reports; without this, differences in task overlap, embodiment factors (e.g., gripper or camera views), or selection bias cannot be ruled out.
  2. [Fine-tuning and generalization experiments (likely §5)] Fine-tuning and generalization experiments (likely §5): The reported 20.4% improvement over Diffusion Policy and strong results in multi-object/multi-language settings rest on fine-tuning evaluations whose details—exact trial counts per task, statistical tests, data splits, and embodiment balancing—are not fully specified. This weakens assessment of whether the gains reflect robust generalization or evaluation-specific factors.
minor comments (2)
  1. [Methods section] The abstract states training on '970k real-world robot demonstrations' but the methods section should more explicitly tabulate the per-embodiment breakdown and any filtering steps from the Open X-Embodiment corpus.
  2. [Figures] Figure captions for qualitative results could clarify the exact robot platforms and camera views shown to aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which has helped us improve the clarity and completeness of our manuscript. We address each of the major comments below and have made corresponding revisions to the paper.

read point-by-point responses

  1. Referee: Abstract and Evaluation section: The headline claim of a 16.5% absolute success-rate gain over RT-2-X (55B) across 29 tasks is load-bearing for the assertion of superior generalist manipulation. The manuscript does not include an explicit side-by-side task list, confirmation of identical success criteria, or verification that RT-2-X was re-run on the same protocol and embodiments rather than citing prior reports; without this, differences in task overlap, embodiment factors (e.g., gripper or camera views), or selection bias cannot be ruled out.

    Authors: We appreciate this observation and agree that explicit documentation of the evaluation protocol is essential. In the revised manuscript, we have included a new table in the Evaluation section that provides a side-by-side comparison of the 29 tasks, including task names, success criteria, robot embodiments, and camera configurations. We confirm that these match the protocol and metrics used in the RT-2-X paper exactly, with results for RT-2-X taken directly from their published reports since the model is closed-source and not available for re-running. This ensures no discrepancies in task overlap or embodiment factors. The tasks were selected as a representative set from the Open X-Embodiment benchmark to evaluate generalist capabilities, mitigating concerns of selection bias. revision: yes

  2. Referee: Fine-tuning and generalization experiments (likely §5): The reported 20.4% improvement over Diffusion Policy and strong results in multi-object/multi-language settings rest on fine-tuning evaluations whose details—exact trial counts per task, statistical tests, data splits, and embodiment balancing—are not fully specified. This weakens assessment of whether the gains reflect robust generalization or evaluation-specific factors.

    Authors: We thank the referee for highlighting the need for additional experimental details. In the updated manuscript, we have expanded the fine-tuning section to specify the number of evaluation trials per task (ranging from 20 to 100 based on task complexity), the use of statistical tests such as Wilcoxon signed-rank tests to confirm significance of the 20.4% improvement, the data split methodology (e.g., held-out tasks and objects), and how multi-embodiment data was balanced during fine-tuning. These details support that the performance gains demonstrate robust generalization in multi-object and multi-language scenarios. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical performance claims rest on external benchmarks and independent prior models

full rationale

The paper describes training OpenVLA on 970k external robot demonstrations using a Llama 2 backbone fused with DINOv2 and SigLIP encoders, then reports direct task success rates on 29 tasks and comparisons to RT-2-X and Diffusion Policy. No equations, uniqueness theorems, or first-principles derivations are presented that could reduce to self-defined quantities or fitted parameters by construction. All performance numbers are obtained from held-out evaluation protocols on standard benchmarks; the architecture and data mixture are explicitly described as combinations of publicly available components rather than internally fitted constructs. Self-citations, if present, support only background methods and do not bear the load of the central empirical claims.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on empirical training outcomes and benchmark comparisons rather than new theoretical derivations or invented constructs.

axioms (1)
  • domain assumption Pretrained DINOv2 and SigLIP visual features can be effectively fused with a Llama 2 language model for action prediction in robotics
    Invoked when describing the model architecture and its expected benefits.

pith-pipeline@v0.9.0 · 5711 in / 1374 out tokens · 48881 ms · 2026-05-10T14:41:44.681348+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

    cs.RO 2026-06 unverdicted novelty 8.0

    HABIT is a large-scale robot demonstration dataset for human-present environments that elicits spatiotemporal synchronization, yielding, and gesture grounding behaviors absent from robot-only training data.

  2. Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

    cs.CV 2026-05 accept novelty 8.0

    Introduces the TVR active viewpoint-matching task and TVRBench indoor simulation benchmark, where foundation models start at low single-digit success rates but reach 51.4% after visual-action SFT and multi-turn GRPO p...

  3. Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics

    cs.RO 2026-04 conditional novelty 8.0

    Open-H-Embodiment is the largest open multi-embodiment medical robotics dataset, used to train GR00T-H, the first open vision-language-action model that achieves end-to-end suturing completion where prior models fail.

  4. FlowHijack: A Dynamics-Aware Backdoor Attack on Flow-Matching Vision-Language-Action Models

    cs.CV 2026-03 unverdicted novelty 8.0

    FlowHijack is the first dynamics-aware backdoor attack on flow-matching VLAs that achieves high success rates with stealthy triggers while preserving benign performance and making malicious actions kinematically indis...

  5. Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

    cs.RO 2026-07 unverdicted novelty 7.0

    Embodied.cpp introduces a portable C++ inference runtime with modular layers for deploying VLA and WAM models on heterogeneous robots, reporting 100% and 91% task success on two models plus memory reduction on a WAM b...

  6. LIME: Learning Intent-aware Camera Motion from Egocentric Video

    cs.RO 2026-07 unverdicted novelty 7.0

    LIME formulates language-conditioned camera motion as predicting SE(3) target poses from RGB and intent text, using mined multi-intent supervision from egocentric video and a flow-matching pose head.

  7. EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

    cs.CV 2026-06 unverdicted novelty 7.0

    EgoSafetyBench shows VLMs reliably spot hazard-containing videos but miss specific contextual hazards and are degraded by misleading in-scene text.

  8. Adapting Generalist Robot Policies with Semantic Reinforcement Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    SARL optimizes language prompt inputs to generalist vision-language-action policies through online RL to solve complex long-horizon tasks by composing existing skills.

  9. Revisiting Parameter Redundancy in Vision-Language-Action Models: Insights from VLM-to-VLA Adaptation

    cs.RO 2026-06 unverdicted novelty 7.0

    VLA models from VLM adaptation can be pruned 12-30% via multi-module joint scheme based on divergence signals while keeping ~90% performance on LIBERO without post-pruning recovery, unlike standard criteria that collapse.

  10. Labimus: A Simulation and Benchmark for Humanoid Dexterous Manipulation in Chemical Laboratory

    cs.RO 2026-06 unverdicted novelty 7.0

    Labimus is the first benchmark for humanoid dexterous manipulation in organic chemistry laboratories, exposing a gap between task completion and required experimental precision.

  11. Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation

    cs.RO 2026-06 unverdicted novelty 7.0

    SWAM jointly generates intermediate RGB-D sequences and action trajectories from monocular RGB start/goal observations for embodied navigation.

  12. SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics

    cs.AI 2026-06 unverdicted novelty 7.0

    SurgVLA-Bench supplies a hierarchical task taxonomy and multi-dimensional evaluation framework for VLA models in laparoscopic robotics simulation, showing autoregressive models excel at semantics while flow-matching m...

  13. The Speedup Paradox: Rethinking Inference Speed-Quality Trade-off in Embodied Tasks

    cs.RO 2026-06 unverdicted novelty 7.0

    TISED framework reveals paradoxical effects where inference optimizations can lengthen task completion time on static tasks or raise success rates on dynamic tasks in embodied AI.

  14. World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays

    cs.RO 2026-06 unverdicted novelty 7.0

    REGEN uses recurrent generative replays from World Action Models to cut catastrophic forgetting by up to 50% in continual imitation learning compared to sequential fine-tuning.

  15. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    ForesightSafety-VLA creates a diagnostic benchmark for VLA safety with taxonomy across physical, language, and visual risks, showing perception and structure variations cause more safety degradation than language chan...

  16. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    ForesightSafety-VLA is a new benchmark with 13 safety categories, cumulative cost and risk exposure metrics, and controlled variations to diagnose safety failures in VLA models rather than aggregate task success.

  17. USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning

    cs.CV 2026-06 unverdicted novelty 7.0

    USS is an end-to-end framework for embodied visual tracking that fuses text, point, box, and mask prompts via modality-specific encoders and hybrid attention, augmented by a latent world model, and demonstrates higher...

  18. LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    LIBERO-Safety supplies a scalable benchmark, data-generation pipeline, and 19,664-demonstration dataset that exposes a generalization-safety tension in current VLA models where diverse training improves collision avoi...

  19. Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLA

    cs.RO 2026-06 unverdicted novelty 7.0

    Masking the end-effector from wrist views during training lets a single-gripper VLA transfer zero-shot to other grippers, arms, and five-fingered hands while keeping original performance.

  20. Geometric Entropy: When Trajectory Diversity Helps and Hurts in Imitation Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    Geometric diversity of demonstration trajectories exhibits an inverted-U effect on imitation learning success, with the peak shifting lower as mastery increases via more data, easier tasks, or stronger priors.

  21. HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

    cs.CV 2026-06 unverdicted novelty 7.0

    Processed egocentric human video outperforms teleoperated real-robot trajectories as pretraining data for embodied foundation models, delivering 24% lower validation loss and 52.5-90% higher task success rates under m...

  22. Frequency-Aware Flow Matching for Continuous and Consistent Robotic Action Generation

    cs.RO 2026-06 unverdicted novelty 7.0

    FAFM performs flow matching in the frequency domain using DCT on action sequences to produce continuous temporally consistent robotic actions with a Sobolev-style smoothness regularizer.

  23. EquiVLA: A General Framework for Rotationally Equivariant Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    EquiVLA is the first general framework for end-to-end SO(2)-equivariant VLA models using EquiPerceptor and EquiActor modules, reporting improved success rates on LIBERO, CALVIN, and real-robot benchmarks.

  24. Start Right, Arrive Right: Asynchronous Execution via Initial Noise Selection

    cs.RO 2026-06 unverdicted novelty 7.0

    PAINT reframes asynchronous flow-based action chunking as an initial noise selection problem solved via backward Euler inversion and a repainting rule.

  25. Mix-QVLA: Task-Evidence-Aware Mixed-Precision Quantization of Vision-Language-Action Models

    cs.CV 2026-06 unverdicted novelty 7.0

    Mix-QVLA is a task-evidence-aware mixed-precision PTQ framework for VLA models that preserves task-relevant evidence via evidence-mass and attribution-distribution metrics to guide bit allocation under memory and BitO...

  26. Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

    cs.LG 2026-06 unverdicted novelty 7.0

    Act2Answer protocol reveals VLA models retain simple concepts but show larger gaps on complex semantics than source VLMs, with VQA co-training linked to better retention and knowledge signals peaking in middle layers.

  27. EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    EBench is a benchmark that evaluates generalist mobile manipulation policies on 26 tasks across 5 capability and 4 generalization dimensions, revealing distinct capability profiles among models with similar success rates.

  28. PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Space

    cs.RO 2026-06 unverdicted novelty 7.0

    PearlVLA achieves SOTA on LIBERO by separating VLM representations into visual grounding and an iterative latent plan branch refined via world model queries and RefineNet with process-reward RL.

  29. HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    HumanoidArena is a new benchmark of 7 leg-critical HOI/HSI tasks that evaluates egocentric hierarchical whole-body policies in humanoids and finds performance is strongly conditioned on the low-level GMT used.

  30. MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 7.0

    MuseVLA adds on-demand sensor selection via tokens and converts readings into grounded sensor images for multimodal fusion, reporting 80.6% average success on real-robot dexterous tasks that need non-visual sensing.

  31. FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation

    cs.RO 2026-06 unverdicted novelty 7.0

    FTP-1 is the first foundation tactile policy pretrained on ~3000 hours of data from 26 sources across 21 sensors that improves performance on seen setups by 17.2% and transfers to unseen sensors with 31% success rate gain.

  32. Trajectory-Level Redirection Attacks on Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    A prompt-only attack called command-preserving trajectory redirection can steer VLA robot behavior to attacker-chosen physical outcomes while the text still appears to match the intended task.

  33. Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Robotics

    cs.RO 2026-06 unverdicted novelty 7.0

    Ambient Diffusion Policy enables better imitation learning from suboptimal robot data by leveraging spectral properties to restrict data usage to specific diffusion times.

  34. Learning Object Manipulation from Scratch via Contrastive Interaction

    cs.RO 2026-06 unverdicted novelty 7.0

    IWR improves CRL sample efficiency and performance in interaction-rich manipulation by interaction-aware resampling that preserves mode boundaries, yielding 19.8% average gains and a real-world air-hockey agent.

  35. LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination

    cs.CV 2026-06 unverdicted novelty 7.0

    Introduces LIBERO-Occ benchmark showing VLA performance drop under occlusion and Viewpoint Imagination method that generates complementary views to improve robustness without extra hardware.

  36. ProbeAct: Probe-Guided Training-Free Failure Recovery in Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    PROBEACT is a plug-and-play intervention framework that combines hidden-state probing, kinematic failure detection, and CBF-based correction to boost success rates of pre-trained VLA models on the LIBERO-plus benchmar...

  37. ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    ReCoVLA improves VLA policy reliability by using a VLM as a semantic reward selector to train residual recovery policies in simulation, raising average success from 36.7% to 66.7% in sim and achieving 61.7% in zero-sh...

  38. Targeting World Models to Compromise Robot Learning Pipelines

    cs.RO 2026-06 unverdicted novelty 7.0

    World models introduce a stealthy poisoning vector into robot learning pipelines where malicious prompts or dynamics in teleoperated data activate only during synthetic trajectory generation, enabling backdoors in dow...

  39. Back to the Familiar Future: Failure Recovery for VLA Policies via Pre-Imagined Milestone Selection

    cs.RO 2026-06 unverdicted novelty 7.0

    B2FF pre-generates a milestone bank of familiar future states from the clean initial observation and uses a recoverability-aware selector to guide VLA policies back from deviations, raising average success rate from 5...

  40. PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback

    cs.RO 2026-06 unverdicted novelty 7.0

    PhysAgent is a simulator-in-the-loop multi-agent system that automates physically grounded 4D synthesis from multimodal prompts by using trajectory feedback from vision models and LLM reasoning to optimize force fields.

  41. X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining

    cs.CV 2026-06 unverdicted novelty 7.0

    X-Tokenizer creates semantic action tokens via asymmetric residual quantization and contrastive pretraining on large trajectory data, outperforming prior methods like FAST on robotic tasks.

  42. ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    ActProbe is an action-space detector that uses temporal consistency error and action chunk magnitude from policy outputs, mapped via LSTM-MLP, to predict failures earlier than baselines across policies and real-robot tasks.

  43. Q-VGM: Q-Guided Value-Gradient Matching for Flow-Matching VLA Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    Q-VGM introduces value-gradient matching via VGG-Flow to improve flow-matching VLA policies with a Cal-QL critic, achieving success rate lifts on LIBERO, RoboTwin, and real-robot tasks.

  44. ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?

    cs.CV 2026-06 unverdicted novelty 7.0

    ChronoPhyBench is a new benchmark and dataset for chronological physical dynamics reasoning that combines video-conditioned next-state prediction with VQA to reduce language bias in MLLM evaluation.

  45. VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation

    cs.RO 2026-06 unverdicted novelty 7.0

    VoLoAgent uses a VLM to steer heterogeneous robot capabilities as interruptible tools for long-horizon manipulation and introduces the RoboVoLo benchmark, claiming substantial outperformance over single VLA/VLM or too...

  46. PiL-World: A Chunk-Wise World Model for VLA Policy-in-the-Loop Evaluation

    cs.RO 2026-06 unverdicted novelty 7.0

    PiL-World introduces a chunk-wise world model for closed-loop VLA policy evaluation that reduces the gap between simulated and real success rates from 63.2% to 12.0% on three dual-arm manipulation tasks by conditionin...

  47. HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning

    cs.RO 2026-06 unverdicted novelty 7.0

    HapTile introduces a visuotactile dataset with haptic-informed teleoperation for language-conditioned contact-rich manipulation tasks and provides baseline policy benchmarks.

  48. NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0

    NextMotionQA benchmark reveals VLMs have critical gaps in fine-grained human motion understanding and align with experts on coarse judgment (κ=0.70) but not fine-grained (κ=0.10).

  49. A Dataset for Dynamic Human Preferences for Vision Language Models

    cs.CV 2026-06 unverdicted novelty 7.0

    Introduces a benchmark dataset with automated pipeline for evaluating VLMs on dynamic in-context human preferences, distinct from static benchmarks.

  50. Same Weights, Different Robot: A Deployment Safety View of VLA Policies

    cs.CR 2026-06 unverdicted novelty 7.0

    The paper identifies a deployment safety gap in VLA policies where identical checkpoints can be executable-inequivalent due to action metadata mismatches, supported by a derived closed-form transform and empirical dri...

  51. BOKBO (Best of K Bad Options): Calibrated Abstention for VLA Policies

    cs.LG 2026-05 unverdicted novelty 7.0

    BOKBO is the first conformal abstention method for K-sample VLA policies that supplies finite-sample distribution-free guarantees on executed violation rates, with global and Mondrian per-task variants.

  52. PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology

    cs.RO 2026-05 unverdicted novelty 7.0

    PhAIL provides an open benchmark and distributional evaluation method for real-robot VLA policies using time-to-success CDF, HRT scoring, and KS significance tests.

  53. {\Omega}-QVLA: Robust Quantization for Vision-Language-Action Models via Composite Rotation and Per-step Scaling

    cs.CV 2026-05 unverdicted novelty 7.0

    Omega-QVLA is a post-training quantization framework achieving uniform W4A4 for VLA models' LLM backbone and DiT action head via composite SVD-Hadamard rotation and per-step scaling, matching FP16 success rates on LIBERO.

  54. How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures

    cs.RO 2026-05 unverdicted novelty 7.0

    VLA architectures exhibit architecture-specific failure signatures at the motor-command level, with direction reversal as a universal predictor and velocity monitoring ineffective for continuous models.

  55. Colosseum V2: Benchmarking Generalization for Vision Language Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    Introduces Colosseum V2 benchmark for evaluating VLA model generalization in robotic manipulation with 28 tasks, revealing limitations in current methods and sim-real correlations.

  56. What-If World: A Causal Benchmark for General World Models in Embodied Scenarios

    cs.CV 2026-05 unverdicted novelty 7.0

    What-If World is a new paired-prompt benchmark showing that nine state-of-the-art video generation models achieve at most 52% on causal intervention tests and cluster near 28% for open-source systems.

  57. Sentinel: Embodied Cooperative Spatial Reasoning and Planning

    cs.CV 2026-05 unverdicted novelty 7.0

    Introduces Sentinel Challenge benchmark and CoSaR framework for cooperative spatial reasoning and planning among 3-5 decentralized embodied agents across 14 city-scale scenes.

  58. Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence

    cs.CV 2026-05 unverdicted novelty 7.0

    GAMSI is a dual-pathway Geometry-Aware MLLM using Metric-Structure Decoupled Queries and Expert-Guided Visual Grounding on RGB inputs alone, trained on a new 152k-sample MTS dataset to reach SOTA on seven spatial benchmarks.

  59. Point Tracking Improves World Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    JOPAT jointly models pixels, point tracks, and actions in a diffusion transformer and reports gains over pixel-only baselines on long-horizon robot tasks with occlusion and off-screen motion.

  60. VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

    cs.CV 2026-05 unverdicted novelty 7.0

    VGenST-Bench is a new video benchmark for MLLM spatio-temporal reasoning built via generative synthesis, a multi-agent pipeline with human oversight, a 3x2x2 taxonomy, and hierarchical tasks separating perception from...

Reference graph

Works this paper leans on

169 extracted references · 169 canonical work pages · cited by 591 Pith papers · 25 internal anchors

  1. [1]

    Open X-Embodiment: Robotic Learning Datasets and RT-X Models

    Open X-Embodiment Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. Fu, C. Devin, D. Driess, D. Pathak, D. Shah, D. Büchler, D. Kalashnikov, ...

    work page internal anchor Pith review arXiv 2023

  2. [3]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023

    work page 2023

  3. [4]

    A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint arXiv:2307.03659, 2023

    work page arXiv 2023

  4. [5]

    Ghosh, H

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https://octo-models.github.io, 2023

    work page 2023

  5. [6]

    Walke, K

    H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, V . Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale, 2023

    work page 2023

  6. [7]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y . Lu, H. Michalewski, I. Mordatch, K. Pe...

    work page internal anchor Pith review arXiv 2023

  7. [8]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , volume 139, pages 8748–8763, 2021

    work page 2021

  8. [9]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In International Conference on Computer Vision (ICCV), 2023

    work page 2023

  9. [10]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [11]

    Khazatsky, K

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y . J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y . Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. ...

    work page 2024

  11. [12]

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual representa- tion for robot manipulation. In CoRL, 2022

    work page 2022

  12. [13]

    Karamcheti, S

    S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. ArXiv, abs/2302.12766, 2023. URL https: //api.semanticscholar.org/CorpusID:257205716

    work page arXiv 2023

  13. [14]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894–906. PMLR, 2022

    work page 2022

  14. [15]

    arXiv preprint arXiv:2303.00905 (2023) 1

    A. Stone, T. Xiao, Y . Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023

    work page arXiv 2023

  15. [16]

    PaLM-E: An Embodied Multimodal Language Model

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    work page internal anchor Pith review arXiv 2023

  16. [17]

    A. S. et al. Introducing rfm-1: Giving robots human-like reason- ing capabilities, 2024. URL https://covariant.ai/insights/ introducing-rfm-1-giving-robots-human-like-reasoning-capabilities/

    work page 2024

  17. [18]

    Lingo-2: Driving with natural language

    Wayve. Lingo-2: Driving with natural language. 2024. URLhttps://wayve.ai/thinking/ lingo-2-driving-with-language/

    work page 2024

  18. [19]

    X. Chen, X. Wang, S. Changpinyo, A. J. Piergiovanni, P. Padlewski, D. M. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. V . Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. Riquelme, A. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. Pa...

    work page internal anchor Pith review arXiv 2022

  19. [20]

    X. Chen, X. Wang, L. Beyer, A. Kolesnikov, J. Wu, P. V oigtlaender, B. Mustafa, S. Goodman, I. M. Alabdulmohsin, P. Padlewski, D. M. Salz, X. Xiong, D. Vlasic, F. Pavetic, K. Rong, T. Yu, D. Keysers, X.-Q. Zhai, and R. Soricut. PaLI-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023

    work page arXiv 2023

  20. [21]

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, and ... Transformers: State-of-the-art natural language processing. In Proceedings of the 6th International Conference on Learning Representations,

    work page

  21. [22]

    URL https://arxiv.org/abs/1910.03771

    work page internal anchor Pith review arXiv 1910

  22. [23]

    LLaMA: Open and Efficient Foundation Language Models

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [24]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [26]

    DINOv2: Learning Robust Visual Features without Supervision

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [27]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  26. [28]

    Dettmers, A

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  27. [29]

    Goyal, T

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Computer Vision and Pattern Recognition (CVPR), 2017

    work page 2017

  28. [30]

    D. A. Hudson and C. D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Computer Vision and Pattern Recognition (CVPR) , 2019

    work page 2019

  29. [31]

    Singh, V

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards VQA models that can read. In Computer Vision and Pattern Recognition (CVPR), 2019

    work page 2019

  30. [32]

    J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh. VizWiz: nearly real-time answers to visual questions. In User Interface Software and Technology (UIST), pages 333–342, 2010

    work page 2010

  31. [33]

    Kazemzadeh, V

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg. ReferItGame: Referring to objects in photographs of natural scenes. In Empirical Methods in Natural Language Processing (EMNLP), pages 787–798, 2014

    work page 2014

  32. [34]

    L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision (ECCV), 2016

    work page 2016

  33. [35]

    Gemma: Open Models Based on Gemini Research and Technology

    T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid...

    work page internal anchor Pith review arXiv 2024

  34. [36]

    Y . Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y . T. Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023

    work page internal anchor Pith review arXiv 2023

  35. [37]

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  36. [38]

    J. Li, D. Li, C. Xiong, and S. C. H. Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), 2022

    work page 2022

  37. [39]

    J. Li, D. Li, S. Savarese, and S. C. H. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), 2023

    work page 2023

  38. [40]

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. A. Li, P. Fung, and S. C. H. Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023

    work page internal anchor Pith review arXiv 2023

  39. [41]

    H. H. Tan and M. Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Empirical Methods in Natural Language Processing (EMNLP), 2019

    work page 2019

  40. [42]

    Laurençon, L

    H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karam- cheti, A. M. Rush, D. Kiela, M. Cord, and V . Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), 2023

    work page 2023

  41. [43]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  42. [44]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    work page internal anchor Pith review arXiv 2023

  43. [45]

    Karamcheti, S

    S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865, 2024

    work page arXiv 2024

  44. [46]

    QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhoucke, et al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018

    work page Pith review arXiv 2018

  45. [47]

    Kalashnkov, J

    D. Kalashnkov, J. Varley, Y . Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021

    work page 2021

  46. [48]

    Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

    F. Ebert, Y . Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396, 2021. 15

    work page internal anchor Pith review arXiv 2021

  47. [49]

    Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world

    K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, K. P. Singh, Y . Kim, W. Han, A. Herrasti, et al. Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. arXiv preprint arXiv:2312.02976, 2023

    work page arXiv 2023

  48. [50]

    Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023.https://arxiv.org/abs/2309.01918

    H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V . Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918, 2023

    work page arXiv 2023

  49. [51]

    Pinto and A

    L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406–3413. IEEE, 2016

    work page 2016

  50. [52]

    Mandlekar, Y

    A. Mandlekar, Y . Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893. PMLR, 2018

    work page 2018

  51. [53]

    Gupta, A

    A. Gupta, A. Murali, D. P. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018

    work page 2018

  52. [54]

    Dasari, F

    S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning. CoRL, 2019

    work page 2019

  53. [55]

    S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y . Aytar, D. Budden, M. Vecerik, O. Sushkov, D. Barker, J. Scholz, M. Denil, N. de Freitas, and Z. Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. RSS, 2019

    work page 2019

  54. [56]

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022

    work page 2022

  55. [57]

    H.-S. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 3:5, 2023

    work page 2023

  56. [58]

    Devin, A

    C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural net- work policies for multi-task and multi-robot transfer. In Proceedings of IEEE International Conference on Robotics and Automation, 2017

    work page 2017

  57. [59]

    E. S. Hu, K. Huang, O. Rybkin, and D. Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. In International Conference on Learning Representations, 2022

    work page 2022

  58. [60]

    J. H. Yang, D. Sadigh, and C. Finn. Polybot: Training one policy across robots while embracing variability. In 7th Annual Conference on Robot Learning , 2023. URL https: //openreview.net/forum?id=HEIRj51lcS

    work page 2023

  59. [61]

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y . Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y . Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent.Transactions on Machine Learning Research, 2022. ISSN 2835-8856

    work page 2022

  60. [62]

    Bridging action space mismatch in learning from demonstra- tions,

    G. Salhotra, I.-C. A. Liu, and G. Sukhatme. Bridging action space mismatch in learning from demonstrations. arXiv preprint arXiv:2304.03833, 2023

    work page arXiv 2023

  61. [63]

    Radosavovic, B

    I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and J. Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning, 2023. 16

    work page 2023

  62. [64]

    D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7226–7233. IEEE, 2023

    work page 2023

  63. [65]

    Bousmalis, G

    K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Y . Zhou, A. Gupta, A. Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706, 2023

    work page arXiv 2023

  64. [66]

    D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning, 2023. URL https://arxiv.org/abs/2306.14846

    work page arXiv 2023

  65. [67]

    J. Yang, C. Glossop, A. Bhorkar, D. Shah, Q. Vuong, C. Finn, D. Sadigh, and S. Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402.19432, 2024

    work page arXiv 2024

  66. [68]

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23171–23181, 2023

    work page 2023

  67. [69]

    Y . Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280, 2023

    work page arXiv 2023

  68. [70]

    Y . J. Ma, V . Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image represen- tations and rewards for robotic control. In International Conference on Machine Learning, pages 23301–23320. PMLR, 2023

    work page 2023

  69. [71]

    Grounding classical task planners via vision-language models.arXiv preprint arXiv:2304.08587,

    X. Zhang, Y . Ding, S. Amiri, H. Yang, A. Kaminski, C. Esselink, and S. Zhang. Grounding classical task planners via vision-language models. arXiv preprint arXiv:2304.08587, 2023

    work page arXiv 2023

  70. [72]

    Sontakke, J

    S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti. Roboclip: One demonstration is enough to learn robot policies.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  71. [73]

    Huang, S

    J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y . Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024

    work page 2024

  72. [74]

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023

    work page internal anchor Pith review arXiv 2023

  73. [75]

    H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y . Du, Y . Hong, and C. Gan. 3d-vla: 3d vision- language-action generative world model. arXiv preprint arXiv:2403.09631, 2024

    work page internal anchor Pith review arXiv 2024

  74. [76]

    Automatic mixed precision

    PyTorch. Automatic mixed precision. URL https://pytorch.org/docs/stable/amp. html

    work page

  75. [77]

    T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    work page internal anchor Pith review arXiv 2023

  76. [78]

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023

    work page internal anchor Pith review arXiv 2023

  77. [79]

    Dorka, C

    N. Dorka, C. Huang, T. Welschehold, and W. Burgard. What matters in employing vision language models for tokenizing actions in robot control? InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024. 17

    work page 2024

  78. [80]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11975–11986, 2023

    work page 2023

  79. [81]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    work page 2021

  80. [82]

    Sharma, N

    P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556–2565, 2018

    work page 2018

Showing first 80 references.