pith. sign in

arxiv: 2605.20992 · v3 · pith:E5BKUHI4new · submitted 2026-05-20 · 💻 cs.CV

CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction

Hao Xu , Yilin Liu , Yinqiao Wang , Chi-Wing Fu , Niloy J. Mitra This is my paper

Pith reviewed 2026-06-30 17:12 UTC · model grok-4.3

classification 💻 cs.CV
keywords hand-object interaction4D reconstructionmonocular videocontact modelingpose estimationshape reconstructionjoint optimization
0
0 comments X

The pith

CHOIR reconstructs 4D hand-object interactions from monocular videos by treating contact as an explicit coupling signal between hands and objects.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents CHOIR as a pipeline that converts everyday monocular videos into reusable 4D interaction models consisting of articulated hand motion, object shape and 6D pose over time, and contact events. It begins with a coarse contact-agnostic initialization drawn from open-world visual priors, then applies a generative rectification step that predicts ray-depth corrections to adjust relative hand-object placement and establish initial contact correspondences. A final contact-aware joint optimization stage enforces geometric, temporal, and contact consistency through dynamically updated constraints. The central motivation is that separate hand and object estimates frequently misalign under occlusion, clutter, and unseen geometries, and explicit contact modeling can mitigate those errors. Experiments on both controlled and challenging videos are reported to show gains in object reconstruction quality, physical plausibility, and temporal coherence relative to prior methods.

Core claim

CHOIR reconstructs articulated hand motion, object shape with 6D pose over time, and contact events from monocular videos by first producing a coarse 4D HOI sequence, then using a generative HOI spatial rectification module to predict ray-depth corrections that rectify hand-object placement and yield initial per-frame contact correspondences, and finally running a contact-aware joint optimization that enforces geometric, temporal, and contact consistency via dynamically updated constraints.

What carries the argument

The generative HOI spatial rectification module, which predicts ray-depth corrections from a coarse initialization to rectify relative hand-object placement and derive initial contact correspondences.

If this is right

  • Object reconstruction quality improves because contact constraints anchor the object geometry to the hand surface.
  • Physical plausibility increases as the optimization penalizes interpenetration and enforces contact consistency across frames.
  • Temporal consistency is strengthened by propagating contact information through the joint optimization rather than relying on independent per-frame estimates.
  • The output 4D primitives become directly usable for downstream tasks such as scene-aware synthesis and planning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same contact-coupling idea could be applied to reconstruct interactions involving multiple objects or full-body motion without major architectural changes.
  • Success on open-world videos suggests the method might scale to large-scale mining of interaction data from internet footage for training generative models.
  • If the rectification step generalizes, the framework could serve as a test-time adaptation layer for existing hand and object trackers.

Load-bearing premise

The generative rectification module can produce sufficiently accurate ray-depth corrections from the coarse initialization so that the resulting contact correspondences remain reliable enough for the joint optimization to refine without propagating large errors.

What would settle it

Videos containing large initial hand-object misalignments where the rectification module produces incorrect ray-depth values, resulting in contact correspondences that cause the joint optimization to converge to visibly implausible or temporally inconsistent 4D reconstructions.

Figures

Figures reproduced from arXiv: 2605.20992 by Chi-Wing Fu, Hao Xu, Niloy J. Mitra, Yilin Liu, Yinqiao Wang.

Figure 1
Figure 1. Figure 1: From a monocular RGB video, CHOIR reconstructs 4D hand–object interactions (HOI) (including 3D hand motion, 3D object shape, 6D pose trajectory, and contact evidence) across open-world in-the-wild scenes. Heatmaps reveal the HOI contact information (see supplementary for videos). We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand moti… view at source ↗
Figure 2
Figure 2. Figure 2: Method overview. Stage 1: From a monocular video, we first obtain 2D HOI cues and initialize 3D hand/object reconstructions to form a coarse 4D HOI [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Problem definition: our goal is to reconstruct 4D HOI from a monoc￾ular RGB video of a complete atomic interaction that includes (i) hand approaching a static object; (ii) grasping and manipulating the object; and (iii) putting the object back. Active-manipulation-only clips are also allowed. (ii) 3D Spatial Ambiguity. Monocular depth and scale ambiguities, especially under hand-object occlusions, make rel… view at source ↗
Figure 4
Figure 4. Figure 4: Our flow-matching-based generative HOI spatial rectification. To mimic monocular ambiguity, we generate a corresponding noisy source grasp for each simulated hand grasping pose by in￾jecting anisotropic ray-aligned perturbations that emphasize depth uncertainty: large variance along the camera ray, mild in-plane noise, and anatomy-preserving perturbations in the MANO param￾eters. Each training pair thus co… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison with HOLD and MagicHOI on in-the-wild videos. image-space masks or object centers. (iv) For contact, our formula￾tion builds contact evidence on contact-bearing frames and updates soft contact constraints during optimization, so the active hand vertices can vary during the interaction. However, we do not handle arbitrary re-grasping or highly non-rigid object deformations. Ablation studies. We c… view at source ↗
Figure 6
Figure 6. Figure 6: Examples from our GraspPair dataset. Each case comprises a posed object point cloud with 4,096 points, a source hand pose (red), a target hand pose [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of Stage 2 rectification and contact correspondences on HO3D, TASTE-Rob, and self-captured videos. We compare the reconstruction before and after Stage 2, together with ground truths when available or the final optimized result otherwise. Self-captured w/o w/o w/o Stage 2 rectification w/o w/o Full Pipeline [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative ablation of core components in CHOIR against the full pipeline. The first row shows the camera view and the second row shows a side view. TASTE-Rob TASTE-Rob Self-captured [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visual results of our CHOIR on the challenging cases in TASTE-Rob and self-captured videos. Each case shows the input view, a novel-view rendering with the estimated contact map, and the rest-pose hand contact map. Temporal results are shown in videos on the supplementary webpage [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative comparison between CHOIR and state-of-the-art methods on HO3D. Red circles indicate regions of interest for comparing baseline reconstructions with ours. See videos on the supplementary webpage for temporal comparisons [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Representative failure cases. Most failures originate from Stage 1 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Visualization of Stage 1 2D assets. The figure shows modal object masks, amodal object masks, 2D hand and object bounding boxes, and 2D hand [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
read the original abstract

We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present CHOIR, a Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper introduces CHOIR, a contact-aware framework for 4D hand-object interaction reconstruction from monocular open-world videos. It proceeds in three stages: (1) coarse contact-agnostic 4D HOI initialization from visual priors, (2) a generative HOI spatial rectification module that predicts ray-depth corrections to rectify relative hand-object placement and derive initial per-frame contact correspondences, and (3) contact-aware joint optimization that enforces geometric, temporal, and contact consistency with dynamically updated constraints. Experiments on controlled and challenging videos are reported to show improvements in object reconstruction, physical plausibility, and temporal consistency over state-of-the-art baselines.

Significance. If the quantitative gains hold under rigorous verification, the work would advance monocular HOI reconstruction by explicitly coupling hands and objects via contact, enabling more scalable extraction of interaction primitives from in-the-wild video. The multi-stage design with rectification and dynamic constraints is a clear technical contribution over prior separate hand/object pipelines.

major comments (1)
  1. [Method (generative rectification)] Method section (generative HOI spatial rectification module): the central claim that the module produces sufficiently accurate ray-depth corrections from coarse initialization to yield reliable initial contact correspondences (enabling the subsequent joint optimization to improve reconstruction and plausibility on challenging videos) lacks independent verification. No quantitative evaluation of the rectification step's depth or contact accuracy against ground-truth depths or contacts is described, which is load-bearing for the error-propagation argument in the skeptic note.
minor comments (2)
  1. [Experiments] The abstract and experiments section should explicitly state the datasets, metrics (e.g., object reconstruction error, contact F1, temporal consistency measures), number of sequences, and whether error bars or statistical tests accompany the reported improvements over baselines.
  2. [Method] Notation for ray-depth corrections and contact correspondences should be formalized with equations in the method section for reproducibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment of the technical contributions and for the constructive comment on the generative rectification module. We address the point below and will strengthen the manuscript accordingly.

read point-by-point responses

  1. Referee: [Method (generative rectification)] Method section (generative HOI spatial rectification module): the central claim that the module produces sufficiently accurate ray-depth corrections from coarse initialization to yield reliable initial contact correspondences (enabling the subsequent joint optimization to improve reconstruction and plausibility on challenging videos) lacks independent verification. No quantitative evaluation of the rectification step's depth or contact accuracy against ground-truth depths or contacts is described, which is load-bearing for the error-propagation argument in the skeptic note.

    Authors: We agree that an independent quantitative evaluation of the rectification module would strengthen the error-propagation argument. The current manuscript reports only end-to-end results and qualitative rectification visualizations. In the revised version we will add a dedicated quantitative analysis of the rectification step on the controlled video sequences (where approximate ground-truth depth and contact can be derived from multi-view capture or synthetic rendering). This will include (i) ray-depth correction error (L1 and relative) before/after rectification, (ii) contact correspondence precision/recall against the derived ground truth, and (iii) an ablation showing that the subsequent joint optimization benefits from the rectified initialization. We will also clarify in the text that open-world videos lack pixel-perfect ground truth, which is why the primary evaluation remains end-to-end. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation is self-contained

full rationale

The described pipeline initializes from external open-world visual priors, applies a new generative rectification module to produce ray-depth corrections and initial contacts, then performs joint optimization with contact constraints. No equations or steps reduce by construction to fitted inputs, self-definitions, or self-citation chains; the method introduces distinct stages evaluated empirically against independent baselines rather than deriving outputs equivalent to inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Abstract-only review yields limited visibility into implementation details; the framework rests on the domain assumption that contact can serve as a reliable coupling signal and that visual priors suffice for coarse initialization.

axioms (2)
  • domain assumption Contact can serve as an explicit coupling signal between hands and objects in monocular video reconstruction
    Stated as the core targeting mechanism in the abstract.
  • domain assumption Open-world visual priors can initialize a usable coarse 4D HOI sequence
    Described as the first step of the pipeline.

pith-pipeline@v0.9.1-grok · 5765 in / 1308 out tokens · 41087 ms · 2026-06-30T17:12:51.930623+00:00 · methodology

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 5 canonical work pages · 2 internal anchors

  1. [1]

    InProceedings of the European Conference on Computer Vision (ECCV)

    Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?. InProceedings of the European Conference on Computer Vision (ECCV). Springer, 36–54. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2023. Flow Matching for Generative Modeling. InInternational Conference on Learning Representations (ICLR). Shaowei ...

    2023

  2. [2]

    Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang

    EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape Generation.ACM Transactions on Graphics (TOG)42, 6 (2023), 1–12. Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. 2025. EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wi...

    work page arXiv 2023

  3. [3]

    MediaPipe: A Framework for Building Perception Pipelines.arXiv preprint arXiv:1906.08172(2019). Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Beno...

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  4. [4]

    In Advances in Neural Information Processing Systems (NeurIPS)

    PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 8024–8035. Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2024. Reconstructing Hands in 3D with Transformers. InPro- ceedings of the IEEE/CVF C...

    2024

  5. [5]

    SAM 2: Segment Anything in Images and Videos

    WiLoR: End-to-End 3D Hand Localization and Reconstruction in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12242–12254. Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. InAdvances in Neural Information Process...

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  6. [6]

    LaS- Comp: Zero-shot 3D completion with latent-spatial consistency.arXiv preprint arXiv:2602.18735, 2026

    MagicHOI: Leveraging 3D Priors for Accurate Hand-Object Reconstruction from Short Monocular Video Clips. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 5957–5968. Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022. ViTPose: Simple Vi- sion Transformer Baselines for Human Pose Estimation. InAdvances in Neural Infor...

    work page arXiv 2022

  7. [7]

    Cos3d: Collaborative open-vocabulary 3d segmentation

    Predicting 4D Hand Trajectory from Monocular Videos. InInternational Conference on 3D Vision (3DV). Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. 2024. G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1911–1920...

    work page arXiv 2024