MoCapAnything V2: End-to-End Motion Capture for Arbitrary Skeletons

Dao Thien Phong; Dongze Lian; Guanli Hou; Hanwang Zhang; Kehong Gong; Mingxi Xu; Mingyuan Zhang; Ning Zhang; Qi Wang; Weixia He

arxiv: 2604.28130 · v3 · pith:TQIHEVKTnew · submitted 2026-04-30 · 💻 cs.CV

MoCapAnything V2: End-to-End Motion Capture for Arbitrary Skeletons

Kehong Gong , Zhengyu Wen , Dao Thien Phong , Mingxi Xu , Weixia He , Qi Wang , Ning Zhang , Zhengyu Li

show 5 more authors

Guanli Hou Dongze Lian Xiaoyu He Mingyuan Zhang Hanwang Zhang

This is my paper

Pith reviewed 2026-07-01 08:27 UTC · model grok-4.3

classification 💻 cs.CV

keywords motion captureend-to-end learningarbitrary skeletonsmonocular videopose estimationrotation predictioninverse kinematics

0 comments

The pith

End-to-end learning resolves rotation ambiguities in arbitrary-skeleton motion capture from monocular video by conditioning on a reference pose-rotation pair.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Prior methods split the task into a video-to-pose network followed by an analytical inverse-kinematics step, but joint positions leave bone twists and other degrees of freedom undetermined. The paper replaces the non-differentiable IK stage with a learnable pose-to-rotation network whose input includes one reference pose-rotation pair taken from the target asset plus the rest pose. This single extra input anchors the local coordinate system and turns the otherwise ill-posed mapping into a supervised conditional task that can be trained jointly with the pose estimator. The resulting system predicts positions directly from video, skips mesh intermediates, and reports lower rotation error on both familiar and unseen skeletons together with substantially faster inference.

Core claim

The first fully end-to-end framework for arbitrary-skeleton motion capture jointly optimizes a Video-to-Pose stage and a Pose-to-Rotation stage; supplying one reference pose-rotation pair from the target asset together with the rest pose supplies the missing coordinate-system information and removes the ambiguities that positions alone cannot resolve.

What carries the argument

A reference pose-rotation pair from the target asset, supplied together with the rest pose, that anchors the rotation coordinate system and converts the pose-to-rotation task into a well-constrained conditional prediction problem.

If this is right

Joint optimization lets the network adapt to noisy pose predictions and directly minimize the final animation objective instead of an intermediate position loss.
Direct video-to-position prediction without mesh reconstruction removes a major source of latency and improves robustness to lighting or texture variation.
The same skeleton-aware attention module works across arbitrary bone counts and topologies because the reference pair normalizes the local frames.
Error on unseen skeletons drops to 6.54 degrees, indicating that the conditioning generalizes beyond the training distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The method suggests that any future monocular capture system for custom rigs will need an explicit mechanism to communicate the asset's local axis conventions rather than relying solely on joint locations.
Replacing the analytical IK solver with a learned conditional module opens the possibility of end-to-end fine-tuning on animation-specific losses such as foot sliding or contact preservation.
Because the reference pair is cheap to supply at inference time, the approach could be integrated into existing animation pipelines without retraining per character.
The reported twenty-fold speed-up relative to mesh-based pipelines indicates that skipping the surface representation stage may be the dominant factor for real-time deployment.

Load-bearing premise

The ambiguity in the pose-to-rotation mapping can be resolved by supplying a single reference pose-rotation pair from the target asset together with the rest pose.

What would settle it

Running the model on the same test sequences while deliberately omitting or mismatching the reference pair and measuring whether rotation error returns to the original ~17-degree baseline.

Figures

Figures reproduced from arXiv: 2604.28130 by Dao Thien Phong, Dongze Lian, Guanli Hou, Hanwang Zhang, Kehong Gong, Mingxi Xu, Mingyuan Zhang, Ning Zhang, Qi Wang, Weixia He, Xiaoyu He, Zhengyu Li, Zhengyu Wen.

**Figure 1.** Figure 1: Overview of MoCapAnything V2. Given an input video of a human or an animal, our method infers a topology-agnostic skeleton sequence across view at source ↗

**Figure 2.** Figure 2: Comparison of MoCapAnything V1 and V2. Unlike V1, which depends on mesh-conditioned video-to-pose estimation and analytical inverse kinematics view at source ↗

**Figure 3.** Figure 3: Framework of MoCapAnything V2. Our method unifies video-to-pose and pose-to-rotation within a single end-to-end trainable architecture. The view at source ↗

**Figure 4.** Figure 4: MoCap V1 vs. V2. Row 1: V1 (traditional IK-based optimization). Row 2: V2 (our learning-based rotation recovery). V1 suffers from joint spinning artifacts, whereas V2 produces stable, temporally consistent rotations. The 8-layer model achieves the best results (6.54◦ on Zoo-Unseen), suggesting a good balance between model capacity and optimization. 4.9 Cross-Attention Depth We study the effect of referen… view at source ↗

**Figure 5.** Figure 5: MoCap demo across domains. Row 1: Objaverse assets; Row 2: Truebones Zoo; Rows 3–4: in-the-wild videos. Results are shown from multiple viewpoints, demonstrating accurate mocap on arbitrary subjects view at source ↗

**Figure 6.** Figure 6: Dance demo. Given a single input video (center), our method performs mocap on a humanoid skeleton (left) and retargets the motion to an animal skeleton (right). Rotation quality: V1 vs. V2 view at source ↗

read the original abstract

Recent methods for arbitrary-skeleton motion capture from monocular video follow a factorized pipeline, where a Video-to-Pose network predicts joint positions and an analytical inverse-kinematics (IK) stage recovers joint rotations. While effective, this design is inherently limited, since joint positions do not fully determine rotations and leave degrees of freedom such as bone-axis twist ambiguous, and the non-differentiable IK stage prevents the system from adapting to noisy predictions or optimizing for the final animation objective. In this work, we present the first fully end-to-end framework in which both Video-to-Pose and Pose-to-Rotation are learnable and jointly optimized. We observe that the ambiguity in pose-to-rotation mapping arises from missing coordinate system information: the same joint positions can correspond to different rotations under different rest poses and local axis conventions. To resolve this, we introduce a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system. This formulation turns rotation prediction into a well-constrained conditional problem and enables effective learning. In addition, our model predicts joint positions directly from video without relying on mesh intermediates, improving both robustness and efficiency. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module for joint-level local reasoning and global coordination. Experiments on Truebones Zoo and Objaverse show that our method reduces rotation error from ~17 degrees to ~10 degrees, and to 6.54 degrees on unseen skeletons, while achieving ~20x faster inference than mesh-based pipelines. Project page: https://animotionlab.github.io/MoCapAnythingV2/

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper replaces analytical IK with a learned, reference-conditioned Pose-to-Rotation stage for arbitrary skeletons, but the single-pair conditioning may leave axial twist under-constrained.

read the letter

The main point is that this work trains Video-to-Pose and Pose-to-Rotation end-to-end instead of stopping at joint positions and handing off to non-differentiable IK. They condition the rotation head on a rest pose plus one reference pose-rotation pair taken from the target asset, which they say supplies the missing coordinate conventions and turns the mapping into a well-posed conditional task. The shared GL-GMHA module handles local bone reasoning and global skeleton coordination, and they avoid mesh intermediates entirely.

That change produces the reported gains: rotation error drops from roughly 17° to 10°, reaches 6.54° on unseen skeletons, and inference runs about 20 times faster than mesh pipelines. Those numbers matter for anyone who needs to retarget to new rigs without per-asset IK tuning.

The soft spot is exactly the one flagged in the stress-test note. A 3D bone direction fixes only two rotational degrees of freedom; the axial twist around the bone remains free. One global reference pair may anchor the overall frame but does not automatically eliminate per-bone twist freedom when bone lengths and hierarchy vary. The abstract asserts that the reference plus rest pose defines the coordinate system, yet without an explicit uniqueness argument or ablation that isolates twist error, it is unclear whether the learned outputs stay animation-valid or simply satisfy position loss while allowing inconsistent twists. That gap directly affects how much weight to give the unseen-skeleton result.

The work is aimed at teams building production animation tools or real-time capture systems that currently rely on factorized pipelines. Readers who care about differentiable retargeting or skeleton-agnostic capture will find the conditioning idea and the graph attention design worth examining. The central departure from prior factorized methods is real, and the experiments use relevant datasets.

I would send it to peer review. The joint-optimization claim is substantive enough to justify referee time, even if the ambiguity resolution needs tighter evidence in revision.

Referee Report

2 major / 2 minor

Summary. The paper claims to introduce the first fully end-to-end learnable framework for monocular motion capture on arbitrary skeletons. It replaces the standard factorized Video-to-Pose + analytical IK pipeline with jointly optimized learnable Video-to-Pose and Pose-to-Rotation stages. A reference pose-rotation pair from the target asset, supplied together with the rest pose, is used to anchor the rotation coordinate system and resolve pose-to-rotation ambiguities. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module. Experiments on Truebones Zoo and Objaverse report rotation error reductions from ~17° to ~10° (6.54° on unseen skeletons) and ~20x faster inference than mesh-based methods.

Significance. If the reference-pair conditioning truly converts the under-constrained rotation problem into a well-posed conditional regression task and the joint optimization yields the stated error reductions, the work would constitute a meaningful technical advance. It directly addresses the non-differentiability and ambiguity limitations of prior factorized pipelines, enables optimization for the final animation objective, and improves efficiency by avoiding mesh intermediates. The reported generalization to unseen skeletons and the GL-GMHA architecture for variable skeletons would be of practical interest to animation and computer vision communities.

major comments (2)

[Section 3] Section 3 (method description): The central assertion that 'a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system' and thereby turns rotation prediction into a 'well-constrained conditional problem' lacks any derivation or uniqueness argument. A 3D bone direction constrains only two rotational degrees of freedom, leaving an axial twist; it is not shown why a single global reference pair suffices to eliminate per-bone twist ambiguities consistently across heterogeneous bone lengths and hierarchies. This is load-bearing for both the claimed error reductions and the generalization result.
[Abstract / Experiments] Abstract / Experiments section: The reported quantitative improvements (~17° to ~10°, 6.54° on unseen skeletons) are presented without any description of error-bar methodology, number of evaluation runs, reference-pair selection protocol during training/testing, or basic dataset statistics. This makes it impossible to determine whether the joint end-to-end training, rather than the external reference input, is responsible for the gains.

minor comments (2)

[Abstract] The abstract refers to 'Section 3' but the provided text contains no numbered sections; the full manuscript should ensure the method description is clearly labeled and cross-referenced.
[Method] Notation for the reference pose-rotation pair, rest pose, and the conditioning mechanism should be introduced with explicit equations to support reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for recognizing the potential technical contribution of the end-to-end framework. We address each major comment below and will revise the manuscript to incorporate the requested clarifications and derivations.

read point-by-point responses

Referee: [Section 3] Section 3 (method description): The central assertion that 'a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system' and thereby turns rotation prediction into a 'well-constrained conditional problem' lacks any derivation or uniqueness argument. A 3D bone direction constrains only two rotational degrees of freedom, leaving an axial twist; it is not shown why a single global reference pair suffices to eliminate per-bone twist ambiguities consistently across heterogeneous bone lengths and hierarchies. This is load-bearing for both the claimed error reductions and the generalization result.

Authors: We agree that the manuscript would be strengthened by a formal derivation. The reference pair supplies the complete rotation matrix (including twist) for a designated reference bone; combined with the rest pose, this fixes the local coordinate frame for that bone. Because the skeleton hierarchy and bone lengths are known, the fixed local axes propagate consistently to all descendant bones, removing per-bone twist freedom. In the revision we will add a new subsection in Section 3 containing (i) a coordinate-frame derivation showing uniqueness under the given conditioning and (ii) a brief discussion of how the same mechanism supports generalization to unseen skeletons whose hierarchies differ from the training set. revision: yes
Referee: [Abstract / Experiments] Abstract / Experiments section: The reported quantitative improvements (~17° to ~10°, 6.54° on unseen skeletons) are presented without any description of error-bar methodology, number of evaluation runs, reference-pair selection protocol during training/testing, or basic dataset statistics. This makes it impossible to determine whether the joint end-to-end training, rather than the external reference input, is responsible for the gains.

Authors: We acknowledge that the current manuscript omits these experimental details. In the revised version we will expand the Experiments section to report: error bars computed over five independent runs with different random seeds; the reference-pair selection protocol (a single fixed pair per skeleton, chosen once from the asset’s animation library and held constant for both training and testing); and basic dataset statistics (number of skeletons, total frames, average bone count, and train/test splits for Truebones Zoo and Objaverse). These additions will make the evaluation protocol transparent and allow readers to better assess the source of the observed gains. revision: yes

Circularity Check

0 steps flagged

No circularity: external reference input conditions learned mapping

full rationale

The paper's central formulation supplies a reference pose-rotation pair from the target asset as an explicit external input, together with the rest pose, to condition the Pose-to-Rotation network. This turns the problem into supervised conditional regression without any equation or step that defines the output as a function of itself or reduces a prediction to a fitted parameter drawn from the same data. No self-citations, uniqueness theorems, or ansatzes appear in the derivation chain; the end-to-end training and reported error metrics are presented as empirical outcomes of joint optimization rather than tautological re-expressions of inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the domain assumption that graph attention can jointly model local joint reasoning and global coordination, plus the paper-specific assumption that a reference pose-rotation pair suffices to anchor the rotation coordinate system. No free parameters or invented entities are described in the abstract.

axioms (2)

domain assumption Graph attention mechanisms can effectively capture both local joint-level reasoning and global skeleton coordination
Invoked for the shared GL-GMHA module
ad hoc to paper A reference pose-rotation pair together with the rest pose anchors the rotation coordinate system and resolves pose-to-rotation ambiguity
Introduced in the abstract as the key conditioning input that turns rotation prediction into a well-constrained problem

pith-pipeline@v0.9.1-grok · 5885 in / 1445 out tokens · 36223 ms · 2026-07-01T08:27:01.747822+00:00 · methodology

Review history (3 revisions) →

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

AnyAct: Towards Human Reenactment of Character Motion From Video
cs.CV 2026-05 unverdicted novelty 7.0

AnyAct generates plausible human reenactments from non-human character videos via conditional motion generation from transferable sparse local 2D articulated cues, using human-only supervision, progressive training, a...
AnyAct: Towards Human Reenactment of Character Motion From Video
cs.CV 2026-05 unverdicted novelty 6.0

AnyAct generates editable human reenactments from character videos via conditional motion generation from transferable sparse local 2D articulated cues, with designs for human-only supervision and global-local decoupling.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

FiLM: visual reasoning with a general conditioning layer. In�� (New Orleans, Louisiana, USA) ��. AAAI Press, P...

2020
[2]

In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Global-to-Local Modeling for Video-Based 3D Human Pose and Shape Estimation . In�� . IEEE Computer Society, Los Alamitos, CA, USA, 8887–8896. doi:10.1109/ CVPR52729.2023.00858 Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan. 2022. End-to-End Multi- Person Pose Estimation with T...

work page doi:10.1109/cvpr52688.2022.01079 2023
[3]

In: 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR)

MagicPony: Learning Articulated 3D Animals in the Wild . In�� . IEEE Computer Society, Los Alamitos, CA, USA, 8792–8802. doi:10.1109/CVPR52729.2023.00849 Yabo Xiao, Kai Su, Xiaojuan Wang, Dongdong Yu, Lei Jin, Mingshu He, and Zehuan Yuan

work page doi:10.1109/cvpr52729.2023.00849 2023
[4]

Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang

Querypose: Sparse multi-person pose regression via spatial-aware part-level query.�� 35 (2022), 12464–12477. Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. 2022a. Pose for Everything: Towards Category-Agnostic Pose Estimation. In�� ...

work page doi:10.1007/978-3-031-20068-7_23 2022

[1] [1]

FiLM: visual reasoning with a general conditioning layer. In�� (New Orleans, Louisiana, USA) ��. AAAI Press, P...

2020

[2] [2]

In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Global-to-Local Modeling for Video-Based 3D Human Pose and Shape Estimation . In�� . IEEE Computer Society, Los Alamitos, CA, USA, 8887–8896. doi:10.1109/ CVPR52729.2023.00858 Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan. 2022. End-to-End Multi- Person Pose Estimation with T...

work page doi:10.1109/cvpr52688.2022.01079 2023

[3] [3]

In: 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR)

MagicPony: Learning Articulated 3D Animals in the Wild . In�� . IEEE Computer Society, Los Alamitos, CA, USA, 8792–8802. doi:10.1109/CVPR52729.2023.00849 Yabo Xiao, Kai Su, Xiaojuan Wang, Dongdong Yu, Lei Jin, Mingshu He, and Zehuan Yuan

work page doi:10.1109/cvpr52729.2023.00849 2023

[4] [4]

Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang

Querypose: Sparse multi-person pose regression via spatial-aware part-level query.�� 35 (2022), 12464–12477. Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. 2022a. Pose for Everything: Towards Category-Agnostic Pose Estimation. In�� ...

work page doi:10.1007/978-3-031-20068-7_23 2022