Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints

Abstract

Reliable co-speech gesture generation requires precise motion representation and consistent structural priors across all joints, especially over long sequences. Existing generative methods typically operate on local joint rotations, which are defined hierarchically based on the skeleton structure. This leads to cumulative errors during generation, manifesting as unstable and implausible motions at end-effectors. We propose GlobalDiff, a diffusion-based framework that operates directly in the space of global joint rotations for the first time, fundamentally decoupling each joint's prediction from upstream dependencies and alleviating hierarchical error accumulation. To compensate for the absence of structural priors in global rotation space, we introduce a multi-level constraint scheme. Specifically, a joint structure constraint introduces virtual anchor points around each joint to better capture fine-grained orientation. A skeleton structure constraint enforces angular consistency across bones to maintain structural integrity. A temporal structure constraint utilizes a multi-scale variational encoder to align the generated motion with ground-truth temporal patterns. These constraints jointly regularize the global diffusion process and reinforce structural awareness. Extensive evaluations on standard co-speech benchmarks show that GlobalDiff generates smooth and accurate motions, improving the performance by 46.0% compared to the current SOTA under multiple speaker identities. The code is available at https://github.com/Xiangyue-Zhang/GlobalDiff.

Demo

Representation Shift

GlobalDiff first changes the motion representation. Instead of predicting local rotations along the kinematic tree, it predicts each joint in global rotation space so downstream joints are not forced to inherit upstream mistakes.

Method

The method is not only a new rotation parameterization. GlobalDiff pairs global diffusion with structure-aware losses so the model remains anatomically plausible.

Structural Constraints

Because global rotations remove the skeleton hierarchy, the model needs explicit structural guidance. The constraints below make the generated motion respect local joint orientation, bone-angle consistency, and temporal dynamics.

Joint structure constraint. Virtual anchor points around a joint help resolve fine-grained orientation and reduce ambiguous rotations.

Skeleton structure constraint. Adjacent bones are regularized together so global rotations still form a coherent human skeleton.

Qualitative Results

The visual examples show the main benefit of the representation shift: hands and arms stay spatially coherent even when the utterance requires large expressive movement.

Quantitative Results

The multi-speaker setting is the key stress test because it exposes whether the representation handles different identities. GlobalDiff achieves the best all-speaker FGD and MSE while keeping beat alignment close to ground truth.

BibTeX

@misc{zhang2025mitigatingerroraccumulationcospeech,
      title={Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints},
      author={Xiangyue Zhang and Jianfang Li and Jianqiang Ren and Jiaxu Zhang},
      year={2025},
      eprint={2511.10076},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.10076},
}

Positioning and Related Work

GlobalDiff: global rotations and structural constraints for co-speech gesture generation.

GlobalDiff targets error growth when predicted local rotations are composed along the skeleton's kinematic hierarchy. The generation model predicts global joint rotations directly. Virtual anchor points supervise joint orientation, bone-angle relations constrain the skeleton, and multi-scale temporal features regularize motion dynamics. These constraints supply structural information that is less explicit when joints are represented by global rotations. See the GlobalDiff paper for the method and original experiments.

EMAGE (paper) approaches holistic gesture generation through audio conditioning and masked gesture reconstruction. GestureLSM (paper) models spatial and temporal interactions among tokenized body regions and uses latent shortcut learning for flow-matching sampling. GlobalDiff's distinguishing focus is global rotation prediction with joint, skeleton, and temporal constraints. Its hierarchical error-accumulation argument concerns the spatial kinematic chain. It does not, by itself, establish a solution to autoregressive streaming drift or chunk-boundary discontinuities.

BEAT2 baseline use. Select the matching Speaker 2 or 25-English-speaker inference archive in the repository's inference-data section. These releases use different training and evaluation protocols, so their results are not a controlled single-speaker versus multi-speaker ablation. Cross-method comparisons also require matching speaker coverage, data splits, and evaluation settings.