EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generation

Abstract

Co-speech gesture generation benefits from masked modeling, but existing strategies struggle to identify semantically significant frames for effective motion masking. We propose EchoMask, a speech-queried attention-based mask modeling framework for holistic co-speech gesture generation. Our key insight is to leverage motion-aligned speech features to guide the masked motion modeling process, selectively masking rhythm-related and semantically expressive motion frames. Specifically, we first propose a motion-audio alignment module (MAM) to construct a latent motion-audio joint space. In this space, both low-level and high-level speech features are projected, enabling motion-aligned speech representation using learnable speech queries. Then, a speech-queried attention mechanism (SQA) is introduced to compute frame-level attention scores through interactions between motion keys and speech queries, guiding selective masking toward motion frames with high attention scores. Finally, the motion-aligned speech features are also injected into the generation network to facilitate co-speech motion generation. Qualitative and quantitative evaluations confirm that our method outperforms existing state-of-the-art approaches, successfully producing high-quality co-speech motion. The code is available at https://github.com/Xiangyue-Zhang/EchoMask.

Demo

Masking Problem

The main question is where masking should happen. Random masking is blind, and loss-based masking can over-focus on hard but uninformative transitions. EchoMask uses speech as the query signal.

Method

EchoMask first aligns audio and motion in a shared latent space, then uses speech-queried attention to decide which motion frames should be masked and reconstructed.

Qualitative Results

The examples show that speech-guided masking improves both body gestures and facial articulation, especially where semantic cues should trigger clearer motion.

Quantitative Results

Lower is better for FGD, MSE, and LVD. BC and DIV follow the benchmark convention shown in each table. EchoMask now provides results for the original Speaker 2 protocol and a released 25-speaker BEAT2 checkpoint.

BibTeX

@inproceedings{zhang2025echomask,
  title={EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generation},
  author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Ren, Jianqiang and Bo, Liefeng and Tu, Zhigang},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  pages={10827--10836},
  year={2025}
}

Positioning and Related Work

EchoMask: speech-guided masked modeling for co-speech gesture generation.

EchoMask's motion-audio alignment module maps speech and motion into a joint latent space and learns speech queries aligned with motion. Speech-queried attention scores motion frames and selects high-attention frames for masking. The aligned speech features also condition the generation network. This connects the masking decision to speech rhythm and semantic content. See the EchoMask paper for the method and original experiments.

EMAGE (paper) provides a related masked-modeling approach through joint audio-to-gesture generation and masked gesture reconstruction. EchoMask focuses on how motion-aligned speech features guide which frames are masked. SemTalk (paper) addresses frame-level semantic emphasis through separately learned base and sparse motions with adaptive fusion. GestureLSM (paper) emphasizes interactions among tokenized body regions and latent shortcut sampling. These comparisons describe mechanisms, without assuming a shared evaluation protocol or ranking results across different settings.

BEAT2 baseline use. Use weights and outputs from the same release. The original Speaker 2 paper protocol and the later 25-speaker release are not a controlled single-speaker versus multi-speaker ablation. See the all-speaker workflow and released metrics and checkpoint metadata for that release's setup.