StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

Abstract

Real-time co-speech gesture generation must produce 3D motion clip by clip as speech streams in. Existing streaming methods are fundamentally open-loop: each clip is synthesized from past context and directly passed to the next window, so small errors compound into long-horizon distributional drift. StreamTalk addresses this with a closed-loop streaming framework. At inference time, Streaming Pose-Guided Generation (SPG) first generates a coarse clip, retrieves a plausible tail key pose from a speaker-specific motion database, and refines the clip with this destination anchor before moving to the next window. During training, Stochastic Anchor Masking (SAM) teaches the model to inpaint complete motion from sparse boundary anchors, while a part-aware DiT separates hands, body, and translation streams. On BEAT2, StreamTalk improves motion quality, suppresses minute-scale drift, and runs in real time.

Demo

Method

StreamTalk uses a generate-retrieve-refine cycle to periodically pull streaming motion back toward a plausible pose distribution.

Closed-Loop Streaming

Core Quantitative Results

The key number is FGD: lower FGD means the generated gesture distribution is closer to real human motion. StreamTalk achieves the best FGD on both the single-speaker streaming setting and the harder all-speaker setting, while keeping beat consistency and diversity close to the ground truth.

The four related project rows use a shared raw-value scale across all project pages; unavailable metrics are marked “—”. BeatAlign/BC and Diversity/DIV are interpreted relative to the ground-truth row rather than simply maximized.

What Makes It Work?

The gains come from matching training and inference: SAM teaches the model to use sparse anchors, and SPG supplies those anchors during streaming inference. Either piece alone helps less than the full closed-loop system.

A random anchor hurts because an arbitrary pose can point the motion in the wrong direction. A retrieved tail anchor works because drift is mainly a destination problem: the clip needs one plausible waypoint near its boundary, not dense constraints at many frames.

Long-Horizon Stability

StreamTalk is designed for streaming, so we care not only about a short clip looking good, but also whether quality decays after many clip-to-clip transitions.

Qualitative and User Study

The tables show distribution-level improvements; the visual comparison explains what those numbers correspond to: smoother transitions, fewer frozen poses, and more stable arm spacing across long speech.

BibTeX

@inproceedings{zhang2026streamtalk,
  title={StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring},
  author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Yang, Kaixing and Hoi, Steven},
  booktitle={European Conference on Computer Vision},
  year={2026},
  eprint={2608.01643},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.01643}
}