---
title: StreamTalk
emoji: 🗣️
colorFrom: gray
colorTo: pink
sdk: gradio
sdk_version: 6.26.0
app_file: app.py
short_description: Speech to streaming 3D co-speech gesture animation
python_version: "3.12"
startup_duration_timeout: 45m
---

# StreamTalk — Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

Gradio demo for [StreamTalk](https://github.com/Xiangyue-Zhang/StreamTalk) (ECCV 2026).
Give it a speech recording; it returns a full-body SMPL-X gesture animation rendered to
video, plus the raw `.npz` SMPL-X sequence at 30 fps.

StreamTalk generates motion one 2-second window at a time. For each window it samples a
coarse motion with a part-aware DiT, retrieves a plausible destination pose from a
speaker-specific motion database via forward kinematics, and then refines the
window toward that anchor before it becomes the context for the next window. That
closed loop is what keeps long sequences from drifting.

* Paper: <https://huggingface.co/papers/2608.01643>
* Code: <https://github.com/Xiangyue-Zhang/StreamTalk> (MIT)
* Weights: [`X-Zhang/StreamTalk`](https://huggingface.co/X-Zhang/StreamTalk) —
  speaker-2 (Scott) checkpoint, CFG 3

## Implementation notes

The pipeline follows the authors' reference script
`Scripts/FM/TestFixedExpressions_keyposes.py` step for step (WavLM Large features →
flow-matching DiT sampling → body FK → L1 key-pose retrieval → anchored refinement →
seed blending). Three operational deviations:

1. The body model is NVIDIA **SOMA-X**, not SMPL-X (see below).
2. The speaker-2 retrieval database (joint positions + global rot6d poses for all 95
   BEAT2 `2_scott_0` training clips, 156 977 frames) is precomputed offline and shipped
   as `assets/retrieval_db_2_scott_0.npz`, instead of being rebuilt on every start.
3. The generated sequence is rasterised to video with PyTorch3D; the upstream repo only
   documents a Blender add-on workflow.

## Body model

StreamTalk's reference code drives the SMPL-X neutral 2020 body. The SMPL-X model file
may **not** be redistributed by third parties, so this Space cannot ship it, download it
from a community mirror, or (without pushing a licence click-through onto every visitor)
fetch it from the official site. Instead the demo runs forward kinematics and rendering
on NVIDIA's [SOMA-X](https://huggingface.co/nvidia/SOMA-X) neutral body — Apache-2.0,
ungated, cleared for commercial use, and built from NVIDIA-licensed body scans rather
than from SMPL/SMPL-X data.

SOMA's unified skeleton contains a joint for every SMPL-X joint, and it parameterises
poses as rotations relative to the T-pose composed down the kinematic chain — exactly
SMPL-X's convention — so StreamTalk's SMPL-X joint rotations drive it unchanged. The one
place the skeletons differ is the fingers: SOMA gives each of the four fingers an extra
metacarpal joint (`<Finger>1`, sitting inside the palm ~3 cm from the wrist), so SMPL-X's
three finger joints drive SOMA's `<Finger>2/3/4` — the MCP, PIP and DIP joints on the
knuckle line — and the metacarpals stay put, which is what SMPL-X assumes anyway.
`soma_body.py` implements the mapping plus a dependency-light FK/LBS pass over
`assets/soma_rig.npz`, a compact extract of the SOMA-X v0.3.0 neutral template rig
(hierarchy, T-pose/bind transforms, skinning weights and the neutral mesh) redistributed
here under Apache-2.0 (`LICENSE-SOMA-X`).

Consequences, all cosmetic:

* the retrieval database's joint positions were regenerated with the same SOMA FK, so
  query and database stay consistent — retrieval behaviour is unchanged;
* the avatar has SOMA's neutral proportions rather than BEAT2 speaker 2's SMPL-X betas;
* SOMA has no FLAME expression blendshapes, so the predicted 100 expression coefficients
  are exported in the `.npz` but are not rendered in the preview (jaw and eye
  articulation still are, since those are skeleton joints).

The downloadable `.npz` is unaffected: it is a standard SMPL-X motion sequence
(`poses`, `betas`, `expressions`, `trans`, `model="smplx2020"`) and can be replayed on a
real SMPL-X body by anyone who has accepted the SMPL-X licence themselves.

## Credits

Example audio clips are BEAT2 speaker 2 (Scott) recordings — the sample bundled with the
StreamTalk repository and one BEAT2 test clip — from
[H-Liu1997/BEAT2](https://huggingface.co/datasets/H-Liu1997/BEAT2). BEAT/BEAT2 is
released for non-commercial research use; please cite the BEAT authors if you reuse them.

`assets/soma_rig.npz` is derived from [`nvidia/SOMA-X`](https://huggingface.co/nvidia/SOMA-X)
v0.3.0 — Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES, Apache-2.0
(`LICENSE-SOMA-X`).