---
license: mit
library_name: litert
pipeline_tag: text-to-speech
tags:
  - text-to-speech
  - tts
  - diffusion
  - autoregressive
  - streaming
  - vibevoice
  - litert
  - tflite
  - on-device
base_model: microsoft/VibeVoice-Realtime-0.5B
base_model_relation: quantized
---

# VibeVoice-Realtime-0.5B — LiteRT (on-device, hybrid GPU/CPU)

[VibeVoice-Realtime-0.5B](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B) (Microsoft, MIT) —
a **streaming, autoregressive next-token-diffusion** text-to-speech model — re-authored to run
on-device with LiteRT `CompiledModel`. Four graphs plus host-side orchestration synthesize 24 kHz
speech with **no FFT anywhere** (VibeVoice's acoustic tokenizer is a convolutional σ-VAE). Verified
on a Pixel 8a (Tensor G3).

The 24-layer Qwen2.5-0.5B backbone is split into a 4-layer text LM and a 20-layer TTS LM; each token,
the TTS LM's hidden state conditions a 4-layer DDPM head that a 5-step DPM-Solver++ loop denoises into
a 64-d acoustic latent, which a convolutional σ-VAE decoder turns into audio. The two LMs keep their
KV cache **host-side** as a packed `[1, L·nkv, Pmax, 64]` tensor fed in/out each step (the ML-Drift-safe
"state as graph I/O" pattern); the voice is a precomputed prompt KV cache.

## Device placement (hybrid — verified on-device, not just by desktop parity)

| graph | file | runs on |
|---|---|---|
| 4-layer text LM (KV step) | `vv_base_lm_kv_fp32.tflite` | CPU (fp32) |
| 20-layer TTS LM (KV step) | `vv_tts_lm_kv_fp32.tflite` | CPU (fp32) |
| DDPM diffusion head | `vv_diffhead_fp16.tflite` | GPU (fp32 precision) |
| σ-VAE decoder | `vv_decoder_fp32.tflite` | CPU (fp32) |

The LMs run on CPU because this build pins LiteRT 2.1.3, whose Mali ML Drift delegate rejects their
KV-step `FULLY_CONNECTED` weights shape — a delegate bug **fixed in 2.1.5** (see the correction
below) — and because fp16 collapses the 20-layer stack on ARM XNNPACK, so they ship as **fp32**
graphs. The
σ-VAE decoder compiles on GPU but ML Drift **miscomputes** it (a graph-assembly buffer bug: every op
is bit-exact in isolation, but the assembled ConvNeXt block is wrong; identical on OpenCL/OpenGL and
at fp32), so it too runs as an **fp32** graph on CPU. Only the tiny diffusion head runs on GPU. Each
graph is bit-exact with the PyTorch reference on its chosen accelerator.

> **Correction (2026-07-10): the LMs are not blocked by the GPU.** An earlier version of this card
> said ML Drift rejects the KV-step `FULLY_CONNECTED` weights shape, so autoregressive attention
> decoders cannot run on it. That is withdrawn. The rejection is real on LiteRT **2.1.3**
> (`INVALID_ARGUMENT: Unsupported weights shape`, `fully_connected.cc:1070`) and **fixed in 2.1.5**,
> where the same graphs delegate fully and compute correctly: `vv_base_lm_kv_fp32` **313/313 nodes at
> 27 ms/step** (hidden corr 0.999964 vs CPU), the 20-layer `vv_tts_lm_kv_fp32` **1559/1559 nodes at
> 65 ms/step** (corr 0.999852). Both timings are `run()` **plus the output readback** —
> `CompiledModel.run()` only enqueues the GPU work, so timing it alone reports a misleading 3 ms and
> 23 ms, which an earlier version of this card did. These graphs still ship for CPU as fp32; a GPU
> build is a follow-up.
>
> The σ-VAE decoder miscompute is **unaffected and still open** — re-verified against the full decoder
> graph on both runtimes, all 1578 nodes delegate on each and the GPU output is bit-identically wrong
> on both (std 0.030584, absmax 0.134888, corr 0.0254 against a CPU fp32 reference whose std is ~0).

Non-graph assets: `embed_tokens.f16` (mmapped fp16 token embeddings), `glue.f32` (connector +
type-embed + EOS weights), `voice_en-Emma_woman.bin` (a precomputed prompt KV cache).

## Minimal usage (Python — one autoregressive + diffusion step)

```python
import numpy as np
from ai_edge_litert.interpreter import Interpreter

# The full generate() loop (BPE, host KV cache, DPM-Solver++, EOS, decode) is in the
# conversion/ reference; this shows loading a graph and running one TTS-LM step.
lm = Interpreter(model_path="vv_tts_lm_kv_fp32.tflite")
lm.allocate_tensors()
ins = lm.get_input_details()          # x[1,1,896], cos/sin[1,1,1,64], mask[1,1,1,385], pk/pv[1,40,384,64]
for d in ins:
    lm.set_tensor(d["index"], np.zeros(d["shape"], np.float32))
lm.invoke()
hidden = lm.get_tensor(lm.get_output_details()[0]["index"])   # [1,1,896]
print(hidden.shape)
```

## Minimal usage (Kotlin — LiteRT CompiledModel)

```kotlin
import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel

// LMs + decoder on CPU (fp32), the head on GPU with fp32 precision.
val opts = CompiledModel.Options(Accelerator.CPU)
val lm = CompiledModel.create("vv_tts_lm_kv_fp32.tflite", opts, null)
val inputs = lm.createInputBuffers()
val outputs = lm.createOutputBuffers()
inputs[0].writeFloat(xEmbedding)      // [1,1,896]
// ... write cos, sin, mask, packed pk/pv ...
lm.run(inputs, outputs)
val hidden = outputs[0].readFloat()   // [1,1,896]
```

## Notes

- **Stochastic, autoregressive**: each frame draws fresh diffusion noise, so two runs differ.
- **Single speaker, English**; the bundled voice is a precomputed prompt KV cache (voices are exported
  offline — the realtime checkpoint is decoder-only and cannot clone voices on-device).
- This is a **correctness-first** placement, not a real-time one: the two fp32 LMs and the fp32 decoder
  make a short sentence take ~30 s on a Pixel 8a. See the conversion notes for the ML Drift decoder bug.

Original model: [microsoft/VibeVoice-Realtime-0.5B](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B) (MIT).

<!-- funnel:v1 -->

---

**Want a different model on-device?** [Open a request](https://github.com/john-rocky/on-device-requests) — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->