---
pipeline_tag: automatic-speech-recognition
license: apache-2.0
base_model: ibm-granite/granite-speech-3.3-8b
library_name: zeromodels
tags:
- keras
- zeromodels
- granite-speech
- speech-llm
- automatic-speech-recognition
- audio
- arxiv:2505.08699
- pytorch
- jax
- tf
---

## ***See [our collection](https://huggingface.co/collections/zeromodels/granite-speech-6a8eaf2e42e4b726c9c08ba8) for all versions of Granite Speech.***

# Run Granite Speech with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-ZeroModels-black?logo=github)](https://github.com/IMvision12/ZeroModels) [![Docs](https://img.shields.io/badge/Docs-Granite%20Speech-blue)](https://imvision12.github.io/ZeroModels/granite_speech/) [![Collection](https://img.shields.io/badge/HF-Granite%20Speech%20collection-yellow)](https://huggingface.co/collections/zeromodels/granite-speech-6a8eaf2e42e4b726c9c08ba8)

# zeromodels/granite_speech_3_3_8b

Paper: [Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities (arXiv:2505.08699)](https://arxiv.org/abs/2505.08699) · [HF Papers](https://huggingface.co/papers/2505.08699)

Granite Speech is a **speech-aware LLM**, not ASR with an LM bolted on. A conformer CTC encoder and BLIP-2 style Q-Former turn mel features into audio embeddings that fill `<|audio|>` placeholders in a Granite decoder. You ask for a transcript, a summary, or an answer in ordinary English; text-only mode keeps the plain Granite decoder.

For more details on the model, please go to the upstream [model card](https://huggingface.co/ibm-granite/granite-speech-3.3-8b).

Pure-**Keras 3** conversion of [`ibm-granite/granite-speech-3.3-8b`](https://huggingface.co/ibm-granite/granite-speech-3.3-8b) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.

This is a **speech LLM** checkpoint (`GraniteSpeechConditionalGenerate`, Granite 3.3 8B). Prefer `load_dtype="bfloat16"`.

## ✨ Quick start

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import keras
import numpy as np
import soundfile as sf
from zeromodels.models.granite_speech import (
    GraniteSpeechConditionalGenerate,
    GraniteSpeechProcessor,
)

model = GraniteSpeechConditionalGenerate.from_weights(
    "zeromodels/granite_speech_3_3_8b", load_dtype="bfloat16"
)
processor = GraniteSpeechProcessor.from_weights("zeromodels/granite_speech_3_3_8b")

audio, sr = sf.read("your_audio.wav", dtype="float32")  # 16 kHz mono

# Ask in words: same audio + different instruction => different answer.
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio"},
            {
                "type": "text",
                "text": "can you transcribe the speech into a written format?",
            },
        ],
    }
]

inputs = processor(conversation=conversation, audio=audio, sampling_rate=sr)
out = model.generate(**inputs, max_new_tokens=64)
ids = np.asarray(keras.ops.convert_to_numpy(out))[0].tolist()
print(repr(processor.tokenizer.decode(ids)))
```

Load any Granite Speech variant the same way with `from_weights("zeromodels/<variant>")`:

| Variant | Hub | Notes |
|---|---|---|
| `granite_speech_3_3_2b` | [`zeromodels/granite_speech_3_3_2b`](https://huggingface.co/zeromodels/granite_speech_3_3_2b) | Granite 3.3 2B |
| `granite_speech_3_3_8b` | [`zeromodels/granite_speech_3_3_8b`](https://huggingface.co/zeromodels/granite_speech_3_3_8b) | Granite 3.3 8B |
| `granite_speech_4_1_2b` | [`zeromodels/granite_speech_4_1_2b`](https://huggingface.co/zeromodels/granite_speech_4_1_2b) | Granite 4.1 2B |
| `granite_4_0_1b_speech` | [`zeromodels/granite_4_0_1b_speech`](https://huggingface.co/zeromodels/granite_4_0_1b_speech) | Granite 4.0 1B |

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Pass audio via `audio=` + `sampling_rate=`; put only an `{"type": "audio"}` marker in the conversation.
- Drop audio for text-only chat; the audio LoRA stays off.
- See [Granite Speech docs](https://imvision12.github.io/ZeroModels/granite_speech/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `GraniteSpeechConditionalGenerate.from_weights("hf:ibm-granite/granite-speech-3.3-8b")`.

## Special Thanks

A huge thank you to the IBM Granite authors for creating and releasing these models.

License: Apache 2.0.