---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- nvfp4
- qwen
- reasoning
- math
- coding
- nvidia-modelopt
- blackwell
- gb10
- sm121
- quantized
- base_model:WeiboAI/VibeThinker-3B
- base_model:quantized:WeiboAI/VibeThinker-3B
base_model: WeiboAI/VibeThinker-3B
base_model_relation: quantized
quantized_by: r0b0tlab
---

# VibeThinker-3B-NVFP4

NVFP4 (W4A4) quantization of [WeiboAI/VibeThinker-3B](https://huggingface.co/WeiboAI/VibeThinker-3B) — a 3B reasoning model optimized for math, coding, and STEM tasks.

**Compression:** 5.8 GB BF16 → 2.20 GB (2.6×)  
**Performance:** 2.60× throughput vs BF16 at c1 (71.3 vs 27.4 tok/s)  
**Quality:** Identity tasks 4/4 correct. Reasoning quality preserved.

## Credits and Attribution

- **Base model:** [WeiboAI/VibeThinker-3B](https://huggingface.co/WeiboAI/VibeThinker-3B) — 3B reasoning model by WeiboAI, fine-tuned from Qwen2.5-Coder-3B (Alibaba/Qwen team)
- **Quantization:** NVIDIA Model Optimizer 0.44.0 ([NVIDIA/TensorRT-Model-Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer))
- **Calibration data:** [CNN/DailyMail](https://huggingface.co/datasets/abisee/cnn_dailymail) by Abisee et al.
- **Inference:** vLLM 0.23.0 ([vllm-project/vllm](https://github.com/vllm-project/vllm)) with FlashInfer CUTLASS NVFP4 backend
- **Hardware:** Tested on NVIDIA DGX Spark (GB10, SM121)
- **Prior art:** NVIDIA official NVFP4 checkpoints, bg-digitalservices Gemma-4 MoE unfuse plugin, FastContext-4B NVFP4 precedent

## Quick Start

```bash
# Download from HuggingFace
hf download r0b0tlab/VibeThinker-3B-NVFP4 --local-dir ./vibethinker-3b-nvfp4

# Serve with vLLM 0.22.0+ (pip or Docker)
vllm serve ./vibethinker-3b-nvfp4 \
    --quantization modelopt \
    --kv-cache-dtype fp8 \
    --attention-backend flashinfer \
    --gpu-memory-utilization 0.85 \
    --max-model-len 32768 \
    --enable-prefix-caching \
    --enforce-eager \
    --trust-remote-code

# Or via Docker
docker run --gpus all -v $(pwd):/mnt/model:ro \
    -p 8000:8000 ghcr.io/r0b0tlab/vibethinker-3b-nvfp4
```

## Quantization Recipe

| Parameter | Value |
|-----------|-------|
| Tool | NVIDIA Model Optimizer 0.44.0 |
| Config | `NVFP4_DEFAULT_CFG` (W4A4) |
| Group size | 16 |
| Calibration dataset | `abisee/cnn_dailymail` 3.0.0 |
| Calibration samples | 512 |
| Sequence length | 1024 |
| Batch size | 16 |
| Export | `export_hf_checkpoint` with `torch.inference_mode()` |
| Quantized layers | 903 (all Linear layers) |
| Exclusions | lm_head (tied with embed_tokens: `tie_word_embeddings=true`) |

### Reproduce

```bash
uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install torch --index-url https://download.pytorch.org/whl/cu130
uv pip install "transformers>=5.4" safetensors accelerate datasets
uv pip install "nvidia-modelopt[hf]>=0.44.0"

python3 -c "
import torch, modelopt.torch.quantization as mtq
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from modelopt.torch.export import export_hf_checkpoint

model = AutoModelForCausalLM.from_pretrained('WeiboAI/VibeThinker-3B',
    torch_dtype=torch.bfloat16, device_map='cpu', low_cpu_mem_usage=True)
for n, p in model.named_parameters(): p.data = p.data.to('cuda')
for n, b in model.named_buffers(): b.data = b.data.to('cuda')
tokenizer = AutoTokenizer.from_pretrained('WeiboAI/VibeThinker-3B')
calib = load_dataset('abisee/cnn_dailymail', '3.0.0', split='train[:512]')
def fwd(m):
    for i in range(0, 512, 16):
        b = calib[i:i+16]['article']
        m(**tokenizer(b, return_tensors='pt', padding=True, truncation=True,
            max_length=1024).to('cuda'))
mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG, fwd)
with torch.inference_mode(): export_hf_checkpoint(model, export_dir='./output')
print('Done!')
"
```

## How to Verify

```bash
# 1. Download and serve (see Quick Start above)
# 2. Test identity/reasoning
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"What is 15 * 7 + 3?"}],"max_tokens":256}'
# Expected: 108 (correct arithmetic through <think> reasoning)
```

## Benchmarks (GB10 / SM121)

**c1 tg128 (prefill=2048, output=128, vLLM 0.23.0, random dataset):**

| Metric | BF16 | NVFP4 | Speedup |
|--------|------|-------|---------|
| Output tok/s | 27.4 | **71.3** | **2.60×** |
| TTFT | 233 ms | 51 ms | 4.57× |
| TPOT | 34.8 ms | 13.6 ms | 2.56× |

**Concurrency ramp (tg128):**

| Concurrency | BF16 tok/s | NVFP4 tok/s | Speedup |
|-------------|-----------|------------|---------|
| c1 | 27.4 | 71.3 | 2.60× |
| c2 | 65.5 | 128.0 | 1.96× |
| c4 | 123.3 | 243.1 | 1.97× |
| c8 | 219.5 | 423.0 | 1.93× |

**Depth stability (c1 tg128):**

| Depth | NVFP4 tok/s | TTFT |
|-------|------------|------|
| d0 | 71.3 | 51 ms |
| d4096 | 67.7 | 76 ms |
| d8192 | 60.5 | 274 ms |
| d16384 | 56.4 | 383 ms |

Full benchmark report: [github.com/r0b0tlab/vibethinker-3b-nvfp4](https://github.com/r0b0tlab/vibethinker-3b-nvfp4)

## `hf_quant_config.json`

```json
{
    "producer": {"name": "modelopt", "version": "0.44.0"},
    "quantization": {
        "quant_algo": "NVFP4",
        "kv_cache_quant_algo": null,
        "group_size": 16,
        "exclude_modules": ["lm_head"]
    }
}
```

## Notes and Limitations

- **This is an NVFP4 quantization of a reasoning model.** The model outputs `<think>` tags — use `--reasoning-parser deepseek_r1` in vLLM for chat use.
- **Quantization scope:** All Linear layers (903 quantizers). `lm_head` excluded (tied with `embed_tokens`). No multimodal exclusion needed (text-only model).
- **Calibration is text-only.** The model has no vision/audio components.
- **Small model dimensions (hidden=2048):** This model benefits significantly from NVFP4 FP4 tensor cores. Larger models with hidden ≥ 4096 may show smaller or even negative speedups — always benchmark your specific model.
- **Not re-tested on full benchmark suites:** Correctness verified on identity tasks. Full benchmark eval (AIME, LiveCodeBench) not yet reproduced on this quantized checkpoint. The original model's reasoning quality should be preserved given the 4/4 identity test pass.

## License

MIT (inherited from base model [WeiboAI/VibeThinker-3B](https://huggingface.co/WeiboAI/VibeThinker-3B)).  
Quantization artifact copyright 2026 r0b0tlab, distributed under the same MIT license.  
Calibration data from [CNN/DailyMail](https://huggingface.co/datasets/abisee/cnn_dailymail) (Apache 2.0).

## Citation

```bibtex
@misc{vibethinker2026,
  title={VibeThinker: Optimizing Post-training for Small Model Reasoning},
  author={WeiboAI},
  year={2026},
  url={https://huggingface.co/papers/2606.16140}
}

@misc{qwen2.5-coder2025,
  title={Qwen2.5-Coder: Code is More Than Language},
  author={Qwen Team, Alibaba Group},
  year={2025},
  url={https://huggingface.co/Qwen/Qwen2.5-Coder-3B}
}

@misc{modelopt2025,
  title={NVIDIA TensorRT Model Optimizer},
  author={NVIDIA},
  year={2025},
  url={https://github.com/NVIDIA/TensorRT-Model-Optimizer}
}

@misc{see2017cnndailymail,
  title={Get To The Point: Summarization with Pointer-Generator Networks},
  author={See, Abigail and Liu, Peter J. and Manning, Christopher D.},
  year={2017},
  journal={ACL},
  url={https://huggingface.co/datasets/abisee/cnn_dailymail}
}

@misc{vllm2025,
  title={vLLM: Easy, Fast, and Cheap LLM Serving},
  author={vLLM Team},
  year={2025},
  url={https://github.com/vllm-project/vllm}
}
```