---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
base_model_relation: quantized
quantized_by: blockblockblock
library_name: exllamav3
pipeline_tag: text-generation
tags:
  - exl3
  - exllamav3
  - quantized

quantization_format: exl3
bits_per_weight: 4.0
---

<div align="center">

# Qwen2.5 · 0.5B · Instruct

<sub><code>EXL3</code> &nbsp;·&nbsp; <b>4.0&nbsp;bpw</b> &nbsp;·&nbsp; 0.6&nbsp;GB &nbsp;·&nbsp; Dense &nbsp;·&nbsp; 24 layers</sub>

<br/>

[![format](https://img.shields.io/badge/format-EXL3-c63010?style=for-the-badge&labelColor=14120e)](https://github.com/turboderp-org/exllamav3)
[![bpw](https://img.shields.io/badge/bpw-4.0-6b8a76?style=for-the-badge&labelColor=14120e)](#quants)
[![size](https://img.shields.io/badge/size-0.6_GB-6b8a76?style=for-the-badge&labelColor=14120e)](#quants)
[![arch](https://img.shields.io/badge/arch-Dense_0.5B-c63010?style=for-the-badge&labelColor=14120e)](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)

[![base model](https://img.shields.io/badge/Base-Qwen%2FQwen2.5--0.5B--Instruct-2a2620?style=flat-square&logo=huggingface&logoColor=white)](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
[![quantized by](https://img.shields.io/badge/Quantized_by-blockblockblock-2a2620?style=flat-square&logo=huggingface&logoColor=white)](https://huggingface.co/blockblockblock)
[![collection](https://img.shields.io/badge/All_bpws-Collection-c63010?style=flat-square&logo=huggingface&logoColor=white)](https://huggingface.co/collections/blockblockblock/qwen25-05b-instruct-exl3-6a18c376ff69506d7e1e32cb)

</div>

---

> [!NOTE]
> An [ExLlamaV3](https://github.com/turboderp-org/exllamav3) build of [`Qwen/Qwen2.5-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) at **4.0 bits per weight**. See [Quants](#quants) for sibling repos at other bit&#8209;widths or browse the [collection](https://huggingface.co/collections/blockblockblock/qwen25-05b-instruct-exl3-6a18c376ff69506d7e1e32cb).

## Quants

<div align="center">

| BPW &nbsp; | &nbsp; Head bits &nbsp; | &nbsp; Calibration rows &nbsp; | &nbsp; Size &nbsp; | &nbsp; KL&nbsp;&divide;&nbsp;fp16 &nbsp; | &nbsp; Status |
| :---: | :---: | :---: | ---: | :---: | :--- |
| 3.0 | 8 | 250 | 0.5&nbsp;GB | 0.1082 | [link](https://huggingface.co/blockblockblock/Qwen2.5-0.5B-Instruct-exl3-3.0bpw) |
| **4.0** | 8 | 250 | **0.6&nbsp;GB** | **0.0277** | <kbd>this repo</kbd> |
| 4.5 | 8 | 250 | 0.6&nbsp;GB | 0.0158 | [link](https://huggingface.co/blockblockblock/Qwen2.5-0.5B-Instruct-exl3-4.5bpw) |
| 5.0 | 8 | 250 | 0.6&nbsp;GB | 0.0074 | [link](https://huggingface.co/blockblockblock/Qwen2.5-0.5B-Instruct-exl3-5.0bpw) |
| 6.0 | 8 | 250 | 0.7&nbsp;GB | 0.0020 | [link](https://huggingface.co/blockblockblock/Qwen2.5-0.5B-Instruct-exl3-6.0bpw) |

<sub>KL&nbsp;&divide;&nbsp;fp16: mean KL-divergence from the fp16 source over wikitext rows &mdash; lower is closer to the original.</sub>

</div>

## Inference

<table>
  <thead>
    <tr>
      <th align="left" width="32%">Loader</th>
      <th align="left">Use it for</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://github.com/theroyallab/tabbyAPI"><b>TabbyAPI</b></a></td>
      <td>OpenAI&#8209;compatible HTTP server. Drop&#8209;in for OpenAI clients.</td>
    </tr>
    <tr>
      <td><a href="https://github.com/oobabooga/text-generation-webui"><b>text&#8209;generation&#8209;webui</b></a></td>
      <td>Local chat UI. Pick the <i>ExLlamaV3</i> loader from the model dropdown.</td>
    </tr>
    <tr>
      <td><a href="https://github.com/turboderp-org/exllamav3"><b>ExLlamaV3</b></a></td>
      <td>Direct Python API for embedding the model in your own code or pipeline.</td>
    </tr>
  </tbody>
</table>

## Download

```bash
pip install -U huggingface_hub

hf download \
  blockblockblock/Qwen2.5-0.5B-Instruct-exl3-4.0bpw \
  --local-dir ./Qwen2.5-0.5B-Instruct-exl3-4.0bpw
```

<details>
<summary><b>Quantization recipe</b> &nbsp;<sub>(advanced, embedded in <code>quantization_config.json</code>)</sub></summary>

<br/>

| Setting | Value |
| :--- | :--- |
| Format | `EXL3` |
| Bits per weight | `4.0` |
| Head bits | `8` |
| Calibration rows | `250` |
| Codebook | `MCG` |
| Out&#8209;scales | `always` |
| Parallel mode | `enabled` |

Loaded automatically by every ExLlamaV3 loader; reproduced here for searchability.

</details>

## License &amp; use

> [!IMPORTANT]
> Use and license **follow the [base model](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)**.
> Quantization adds no additional restrictions. Refer to the upstream repository for terms, citation, and safety documentation.

---

<div align="center">
  <sub><i>Quantized with <a href="https://github.com/Honkware/blockquant"><b>BlockQuant</b></a> &nbsp;·&nbsp; convention&nbsp;<code>{org}/{model}-exl3-{bpw}bpw</code></i></sub>
</div>