---
library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3.6-27B
base_model_relation: quantized
tags:
- compressed-tensors
- qwen3_6
- int8
- autoround
---

# Qwen3.6-27B INT8 AutoRound

This is an unofficial INT8 quantized version of the Qwen3.6-27B. 
It was created using [AutoRound](https://github.com/intel/auto-round).

## Available versions

* There are two versions.
* Main branch one is a little bit smaller by quantizing self_attn, and disabling group_size at the cost of the model's intelligence.
* For users with 48GB VRAM, just using Main branch is recommended. If you have more than that, gs128 branch might be better. The performance difference in practical use is minimal.

## Quantization details

| Field | Main branch | gs128 branch | 
|------|------|------|
| Base | `Qwen/Qwen3.6-27B` | `Qwen/Qwen3.6-27B` |
| Method | AutoRound (`intel/auto-round`), **custom** recipe | AutoRound (`intel/auto-round`), default recipe |
| Scheme | W8A16 | W8A16 |
| Bits | 8 | 8 |
| Group size | **-1** | 128 |
| Symmetric | yes | yes |
| Unquantized layers | `visual`, `mtp`, `linear_attn`, `embed_tokens`, `lm_head`  | `visual`, `mtp`, <code><strong>self_attn</strong></code>, `linear_attn`, `embed_tokens`, `lm_head`  |
| Calibration samples | 128 | 128 |
| Iterations | **1000** | 200 |
| Batch size | 8 | 8 |
| torch.compile | enabled | enabled |
| Size | **36.8GB** | **38.8GB** |
| GPU used for quant | 2× RTX 3090 | 2× RTX 3090 |


* For more information, please check quantize.py.

## Evaluation Results (KLD)

~~This model's KLD was measured using vLLM because Transformers couldn't load this model at the time. It's good enough for a sanity check, but it's not precise.~~ It's now fixed and re-evaluated.

Lower values indicate less degradation caused by quantization.
The main branch is used for the evaluation.

The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.

| Metric @ctx 512 | Value |
|---|---|
| Mean KLD | 0.00167 ± 0.00022 |
| Median KLD | 0.000567 |
| P90 / P95 / P99 KLD | 0.00207 / 0.00297 / 0.00781 |
| P99.9 / Max KLD | 0.0490 / 18.72 |
| Top-1 Agreement | 98.51% |
| PPL (Base → Quantized) | 7.7415 → 7.7410 |
| ln(PPL Ratio) | -0.000076 |

* Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats. 


## Quantization log

* Please check log.txt. 

## How to use

* This model is tested on latest docker.io/vllm/vllm-openai:cu130-nightly.
* vLLM is recommended.

* Example args (For 2x 3090 Users) :
```
vllm serve ./Qwen3.6-27B-INT8-AutoRound \
  --tensor-parallel-size 2 \
  --attention-backend FLASHINFER \
  --performance-mode interactivity \
  --max-model-len auto \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.932 \
  --compilation-config '{"mode":"VLLM_COMPILE","cudagraph_capture_sizes":[3]}' \
  -O3 \
  --async-scheduling \
  --language-model-only \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --default-chat-template-kwargs.preserve_thinking true \
  --mamba-cache-mode all \
  --mamba-block-size 8 \
  --enable-prefix-caching \
  --enable-chunked-prefill
```
* With these settings, you get around 129k context. You can also add --kv-cache-dtype fp8_e4m3 ~~--calculate-kv-scales~~ args to get about 252k tokens. Edit: --calculate-kv-scales silently corrupts kv cache, so please don't use it for now. For more information, please see [this huggingface discussion](https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/discussions/2).
* You can add --enforce-eager (you might need to remove --compilation-config) or set the PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False environment variable (requires --disable-custom-all-reduce) to allocate more VRAM to the KV cache, but the tk/s will be noticeably lower.
* Remove --speculative-config if you really want more context, but I highly recommend keeping it.
* Note: This information is based on my current understanding and testing. Optimal configurations may vary depending on your specific hardware setup. For further details, please refer to the official vLLM documentation.


## Acknowledgements
- [Lorbus](https://huggingface.co/Lorbus) for the README.md format
- [Alibaba / Qwen team](https://huggingface.co/Qwen) for the base [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) model
- [Intel AutoRound](https://github.com/intel/auto-round) team for the quantization framework
- [vLLM project](https://github.com/vllm-project/vllm) for the inference engine and Qwen3_5 MTP support