aleada/Phi-4-mini-instruct-W4A16

Phi-4-mini-instruct — W4A16 (compressed-tensors)

Standard W4A16 quantization of microsoft/Phi-4-mini-instruct, produced with llm-compressor (the official vLLM-team quantization toolkit) inside a reproducible Docker container. The artifact saves in compressed-tensors format; vLLM auto-detects the quantization config from the embedded config.json at load time. It is drop-in loadable by vLLM — no upstream patches, no client-side shims.

This release is part of an ongoing series of vLLM-friendly quantized packs maintained by atlas, a self-evolving agent project run by Alex Adamopoulos at assert.gr.

Reproducibility

Parameter	Value
Source model	microsoft/Phi-4-mini-instruct
Quantization tool	llm-compressor 0.12.0 (Neural Magic / vLLM team)
Quantization recipe	GPTQModifier
scheme	W4A16
targets	Linear
ignore	re:.*lm_head
graft (kept in source dtype)	—
sequential_targets	—
group_size	128 (scheme preset)
dampening_frac	0.01
Calibration dataset	ultrachat-200k
Calibration samples	256
max_seq_length	2048
Quantized size	2.71 GiB
Quantization time	—

What the quantization cost

Both arms were run identically — same tasks, same harness, same machine — against microsoft/Phi-4-mini-instruct. Recovery is this pack's score over the source model's.

Task	How it scores	Source	This pack	Recovery
arc_challenge · acc	likelihood, multiple choice — forgiving	0.5674	0.5486	96.7 %
arc_challenge · acc_norm	likelihood, multiple choice — forgiving	0.5896	0.5751	97.5 %
gsm8k · exact_match (flexible-extract)	generative, multi-step arithmetic — the sensitive one	0.8180	0.7521	91.9 %
gsm8k · exact_match (strict-match)	generative, multi-step arithmetic — the sensitive one	0.8180	0.7513	91.8 %
hellaswag · acc	likelihood, multiple choice — forgiving	0.5452	0.5376	98.6 %
hellaswag · acc_norm	likelihood, multiple choice — forgiving	0.7276	0.7139	98.1 %
mmlu · acc	likelihood, broad knowledge	0.6665	0.6457	96.9 %
winogrande · acc	likelihood, multiple choice — forgiving	0.7080	0.7238	102.2 %

• Generative tasks: 91.8–91.9 % — the ones that write an answer and are graded on it.
• Multiple-choice tasks: 96.7–102.2 % — ranking fixed options, where rounding error rarely changes the winner.

There is no single figure here on purpose. The two kinds of task answer different questions, and averaging them hides the answer that matters: on one of our packs the mean came to 100.1% while the generative task had lost nearly nine points. Multiple-choice scoring asks the model to rank options it is given — error that ruins a calculation rarely reverses such a ranking, and a row above 100% is noise, not improvement. A generated answer graded on its final number is where 4-bit rounding compounds.

Measured with lm_eval 0.4.13, torch 2.12.0, transformers 5.10.1, full standard task configurations, on 2026-09-05T13:25:42Z.

License

Inherits the license of the base model. By using this artifact you agree to the original license at the source link above. Atlas / assert.gr adds no additional restrictions on the quantized weights.

Usage with vLLM

docker run --runtime=nvidia --gpus all \
    -p 8000:8000 \
    -e HF_TOKEN=hf_XXX \
    vllm/vllm-openai:latest \
    aleada/Phi-4-mini-instruct-W4A16 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching

The model is the first positional argument — vLLM's --model flag is deprecated and slated for removal.

vLLM auto-detects compressed-tensors from the model's config — no --quantization flag required (it is accepted as a redundant hint). vLLM also picks the model's full native context window from config.json. If you hit KV-cache OOM on a smaller GPU, pin a shorter window with --max-model-len 16384 (or smaller) — leave it off to get the maximum the model was trained for. Once vLLM is running, hit it with any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="aleada/Phi-4-mini-instruct-W4A16",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Hardware target

Requires CUDA compute-capability ≥ 8.0 (Ampere or newer). Verified on NVIDIA RTX 3090 (compute 8.6) where the W4A16 path runs the language tower at INT4 weights / BF16 activations through vLLM's compressed-tensors kernels.

Weight-only INT4 is the point on this class of card: FP8 and NVFP4 checkpoints are native on Hopper and Blackwell but emulated or unusable on Ampere, where the INT4 Marlin kernels are what actually run fast.

Check this pack yourself

Quantization can drop or disable part of a model without failing: the pack loads, serves, and answers correctly while something its card says it kept is absent, or present and ignored by the runtime. Nothing errors, and the card still promises it.

Pack integrity check reads any published repo's metadata — safetensors headers and config.json, no weights — and reports whether its exclusion entries name real modules, whether anything from the source model failed to reach it, and whether anything is left at source precision without being declared. It runs entirely in your browser, so it reads exactly what you could read yourself.

Point it at this pack. Point it at someone else's.

About the maintainer

Alex Adamopoulos is the founder of assert.gr and the engineer behind the atlas self-evolving AI agent platform. Atlas runs a planner→executor→supervisor loop over a skill registry, backed by Postgres, Redis, Qdrant, and a multi-LLM vLLM deployment. Quantization releases like this one keep the open-source model ecosystem usable on consumer-grade hardware for self-hosted agent research.

Connect:

• HuggingFace: @aleada
• Company: assert.gr