andreaborio/DeepSeek-V4-Flash-Hebrus-GGUF

DeepSeek V4 Flash 0731 — Hebrus GGUF releases

This repository is the single release container for the versioned DeepSeek V4 Flash artifacts used by the Hebrus inference engine. The current release targets the 0731 DeepSeek V4 Flash checkpoint. Hebrus began as a fork of Salvatore Sanfilippo's antirez/ds4. Historical DS4 names remain only in filenames and serialized compatibility fields.

The runtime artifact embeds its routed MoE weights once as a self-describing ds4.expert_major.v2 store. It needs no sidecar, runtime repack, or second routed-weight copy.

Compatibility: existing filenames, the dsbox.json manifest, and the ds4.expert_major.v2 marker remain unchanged serialized contracts.

Versions

File	Checkpoint	Status	Runtime
DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-DS4-ExpertMajor-v2.gguf	0731	Current, opt-in	validated with Hebrus main at d39f841
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-DS4-ExpertMajor-v2.gguf	pre-0731	Previous release	Hebrus main at 57acfd4 or newer
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf	pre-0731	Legacy canonical source and conversion input	Generic GGUF tooling; rejected by current Hebrus inference

The 0731 artifact is currently opt-in: dsbox.json and the repository downloader still select the previous release by default. Pass the 0731 file path explicitly to use it.

ExpertMajor v1, sidecars, canonical MoE inference, and non-Metal inference are intentionally unsupported. The ExpertMajor v2 files use a Hebrus extension rather than a portable GGUF layout: generic GGUF runtimes cannot reconstruct or execute the opaque routed store.

Current artifact — 0731

Property	Value
Base model	deepseek-ai/DeepSeek-V4-Flash, 0731 checkpoint
File	DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-DS4-ExpertMajor-v2.gguf
Size	86,720,114,240 bytes (80.76 GiB)
SHA-256	d89dd628ed786ecf14285cb886459eec01df89fd7e7bf3cbff1416a551bcd966
Container	GGUF v3 with one embedded ds4.expert_major.v2 store
Routed geometry	43 routed layers, 256 experts per layer, top-6
Expert record	gate + up + down, 7,077,888 contiguous bytes
Quantization	IQ2_XXS routed gate/up, Q2_K routed down, selected Q8_0/F16/F32 dense tensors
Qualified hardware	Apple Silicon Metal, at least 64 GiB unified memory

The ExpertMajor v2 conversion changes only routed-weight storage order and the GGUF tensor directory. It does not requantize model values. Non-routed tensors are copied byte-for-byte from the canonical source and the routed payload is stored once.

Run with Hebrus

git clone https://github.com/andreaborio/hebrus.git
cd hebrus
make -j

./hebrus \
  -m /absolute/path/to/DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-DS4-ExpertMajor-v2.gguf \
  --ctx 8192

The ExpertMajor v2 store, Metal backend, and memory policy are selected automatically. No Metal, SSD-streaming, power, sidecar, pack, or format flag is needed. The qualified minimum is 64 GiB unified memory; 128 GiB is the more comfortable tier for this 80.76 GiB artifact.

Do not pass --ssd-streaming-cache-experts with this model: an explicit expert cache budget bypasses the phase cache schedule. Current Hebrus refuses the combination; older builds accepted it and produced wrong output silently.

Measured performance — 0731

Apple M5 Pro, 64 GiB unified memory, AUTO residency resolving to SSD streaming, prose prompt, 2026-08-04. Steady-state decode, meaning the expert cache has re-warmed after prefill:

Context	Prefill	Decode	TPOT p50	Expert cache hit
8K + 512	185.3 t/s	10.7 t/s	86.7 ms	0.862
2K + 512	126.1 t/s	12.0 t/s	76.1 ms	0.871
512 + 512	n/a	12.8 t/s	72.6 ms	0.891

A code-heavy prompt at 8K measured 11.0 t/s decode with hit rate 0.892.

Long-context lanes, same host and method (512-token steady-state decode, output validity checked on every lane):

Context	Prefill	TTFT	Decode	TPOT p50/p95	Expert cache hit
32K + 512	169.8 t/s	193 s	11.0 t/s	82.2 / 108.1 ms	0.925
65K + 512	147.1 t/s	446 s	8.8 t/s	108.0 / 129.0 ms	0.687
100K + 512	130.7 t/s	765 s	8.3 t/s	114.8 / 133.2 ms	0.691

32K is the best measured operating point — decode there is faster than at 8K, because the adaptive cache reaches its highest hit rate. The step down at 65K+ is a deliberate, measured anti-swap tier of the expert cache, not a regression. TTFT scales with prefill length; for repeated prefixes use the engine's KV-disk restore, measured below.

Read short generations carefully. Large prefill deliberately shrinks the expert cache to its correctness floor, then grows it back. A 128-token decode therefore measures the re-warming ramp, not steady state: the same 8K lane reports 6.3 t/s at 128 tokens and 10.7 t/s at 512. Compare only equal generation lengths.

The pre-0731 numbers previously published in this card do not carry over and have been replaced by the tables above.

Cutting TTFT on repeated prefixes — KV disk cache

The long TTFT above is the price of prefilling a long prompt once. When the same prefix comes back — a continued conversation, a re-run against the same document, a restarted server — the engine can restore the KV state from disk instead of recomputing it. Same host and method, measured 2026-08-05 across a full server restart:

Prompt	Cold	Warm (after restart)	Restore
~26.6k tokens	185.4 s	23.0 s	74 ms
~1.7k tokens	17.8 s	2.1 s	8 ms

hebrus-server -m <model.gguf> --ctx 32768   --kv-disk-dir ~/.hebrus/server-kv --kv-disk-space-mb 8192

Restore is automatic on the first prompt whose text prefix matches; the key is byte-exact, so a single changed character misses. Disk cost is about 14 KB per token (372 MiB for 26.6k tokens), and the server also writes intermediate and shutdown checkpoints, so size the budget above a single snapshot.

The warm number is not the restore time: the restore itself is 74 ms. The remaining seconds are the prompt suffix beyond the saved boundary — here about 1.6k tokens — plus the expert cache re-warming in the fresh process. Aligning a prompt to the saved boundary reduces it further.

Verification

The deterministic converter verifies source identity, GGUF metadata, non-routed tensors, every routed expert component, alignment, embedded manifest, and the complete ExpertMajor payload. The published SHA-256 above is the converter's pinned output digest and was re-verified against the uploaded file.

Limitations

• Apple Silicon Metal is the only qualified inference backend.
• At least 64 GiB unified memory is required.
• Current Hebrus deliberately rejects ExpertMajor v1, sidecars, CPU, CUDA, ROCm, and distributed inference for this model.
• Performance depends on storage state, memory pressure, context length, routing locality, and thermals.

License and attribution

The source repository declares MIT. DeepSeek retains the base-model copyright; the GGUF is redistributed under the base model's release terms. Quantization credit belongs to antirez and the DS4 project. The ExpertMajor v2 layout conversion is Hebrus.