bertholomus/DeepSeek-V4-Flash-0731-DSpark-Graph8

DeepSeek-V4-Flash-0731 DSpark Graph-8 on two GB10 nodes

A reproducible deployment and benchmark package for the official deepseek-ai/DeepSeek-V4-Flash-0731 checkpoint on two NVIDIA GB10-class nodes using vLLM tensor parallelism, DSpark speculative decoding, NVFP4 DS-MLA KV cache, FlashInfer B12X MoE, RoCE, and ordinary CUDA Graph capture through batch shape 8.

Independent community recipe. This repository is not an official DeepSeek release and contains no model weights. Download the official checkpoint from DeepSeek-AI. DSpark accelerates target-verified generation; it does not change the target model's knowledge or semantics.

Headline result

Stabilized balanced profile

The current safer production profile runs two concurrent sequences with GPU_MEMORY_UTILIZATION=0.75, 4,096 batched tokens, Graph-8, and the UCX registration-cache safeguards documented in docs/STABILITY.md. In a simultaneous two-request proof, both requests stayed scheduled (running=2, waiting=0) and each generated 1,005 exact bounded tokens in 21.05 seconds:

Concurrency	Per-stream decode	Aggregate decode	Post-baseline allocation errors
2	47.74 tok/s	95.48 tok/s	0 on both nodes

At the end of that proof the nodes retained approximately 20 GiB and 18 GiB MemAvailable, memory PSI avg10 was 0.00, and the service had zero restarts. This is the recommended default profile; it does not imply two simultaneous full-1M-token requests because the KV pool is shared.

Initial warm single-request qualification

On a two-node production pair, three warm single-request repetitions per shape produced:

Workload	Median decode	Median TTFT	DSpark acceptance
Count 1–300, exact bounded output	91.89 tok/s	0.186 s	98.81%
Twenty Python dataclasses	84.43 tok/s	0.249 s	98.66%
Low-predictability technical prose	40.08 tok/s	0.235 s	32.10%

These medians came from the initial Graph-8 qualification profile before the headroom hardening above. The replaced GGUF/llama.cpp lane delivered about 9.3 tok/s on the same two-node class, so the measured uplift was approximately 9.9×, 9.1×, and 4.3× respectively. These are workload-specific measurements, not a universal throughput floor, and should not be compared directly with the two-stream aggregate measurement.

Graph-8 improved matched decode over Graph-4 by:

Workload	Graph-4	Graph-8	Gain
Count 1–300	74.28	93.90 tok/s	+26.42%
Dataclasses	75.22	86.33 tok/s	+14.78%
Technical prose	32.12	40.87 tok/s	+27.24%

The K5 proposer verifies six-token shapes. Capturing [1,2,4,8] rather than stopping at 4 covers that shape.

Qualification gates

• Official checkpoint revision: 9e165c30e2704aec5d9d593cce3eebd58bbef1cb
• 48/48 safetensor shards verified independently on both ranks
• 96/96 draft parameters loaded; no skipped tensors
• Configured context: 1,048,576 tokens
• Stabilized-profile KV capacity observed at startup: 1,372,579 tokens (1.31× the configured full-context ceiling, shared across active requests)
• Stabilized two-request proof: 47.74 tok/s per stream, 95.48 tok/s aggregate, running=2, waiting=0
• Stabilized post-proof headroom: approximately 20 GiB / 18 GiB MemAvailable, memory PSI avg10=0.00
• Production exact uncached retrieval: 159,979 prompt tokens, 97.50 s, cached tokens 0
• Equivalent pre-promotion qualification pair: 949,896 prompt tokens, exact key retrieval, 1,057.98 s TTFT, cached tokens 0
• Native OpenAI-format tool call: passed
• Tool-result continuation: passed
• Closed-world strict JSON schema: passed
• Production service restart and post-restart smoke: passed

One unconstrained prose parity sample differed by one output token between speculative and non-speculative runs (256 vs. 255 tokens). Bounded outputs, retrieval keys, tool names/arguments, and strict JSON remained exact. Do not interpret this package as a claim of byte-identical unconstrained prose.

Canonical runtime profile

Model: DeepSeek-V4-Flash-0731
Tensor parallel: 2 nodes
DSpark draft tokens: 5 (K5)
KV cache: nvfp4_ds_mla
MoE backend: flashinfer_b12x
VLLM_USE_BREAKABLE_CUDAGRAPH=0
CUDA graph capture set observed: [1, 2, 4, 8]
--max-cudagraph-capture-size 8
--max-model-len 1048576
--max-num-seqs 2
--max-num-batched-tokens 4096
--gpu-memory-utilization 0.75
--block-size 256
UCX_MEM_MMAP_HOOK_MODE=none
UCX_RCACHE_MAX_UNRELEASED=1024
RoCE v2 / NCCL IB

Pinned runtime image:

ghcr.io/anemll/dspark-vllm-gx10:0.1.1
registry digest: sha256:a83948492cf13df455170fb42885f5ef4db54fefe0feff0f841ecbff464ac9d8
validated local image ID: sha256:3430d6614a8e2925f34d059af6caf05aff42387326db4d05639a60f10f2654d8

Quick start

1. Prepare both nodes

Each node needs:

• one GB10-class CUDA GPU;
• Docker with Compose and NVIDIA Container Toolkit;
• RoCE-capable connectivity visible under /dev/infiniband;
• the complete official 0731 checkpoint at the same local path;
• passwordless SSH only if you add your own coordinator wrapper.

Verify all 48 model shards against a trusted manifest before launch.

2. Configure node-specific environments

Copy runtime/.env.example to .env.worker and .env.head. Set a different VLLM_HOST_IP and NCCL_IB_GID_INDEX on each node. MASTER_ADDR must be the head node's fabric address on both ranks.

Do not reuse one GID index across nodes without checking sysfs: RoCEv2 GID indices can differ and can move after reboot or link changes.

Do not restore the earlier 0.80 GPU-utilization / 8,192-batched-token defaults without a fresh pressure-qualified experiment. Read docs/STABILITY.md before changing memory, sequence, graph, or watchdog settings.

3. Start worker, then head

Worker rank:

COMPOSE_DISABLE_ENV_FILE=1 NODE_RANK=1 HEADLESS=1 \
  docker compose --env-file runtime/.env.worker \
  -f runtime/docker-compose.yml -p deepseek-v4-flash-0731 up -d

Head rank:

COMPOSE_DISABLE_ENV_FILE=1 NODE_RANK=0 HEADLESS= \
  docker compose --env-file runtime/.env.head \
  -f runtime/docker-compose.yml -p deepseek-v4-flash-0731 up -d

Keep the API loopback-only unless you deliberately provide authentication and firewalling. If remote clients need access, put an authenticated reverse proxy in front of vLLM.

4. Verify

BASE_URL=http://127.0.0.1:8000 MODEL=deepseek-v4-flash-0731 \
  python3 scripts/production_proof.py --context-tokens 160000

BASE_URL=http://127.0.0.1:8000 MODEL=deepseek-v4-flash-0731 \
  python3 scripts/benchmark.py --repetitions 3 --output benchmark.json

Repository contents

• runtime/docker-compose.yml — two-node vLLM/DSpark Graph-8 profile
• runtime/.env.example — node-specific configuration template
• scripts/production_proof.py — model, tools, strict JSON, and long-context gates
• scripts/benchmark.py — streaming TTFT/decode/acceptance harness
• benchmarks/production-summary.json — sanitized measured results
• docs/METHODOLOGY.md — definitions, test conditions, caveats, and parity disclosure
• docs/STABILITY.md — OOM/headroom incident, stabilized defaults, startup-baseline rule, and promotion gates
• NOTICE — upstream projects and pinned revisions

Attribution

This package builds on the official DeepSeek model and community GB10 work from Anemll, MiaAI-Lab, Tony Deangelo, and Weschera. Exact repositories and commits are recorded in NOTICE.

License

Code and documentation in this package are MIT-licensed. The official model and weights are separately provided by DeepSeek-AI under MIT; obtain them from the upstream repository. Preserve all upstream notices when redistributing upstream code.