jarrelscy/deepseek-v4-flash-0731-vision

DeepSeek V4 Flash 0731 — Vision (projector-only graft)

STATUS: WORK IN PROGRESS. The projector here is the current best from a reading-focused SFT run (sft_wb_read_trace4, projector-only, 8×B200), best mean reward 0.324 @ cycle 1620 on the held-out reading/click val set (T=1.0). This supersedes the earlier GRPO cycle 555 checkpoint. Training is ongoing (a wider multi-scale corpus is in progress); expect further updates.

Note on architecture change: this checkpoint adds 2 residual MLP blocks on top of the base patchmerger projector (see below), so it is not loadable by the pre-patchmerger_resmlp serving code — the residual stack would be silently skipped. The updated vLLM overlay in this repo (mm_projector_type: patchmerger_resmlp) runs them. The residual-block forward was verified bit-exact against the training projector; it has not yet been boot-tested on the serving rig (see Serving).

This repo contains only the vision components of a projector-only vision graft onto DeepSeek V4 Flash:

• frozen text backbone — deepseek-ai/DeepSeek-V4-Flash-0731 (revision 9e165c30e2704aec5d9d593cce3eebd58bbef1cb), referenced, not included (~176 GB — download it separately)
• frozen MoonViT vision tower (vision_tower.safetensors, 329 tensors, 834 MB) — Kimi-K2.6 MoonViT, filtered from the exolabs/Kimi-K2.6-vision export
• trainable ~107 M parameter projector (mm_projector.safetensors, 215 MB, bf16) — the only trained part: LayerNorm(1152) → 2×2 patch-merge concat (4608) → Linear 4608×4608 → GELU → Linear 4608→4096 ("patchmerger" base) → 2 × zero-init residual MLP blocks (x + linear_2(GELU(linear_1(LN(x)))), 4096-dim each), spliced into the prompt embeddings at image positions

What does "0731" mean?

The backbone is the DeepSeek V4 Flash 0731 GA release (July 31 checkpoint of the text-only MoE): 43 layers (2 sliding-window + 21 compressed-sparse + 20 heavily-compressed attention), shared-KV MQA (1 KV head / 64 query heads / head_dim 512), MoE top-6 of 256 experts, fp8 attention + FP4 experts (expert_dtype: fp4, served through W4A16 fused MoE kernels). It is completely frozen here — this repo ships no backbone weights; the serving directory is assembled by symlinking the base-model shards next to the files in this repo (instructions below).

Attribution

• WebBrain — the projector architecture, its warm-start weights, the 64-id routing palette, and the NaViT featurization all derive from webbrain-one/DeepSeek-V4-Flash-0731-Vision-NVFP4. The projector here was initialized from their mm_projector.safetensors and then trained further with GRPO.
• Original method — the projector-graft recipe originates from Harry Partridge (@part_harry_, "GLM 5.2 With Vision"), via the community glm-vision reproduction lineage: 0xSero/fable-glm-vision (weights) and 0xSero/glm-vision-sft-mix (data mix).
• MoonViT tower — Moonshot AI's Kimi-K2.6 vision encoder, via the exolabs/Kimi-K2.6-vision export. The kimi_k25_*.py / media_utils.py processor files are the Kimi processor code.
• Base model — deepseek-ai/DeepSeek-V4-Flash-0731 (MIT license, inherited here).

Training

Reading-focused SFT (run sft_wb_read_trace4), projector-only — the optimizer holds projector parameters exclusively; backbone and tower cannot move by construction. This run follows a GRPO warm start (the earlier cycle 555 upload) and targets the projector's weakest axis: pixel-accurate reading of dense and small text. Two supervision levers drive it:

• an align-MSE box loss (--align-boxes, --align-per-token) that forces each output token to attend to the image cell its word actually occupies, so the projector reads rather than confabulates;
• a featurize-time adaptive upscale (small-glyph images enlarge so sub-cell glyphs cross the 28-px merge cell) plus per-source sampling reweight toward starved text sources (IAM / TextVQA / OCRVQA).

The projector was grown with 2 zero-init residual MLP blocks (added capacity that reproduces the blockless checkpoint bitwise at init and moves off identity only as training proceeds). Data mix (web/art/OCR): multiple-choice art (WikiArt via the glm-vision mix), OCRVQA / TextVQA / IAM handwriting / rendered text (the_cauldron), UI screenshots with a dense click-distance reward (wave-ui-25k, ShowUI-desktop), and a synthetic multi-scale rendered-text corpus carrying word boxes for the align loss.

Best mean reward 0.324 @ cycle 1620 on the held-out val set (T=1.0), with reading sources (IAM / art / OCRVQA / TextVQA) measurably improved over the GRPO warm start; screenshot click reward stays strong (~0.6). Uploaded checkpoints:

file	keys	use
mm_projector.safetensors	mm_projector.pre_norm.* / linear_1.* / linear_2.* / res_blocks.N.*	drop-in for vLLM serving (patchmerger_resmlp overlay)
training_checkpoint/projector_sft_wb_read_trace4_best.safetensors	pre_norm.* / linear_1.* / linear_2.* / res_blocks.N.*	raw trainer checkpoint (same tensors, trainer key names)
training_checkpoint/projector_grpo_wb_partial_cycle555.safetensors	pre_norm.* / linear_1.* / linear_2.*	prior GRPO warm-start projector (base geometry, no residual blocks)

Architecture notes you must not break

• Hash routing / routing palette. The first 3 MoE layers of DSV4 route by token id (tid2eid[input_ids]), not by a learned gate. The token ids placed at image positions therefore decide which experts process image tokens in layers 0–2. This checkpoint cycles the 64-id routing_palette from config.json across image positions (WebBrain scheme). Train and serve must assign identical image-position ids — the vLLM model code in this repo does this for you; do not modify it.
• Image span markers: begin_of_image_token_id 128129, media_placeholder_token_id 128130, end_of_image_token_id 128131.
• Featurization is NaViT-style: bicubic resize on 0–255 floats, zero-pad to 28-px multiples, Kimi normalization stats, ~2 M pixel budget (preprocessor_config.json + dsv4_vision_processing.py are the exact training-side processors).

Serving in vLLM — exact reproduction

Boot-test caveat for this checkpoint. The end-to-end boot below was verified for the base patchmerger projector. This upload adds 2 residual MLP blocks and a new mm_projector_type: patchmerger_resmlp path in the overlay (DeepseekV4VisionPatchMergerResMLP). That residual-block forward is verified bit-exact against the training projector (offline, plain-torch), but has not yet been re-booted on the serving rig. If you serve this checkpoint, confirm startup + a read before relying on it; the base-projector serving path is unchanged and still boot-tested.

Verified end-to-end (base projector) on 4× RTX PRO 6000 Blackwell Max-Q 96 GB (SM 12.0), TP4: boots to Application startup complete, serves image+text requests including a 896×896 image (~1027 image tokens) through the MoE prefill with no kernel errors. config.json selects the projector via vision_config.mm_projector_type (patchmerger_resmlp here, with projector_res_blocks: 2); the overlay falls back to the base patchmerger / mean-pool paths for the other values.

1. Assemble the model directory

Download the base model, then symlink its shards next to this repo's files:

export HF_HOME=/data/huggingface   # or wherever you keep blobs
BASE=$(hf download deepseek-ai/DeepSeek-V4-Flash-0731 --revision 9e165c30e2704aec5d9d593cce3eebd58bbef1cb)
THIS=$(hf download jarrelscy/deepseek-v4-flash-0731-vision)

M=/data/huggingface/dsv4flash-models/0731-vision-wb
mkdir -p "$M" && cd "$M"
ln -s "$BASE"/model-000*.safetensors .
ln -s "$BASE"/encoding "$BASE"/tokenizer.json "$BASE"/tokenizer_config.json \
      "$BASE"/generation_config.json "$BASE"/LICENSE .
cp "$THIS"/config.json "$THIS"/preprocessor_config.json \
   "$THIS"/model.safetensors.index.json "$THIS"/chat_template.jinja \
   "$THIS"/dsv4_vision_processing.py "$THIS"/kimi_k25_processor.py \
   "$THIS"/kimi_k25_vision_processing.py "$THIS"/media_utils.py \
   "$THIS"/mm_projector.safetensors "$THIS"/vision_tower.safetensors .

model.safetensors.index.json (included here) is the merged index mapping all 72,652 tensors — backbone tensors to the symlinked shards, vision tensors to the two safetensors files above.

2. vLLM code: image + overlay + patches

Serving runs the production vLLM container image voipmonitor/vllm:gilded-gnosis-v20-vllm749050e-si8110e3e-fi801d57a-cu132-20260730-r14 (vLLM base commit 749050e per the tag) with two modifications, both shipped in this repo:

1. Vision model overlay (vllm/overlay/) — 7 files bind-mounted read-only over the installed vLLM package (see the launch script for the exact mount targets). These add DeepseekV4VisionForConditionalGeneration (model_type: deepseek_v4_vision): MoonViT + WebBrain patchmerger projector grafted onto the DSV4 text model, text weights delegated to the native DSV4 loader, 64-id palette hash routing at image positions (with the int32/int64 palette-dtype fix for vLLM V1), and the fp8 → deepseek_v4_fp8 quant-method promotion for the composite vision config.
2. W4A16 MoE kernel fix (vllm/patches/sparkinfer_w4a16_tile_k.patch) — applied to the in-image sparkinfer/moe/_shared/kernels/w4a16/kernel.py at container start. Without it, request-time MoE prefill crashes with force_tile_config fc2 tile (tile_k=32, tile_n=512) does not fit problem N/K=4096/512: the FC2 "ultra-wide" tile override hard-codes tile_k=32, which is rejected by the launch-boundary _candidate_tile_fits floor (tile_k >= 64). The patch gates the override behind that same floor so it falls back to the fit-checked (fc2_tile_n=256, fc2_tile_k=64) slab — numerically identical, only the FC2 output-tile width changes. This bug is independent of the vision path (it hits pure-text MTP too).

Provenance patches: the overlay code was developed on a local vLLM fork, branch dsv4flash-vision, head commit 53963a0c5 — vllm/patches/0001…0004-*.patch are those commits (git format-patch), and 0005-dsv4flash-vision-worktree-uncommitted.patch is the final uncommitted fix set (palette dtype + quant-method promotion). The overlay files are the authoritative, boot-tested code — they were regenerated against the container's vLLM version (vllm/regenerate.sh shows how); the patches are reference material if you want to port the graft onto a different vLLM tree.

3. Launch

vllm/launch_wb_rollout.sh is the exact boot-tested launcher. Edit the paths at the top (OVERLAY, MODEL, the kernel.py.patch mount) to your locations, then:

PORT=8001 TP=4 bash vllm/launch_wb_rollout.sh

Core of what it runs (see the script for the full env — the B12X_* / SPARKINFER_* / CUTE_DSL_ARCH=sm_120a variables are tuned for SM 12.0 Blackwell workstation GPUs; adjust arch-specific ones for other hardware):

# inside the container, after applying the kernel patch (the script exits if the patch fails):
vllm serve /models/dsv4-vision-wb \
  --served-model-name deepseek/v4flash \
  --host 0.0.0.0 --port 8001 \
  --trust-remote-code \
  --kv-cache-dtype fp8 --block-size 256 --load-format safetensors \
  --tensor-parallel-size 4 --moe-backend b12x --linear-backend b12x \
  --gpu-memory-utilization 0.90 --max-model-len 16384 --max-num-seqs 16 \
  --max-num-batched-tokens 4096 --max_cudagraph_capture_size 256 --enable-chunked-prefill \
  --enable-prefix-caching \
  --attention-backend B12X_MLA_SPARSE \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --no-enable-flashinfer-autotune \
  --enforce-eager

Note: the palette rewrite at image positions is position-dependent — keep the launch script's prefill/batching settings as shipped (--max-num-batched-tokens 4096 covers a full ~2 M px image prompt in one prefill chunk).

4. Sending images

Standard OpenAI chat completions with image content; the chat template (chat_template.jinja) expands the image span, the processor produces the patches:

curl http://localhost:8001/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "deepseek/v4flash",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "What does the sign say?"}
  ]}],
  "max_tokens": 128
}'

Sanity check that image ingestion works: prompt token count must scale with image size (text-only ~16 → 448 px ~275 → 896 px ~1043 tokens).

What works / known limits

Works

• Image+text prefill through the full MoE stack (verified up to ~1027 image tokens in one prompt)
• Reading is the focus of this run: after the align-MSE + upscale + reweight levers, the model reads dense/small text and cursive (IAM) by attending to the pixels rather than confabulating, and failures read as honest perceptual errors or refusals rather than fluent wrong answers
• UI click grounding ("respond with the click coordinate as (x, y)") stays strong (click reward ~0.6)

Limits

• Training is ongoing — this is a research checkpoint, not a release; a wider multi-scale reading corpus is in progress
• Reading of extreme scales/aspects still off the training band can degrade; mirrored/reversed text is a known hard case (augmentation for it is being added to the corpus)
• Synthetic inputs far from the web/art/OCR/reading training mix (e.g. solid-color fields) give unreliable answers
• Video path untested; the config's video plumbing is inherited from the Kimi processor and unused

License

The backbone is MIT (DeepSeek). The MoonViT tower inherits Moonshot AI's Kimi-K2.6 license terms; the projector weights and serving code in this repo are released under MIT.