Qwen3.6-35B-A3B-DSV4Pro-FP4

Model Summary

This is the NVFP4 (W4A16) quantized version of the Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill model.

Key Characteristics:

Vision Tower Kept: Both the speculative decoding MTP heads have been pruned to reduce model overhead, but the visual tower (model.visual) is fully preserved. To guarantee multimodal capability and high accuracy, the visual tower is kept unquantized in high-precision bfloat16.
NVIDIA ModelOpt FP4 (GS16) Quantization: Quantized using NVIDIA ModelOpt with Group Size 16 (GS16).
- Note on Group Size: While we tested Group Size 128 (GS128) to shrink the footprint further, the vLLM Marlin FP4 CUDA kernel (marlin_mm) only supports a group size of 16. Attempts to serve GS128 will result in a serving engine crash (Invalid thread config). Therefore, GS16 is the only viable serving configuration.
Disk Footprint: ~21 GB.

Evaluation Benchmarks (GS16, thinking-on)

Evaluated using the local test suite (temperature 0.6, top_p 0.95):

GSM8K Accuracy: 85.00% (17/20 correct)
ARC-Challenge Accuracy: 80.00% (16/20 correct)
GPU KV Cache (RTX 5060 Ti / vLLM): 534,588 tokens (FP8 KV Cache enabled). At a target concurrency of 32, this allocates a physical slice of 16,705 tokens per concurrent request (prefix caching enabled).

Original Distill Model Information (`nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill`)

On Qwen3.6-35B-A3B (MoE, 3B active), we use LoRA to distill the way DeepSeek-V4-Pro reasons (with thinking-on) plus its agentic behavior — purpose-built as a fast task orchestrator (decompose → delegate → verify) for Lynn Agent.

This is the MoE counterpart of the 27B Dense sister model: same R6000 GPU, same teacher, same recipe, on a sparse architecture.

⚠️ Distilling a thinking style ≠ distilling knowledge/capability: the goal is "learn how to reason and how to converge", not to inject knowledge or raise the capability ceiling.

Training details

Base: Qwen3.6-35B-A3B (MoE, 3B active, BF16 base)
Method: LoRA, r = 64, α = 128, dropout = 0.05, targets = all attention + MLP projections
Optim: paged_adamw_8bit, cosine LR, warmup 0.03, ~1 epoch
Teacher: DeepSeek-V4-Pro (thinking-on + agentic)
Data: ~1,842 distillation samples (lynn_prod spec). Trajectories = DS-V4-Pro multi-step reasoning under thinking-on (<think>) + ReAct-style tool calls (think one step → call one tool → observe → loop).
- The tool "execution results" are SIMULATED, not actually run — each result line is improvised by a small fast model (DeepSeek-V4-Flash) role-playing the runtime, not real sandbox execution.
- Training masks those fabricated results — the model learns only "how to think / how to call tools", never the made-up outputs (so it doesn't learn to fabricate tool returns).

Attribution (the method is not original — it is a combination of published techniques)

ReAct (interleaved reasoning + acting): Yao et al., 2022, arXiv:2210.03629 (ICLR 2023)
STaR (bootstrapping reasoning traces): Zelikman et al., 2022, arXiv:2203.14465
Self-Instruct / Baize self-chat: Wang et al., 2022; Xu et al., 2023, arXiv:2304.01196
AgentTuning: Zeng et al., 2023, arXiv:2310.12823
ToolBench / ToolLLM: Qin et al., 2023, arXiv:2307.16789
DeepSeek-R1 reasoning distillation: DeepSeek-AI, 2025, arXiv:2501.12948

Serving & Inference with vLLM

Load using the standard compressed-tensors path inside vLLM. Here is the recommended serving configuration:

vllm serve /path/to/Qwen3.6-35B-A3B-DSV4Pro-Vision-FP4-GS16 \
  --served-model-name nvidia/Agents-A1-FP4 \
  --enforce-eager \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --max-model-len 128000 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching

🇨🇳 中文版 (Chinese Version)

模型摘要

这是 Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 模型的 NVFP4 (W4A16) 量化版本。

保留视觉塔 (Visual Tower): 我们移除了投机解码的 MTP 头以降低运行开销，但**完整保留了视觉塔 (model.visual)**。视觉塔保持未量化的 bfloat16 高精度格式，以确信优秀的多模态理解能力。
NVIDIA ModelOpt FP4 (GS16) 量化: 使用 NVIDIA ModelOpt 以 Group Size 16 (GS16) 进行 PTQ 校准。
- 关于组大小的说明: 虽然我们测试了 Group Size 128 (GS128) 以进一步减小模型体积，但在 vLLM 中，Marlin FP4 CUDA 算子 (marlin_mm) 目前仅支持 group size 为 16。尝试加载 GS128 模形将导致服务崩溃（提示 Invalid thread config）。因此，GS16 是唯一可用的运行服务配置。
磁盘大小: ~21 GB。