Does it support MTP?
Does it support MTP?
vllm serve nvidia/Qwen3.6-27B-NVFP4
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
--trust-remote-code
--kv-cache-dtype fp8
--attention-backend flashinfer
--moe-backend marlin
--gpu-memory-utilization 0.4
--max-model-len 262144
--max-num-seqs 4
--max-num-batched-tokens 8192
--enable-chunked-prefill
--async-scheduling
--enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
--load-format fastsafetensors
--reasoning-parser qwen3
--tool-call-parser qwen3_xml
--enable-auto-tool-choice
Deleting - Incorrect
yes, MTP works on eugr's vllm. Tested.
model also heavily hallucinated on my setup so I switched back to Qwopus.
My DGX Spark Service Deployment:
[Unit]
Description=vLLM Qwen3.6-27B-NVFP4
After=network-online.target
[Service]
Type=simple
User=darkmatter2222
Environment=PATH=/home/darkmatter2222/vllm-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=CUDA_VISIBLE_DEVICES=0
Environment=PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
Environment=VLLM_ENGINE_READY_TIMEOUT_S=1800
Restart=on-failure
RestartSec=10
TimeoutStartSec=infinity
ExecStart=/home/darkmatter2222/vllm-env/bin/vllm serve
/home/darkmatter2222/models/Qwen3.6-27B-NVFP4
--host 0.0.0.0
--port 8006
--served-model-name qwen3.6
--trust-remote-code
--quantization compressed-tensors
--dtype auto
--kv-cache-dtype fp8
--language-model-only
--gpu-memory-utilization 0.80
--max-model-len 131072
--max-num-seqs 6
--max-num-batched-tokens 16384
--enable-prefix-caching
--enable-chunked-prefill
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
--reasoning-parser qwen3
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
[Install]
WantedBy=multi-user.target
You might be able to get more batch, haven't tested yet.