Devstral-Small-2-24B-AWQ-EAGLE3

An EAGLE3 speculative-decoding draft for mattbucci/Devstral-Small-2-24B-AWQ (Mistral/Ministral3 dense 24B coder). Format: generic LlamaForCausalLMEagle3 (1 draft layer, hidden 5120, draft_vocab 32000). The draft ships without embed_tokens — the serving stack supplies the target's embedding (standard EAGLE3).

Trained on the 3090 stack for the R9700 cross-team (they serve it on 32 GB RDNA4); also serviceable on the AWQ target directly.

Measured (2×RTX 3090 TP=2, single-user coding decode)

• accept_len ≈ 3.33 at --speculative-num-steps 3 (ceiling 4 → ~83% efficiency).
• Decode speedup: short 91.9 → 207.5 tok/s = 2.26×; ~16K 80.2 → 153.4 tok/s = 1.91× (accept_len 3.32 / 2.86).
• Best of the training checkpoints (1-pass 2.79 / 2-pass 3.33 / 3-pass 2.95 — 2 epochs is the sweet spot; 3 overfit).
• ttt/num-steps is capped at 3 by 24 GB training memory (the TTT-unroll activation makes ttt≥5 exceed 24 GB regardless of context). Serve at --speculative-num-steps 3; deeper serving does not help (the draft is trained to depth 3).
• ⚠ Spec-decode is a ≤~64K win; at true 256K decode depth speculative throughput collapses — use no-spec at depth.

Training

• SpecForge online EAGLE3, target run live (text-only AWQ Devstral on cuda:1, draft+optimizer on cuda:0).
• Data: ~6000 long multi-turn code conversations packed to ~15K tokens from OpenCodeInstruct (--max-length 16384), 2 epochs, lr 1e-4, 8-bit AdamW.
• Plain EAGLE3 (post-norm; no per-aux fc_norm) — the serving loader treats fc_norm as optional.

Serve (SGLang)

⚠ Serve against the Devstral text decoder (Ministral3ForCausalLM), not the full VLM wrapper. In sglang the full mattbucci/Devstral-Small-2-24B-AWQ loads as LlavaForConditionalGeneration, whose top-level class lacks set_eagle3_layers_to_capture → EAGLE3 fails to attach. The text-decoder weights are identical, so serve the decoder (or apply a wrapper-delegation patch).

# v0.5.13: set these to dodge an overlap-spec-v2 tvm_ffi ROCm-misdetect crash on NVIDIA
export TVM_FFI_GPU_BACKEND=cuda SGLANG_ENABLE_SPEC_V2=0
python -m sglang.launch_server \
  --model-path <Devstral text-decoder (Ministral3ForCausalLM) AWQ> --quantization awq_marlin \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path mattbucci/Devstral-Small-2-24B-AWQ-EAGLE3 \
  --speculative-num-steps 3 --speculative-eagle-topk 4 --speculative-num-draft-tokens 8 \
  --speculative-draft-model-quantization unquant --tp-size 2