---
license: apache-2.0
base_model: mattbucci/Devstral-Small-2-24B-AWQ
tags:
  - eagle3
  - speculative-decoding
  - sglang
library_name: sglang
---

# Devstral-Small-2-24B-AWQ-EAGLE3

An EAGLE3 speculative-decoding **draft** for [`mattbucci/Devstral-Small-2-24B-AWQ`](https://huggingface.co/mattbucci/Devstral-Small-2-24B-AWQ) (Mistral/Ministral3 dense 24B coder). Format: generic `LlamaForCausalLMEagle3` (1 draft layer, hidden 5120, draft_vocab 32000). The draft ships **without** `embed_tokens` — the serving stack supplies the target's embedding (standard EAGLE3).

Trained on the 3090 stack for the R9700 cross-team (they serve it on 32 GB RDNA4); also serviceable on the AWQ target directly.

## Measured (2×RTX 3090 TP=2, single-user coding decode)

- **accept_len ≈ 3.33** at `--speculative-num-steps 3` (ceiling 4 → ~83% efficiency).
- **Decode speedup:** short **91.9 → 207.5 tok/s = 2.26×**; ~16K **80.2 → 153.4 tok/s = 1.91×** (accept_len 3.32 / 2.86).
- Best of the training checkpoints (1-pass 2.79 / **2-pass 3.33** / 3-pass 2.95 — 2 epochs is the sweet spot; 3 overfit).
- **`ttt`/num-steps is capped at 3** by 24 GB training memory (the TTT-unroll activation makes ttt≥5 exceed 24 GB regardless of context). Serve at `--speculative-num-steps 3`; deeper serving does not help (the draft is trained to depth 3).
- ⚠ Spec-decode is a **≤~64K** win; at true 256K decode depth speculative throughput collapses — use no-spec at depth.

## Training

- SpecForge online EAGLE3, target run live (text-only AWQ Devstral on cuda:1, draft+optimizer on cuda:0).
- Data: ~6000 long multi-turn code conversations packed to ~15K tokens from OpenCodeInstruct (`--max-length 16384`), 2 epochs, lr 1e-4, 8-bit AdamW.
- Plain EAGLE3 (post-`norm`; no per-aux `fc_norm`) — the serving loader treats `fc_norm` as optional.

## Serve (SGLang)

⚠ **Serve against the Devstral text decoder (`Ministral3ForCausalLM`), not the full VLM wrapper.** In sglang the full `mattbucci/Devstral-Small-2-24B-AWQ` loads as `LlavaForConditionalGeneration`, whose top-level class lacks `set_eagle3_layers_to_capture` → EAGLE3 fails to attach. The text-decoder weights are identical, so serve the decoder (or apply a wrapper-delegation patch).

```bash
# v0.5.13: set these to dodge an overlap-spec-v2 tvm_ffi ROCm-misdetect crash on NVIDIA
export TVM_FFI_GPU_BACKEND=cuda SGLANG_ENABLE_SPEC_V2=0
python -m sglang.launch_server \
  --model-path <Devstral text-decoder (Ministral3ForCausalLM) AWQ> --quantization awq_marlin \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path mattbucci/Devstral-Small-2-24B-AWQ-EAGLE3 \
  --speculative-num-steps 3 --speculative-eagle-topk 4 --speculative-num-draft-tokens 8 \
  --speculative-draft-model-quantization unquant --tp-size 2
```