File size: 3,695 Bytes
f0f7133 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | # Reproducing SlothE-T 25M
End-to-end recipe for the ternary Zhuyin→Traditional-Chinese model
`slothe_t_25m_ce_ls32_ep24` and its GGUF. Trained on 2× RTX 5090 (DDP).
## 0. Inputs
| artifact | role |
|---|---|
| `train_e_g2pw.bin` | packed training set: zh-TW sentences, g2pW-labeled (syllable→char aligned) |
| `syl_vocab.json` | 1539-entry syllable (input) vocab |
| `tokenizer/` | char tokenizer (8342 chars) |
| `phonetic_table.tsv` | syllable→legal-character table (Taiwan readings) |
| `syl2legal.npz` | the same table as a dense `[1539 × 8342]` bool mask used by the legality-masked head |
Data prep: raw zh-TW corpus → g2pW phonetic labeling → aligned `(syllable, char)`
pairs packed into `train_e_g2pw.bin`. The held-out eval sets
(`eval/reference_heldout.tsv`, `eval/testset.tsv`) are **filtered to exclude any
sentence present in the training corpus** — this is what makes the reported
numbers honest (see the leakage note in the model card).
## 1. Train (teacher-free CE + label smoothing, long schedule)
`run_ce_ls_32.sh`:
```bash
python3 -m torch.distributed.run --nproc_per_node=2 train_slothe_ternary.py \
--data train_e_g2pw.bin --vocab syl_vocab.json --tokenizer tokenizer \
--out slothe_t_25m_ce_ls32 \
--dim 352 --depth 16 --heads 8 --kv-heads 2 --ffn 960 --embed-norm \
--quant ternary --weight-quant median --pre-norm \
--label-smoothing 0.1 \
--batch 384 --epochs 32 --save-every 4 --lr 2.5e-3
```
- `--quant ternary --weight-quant median` → W1.58A8 QAT: ternary weights
{−1,0,+1} × per-output-channel **absmedian** scale, int8 activations, STE.
- `--pre-norm` → SubLN RMSNorm before each ternary linear (stability).
- boundary blocks stay fp16 (`fp_boundary=1`, the default).
- `--save-every 4` snapshots every 4 epochs → `slothe_t_25m_ce_ls32_ep{4,8,…,32}`.
- **No `--teacher`** — teacher-free. Distillation was tried and only matched this.
- effective batch = 384 × 2 GPUs = 768.
## 2. Select the peak epoch (early stopping on held-out)
`gate_cels32_snap.sh` gates each snapshot on the held-out sets and prints the
curve. The model **peaks at epoch 24** and overfits after:
```bash
python3 gate_slothe_ternary.py --model slothe_t_25m_ce_ls32_ep24 \
--tokenizer tokenizer --table phonetic_table.tsv \
--testset ../eval/testset.tsv --mspy ../eval/reference_heldout.tsv
```
Expected held-out: **免選字 76 % · homophone-hard 86 % · toneless 77 %**
(ep32 regresses to ~73 % 免選字 — take **ep24**).
## 3. Convert to ternary GGUF
Two steps (torch only needed for extraction):
```bash
# a) extract effective ternary weights + fp tensors from the checkpoint (needs torch)
python3 extract_slothe.py slothe_t_25m_ce_ls32_ep24/slothe.pt \
-> slothe_tensors.npz + slothe_config.json + roles.json
# b) pack GGUF (numpy + gguf-py only): ternary linears -> TQ2_0 (256-padded),
# fp tensors -> f16, custom "slothe" arch metadata + syllable vocab
python3 pack_gguf.py -> slothe-t-25m.gguf
```
The ternarization baked into the GGUF is exactly the trainer's inference-time
quant at `quant_alpha=1.0` (fully annealed): `code = round(clamp(w/scale, −1, 1))`,
`scale = median(|w|)` per output channel. Because the effective weights are exact
ternary multiples, requantizing them to TQ2_0 is **loss-free** (verified by
round-trip: `max|dequant − effective| < 1e-3`). See `NAMES.md` for the
GGUF-tensor ↔ checkpoint-tensor name map and the 256-padding of in-features.
## Environment
- PyTorch (CUDA) for training/extraction; `numpy` + `gguf` (gguf-py) for packing.
- 2× RTX 5090 for the DDP recipe above; a single GPU works with `--nproc_per_node=1`
(halve the effective batch or double `--batch`).
|