BabyLM 2026 Strict-Small Submission — Baseline (FP32, 10 epochs)
Winning configuration from a systematic 15-way ablation of common training optimizations (mixed precision, Flash Attention, curriculum learning, dynamic batching, post-training INT8 quantization) on the BabyLM Strict-Small track. The plain FP32 baseline — no optimizations — won outright at a 10-epoch budget, statistically tied with FP16+Flash Attention.
Paper: How Much Do Common Training Optimizations Cost You? A Systematic Ablation Study on the BabyLM Strict-Small Track (BabyLM Workshop, EMNLP 2026).
Model details
| Architecture | GPT-2 decoder, 33.6M params (embed=384, heads=6, layers=8, context=256) |
| Tokenizer | GPT-2 BPE, 50,257 vocab |
| Training data | BabyLM-community/BabyLM-2026-Strict-Small (10M words, detoxified) |
| Training budget | 10 epochs (100M word exposure), plain FP32, no optimizations |
| Seed | 789 (best of 5 validated seeds) |
| Optimizer | AdamW, lr=3e-4, CosineAnnealingLR (eta_min=3e-5), grad_clip=1.0 |
Branches
main— final checkpoint (step 10,500, epoch 9), used for all full-eval and GLUE results below.chck_1M...chck_9M,chck_10M...chck_100M— 19 official word-count-milestone checkpoints (same seed/run), provided for AoA and fast-eval-across-training-steps benchmarks.
Results (n=5 seeds unless noted; see paper for full detail)
| Benchmark | Score |
|---|---|
| Validation perplexity | 136.786 ± 0.838 (seed 789: 136.07) |
| BLiMP | 62.44% ± 0.47 |
| BLiMP Supplement | 58.13% ± 0.81 |
| COMPS (base) | 54.11% ± 0.35 |
| WUGs | 50.16% ± 0.22 |
| EWoK | 50.89% ± 0.71 |
| Entity Tracking | 18.54% ± 0.25 |
| GlobalPIQA (non-parallel / parallel) | 46.80% ± 3.90 / 20.00% ± 2.13 |
| GLUE average | 62.57% |
| AoA correlation (Pearson r) | −0.031 ± 0.070 |
Intended use
Research artifact for the BabyLM Challenge 2026 (Strict-Small track). Trained on a 10M-word, developmentally-plausible corpus for sample-efficiency research, not intended for production use.
- Downloads last month
- 123