---
license: mit
base_model: karpathy/nanochat-d34
language:
- en
tags:
- nanochat
- text-generation
- karpathy
- single-gpu
- rtx-5090
datasets:
- HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
- HuggingFaceTB/smol-smoltalk
- cais/mmlu
- ai2_arc
- openai/gsm8k
pipeline_tag: text-generation
model-index:
- name: nanochat-d24
  results:
  - task:
      type: text-generation
    dataset:
      name: GSM8K
      type: openai/gsm8k
    metrics:
    - name: pass@1
      type: accuracy
      value: 0.1645
      verified: false
---

# nanochat-d24

Train an LLM from scratch of 1.38B param, Pretrain+SFT+RL (GRPO/GSM8K) on a **single RTX 5090** (32GB). Ported from [Karpathy's nanochat](https://github.com/karpathy/nanochat) ([`6ed7d1d`](https://github.com/karpathy/nanochat/commit/6ed7d1d82cee16c2e26f45d559ad3338447a6c1b), Mar 9 2026) into a [single ~2K line Python file](https://gist.github.com/Katehuuh/34c631b9a9ca5e930eba3903937d01e9/8ccb21a677b1da1682fa2a66e6d7ecaaaf30de93).

## Changelog

- **2026-04-22**: Add <a href="https://huggingface.co/spaces/Luminia/nanochat-d24-CPU"><img src="https://img.shields.io/badge/🤗%20Demo-Hugging%20Face-ffc107?logoColor=white" alt="Demo" style="position: relative; top: 2px;"></a>
- **2026-03-27**: [exl3 bpw4](https://huggingface.co/Nekochu/nanochat-d24/tree/bpw4) and [bpw8](https://huggingface.co/Nekochu/nanochat-d24/tree/bpw8) via [exllamav3 v0.0.27](https://github.com/turboderp-org/exllamav3/releases/tag/v0.0.27)
- **2026-03-23**: Initial release

## Evaluation Results

<!-- ChatCORE* = without HumanEval (5/6 tasks). val_bpb = base model metric only. -->

| Benchmark | Base (step 24K) | SFT (step 1942) | RL (step 300) | d20 SFT (official) | d30 (official) |
|-----------|:-:|:-:|:-:|:-:|:-:|
| ARC-Easy | 23.9% | 52.6% | **52.9%** | 38.8% | ~70% |
| ARC-Challenge | 25.0% | 42.5% | **41.9%** | 28.1% | - |
| MMLU | 23.6% | 35.3% | **35.6%** | 31.5% | ~40% |
| GSM8K (pass@1) | 0.0% | 10.2% | **16.5%** | 4.6% | ~20% |
| SpellingBee | 0.0% | 79.7% | **87.1%** | N/A | - |
| **ChatCORE**\* | -0.6% | 32.7% | **35.5%** | 8.8% | - |
| val_bpb | 0.830 | - | - | - | 0.705 |

*\*ChatCORE without HumanEval (5/6 tasks). Official d20 SFT from [Discussion #1](https://github.com/karpathy/nanochat/discussions/1). d30 approximate, community reports.*

## Architecture

24 layers, 1536 hidden dim, 12 MHA heads (head_dim=128), 32K BPE vocab, 2048 context. RoPE (theta=100k), QK-norm, unweighted RMSNorm, untied embeddings, ReLU² activation, logit softcap 15.0, value embeddings on alternating layers (~604M params at near-zero FLOPs), per-layer residual scalars, smear mechanism, backout mechanism, cuDNN SDPA full context.

<a href="https://hfviewer.com/Nekochu/nanochat-d24?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=Nekochu__nanochat-d24_card&amp;utm_content=embedded_card_open_viewer&amp;from=embedded-model-card" target="_blank" rel="noopener">
  <img
    src="https://hfviewer.com/api/card.svg?source=Nekochu%2Fnanochat-d24&amp;granularity=auto&amp;v=20260516-title-pills-card"
    alt="Architecture graph for Nekochu/nanochat-d24. Open in hfviewer"
    width="100%"
  />
</a>

### Differences from official nanochat d24

| Feature | Official (8xH100) | This model (1xRTX5090) |
|---------|-------------------|------------------------|
| Window pattern | SSSL | L (full context) |
| VE gate channels | 32 | 12 |
| Smear/Backout | Not used | Active |
| Attention | Flash Attention 3 | cuDNN SDPA |
| FP8 | Optional | Default (pretraining) |
| Pretraining data | [NVIDIA ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix) | [finepdfs_edu/dclm/fineweb_edu](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled) (8/100 shards) |

## Training

| Stage | Data | Duration | Steps | Details |
|-------|------|----------|-------|---------|
| **Pretrain** | see above | ~3.8 days | 29,232 | FP8, MuonAdamW, grad ckpt, B=4x32 |
| **SFT** | SmolTalk 460K + MMLUx3 + SimpleSpelling 200K + SpellingBee 80K + GSM8Kx4 | ~7.5h | 1,942 | BF16, bestfit packing, torch.compile |
| **RL** | GSM8K (16 samples/question) | ~5h | 300 | BF16, cosine LR (T_max=400), KV-cache |

![Training Pipeline](https://i.imgur.com/6Mg7Otx.png)

Notes:
- Single-GPU RL needs cosine LR decay to avoid policy collapse (linear LR diverges after step 400)
- FP8 on SM120 needs a 1-line Triton inductor patch (`getattr(binary.metadata, 'cluster_dims', (1, 1, 1))`)
- Entropy bonus (GTPO-style) made collapse worse, gradient noise is the real issue

## Usage

```python
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
```

Interactive chat ([script](https://gist.github.com/Katehuuh/34c631b9a9ca5e930eba3903937d01e9/8ccb21a677b1da1682fa2a66e6d7ecaaaf30de93)):
```bash
set TRAIN_STAGE=rl
set NANOCHAT_FP8=0
python nanochat_gpt-1384M.py --chat
```

Root = RL (step 300, best pass@4). `pt/` and `sft/` folders have base and SFT checkpoints.

## Chat Format

```
<|bos|><|user_start|>If a store sells 3 apples at $2 each and 5 oranges at $1 each, what is the total cost?<|user_end|><|assistant_start|>The cost of 3 apples is 3 * $2 = $<|python_start|>3*2<|python_end|><|output_start|>6<|output_end|>6.
The cost of 5 oranges is 5 * $1 = $<|python_start|>5*1<|python_end|><|output_start|>5<|output_end|>5.
The total cost is $6 + $5 = $<|python_start|>6+5<|python_end|><|output_start|>11<|output_end|>11.
#### 11<|assistant_end|>
```

## Citation

```bibtex
@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}
```