Initial commit

README.md

@@ -0,0 +1,123 @@
+---
+license: mit
+base_model: karpathy/nanochat-d34
+language:
+- en
+tags:
+- nanochat
+- text-generation
+- karpathy
+- single-gpu
+- rtx-5090
+datasets:
+- HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
+- HuggingFaceTB/smol-smoltalk
+- cais/mmlu
+- ai2_arc
+- openai/gsm8k
+pipeline_tag: text-generation
+model-index:
+- name: nanochat-d24
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      name: GSM8K
+      type: openai/gsm8k
+    metrics:
+    - name: pass@1
+      type: accuracy
+      value: 0.1645
+      verified: false
+---
+
+# nanochat-d24
+
+1.38B param, Pretrain+SFT+RL (GRPO/GSM8K) on a **single RTX 5090** (32GB). Ported from [Karpathy's nanochat](https://github.com/karpathy/nanochat) ([`6ed7d1d`](https://github.com/karpathy/nanochat/commit/6ed7d1d82cee16c2e26f45d559ad3338447a6c1b), Mar 9 2026) into a [single ~2K line Python file](https://gist.github.com/Katehuuh/34c631b9a9ca5e930eba3903937d01e9/8ccb21a677b1da1682fa2a66e6d7ecaaaf30de93).
+
+## Changelog
+
+- **2026-03-23**: Initial release
+
+## Evaluation Results
+
+<!-- ChatCORE* = without HumanEval (5/6 tasks). val_bpb = base model metric only. -->
+
+| Benchmark | Base (step 24K) | SFT (step 1942) | RL (step 300) | d20 SFT (official) | d30 (official) |
+|-----------|:-:|:-:|:-:|:-:|:-:|
+| ARC-Easy | 23.9% | 52.6% | **52.9%** | 38.8% | ~70% |
+| ARC-Challenge | 25.0% | 42.5% | **41.9%** | 28.1% | - |
+| MMLU | 23.6% | 35.3% | **35.6%** | 31.5% | ~40% |
+| GSM8K (pass@1) | 0.0% | 10.2% | **16.5%** | 4.6% | ~20% |
+| SpellingBee | 0.0% | 79.7% | **87.1%** | N/A | - |
+| **ChatCORE**\* | -0.6% | 32.7% | **35.5%** | 8.8% | - |
+| val_bpb | 0.830 | - | - | - | 0.705 |
+
+*\*ChatCORE without HumanEval (5/6 tasks). Official d20 SFT from [Discussion #1](https://github.com/karpathy/nanochat/discussions/1). d30 approximate, community reports.*
+
+## Architecture
+
+24 layers, 1536 hidden dim, 12 MHA heads (head_dim=128), 32K BPE vocab, 2048 context. RoPE (theta=100k), QK-norm, unweighted RMSNorm, untied embeddings, ReLU² activation, logit softcap 15.0, value embeddings on alternating layers (~604M params at near-zero FLOPs), per-layer residual scalars, smear mechanism, backout mechanism, cuDNN SDPA full context.
+
+### Differences from official nanochat d24
+
+| Feature | Official (8xH100) | This model (1xRTX5090) |
+|---------|-------------------|------------------------|
+| Window pattern | SSSL | L (full context) |
+| VE gate channels | 32 | 12 |
+| Smear/Backout | Not used | Active |
+| Attention | Flash Attention 3 | cuDNN SDPA |
+| FP8 | Optional | Default (pretraining) |
+| Pretraining data | [NVIDIA ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix) | [finepdfs_edu/dclm/fineweb_edu](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled) (8/100 shards) |
+
+## Training
+
+| Stage | Data | Duration | Steps | Details |
+|-------|------|----------|-------|---------|
+| **Pretrain** | see above | ~3.8 days | 29,232 | FP8, MuonAdamW, grad ckpt, B=4x32 |
+| **SFT** | SmolTalk 460K + MMLUx3 + SimpleSpelling 200K + SpellingBee 80K + GSM8Kx4 | ~7.5h | 1,942 | BF16, bestfit packing, torch.compile |
+| **RL** | GSM8K (16 samples/question) | ~5h | 300 | BF16, cosine LR (T_max=400), KV-cache |
+
+![Training Pipeline](https://i.imgur.com/6Mg7Otx.png)
+
+Notes:
+- Single-GPU RL needs cosine LR decay to avoid policy collapse (linear LR diverges after step 400)
+- FP8 on SM120 needs a 1-line Triton inductor patch (`getattr(binary.metadata, 'cluster_dims', (1, 1, 1))`)
+- Entropy bonus (GTPO-style) made collapse worse, gradient noise is the real issue
+
+## Usage
+
+```python
+from safetensors.torch import load_file
+state_dict = load_file("model.safetensors")
+```
+
+Interactive chat ([script](https://gist.github.com/Katehuuh/34c631b9a9ca5e930eba3903937d01e9/8ccb21a677b1da1682fa2a66e6d7ecaaaf30de93)):
+```bash
+set TRAIN_STAGE=rl
+set NANOCHAT_FP8=0
+python nanochat_gpt-1384M.py --chat
+```
+
+Root = RL (step 300, best pass@4). `pt/` and `sft/` folders have base and SFT checkpoints.
+
+## Chat Format
+
+```
+<|bos|><|user_start|>If a store sells 3 apples at $2 each and 5 oranges at $1 each, what is the total cost?<|user_end|><|assistant_start|>The cost of 3 apples is 3 * $2 = $<|python_start|>3*2<|python_end|><|output_start|>6<|output_end|>6.
+The cost of 5 oranges is 5 * $1 = $<|python_start|>5*1<|python_end|><|output_start|>5<|output_end|>5.
+The total cost is $6 + $5 = $<|python_start|>6+5<|python_end|><|output_start|>11<|output_end|>11.
+#### 11<|assistant_end|>
+```
+
+## Citation
+
+```bibtex
+@misc{nanochat,
+  author = {Andrej Karpathy},
+  title = {nanochat: The best ChatGPT that $100 can buy},
+  year = {2025},
+  publisher = {GitHub},
+  url = {https://github.com/karpathy/nanochat}
+}
+```

config.json

@@ -0,0 +1,23 @@
+{
+  "architectures": [
+    "NanoChatForCausalLM"
+  ],
+  "model_type": "nanochat",
+  "hidden_size": 1536,
+  "num_attention_heads": 12,
+  "num_key_value_heads": 12,
+  "num_hidden_layers": 24,
+  "intermediate_size": 6144,
+  "hidden_act": "relu2",
+  "vocab_size": 32768,
+  "max_position_embeddings": 2048,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 100000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "final_logit_softcapping": 15.0,
+  "torch_dtype": "float16",
+  "ve_gate_channels": 12
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccc646e6fe7d7b0fd9708b6d82676a5db7c8074cb1dc94bcc125ab683c558261
+size 5536507416

pt/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:510db0d091bd0495834be5f3229dd89d42985568dcb3e47edd4898af2601bc63
+size 4227884224

sft/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6be0fac8fb4ebeec8c89c0d5ed5901ed30414cd97d1e5f49a4566a664ace350d
+size 5536507416

special_tokens_map.json

@@ -0,0 +1,14 @@
+{
+  "bos_token": "<|bos|>",
+  "eos_token": "<|assistant_end|>",
+  "additional_special_tokens": [
+    "<|user_start|>",
+    "<|user_end|>",
+    "<|assistant_start|>",
+    "<|assistant_end|>",
+    "<|python_start|>",
+    "<|python_end|>",
+    "<|output_start|>",
+    "<|output_end|>"
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 2048,
+  "bos_token": "<|bos|>",
+  "eos_token": "<|assistant_end|>",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'user' %}<|user_start|>{{ message['content'] }}<|user_end|>{% elif message['role'] == 'assistant' %}<|assistant_start|>{{ message['content'] }}<|assistant_end|>{% endif %}{% endfor %}<|assistant_start|>"
+}