---
license: mit
library_name: nanofable
pipeline_tag: text-generation
language:
- en
datasets:
- roneneldan/TinyStories
tags:
- tinystories
- ternary
- bitnet
- quantization
- research
---

# NanoFable-28M-fp16

A 27.8M-parameter decoder-only transformer trained on TinyStories with fp16 weights —
one point on the coherence-vs-bytes frontier study
([spec](https://github.com/adit-rah/nanofable/blob/main/specs/idea2_ternary_emergence_frontier_spec.md)).

| | |
|---|---|
| Parameters | 27,795,968 |
| **Total bytes (packed, spec §6)** | **55,574,528 (53.00 MiB)** |
| Layers / width / heads | 8 / 512 / 8 |
| Context | 512 |
| Vocab | 4,096 (custom BPE, ByteLevel) |
| In-block linear weights | fp16 (2 bytes/weight) |
| Embeddings / LM head (tied) | fp16 |
| Seeds | 0, 1 (both published, see below) |
| Training tokens | 500M |

## Training

TinyStories, single GPU (T4), fp16 autocast. AdamW (β 0.9/0.95, wd 0.1), peak LR 3e-4,
cosine to 10% after 3% warmup, 65,536 tokens/step (~7,630 steps). All hyperparameters
frozen across the 16-run sweep ([frozen config](https://github.com/adit-rah/nanofable/blob/main/docs/frozen_config.md)).

## Evaluation

Val perplexity on held-out TinyStories + LLM-judge coherence (Qwen2.5-7B-Instruct,
frozen rubric, 200 frozen prefixes, greedy decoding; instrument calibration in
[calibration.md](https://github.com/adit-rah/nanofable/blob/main/eval/calibration.md) — reference anchor: TinyStories-8M at 4.232).

- Val PPL: **5.94 / 5.39** (seed 0 / seed 1)
- Judge score (0–5, greedy, n=400 pooled): **3.11 ±0.09** (grammar 3.83 · consistency 3.23 · completes 2.26)
- Capability gates: coherence (≥4.0) **fail** · PPL (≤8.09) **pass** · T1 surface-fluency (grammar ≥4.55, TS-8M anchor) **fail** — clears the PPL gate; no sweep config cleared the coherence or T1 gates (pre-registered null; see [eval/calibration.md](https://github.com/adit-rah/nanofable/blob/main/eval/calibration.md))


## Usage

Weights ship as `model.safetensors` (fp16). Ternary in-block linears are stored
**dequantized** — `scale * {-1,0,+1}` already materialized in fp16 — so both arms load
into the same fp16 model class.

```python
from safetensors.torch import load_file
from nanofable.config import TIERS
from nanofable.model import build_model
from nanofable.tokenizer import load_tokenizer
from nanofable.generate import generate

model = build_model(TIERS["large"], "fp16")
model.load_state_dict(load_file("model.safetensors"), strict=False)  # lm_head re-ties to tok_emb
model.eval()

tok = load_tokenizer("tokenizer.json")
print(generate(model, tok, "Once upon a time", temperature=1e-4, top_k=0))
```

## Repository layout

```
model.safetensors         # seed 1 — the weights this card describes
config.json
model.tpack               # packed browser payload (what the demo site streams)
meta.json  eval.json  metrics.csv
tokenizer.json            # 4,096-vocab custom BPE
seed0/                    # second training replica, same files
```

Both training replicas are published. The root weights are **seed 1**, selected by judge
score. Do not read that as a quality ranking — the two seeds are **statistically
indistinguishable**: their 95% judge CIs overlap, and val PPL disagrees with judge score on
which seed leads in most configs of this sweep. The choice of which sits at the root is
effectively arbitrary.

| | seed 0 | seed 1 (root) |
|---|---|---|
| Val PPL | 5.94 | 5.39 |
| Judge (0-5, n=200) | 3.103 [2.97, 3.23] | 3.108 [2.98, 3.24] |

`model.tpack` is the packed browser payload (pack format v1), fp16 throughout. **This is
the artifact the byte accounting above refers to**, and the one the demo site streams for
in-browser inference. It carries the same weights as `model.safetensors`, in the container
the browser runtime reads.

Training checkpoints (optimizer state, fp32 latent weights) are not published here; they
live in the [source repo](https://github.com/adit-rah/nanofable).

## Limitations

Children's-story English only (TinyStories distribution) — not a general language model.
512-token context. Research artifact of a controlled precision study; not safety-tuned.