---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- pretrained
- base-model
- small-language-model
---

# TinyBrainBot 303M — Base

A **303M-parameter, LLaMA-architecture** language model trained **from scratch on a home server** (2× NVIDIA Tesla P100). This is the **base (pretrained) model** — it has *not* been instruction-tuned, so it **completes text** rather than following chat instructions.

👉 For the chat/assistant version, see **TinyBrainBot 303M Instruct**.

## Model details

| | |
|---|---|
| Parameters | ~303M |
| Architecture | LLaMA-style decoder (RoPE, RMSNorm, SwiGLU, pre-norm) |
| Layers | 24 |
| Hidden size | 1024 |
| Attention heads | 16 (GQA, 4 KV heads) |
| Head dim | 64 |
| FFN size | 2816 |
| Vocab | 32,000 (custom SentencePiece BPE) |
| Context length | 1024 |
| Tied embeddings | Yes |
| Precision | fp16 |

It re-expresses cleanly as a `LlamaForCausalLM`, so it loads with standard 🤗 Transformers.

## Training

Pretrained on a mix of open English text plus synthetic distillation data:

- **Web / educational text:** FineWeb-Edu, English Wikipedia, OpenWebText2
- **Narrative:** TinyStories
- **Math/reasoning:** Orca-Math
- **Synthetic fact distillation:** Q&A and short-fact data generated by a stronger teacher model (Ling 2.6 Flash) to densify factual coverage.

> Total training tokens are on the order of a few billion — **far fewer** than models like SmolLM/Pythia (hundreds of billions). This is a hobbyist/research model; capability scales with tokens, and this one is intentionally token-limited. *(4.7B)*

## Intended use

- A **starting point for continued pretraining or fine-tuning** at small scale.
- Text completion and research on small language models.
- **Not** for chat/instruction following — use the Instruct version for that.

## Limitations

- **Shallow, fragile factual knowledge.** It knows common facts but is sensitive to phrasing and **capitalization** (e.g. "France" vs "france" can flip the answer).
- **Weak at reasoning and math**; roughly random on knowledge benchmarks like MMLU (typical for this size).
- Will produce fluent **plausible-but-wrong** text confidently. Pair with retrieval (RAG) for anything factual.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("your-username/tinybrainbot-303m-base")
model = AutoModelForCausalLM.from_pretrained("your-username/tinybrainbot-303m-base", torch_dtype=torch.float16)

ids = tok("The water cycle is the process by which", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
```

## License

Apache-2.0 *(change if you prefer a different license)*.