---
library_name: transformers
pipeline_tag: text-generation
base_model: allenai/Olmo-3-1025-7B
tags:
- klettermix
- ablation
- causal-language-modeling
---

# OLMo 3 7B — ClimbMix only

Annealing-only run from the pretrained Stage 1 weights. The HF optimizer is not imported; a fresh optimizer and scheduler are used.

## Base model

[allenai/Olmo-3-1025-7B](https://huggingface.co/allenai/Olmo-3-1025-7B) at revision `stage1-step1413814`. These checkpoints are **base models**, not instruction-tuned or chat-tuned models.

## Training data

- **annealing data:** [ClimbMix only](https://huggingface.co/datasets/RuHae/KletterMix-Ablations-OLMo3-Tokenized-Data/tree/climbmix-only) — Exactly 12B tokens: 100% ClimbMix.

## Training configuration

| Context length | 8,192 tokens |
| Global batch | 256 sequences |
| Micro batch | 2 sequences per rank |
| Steps / target | 5,723 steps / 12,002,000,896 processed tokens |
| Optimizer | Distributed Adam, weight decay 0.1 |
| Learning rate | 1e-5 peak; linear warmup and linear decay to zero |
| Precision | BF16 |
| GPUs | 8, data parallelism 8 |
| Seed | 42 |
| Evaluation during training | Disabled |

The repository contains checkpoint revisions named `step-XXXXXXX`; `main` is the
final checkpoint. Optimizer states are not part of these HF exports. The exact
source paths, revision mapping, and publication code are maintained in the
private [KletterMix_Ablations](https://github.com/ml-research/KletterMix_Ablations)
repository.