# Bantam Tokenizer

BPE tokenizer with 128,016 vocabulary entries (128,000 base + 16 special tokens).

## Special tokens

| Token | ID | String |
|---|---|---|
| `unk_token` | 128000 | `<\|UNK\|>` |
| `pad_token` | 128001 | `<\|PAD\|>` |
| `bos_token` | 128002 | `<\|BOS\|>` |
| `eos_token` | 128003 | `<\|EOS\|>` |
| — | 128004 | `<\|SYSTEM_START\|>` |
| — | 128005 | `<\|SYSTEM_END\|>` |
| — | 128006 | `<\|USER_START\|>` |
| — | 128007 | `<\|USER_END\|>` |
| — | 128008 | `<\|AGENT_START\|>` |
| — | 128009 | `<\|AGENT_END\|>` |
| — | 128010 | `<\|THINK_START\|>` |
| — | 128011 | `<\|THINK_END\|>` |
| — | 128012 | `<\|COMPUTE_START\|>` |
| — | 128013 | `<\|COMPUTE_END\|>` |
| — | 128014 | `<\|IMAGE_START\|>` |
| — | 128015 | `<\|IMAGE_END\|>` |

## Config quick-reference

```yaml
pad_token_id: 128001
bos_token_id: 128002
eos_token_id: 128003
```

## Usage

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("path/to/bantam-tokenizer")
```