---
language:
- ne
tags:
- embeddings
- sentence-similarity
- nepali
- transformer
- chatbot
- retrieval
license: mit
---

# 🇳🇵 Nepali Chatbot Embeddings Model

A Transformer trained **from scratch** on 40,000 Nepali QA pairs.  
Produces **128-dimensional sentence embeddings** for Nepali (Devanagari) text,  
used for semantic retrieval in a chatbot system.

## Model Details

| Property | Value |
|---|---|
| Architecture | Transformer Encoder-Decoder |
| Embedding dim | 128 |
| Attention heads | 4 |
| FF dim | 256 |
| Layers | 2 |
| Tokenizer | Character-level (Devanagari) |
| Vocab size | 127 |
| Max seq len | 64 |
| Language | Nepali (ne) |
| Dataset | [bashyaldhiraj2067/50k_nepali_chatbot_dataset](https://huggingface.co/datasets/bashyaldhiraj2067/50k_nepali_chatbot_dataset) |

## Evaluation Results (Autoregressive — No Teacher Forcing)

| Metric | Score |
|---|---|
| Token Accuracy | 79.32% |
| Precision (macro) | 82.11% |
| Recall (macro) | 75.21% |
| F1 (macro) | 77.69% |
| BLEU-1 (corpus) | 0.9315 |
| BLEU-2 (corpus) | 0.9040 |
| BLEU-3 (corpus) | 0.8845 |
| BLEU-4 (corpus) | 0.8703 |
| ROUGE-1 | 0.1976 |
| ROUGE-2 | 0.1108 |
| ROUGE-L | 0.1976 |

## Training Curves

![Training Curves](training_curves.png)

## Files

| File | Description |
|---|---|
| `pytorch_model.bin` | Full model weights (PyTorch) |
| `config.json` | Model architecture & eval config |
| `tokenizer.json` | Character-level vocab (token2id) |
| `embeddings_index.pt` | Pre-computed embeddings + QA pairs for retrieval |
| `training_curves.png` | Loss / Accuracy / BLEU / ROUGE plots |

## Usage

```python
import torch, json
import torch.nn.functional as F

# 1. Load config
with open("config.json") as f:
    cfg = json.load(f)

# 2. Load tokenizer
with open("tokenizer.json") as f:
    tok_data = json.load(f)
token2id = tok_data["token2id"]

def encode(text, max_len=64):
    ids = [1] + [token2id.get(c, 3) for c in list(text.strip())] + [2]
    ids = ids[:max_len]
    ids += [0] * (max_len - len(ids))
    return ids

# 3. Load pre-built index
index = torch.load("embeddings_index.pt", map_location="cpu")
all_embs = index["embeddings"]   # (N, 128) normalized
questions = index["questions"]
answers   = index["answers"]

# 4. Query
def get_answer(query):
    ids  = torch.tensor([encode(query)], dtype=torch.long)
    # (For full model inference, load pytorch_model.bin + TransformerChatbot class)
    # Quick cosine retrieval using index:
    # ... embed query with model.get_embedding(ids) ...
    # sims = (all_embs @ q_emb.T).squeeze()
    # return answers[sims.argmax()]
    pass
```

## License
MIT