--- language: - el - en license: other license_name: nvidia-open-model-license base_model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 pipeline_tag: text-generation library_name: transformers tags: - greek - english - rag - grounded-generation - citations - abstention - nemotron - long-context model-index: - name: Sophea-RAG-Nemo3 results: - task: type: text-generation name: Grounded RAG reading (Greek long-retrieval benchmark, n=4946) dataset: type: rag name: Greek Wikipedia long-retrieval RAG benchmark (in-house) metrics: - type: accuracy name: Answer correctness (LLM-judge) value: 0.669 - type: accuracy name: Faithfulness / groundedness value: 0.845 - type: f1 name: Citation F1 value: 0.678 ---

Sophea-RAG-Nemo3

Greek + English grounded RAG reader — fine-tuned NVIDIA Nemotron-3 Nano 30B-A3B

**Sophea-RAG-Nemo3** is a Greek + English **RAG reader**: it answers strictly from the retrieved documents you put in its context, **cites** the documents it used as `[n]`, and **abstains** when the answer is not there. It is the generation stage of a Greek RAG stack whose retrieval stages are **[Sophea-Nemo-Embedding](https://huggingface.co/KIEFERSA/Sophea-Nemo-Embedding)** and **[Sophea-Nemo-Reranker](https://huggingface.co/KIEFERSA/Sophea-Nemo-Reranker)**. - **Creator:** Kiefer SA - **Base model:** `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` (`NemotronHForCausalLM`, hybrid Mamba + attention MoE, 30B total / A3B active; `trust_remote_code`) - **Languages:** Greek + English - **Behaviour:** grounded answer → `[n]` citation → abstain when unsupported - **Training:** LoRA (r 16, α 32) on attention + MLP + Mamba `in_proj`, 1 epoch, 8k context, assistant-only loss, bf16, cosine LR 1e-4, grad-clip 0.3, 4×B200 — **merged into the base weights**, so this is a standalone checkpoint (no adapter needed) > **The base model is not a grounded reader.** Off the shelf it answers from parametric memory: it > hallucinates instead of abstaining (abstains on ~1% of unanswerable questions) and its answers are > faithful to the context only a quarter of the time. This fine-tune teaches grounding, citation and > selective prediction. ## Why it matters — base vs fine-tuned Measured on a **Greek Wikipedia long-retrieval RAG benchmark** (4,946 questions, up to 40 documents / ~24k tokens of context, 25% deliberately unanswerable, L1 factoid → L5 multi-hop synthesis). Identical prompts and greedy decoding for both models; correctness and faithfulness are LLM-judged.

metric	base	fine-tuned	Δ
Answer correctness (LLM-judge)	29.4%	66.9%	+37.5
Faithfulness / groundedness	25.2%	84.5%	+59.3
Citation F1	41.1%	67.8%	+26.7
— citation precision	43.2%	73.4%	+30.2
— citation recall	46.8%	65.0%	+18.2
Cited all gold documents	41.0%	56.5%	+15.5
Exact citation set	30.1%	56.5%	+26.4
Abstention recall (unanswerable refused)	1.2%	30.5%	+29.3

**Read.** Correctness **29.4% → 66.9%** and faithfulness **25.2% → 84.5%** — the fine-tune answers from the documents instead of from memory. Citation F1 rises **41.1 → 67.8** and the model returns the **exact** gold citation set on 56.5% of items (base 30.1%). Abstention recall goes from ~zero to 30.5%: the base essentially never refuses, so its "faithfulness" on unanswerable questions is accidental. ### Robustness to retrieval noise More retrieved documents = more distractors. Correct / faithful (%), by number of documents in context:

metric (documents in context)	8	12	16	20	28	40
correct — base	30.8	32.4	29.1	29.0	26.1	29.8
correct — fine-tuned	76.5	72.7	67.2	67.8	63.6	59.3
faithful — base	24.3	23.0	24.1	23.7	28.1	29.0
faithful — fine-tuned	86.1	85.2	86.7	86.5	82.1	79.4

**Read.** The fine-tune leads at every context size, from **76.5%** correct at 8 documents to **59.3%** at 40 — it degrades gracefully as noise grows, while faithfulness stays ≥79% throughout. The base is flat at ~29% because it is not reading the context in the first place. ### Position of the gold document ("lost in the middle")

metric (gold position in context)	early	middle	late
correct — base	31.9	29.4	26.1
correct — fine-tuned	65.8	65.0	70.3
faithful — base	21.9	27.9	26.7
faithful — fine-tuned	84.2	83.1	86.4

**Read.** The base is best when the gold document is **early** (31.9%) and worst when it is **late** (26.1%) — the classic position bias. The fine-tune is strongest on **late** gold (70.3%) and varies by only ~5 points across positions. ### By question difficulty (L1 factoid → L5 multi-hop synthesis)

metric (question difficulty)	L1	L2	L3	L4	L5
correct — base	33.9	28.3	22.1	32.2	34.6
correct — fine-tuned	77.7	73.6	60.0	54.3	57.3
faithful — base	30.0	25.2	23.1	20.4	24.1
faithful — fine-tuned	91.3	90.3	85.4	68.6	70.2

**Read.** The gain holds at every level — largest on the retrieval-shaped levels (L1 **+43.8**, L2 **+45.3**) and still **+22** on the hardest multi-hop synthesis questions, where faithfulness rises from ~20-24% to ~69-70%. ## Domain RAG evaluation A second, independent evaluation on in-house Greek + English domain documents (800 questions across energy, finance, legal and medical; 187 of them unanswerable):

metric	base	fine-tuned	Δ
Answer correctness (LLM-judge)	31.2%	61.0%	+29.8
Faithfulness / groundedness	23.0%	66.2%	+43.2
Citation hit (cited a gold document)	40.0%	60.1%	+20.1
Abstention on unanswerable items (n=187)	8.6%	80.2%	+71.6

metric	Energy	Finance	Legal	Medical
correct — base	37.5	22.5	42.0	23.0
correct — fine-tuned	62.0	57.0	72.0	53.0
faithful — base	25.0	21.0	23.0	23.0
faithful — fine-tuned	67.0	64.5	70.5	63.0
cite-hit — base	35.5	40.5	44.0	40.0
cite-hit — fine-tuned	60.0	57.5	71.0	52.0

**Read.** The fine-tune wins on every domain and every metric, strongest on **legal** (72.0% correct / 70.5% faithful) and weakest on **medical** (53.0% / 63.0%) — the same ordering the retrieval stages show on these domains. ### Selective prediction On the 187 unanswerable items the correct action is to refuse. The base answers anyway **91%** of the time; the fine-tune abstains on **80.2%** of them — this is the mechanism behind the faithfulness gain. ### Position invariance (answerable subset, n=613)

metric (gold document position)	1	2	3	4	5	6
correct — base	45.4	38.6	48.5	32.0	36.2	32.4
correct — fine-tuned	78.3	77.3	84.3	76.7	84.5	76.5
faithful — base	33.6	30.3	34.3	20.4	25.9	32.4
faithful — fine-tuned	84.2	85.6	90.3	86.4	86.2	85.3

**Read.** The base decays as the gold document moves deeper into the context (45.4% at position 1 → 32.4% at position 6). The fine-tune is essentially **flat at 77-85%** across all six positions. ## Usage (transformers) ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM name = "KIEFERSA/Sophea-RAG-Nemo3" tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( name, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True).eval() SYSTEM = ( "Είσαι βοηθός που απαντά ΑΠΟΚΛΕΙΣΤΙΚΑ με βάση τα παρεχόμενα έγγραφα. Χρησιμοποίησε μόνο τις πληροφορίες των εγγράφων, ΜΗΝ επινοείς. Ανέφερε σε αγκύλες τον αριθμό του εγγράφου/των εγγράφων που χρησιμοποίησες, π.χ. [2]. Αν η απάντηση ΔΕΝ υπάρχει στα έγγραφα, πες το καθαρά. Απάντησε στα ελληνικά, σύντομα και ουσιαστικά." ) docs = ["Η Αθήνα είναι η πρωτεύουσα της Ελλάδας.", "Το Παρίσι είναι η πρωτεύουσα της Γαλλίας."] question = "Ποια είναι η πρωτεύουσα της Ελλάδας;" context = "\n".join(f"[{i}] {d}" for i, d in enumerate(docs, 1)) msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"Έγγραφα:\n{context}\n\nΕρώτηση: {question}"}] text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True) enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device) out = model.generate(**enc, max_new_tokens=200, do_sample=False) print(tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True)) # -> "Η Αθήνα. [1]" ``` Number the documents `[1..n]` in the user turn and keep the system message above — the model was trained to cite those numbers and to say the answer is absent when it is. Greedy decoding (`do_sample=False`) is what the numbers above were measured with. Needs `trust_remote_code=True`. ## Serving with vLLM ```bash vllm serve KIEFERSA/Sophea-RAG-Nemo3 --trust-remote-code --max-model-len 40960 ``` ```bash curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "KIEFERSA/Sophea-RAG-Nemo3", "temperature": 0, "messages": [ {"role": "system", "content": "Απάντησε μόνο από τα έγγραφα και ανέφερε [n]."}, {"role": "user", "content": "Έγγραφα:\n[1] Η Αθήνα είναι η πρωτεύουσα της Ελλάδας.\n\nΕρώτηση: Ποια είναι η πρωτεύουσα της Ελλάδας;"} ] }' ``` ## License Fine-tune of `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` — governed by the **NVIDIA Open Model License**; commercial use permitted under those terms (note: **not** Apache-2.0). Review the base model's license before use.