| --- |
| language: |
| - el |
| - en |
| license: other |
| license_name: nvidia-open-model-license |
| base_model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 |
| pipeline_tag: text-generation |
| library_name: transformers |
| tags: |
| - greek |
| - english |
| - rag |
| - grounded-generation |
| - citations |
| - abstention |
| - nemotron |
| - long-context |
| model-index: |
| - name: Sophea-RAG-Nemo3 |
| results: |
| - task: |
| type: text-generation |
| name: Grounded RAG reading (Greek long-retrieval benchmark, n=4946) |
| dataset: |
| type: rag |
| name: Greek Wikipedia long-retrieval RAG benchmark (in-house) |
| metrics: |
| - type: accuracy |
| name: Answer correctness (LLM-judge) |
| value: 0.669 |
| - type: accuracy |
| name: Faithfulness / groundedness |
| value: 0.845 |
| - type: f1 |
| name: Citation F1 |
| value: 0.678 |
| --- |
| |
| <div style="display:flex;align-items:center;gap:18px;padding:16px 20px;margin-bottom:16px;border-radius:14px;background:linear-gradient(100deg,#e8f5ec 0%,#f7fbf8 62%);border:1px solid #c6e7d1"><img alt="KIEFERSA" style="flex:0 0 auto;height:58px;width:58px;border-radius:12px;object-fit:contain" src="data:image/webp;base64,UklGRgQNAABXRUJQVlA4WAoAAAAQAAAAxwAAxwAAQUxQSIsKAAABsMf/nyE52q+qZjd2nluEh00OsY2zbdsOH51t27Zt2744F1vrrqrf69X1m+mumq4K/ooISG4bSZKUmIIOeag1w1Ud8wFwPDgH2Om9zYFD2AcXDIa/hdWdgAff+r6oNK7pDDzw1uPxRlQRVoccFlu3e+pj0zrksByD8huqjSFSAm0c2l+8ilrIEQxaT1qC2ljAYYJD89PnxaYQQ05spcdMp5ZEcI0f/CtqGVsygTXY4+vYJGLI4QJg7CeIxkIOFwyGvE0tLeG0Xs8rVLGFHBZb1aORCXAYcniOQdfb61FJjXaEcEb5NeuoWRI+63DhShrgrAmdtZy4iAa4kJPj0PSMOdRCjuCQO3oaDXAhR3Dg+/9Ko0/IERxgl29pDA05QgCM+ohGn7DDYPDr1EJPr2cUiT7hp5aEhvUBZMZ6AY3rC3BjkLVdga0PkFjTGUqCDykOMw9kwEVwKIL9c1zX1d/saIJUsCFvE4sntILyFYjvjgQmeFBwbCsuaAMCyldhhOqVAYERp7bm2s3MKTMoKTF6apugiEOrvaMr+XbADJq8JtXf391IQImt4aEqYpBH3kmtuaXS7E4gkRL1s73pBS8IEY1rLusYCHFi+PoQurwnkjdh+dQ2wHP+i71p/GgM/XaQggITFpzdIpaQoWL7ZlcaGtKD5LPnrBOaeB8tbO3XAzg5LylIsYe/HyY8FwuLNE4/uoSck5SkkO/38jsk2djc05uSK5uSlOvQZzsAcBESdGyLJrckP4hZkFLeGw7AuZ+kDnAXt6fVyIa0ol7u62u0SGfrri2nM5xDQpJ8oqefksbq7+hOk4Eddgmw4e7uwHLMa6TC6NEe9DLaYyfV13sY95JMv5AX4FxgK6sv7ZiFCVwkDFdIifj2EBp93GAvS6a29ivuFbRPx1Jzi135nndWsyKXMdZvxLDCo7kDlNT43Z5AjjYTFDiW6ceVFFEYNJmOqtCthdgLuCXSBLiDaIBzi5sJv5PjKRalf6PU+UNptZ0lKrZpx5WSo8wA6eLeHhBLkZiWsNGWG/P8nEmfnwySd499Og5AsMyDSye3ISuzexyKQnx7KGSfSzvS6FMEHIt+ui2wjEPrffFxUcaGg8g09IHJPqg1DgGe9Q34AeLQTSF0Wr5egOfMJlKBh5S4QQ+f9zNqqfwkS9bzMV3TCU4yHXsq1LDYut9Ti7i6G0Dzc0gHpX9kxcpvXIeqEas7QQmDNv9bhipSgYUE2PaXrUIVadJHwUhv7pWrUUU6oJAvHi0nLzUnK78fhJ7STrdWo4rCSWzNzphvFpSkvhYuGGxxv+ldDyM5DqXHzEAtdZr+HBZLr6ckShk+BAdx0O+mTCX1GSUUslcwFk8oprE9v0EtpU2/lBAAI99DLWW44AJg/EfmzKTr+0qSnb4y0SJMCMFg2FsapXTRvyY48L1/MhPCA4+t9wsKpXTVhyc4iCOmmZAUFlhsVY81oops+glTSZNTZpv1NBzw2LreVYdKard9kSyWFhMWmLgXAmg+KLtuHSpZhP5OJhi0vWC5CUn+w3Ic2l9IAhzaYXfyr1pjdttzBIfWE5bQ/OwAiwldbiMPQyZxGeBO+we1MQdYylYPNpoJGcSZ5Y6ZRs0BDhao7Z5VqKSPCA7soLz3fUc4kIFvIErpG5wD7JYX4LIB/WFw9IdGMkRfKM1ZjPxgNuZj+tOCS9xck12/NtckM6T+HSwh8r9F7z23uHpGDjTPiMwCiPjQ1TdcbzGM9XqOrgWOcSe5o/5CFRURdyNvbZYa3eO0hsxGHJIFZGQ1utxeRz+2uMettJywbCSIjG9WUssIViGpvD2wjKONFQfXAgCQdRB9AICxzIO+ALAhy7ouwNYHSB1vSsKKRSvuzsAEDx8OOgWnbr38jQFG/CC7tvaazaB8BTY+uV0RGkkg43nIsdXe0RkEVKzECOvv28JMCMbGsTU8vCUwIaB8FfkAUn1LJbAc84mbJ/33PxbD2eIoUT5rbmGe10dBH9ErOwITzBM02vYTOjKF+PpQc/sW7AchE5b9p7VLyfj7tKMZ+CFpxU3ua9GRxoVnN3fXHpzx7xsuTOPXuwFpnEyETNA4+/gmwEXoULH9sj8n5yeBhD3745CSWHzHQQvzkSXAcwCJJO7dD3uzWEKFMq24pzUl5yUVifLpjgBChAgd25KJLcnVTUWqNejdkaYgBwZSlVZc0o7U15SkrGwv9TfiK+ltzfVltCKlJW11i57sacRPUge4O7tScwpipLD+/m5mh4OAVBg90oNeTlvSnvrqGyqM+I6WCvWzfelN6Yo0O77mso4muHqNlIhvDqXRxwU2E5ZNbm1CkrdIqfGTsdTcYFcq5p3ZHLjwExXbd7vRIuoK29I9/YQmsfiHijT+diAnR+wKF/LrISIWbyh03Y7LkaMtOkn31V4mJHlC3oow70wa4IpOitXq0/EAnHsBscVTWpF1zS3u5O2hRjwg0rjq4va0YjrGoegX+5jDyv6m+vpKenjOcSgK5WNVkH3u7kZrvnPcf6q6vSWwjEPLbVFwLsNAZAJbyz6oNQ4BnvmNHyAO3RRCp+XrBXgOylZiowo8pFBvddMlc1BL7SdZsorra+sqm/5vuZFQw3IcOly2EnFNV4COl5Om2gBDvni0nLIEVSNWd4ISBpW31hjJNn+j1PlDaRWTDWt21jyTbEgfBeOCQff761FGOqib2EqOm2XyWaF+ECPbPtWISnpF8Y0f+DtqqQr1tRRoqn1Vo5TucC7de/YoPEozYHt8h1qqpP4cKgCj3sVYHOH14AJg3MeopUzbZyQEwM5foJYqk/CkUWwb/qa5zjb9UoID3+dH1FKGa8MFg74vmOfOtu9LcMgd9reZECZYbD0fbUQVpe1fS5QmJ80166gd/lrXu2tNhU3Zh5cqYzQ/b6GRoEC+eJRdvxaV1O76CZlg0OaC5SZaeEyqi9ruEtJQbNUXmVI6Xk3iXiAQDFpNWYramAUWt08nGvcCQDyj2anz6NuANaknVD3QgCryHcGh9OgZtJxaYlkkej2tUEmfEQLEAb/QUOAASxn0GqKUviI4wO7f0EBQLAqHpFEfGfERYQJc3jEXHwATwnb+2lwL3+CCweA3dGwai0tS3Nv/F1OQC+CJ9Xpe0hcYd7gRceTfRnyBxbblIzTAOcRd7Why2hwzgeDBjM531JHo4xSXnxNaTFiEKvIABmXXVNNPR25xm63aX7AccUjmaX/BSmqucZ51r6kbASLjzKMBrhi4lqoyYBkHpUbMOgCMzMg4CtEHADjLPOgLABuyrOu0fkDpmi7rAUi2WfgvYIGHvNnV3do5SH/gfEbjw1saCxGOZ6hntzWZM+xIifqNwdTChEPDD8cA8NgCDnnL/nJXaiFHxfbL/oy0f4YMNzbtqFJiIUdFCv85tTmxgEPeTxdPaEUs5OhI4coL2pJ/WAghdrbu2jLyNSqMw8Zq7+pCLeSYAPdYFbWQIyXK5/rQ6BNypER8fQi1oJLKPhoPYCzkKKnxm92AFKrgkmi/HSiIBZgEm3F0KbGQoyKN805tSizk6EjhkqktSfQJNvkB7pJ2wIyFG2JrbyynoSHkmC9hd3WjFnYweqQnDQ2h59W+1IIPAABWUDggUgIAAHAdAJ0BKsgAyAA+MRiJRCIhiBJcEAGCWlu4MAAZsUGEUQU6Vr/HVACel73mNKbZii7/0o6X9Y+s6FoAm3srqdXIcWd/45kb7mdOviNuuq60W3qXx/nI9sR42ICkrOwkSyYty0cun8cMpqXF0fh8fW3dSwaV0akfevRFNfnTLgy9bQM6IEWlj9O95hHRhKjNbDCS4qWJnQbJto47spFiqntPEoRO945gp7f+FzLmeAJQakEjD+xgQDxMOYrHw79WEkLQkKplqTo/uTX4n54eEP0N4epeSNwNE7Cl7q0Xi32YrSRma3RieLfasZELkHnrVj60WkAA/uouoblbKpSNm/wu8g0tISmLMdPrSIKVi8oMaNvyXkZc6zskreB0cAODf99ubzUwj0N8IJywGgCWpByLV1Kx+G8TMEVJa0HKGlyheflfEABBIEOGipfrRMJEDIiV7tNyTCRkHRlacBBPJNrPF6gu5KqHk0mGh2xBZYDKEWXI1A8R9WyRig0s6CIELwgJcqvHXXixg1yTYYd3Ja2rm0FNGHOb8MdNErR6nH3DYmradAb7JnP/RNqQ3gvhl/pA7ygUm6CIWAO/JF0d93ZViT6KqEgvSVdzZIwTzPR1TQ5YDMPK1u42qqN2SqTRwtzHrtold7EeIrx7plafSjlNtoSSwuC99MOigEHtk+7uPfbMdQA6uO4+wAsQ6bfVGy+dpyM63pbfBdYa2IbgqA8ZA3EUsyDx6GwlVker0XVZhE/HSCyOiMAVqYYDYXpIECKSg6k4BD3OKMJ7nJlAAAAAAA=="/><div><div style="font-size:1.24rem;font-weight:700;color:#003713;line-height:1.2">Sophea-RAG-Nemo3</div><div style="font-size:.86rem;color:#3a5a47;margin-top:2px">Greek + English grounded RAG reader — fine-tuned NVIDIA Nemotron-3 Nano 30B-A3B</div></div></div> |
|
|
| **Sophea-RAG-Nemo3** is a Greek + English **RAG reader**: it answers strictly from the retrieved documents you |
| put in its context, **cites** the documents it used as `[n]`, and **abstains** when the answer is not |
| there. It is the generation stage of a Greek RAG stack whose retrieval stages are |
| **[Sophea-Nemo-Embedding](https://huggingface.co/KIEFERSA/Sophea-Nemo-Embedding)** and |
| **[Sophea-Nemo-Reranker](https://huggingface.co/KIEFERSA/Sophea-Nemo-Reranker)**. |
|
|
| - **Creator:** Kiefer SA |
| - **Base model:** `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` (`NemotronHForCausalLM`, hybrid Mamba + attention MoE, 30B total / A3B |
| active; `trust_remote_code`) |
| - **Languages:** Greek + English |
| - **Behaviour:** grounded answer → `[n]` citation → abstain when unsupported |
| - **Training:** LoRA (r 16, α 32) on attention + MLP + Mamba `in_proj`, 1 epoch, 8k context, |
| assistant-only loss, bf16, cosine LR 1e-4, grad-clip 0.3, 4×B200 — **merged into the base weights**, |
| so this is a standalone checkpoint (no adapter needed) |
|
|
| > **The base model is not a grounded reader.** Off the shelf it answers from parametric memory: it |
| > hallucinates instead of abstaining (abstains on ~1% of unanswerable questions) and its answers are |
| > faithful to the context only a quarter of the time. This fine-tune teaches grounding, citation and |
| > selective prediction. |
|
|
| ## Why it matters — base vs fine-tuned |
|
|
| Measured on a **Greek Wikipedia long-retrieval RAG benchmark** (4,946 questions, up to 40 documents |
| / ~24k tokens of context, 25% deliberately unanswerable, L1 factoid → L5 multi-hop synthesis). |
| Identical prompts and greedy decoding for both models; correctness and faithfulness are LLM-judged. |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">base</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">fine-tuned</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">Δ</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Answer correctness (LLM-judge)</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">29.4%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">66.9%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+37.5</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Faithfulness / groundedness</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">25.2%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">84.5%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+59.3</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Citation F1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">41.1%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">67.8%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+26.7</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">— citation precision</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">43.2%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">73.4%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+30.2</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">— citation recall</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">46.8%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">65.0%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+18.2</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Cited all gold documents</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">41.0%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">56.5%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+15.5</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Exact citation set</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">30.1%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">56.5%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+26.4</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Abstention recall (unanswerable refused)</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">1.2%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">30.5%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+29.3</td></tr> |
| </tbody></table></div> |
|
|
| **Read.** Correctness **29.4% → 66.9%** and faithfulness **25.2% → 84.5%** — the fine-tune answers |
| from the documents instead of from memory. Citation F1 rises **41.1 → 67.8** and the model returns the |
| **exact** gold citation set on 56.5% of items (base 30.1%). Abstention recall goes from ~zero to |
| 30.5%: the base essentially never refuses, so its "faithfulness" on unanswerable questions is |
| accidental. |
|
|
| ### Robustness to retrieval noise |
|
|
| More retrieved documents = more distractors. Correct / faithful (%), by number of documents in context: |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric (documents in context)</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">8</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">12</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">16</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">20</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">28</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">40</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">correct — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">30.8</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">32.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">29.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">29.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">26.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">29.8</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">correct — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">76.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">72.7</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">67.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">67.8</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">63.6</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">59.3</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">faithful — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">24.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">24.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.7</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">28.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">29.0</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">faithful — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">86.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">85.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">86.7</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">86.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">82.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">79.4</td></tr> |
| </tbody></table></div> |
|
|
| **Read.** The fine-tune leads at every context size, from **76.5%** correct at 8 documents to |
| **59.3%** at 40 — it degrades gracefully as noise grows, while faithfulness stays ≥79% throughout. |
| The base is flat at ~29% because it is not reading the context in the first place. |
|
|
| ### Position of the gold document ("lost in the middle") |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric (gold position in context)</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">early</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">middle</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">late</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">correct — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">31.9</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">29.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">26.1</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">correct — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">65.8</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">65.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">70.3</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">faithful — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">21.9</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">27.9</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">26.7</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">faithful — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">84.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">83.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">86.4</td></tr> |
| </tbody></table></div> |
|
|
| **Read.** The base is best when the gold document is **early** (31.9%) and worst when it is **late** |
| (26.1%) — the classic position bias. The fine-tune is strongest on **late** gold (70.3%) and varies |
| by only ~5 points across positions. |
|
|
| ### By question difficulty (L1 factoid → L5 multi-hop synthesis) |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric (question difficulty)</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">L1</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">L2</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">L3</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">L4</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">L5</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">correct — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">33.9</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">28.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">22.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">32.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">34.6</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">correct — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">77.7</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">73.6</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">60.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">54.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">57.3</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">faithful — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">30.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">25.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.1</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">20.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">24.1</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">faithful — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">91.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">90.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">85.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">68.6</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">70.2</td></tr> |
| </tbody></table></div> |
|
|
| **Read.** The gain holds at every level — largest on the retrieval-shaped levels (L1 **+43.8**, |
| L2 **+45.3**) and still **+22** on the hardest multi-hop synthesis questions, where |
| faithfulness rises from ~20-24% to ~69-70%. |
|
|
| ## Domain RAG evaluation |
|
|
| A second, independent evaluation on in-house Greek + English domain documents (800 questions across |
| energy, finance, legal and medical; 187 of them unanswerable): |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">base</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">fine-tuned</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">Δ</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Answer correctness (LLM-judge)</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">31.2%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">61.0%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+29.8</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Faithfulness / groundedness</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.0%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">66.2%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+43.2</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Citation hit (cited a gold document)</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">40.0%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">60.1%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+20.1</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">Abstention on unanswerable items (n=187)</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">8.6%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">80.2%</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">+71.6</td></tr> |
| </tbody></table></div> |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">Energy</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">Finance</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">Legal</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">Medical</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">correct — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">37.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">22.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">42.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.0</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">correct — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">62.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">57.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">72.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">53.0</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">faithful — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">25.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">21.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">23.0</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">faithful — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">67.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">64.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">70.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">63.0</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">cite-hit — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">35.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">40.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">44.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">40.0</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">cite-hit — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">60.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">57.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">71.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">52.0</td></tr> |
| </tbody></table></div> |
|
|
| **Read.** The fine-tune wins on every domain and every metric, strongest on **legal** |
| (72.0% correct / 70.5% faithful) and weakest on **medical** (53.0% / 63.0%) — the same ordering the |
| retrieval stages show on these domains. |
|
|
| ### Selective prediction |
|
|
| On the 187 unanswerable items the correct action is to refuse. The base answers anyway **91%** of the |
| time; the fine-tune abstains on **80.2%** of them — this is the mechanism behind the faithfulness gain. |
|
|
| ### Position invariance (answerable subset, n=613) |
|
|
| <div style="overflow-x:auto"> |
| <table style="width:100%;border-collapse:collapse;font-size:13px;font-variant-numeric:tabular-nums;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif"> |
| <thead><tr> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:left;">metric (gold document position)</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">1</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">2</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">3</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">4</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">5</th> |
| <th style="padding:10px 8px;font-weight:500;border-bottom:2px solid #0a8043;color:#0a8043;font-size:14px;text-align:center;">6</th> |
| </tr></thead><tbody> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">correct — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">45.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">38.6</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">48.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">32.0</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">36.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">32.4</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">correct — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">78.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">77.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">84.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">76.7</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">84.5</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">76.5</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);">faithful — base</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">33.6</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">30.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">34.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">20.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">25.9</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);">32.4</td></tr> |
| <tr><td style="padding:7px 8px;padding-left:20px;text-align:left;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">faithful — fine-tuned</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">84.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">85.6</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">90.3</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">86.4</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">86.2</td><td style="padding:7px 8px;text-align:center;border-bottom:1px solid rgba(128,128,128,.15);font-weight:700;color:#0a8043;">85.3</td></tr> |
| </tbody></table></div> |
|
|
| **Read.** The base decays as the gold document moves deeper into the context (45.4% at position 1 → |
| 32.4% at position 6). The fine-tune is essentially **flat at 77-85%** across all six positions. |
|
|
| ## Usage (transformers) |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| name = "KIEFERSA/Sophea-RAG-Nemo3" |
| tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| name, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True).eval() |
| |
| SYSTEM = ( |
| "Είσαι βοηθός που απαντά ΑΠΟΚΛΕΙΣΤΙΚΑ με βάση τα παρεχόμενα έγγραφα. Χρησιμοποίησε μόνο τις πληροφορίες των εγγράφων, ΜΗΝ επινοείς. Ανέφερε σε αγκύλες τον αριθμό του εγγράφου/των εγγράφων που χρησιμοποίησες, π.χ. [2]. Αν η απάντηση ΔΕΝ υπάρχει στα έγγραφα, πες το καθαρά. Απάντησε στα ελληνικά, σύντομα και ουσιαστικά." |
| ) |
| |
| docs = ["Η Αθήνα είναι η πρωτεύουσα της Ελλάδας.", "Το Παρίσι είναι η πρωτεύουσα της Γαλλίας."] |
| question = "Ποια είναι η πρωτεύουσα της Ελλάδας;" |
| context = "\n".join(f"[{i}] {d}" for i, d in enumerate(docs, 1)) |
| |
| msgs = [{"role": "system", "content": SYSTEM}, |
| {"role": "user", "content": f"Έγγραφα:\n{context}\n\nΕρώτηση: {question}"}] |
| text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True) |
| enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device) |
| out = model.generate(**enc, max_new_tokens=200, do_sample=False) |
| print(tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True)) |
| # -> "Η Αθήνα. [1]" |
| ``` |
|
|
| Number the documents `[1..n]` in the user turn and keep the system message above — the model was |
| trained to cite those numbers and to say the answer is absent when it is. Greedy decoding |
| (`do_sample=False`) is what the numbers above were measured with. Needs `trust_remote_code=True`. |
|
|
| ## Serving with vLLM |
|
|
| ```bash |
| vllm serve KIEFERSA/Sophea-RAG-Nemo3 --trust-remote-code --max-model-len 40960 |
| ``` |
|
|
| ```bash |
| curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ |
| "model": "KIEFERSA/Sophea-RAG-Nemo3", |
| "temperature": 0, |
| "messages": [ |
| {"role": "system", "content": "Απάντησε μόνο από τα έγγραφα και ανέφερε [n]."}, |
| {"role": "user", "content": "Έγγραφα:\n[1] Η Αθήνα είναι η πρωτεύουσα της Ελλάδας.\n\nΕρώτηση: Ποια είναι η πρωτεύουσα της Ελλάδας;"} |
| ] |
| }' |
| ``` |
|
|
| ## License |
|
|
| Fine-tune of `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` — governed by the **NVIDIA Open Model License**; commercial use permitted under |
| those terms (note: **not** Apache-2.0). Review the base model's license before use. |
|
|
