Byrne Models
Model Link: https://huggingface.co/Quazim0t0/Byrne-86M
Model Link: https://huggingface.co/Quazim0t0/Byrne-86M-Base
| Task | Metric | Byrne-86M | Byrne-86M-Base |
|---|---|---|---|
| ARC-Easy | acc_norm | 0.3468 | 0.3931 |
| ARC-Challenge | acc_norm | 0.2355 | 0.2389 |
| HellaSwag | acc_norm | 0.2858 | 0.2927 |
| WinoGrande | acc | 0.5201 | 0.5193 |
| PIQA | acc_norm | 0.5593 | 0.5860 |
| OpenBookQA | acc_norm | 0.2440 | 0.2820 |
| BoolQ | acc | 0.3865 | 0.6171 |
| ArithMark-2.0 | acc | 0.3096 | 0.2732 |
| WikiText-2 byte-ppl ↓ | — | 2.6839 | 2.3753 |
| BLiMP ↑ | acc | 0.7033 | 0.7356 |
I also wanted to ask if it was possible to update one of my other models scores, I did a quick SFT training during the Small Models, Big Adventures Hackathon for the Escarda-86M and I know the scores can be better. I just have to redo it properly. The model: https://huggingface.co/Quazim0t0/Escarda-86M, the Escarda-86M-Base would still be the same scores.
Yeah sure happy to update the instruct scores
@Datdanboi25 Thank you! It may not be for a day or two for the Escarda-86M, but I will let you know as soon as I update the repo. The Byrne Models will stay as is.
For Byrne-86M & Base there won't be any future changes, I had time to get it right for the instruct.
I also just noticed that I was supposed to do a pull request, I really do apologize for doing that twice now. I can open a pull request for Byrne models.
Oh haha not a problem, both Bryne models up! LMK when the updated escarda is out and Ill update it.
I just seen this now, thank you, does it usually appear after the Space restarts?
Ahh sorry should be available now!
@Datdanboi25 Hey Dan, I just tried to beat the last Escarda training and even though it wasn't very well planned out and also quickly done, it seems I cannot defeat those scores. So, in turn I uploaded another version that aligns more with what I wanted the model to be: https://huggingface.co/Quazim0t0/Escarda-86M-Identity
Evaluation
Zero-shot, full validation/test splits (acc = raw continuation log-likelihood,acc_norm = byte-length-normalized).
| Task | acc | acc_norm |
|---|---|---|
| ARC-Easy | 0.3262 | 0.3380 |
| ARC-Challenge | 0.2048 | 0.2415 |
| HellaSwag | 0.2785 | 0.2818 |
| WinoGrande | 0.5020 | — |
| PIQA | 0.5539 | 0.5462 |
| OpenBookQA | 0.1360 | 0.2440 |
| BoolQ | 0.4174 | — |
ArithMark-2.0 (AxiomicLabs)
— official metric is raw acc: 0.3628 (the strongest of the Escarda family).
Language modeling: WikiText-2 byte-ppl ↓ 2.7062 · BLiMP ↑ 0.7133.
Sorry it took me a little to get around to this, my computer was in the middle of doing my Byrne-Embed project, so I had to let it be for a while. Also thank you for following my projects!
Dayum 36% on arithmark is crazy, will update it shortly.
I can test again to be sure. I saw it had really low scores for everything else, so I didn't think much of it. This was mostly to train in the identity of the model.
Its up!