Thanks for all the suggestions! Good luck with your future models!
Haha, I’ve actually been following AI/ML developments for a year or two, and I only got seriously interested in the SLM space a couple months ago. I really only started building models myself about a month ago with MetaDiffusion though, so I’m still learning a ton and figuring things out as I go.
Thanks! I’m still only about a week into developing SLMs, so I still have a lot to learn.
That’s really generous, thank you! I think I’d rather keep the training on my end, though. I like being able to run everything myself, and I’d feel bad putting any pressure on you to spend your hardware and time on it. I really appreciate the offer!
That’s totally okay! I’m still looking for some more datasets to see where I can improve the model. I’m pretty compute and time-limited, so I usually can’t train for more than 2–3 billion tokens at this model size.
Yeah, my main concern with the benchmarks is that Sorbet-v2 is generally trailing other models in the same parameter range. For example, it scores below Supra2 Medium (25M), BananaMind 2 Mini (25M), and Veyra2 Mango (15M) on most of the benchmarks I’ve tested, although it barely passes them on HellaSwag. On ARC Easy, for example, it’s around 15% behind Supra2 Medium.
That said, you might be right that I’m simply hitting the limit of what I can squeeze out of a ~25M-parameter dense Transformer. That’s actually part of why I’m interested in trying more unusual architectures rather than just making the model bigger.
I just added an option to choose response length (64 to 512.) Thank you again for the suggestion!
Now that it's a ZeroGPU space, I could definitely let the user pick the max response length. Thank you for the suggestion, I'll implement it when I get a chance!
As much as I would like to keep it a CPU Basic space, it isn't enough processing power. Even with only four models, response time was around 40 seconds, so moving to ZeroGPU was necessary unfortunately. Thank you for the suggestion!
why do bots keep popping up everywhere
I'm working on adding a separate base model section right now! All of the data from the battles is available in the CodeSoft/slm-arena-data bucket (which I might make a filtered version published as a dataset later.)
