Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Hoglet-33 
posted an update 1 day ago
Post
2081
Today, we planned to release Pebble-50M and Pebble-50M-Chat to the world. Unfortunately, due to a few issues, that didn't go quite as planned.

What happened:
- Some data and benchmark results were lost or corrupted
- The models performed worse on benchmarks than our other Pebble models

Despite that, you can still find both models here:
Pebble-50M-beta: basically-experimental/Pebble-50M-beta
Pebble-50M-Chat-beta: basically-experimental/Pebble-50M-Chat-beta

There are still some interesting improvements in these models:
- Compatible with non-CUDA devices
- Vocabulary increased to 16K tokens
- Context length increased to 16K tokens

For now, there won't be any more Pebble releases for a while. We're going to take some time to experiment with other approaches and hopefully make the next generation a monumental leap over this one.

Follow for updates:
@Hoglet-33
basically-ai
basically-experimental
Pankaj8922
•
edited 1 day ago

why don't you guys try making a simplifer first? simplify pretraining data as much as possible then train the small models on it? Small models can't learn much from sophisticated datasets. Yes it's a extensive thing to do but that's what really can make a difference otherwise you guys will keep training small models and they won't improve a bit.

·
Hoglet-33

that could work, we are going to start a series of tests that will hopefully give some useful data, simplifying data will probably be part of it.
In this post