On-Device SLM Rise: SmolLM2 & LLaMA 3.2 Embedded Edge Deployment (MLC-LLM / llama.cpp)

📱 The rise of small end-side models and high-energy data refinement

Hugging Face's officially released SmolLM2 (including 135M, 360M, 1.7B) and Meta's LLaMA 3.2 1B/3B prove that under the training of massive refined data (such as FineMath, Cosmopedia v2), the 1B~3B small model is fully equipped with excellent instruction following, text summarization and simple logical reasoning capabilities, and the video memory/memory footprint only requires 1GB~2GB!

⚡ Image pulling and llama.cpp for ultimate lightweight deployment

Compile and run at extremely high speed on the client side

# Image pull SmolLM2 1.7B command version
export HF_ENDPOINT="https://hf-mirror.net"
huggingface-cli download HuggingFaceTB/SmolLM2-1.7B-Instruct \
  --local-dir /data/models/SmolLM2-1.7B-Instruct \
  --local-dir-use-symlinks False

# Compile with llama.cpp and generate at 150+ tokens/s on CPU
./llama-cli -m /data/models/SmolLM2-1.7B-Instruct/smollm2-1.7b-q4_k_m.gguf \
  -p "Hello! Please summarize the advantages of edge AI." -n 128