Google Gemma 2 Architecture: Sliding Window Attention (SWA), Logit Soft-Capping & Unsloth Fine-Tuning

💎 Google Gemma 2 core design philosophy

Alternating Sliding Window Attention (SWA): Even-numbered layers use 4096 window local attention, and odd-numbered layers use global attention, reducing long text KV memory by nearly half.
Logit Soft-Capping: Use tanh after attention projection to scale and truncate the activation value to a specified range to prevent extreme gradients from causing generation crashes.

⚡ Image pulling and Unsloth fast fine-tuning

Download Gemma 2 9B weights

export HF_ENDPOINT="https://hf-mirror.net"
huggingface-cli download google/gemma-2-9b-it \
  --local-dir /data/models/gemma-2-9b-it \
  --local-dir-use-symlinks False

Fine-tune sample code

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="/data/models/gemma-2-9b-it",
    max_seq_length=4096,
    load_in_4bit=True
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16
)
print("Gemma 2 LoRA adapter ready!")