Upload folder using huggingface_hub

README.md

@@ -0,0 +1,36 @@
+---
+tags:
+- sae
+- saelens
+- phi-3
+---
+
+# Sparse Autoencoders for Phi-3-mini-4k-instruct
+
+These are trained Sparse Autoencoders (SAEs) for the `microsoft/Phi-3-mini-4k-instruct` model, compatible with the `sae_lens` library.
+
+## Model Details
+* **Model**: `microsoft/Phi-3-mini-4k-instruct`
+* **Layers**: [21, 22, 23]
+* **Architecture**: Standard (ReLU) with pre-encoder centring (`b_dec` applied to input).
+* **Expansion Factor**: 16x (49152 features)
+* **Tokens Trained**: ~550M
+
+## Datasets
+* NuminaMath-CoT: 40%
+* HH-RLHF: 20%
+* FineWeb: 20%
+* JBB/HarmBench/AdvBench: 20%
+
+## Usage with SAELens
+
+```python
+from sae_lens import SAE
+
+# Load the SAE for a specific layer, e.g., layer 21
+sae, cfg_dict, sparsity = SAE.from_pretrained(
+    release="analist/SAE-Phi-3-mini-4k-instruct",
+    sae_id="layer_21",
+    device="cuda"
+)
+```

layer_21/cfg.json

@@ -0,0 +1,18 @@
+{
+  "architecture": "standard",
+  "d_in": 3072,
+  "d_sae": 49152,
+  "activation_fn_str": "relu",
+  "apply_b_dec_to_input": true,
+  "finetuning_scaling_factor": false,
+  "context_size": 1024,
+  "model_name": "microsoft/Phi-3-mini-4k-instruct",
+  "hook_name": "blocks.21.hook_resid_post",
+  "hook_layer": 21,
+  "hook_head_index": null,
+  "prepend_bos": true,
+  "dataset_path": "mixed-datasets",
+  "dataset_trust_remote_code": true,
+  "normalize_activations": "none",
+  "dtype": "bfloat16"
+}

layer_21/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b24d2e2331fb7eb66008fa8a7ee7ae768817b62be6a69fd30257aeeea40e074
+size 604477848

layer_22/cfg.json

@@ -0,0 +1,18 @@
+{
+  "architecture": "standard",
+  "d_in": 3072,
+  "d_sae": 49152,
+  "activation_fn_str": "relu",
+  "apply_b_dec_to_input": true,
+  "finetuning_scaling_factor": false,
+  "context_size": 1024,
+  "model_name": "microsoft/Phi-3-mini-4k-instruct",
+  "hook_name": "blocks.22.hook_resid_post",
+  "hook_layer": 22,
+  "hook_head_index": null,
+  "prepend_bos": true,
+  "dataset_path": "mixed-datasets",
+  "dataset_trust_remote_code": true,
+  "normalize_activations": "none",
+  "dtype": "bfloat16"
+}

layer_22/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e63d6872a342300c59ca25a16b593c67a73e6dc8e62b401daa7e8bcc38396cbf
+size 604477848

layer_23/cfg.json

@@ -0,0 +1,18 @@
+{
+  "architecture": "standard",
+  "d_in": 3072,
+  "d_sae": 49152,
+  "activation_fn_str": "relu",
+  "apply_b_dec_to_input": true,
+  "finetuning_scaling_factor": false,
+  "context_size": 1024,
+  "model_name": "microsoft/Phi-3-mini-4k-instruct",
+  "hook_name": "blocks.23.hook_resid_post",
+  "hook_layer": 23,
+  "hook_head_index": null,
+  "prepend_bos": true,
+  "dataset_path": "mixed-datasets",
+  "dataset_trust_remote_code": true,
+  "normalize_activations": "none",
+  "dtype": "bfloat16"
+}

layer_23/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8158d8832404c76e0747d3eedcc9ceada04c9081a53f7e8f5c19e7ef3a751fad
+size 604477848