[mergeheal] 02:57:34 :: ######## SH run (Candidate B: min_rel_delta=0.01) — 2026-05-20T02:57:34Z ########
name, memory.total [MiB], driver_version
NVIDIA H200, 143771 MiB, 580.126.09
[mergeheal] 02:57:34 :: code HEAD     = 7c686be
[mergeheal] 02:57:34 :: min_rel_delta =   merge_heal_min_rel_delta: 0.01   # patience resets only if h < best*(1 - this).
[mergeheal] 02:57:34 :: max_steps     =   merge_heal_max_steps: 20000      # real safety cap; convergence is min-rel-delta-
[mergeheal] 02:57:34 :: seeding FP8 kernels from /opt/kernels-hub
[mergeheal] 02:57:34 :: STEP 1 — Stage-1 preflight (separate process)
2026-05-20 02:57:36,771 INFO __main__ :: ========== Stage 2 v2 Ablation Harness ==========
2026-05-20 02:57:36,791 INFO __main__ :: Will run 29 ablation(s): ['A0', 'A1', 'A1_oldkd', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8', 'A9', 'A10', 'A11', 'R0', 'R1', 'R2', 'R3', 'R4', 'R5', 'R6', 'R7', 'S0', 'SA', 'SAB', 'SC', 'SCD', 'SE', 'SH', 'SHR']
2026-05-20 02:57:36,791 INFO __main__ :: Pre-flight Stage 1 already complete in /cache/ablations/_shared
2026-05-20 02:57:36,791 INFO __main__ :: --preflight-only: exiting after pre-flight. _shared/ ready at /cache/ablations/_shared
[mergeheal] 02:57:37 :: STEP 2 — row SH
2026-05-20 02:57:39,056 INFO __main__ :: ========== Stage 2 v2 Ablation Harness ==========
2026-05-20 02:57:39,076 INFO __main__ :: Will run 1 ablation(s): ['SH']
2026-05-20 02:57:39,076 INFO __main__ :: Pre-flight Stage 1 already complete in /cache/ablations/_shared
2026-05-20 02:57:39,076 INFO __main__ :: [1/1] === SH ===
2026-05-20 02:57:39,076 INFO __main__ :: [SH] starting (deltas={'merge_heal_enabled': True, 'merge_heal_train_router': True})
* Trackio project initialized: moe-compress-strategy-a
* Trackio metrics will be synced to Hugging Face Bucket: https://huggingface.co/buckets/pirola/trackio-bucket
* Found existing space: https://huggingface.co/spaces/pirola/trackio
* NVIDIA GPU detected, enabling automatic GPU metrics logging
* Created new run: ablation-SH
2026-05-20 02:57:46,358 INFO __main__ :: Artifacts directory: /cache/ablations/SH
2026-05-20 02:57:46,358 INFO __main__ :: Pipeline target: 35.0% total parameter reduction
2026-05-20 02:57:48,832 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:57:48,859 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/config.json "HTTP/1.1 200 OK"
2026-05-20 02:57:48,873 INFO moe_compress.utils.model_io :: Loading Qwen/Qwen3.6-35B-A3B with AutoModelForImageTextToText (dtype=torch.bfloat16, device_map=auto)
2026-05-20 02:57:49,088 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:57:49,116 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/config.json "HTTP/1.1 200 OK"
2026-05-20 02:57:49,318 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/adapter_config.json "HTTP/1.1 404 Not Found"
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
[WARNING] Field "span" in type "ir.TupleType" duplicates an ancestor field in "ir.Type". Child types should not re-register inherited fields.
[WARNING] Field "span" in type "ir.FuncType" duplicates an ancestor field in "ir.Type". Child types should not re-register inherited fields.
[WARNING] Field "span" in type "ir.TensorMapType" duplicates an ancestor field in "ir.Type". Child types should not re-register inherited fields.
[WARNING] Field "binding_blocks" in type "script.ir_builder.relax.FunctionFrame" duplicates an ancestor field in "script.ir_builder.relax.SeqExprFrame". Child types should not re-register inherited fields.
[WARNING] Field "output" in type "script.ir_builder.relax.FunctionFrame" duplicates an ancestor field in "script.ir_builder.relax.SeqExprFrame". Child types should not re-register inherited fields.
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'span' in 'ir.TupleType' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'span' in 'ir.FuncType' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'span' in 'ir.TensorMapType' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'name' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'tag' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'attrs' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'shape' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'dtype' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
[WARNING] Field "policy_type" in type "tl.GemmSPWarpPolicy" duplicates an ancestor field in "tl.GemmWarpPolicy". Child types should not re-register inherited fields.
[WARNING] Field "m_warp" in type "tl.GemmSPWarpPolicy" duplicates an ancestor field in "tl.GemmWarpPolicy". Child types should not re-register inherited fields.
[WARNING] Field "n_warp" in type "tl.GemmSPWarpPolicy" duplicates an ancestor field in "tl.GemmWarpPolicy". Child types should not re-register inherited fields.
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'policy_type' in 'tl.GemmSPWarpPolicy' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'm_warp' in 'tl.GemmSPWarpPolicy' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'n_warp' in 'tl.GemmSPWarpPolicy' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
2026-05-20 02:57:52,282 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:57:52,309 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/config.json "HTTP/1.1 200 OK"
2026-05-20 02:57:52,507 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/model.safetensors "HTTP/1.1 404 Not Found"

Loading weights:   0%|          | 0/1026 [00:00<?, ?it/s]
Loading weights:   0%|          | 2/1026 [00:00<01:41, 10.04it/s]
Loading weights:   1%|▏         | 14/1026 [00:00<00:26, 38.46it/s]
Loading weights:   3%|▎         | 31/1026 [00:00<00:17, 57.85it/s]
Loading weights:   5%|▍         | 50/1026 [00:00<00:13, 71.21it/s]
Loading weights:   6%|▌         | 59/1026 [00:01<00:15, 62.11it/s]
Loading weights:   8%|▊         | 83/1026 [00:01<00:11, 79.57it/s]
Loading weights:  10%|▉         | 101/1026 [00:01<00:11, 81.12it/s]
Loading weights:  12%|█▏        | 119/1026 [00:01<00:11, 82.29it/s]
Loading weights:  12%|█▏        | 128/1026 [00:01<00:12, 71.08it/s]
Loading weights:  15%|█▍        | 151/1026 [00:01<00:08, 99.07it/s]
Loading weights:  16%|█▌        | 163/1026 [00:02<00:09, 95.29it/s]
Loading weights:  17%|█▋        | 174/1026 [00:02<00:10, 80.05it/s]
Loading weights:  18%|█▊        | 188/1026 [00:02<00:11, 75.39it/s]
Loading weights:  19%|█▉        | 197/1026 [00:02<00:12, 65.57it/s]
Loading weights:  22%|██▏       | 221/1026 [00:02<00:10, 78.07it/s]
Loading weights:  23%|██▎       | 239/1026 [00:03<00:10, 77.63it/s]
Loading weights:  25%|██▌       | 257/1026 [00:03<00:09, 78.21it/s]
Loading weights:  26%|██▌       | 265/1026 [00:03<00:11, 65.36it/s]
Loading weights:  28%|██▊       | 289/1026 [00:03<00:09, 76.50it/s]
Loading weights:  30%|███       | 308/1026 [00:04<00:09, 79.17it/s]
Loading weights:  32%|███▏      | 326/1026 [00:04<00:08, 79.05it/s]
Loading weights:  33%|███▎      | 335/1026 [00:04<00:09, 70.55it/s]
Loading weights:  35%|███▍      | 359/1026 [00:04<00:08, 81.83it/s]
Loading weights:  37%|███▋      | 377/1026 [00:04<00:07, 82.43it/s]
Loading weights:  38%|███▊      | 395/1026 [00:05<00:07, 83.08it/s]
Loading weights:  39%|███▉      | 404/1026 [00:05<00:08, 71.83it/s]
Loading weights:  42%|████▏     | 427/1026 [00:05<00:06, 97.95it/s]
Loading weights:  43%|████▎     | 439/1026 [00:05<00:06, 94.01it/s]
Loading weights:  44%|████▍     | 450/1026 [00:05<00:07, 76.83it/s]
Loading weights:  45%|████▌     | 464/1026 [00:06<00:07, 73.95it/s]
Loading weights:  46%|████▌     | 473/1026 [00:06<00:08, 64.88it/s]
Loading weights:  48%|████▊     | 496/1026 [00:06<00:05, 93.75it/s]
Loading weights:  50%|████▉     | 508/1026 [00:06<00:05, 90.34it/s]
Loading weights:  51%|█████     | 519/1026 [00:06<00:06, 77.89it/s]
Loading weights:  52%|█████▏    | 533/1026 [00:06<00:06, 77.36it/s]
Loading weights:  53%|█████▎    | 542/1026 [00:07<00:07, 66.92it/s]
Loading weights:  55%|█████▌    | 566/1026 [00:07<00:05, 81.43it/s]
Loading weights:  57%|█████▋    | 584/1026 [00:07<00:05, 82.81it/s]
Loading weights:  59%|█████▊    | 601/1026 [00:07<00:05, 81.69it/s]
Loading weights:  60%|█████▉    | 611/1026 [00:07<00:05, 71.80it/s]
Loading weights:  62%|██████▏   | 635/1026 [00:08<00:04, 81.87it/s]
Loading weights:  64%|██████▎   | 653/1026 [00:08<00:04, 83.54it/s]
Loading weights:  65%|██████▌   | 671/1026 [00:08<00:04, 85.38it/s]
Loading weights:  66%|██████▋   | 680/1026 [00:08<00:04, 73.81it/s]
Loading weights:  84%|████████▎ | 857/1026 [00:08<00:00, 367.42it/s]
Loading weights: 100%|█████████▉| 1021/1026 [00:08<00:00, 618.53it/s]
Loading weights: 100%|██████████| 1026/1026 [00:08<00:00, 114.01it/s]
2026-05-20 02:58:04,572 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:58:04,593 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/generation_config.json "HTTP/1.1 200 OK"
2026-05-20 02:58:04,784 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:58:04,805 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/config.json "HTTP/1.1 200 OK"
2026-05-20 02:58:05,014 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:58:05,037 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/tokenizer_config.json "HTTP/1.1 200 OK"
2026-05-20 02:58:05,237 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-05-20 02:58:05,259 INFO httpx :: HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Qwen/Qwen3.6-35B-A3B/995ad96eacd98c81ed38be0c5b274b04031597b0/tokenizer_config.json "HTTP/1.1 200 OK"
2026-05-20 02:58:05,451 INFO httpx :: HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.6-35B-A3B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-05-20 02:58:05,649 INFO httpx :: HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.6-35B-A3B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-05-20 02:58:05,842 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/added_tokens.json "HTTP/1.1 404 Not Found"
2026-05-20 02:58:06,034 INFO httpx :: HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3.6-35B-A3B/resolve/main/special_tokens_map.json "HTTP/1.1 404 Not Found"
2026-05-20 02:58:06,762 INFO httpx :: HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.6-35B-A3B "HTTP/1.1 200 OK"
2026-05-20 02:58:06,768 INFO moe_compress.utils.model_io :: Model loaded: Qwen3_5MoeForConditionalGeneration on devices=cuda:0:35.1B
2026-05-20 02:58:06,821 WARNING moe_compress.utils.trackio_log :: trackio.log failed; subsequent log failures will not emit a warning
Traceback (most recent call last):
  File "/cache/code/moe_compress/max_quality/src/moe_compress/utils/trackio_log.py", line 190, in trackio_log
    mod.log(metrics)
  File "/usr/local/lib/python3.12/dist-packages/trackio/__init__.py", line 700, in log
    raise RuntimeError("Call trackio.init() before trackio.log().")
RuntimeError: Call trackio.init() before trackio.log().
2026-05-20 02:58:06,835 INFO __main__ :: === Stage 2 — REAP + REAM ===
2026-05-20 02:58:06,835 INFO moe_compress.stage5_router_kd :: Stage 5: MoE experts_implementation = 'grouped_mm' (forward dispatch via transformers.integrations.moe.ExpertsInterface)
2026-05-20 02:58:06,849 INFO moe_compress.utils.calibration :: Building calibration tensor: 1000 x 2048 tokens, source=nvidia-cascade
2026-05-20 02:58:06,849 INFO moe_compress.utils.calibration ::   per-subset sequence counts: {'math': 210, 'science': 110, 'chat': 560, 'instruction_following': 34, 'conversational_agent': 33, 'swe': 20, 'terminal_agent': 33}
2026-05-20 02:58:07,073 INFO moe_compress.utils.calibration :: Streaming 210 math samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=444731)
2026-05-20 02:58:41,853 INFO moe_compress.utils.calibration :: Streaming 110 science samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=772340)
2026-05-20 02:58:51,162 INFO moe_compress.utils.calibration :: Streaming 560 chat samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=492622)
2026-05-20 02:59:01,433 INFO moe_compress.utils.calibration :: Streaming 34 instruction_following samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=307652)
2026-05-20 02:59:06,292 INFO moe_compress.utils.calibration :: Streaming 33 conversational_agent samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=234607)
2026-05-20 02:59:18,177 INFO moe_compress.utils.calibration :: Streaming 20 swe samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=976611)
2026-05-20 03:01:04,062 INFO moe_compress.utils.calibration :: Streaming 33 terminal_agent samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=435928)
2026-05-20 03:01:37,072 INFO moe_compress.utils.calibration :: Cached calibration tensor: /cache/ablations/_shared/_calibration_cache/calib_0c65ad6ae83119b9.pt (shape=(1000, 2048))
2026-05-20 03:01:37,076 INFO moe_compress.stage2_reap_ream :: Stage-2 merge-heal enabled: self-distillation, pool=262144 tokens/layer, train_router=True, lr=0.0001
2026-05-20 03:01:37,077 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 1/40 (idx=0) — profiling then merging to 256 experts
2026-05-20 03:01:37,089 INFO moe_compress.stage2_reap_ream :: DIAG layer 0: entering batch loop (calibration tensor + early-exit forwards) | vram=70.9/150GB host_free=184GB gpu_util=0% temp=28C
2026-05-20 03:02:11,800 INFO moe_compress.stage2_reap_ream :: DIAG layer 0 batch 1: total=34.71s fwd=34.56s hooks: input=0.03s intermed=0.07s down=0.08s (n_cb=256) | cum=34.7s | vram=92.0/150GB host_free=183GB gpu_util=13% temp=32C
2026-05-20 03:02:12,221 INFO moe_compress.stage2_reap_ream :: DIAG layer 0 batch 2: total=0.42s fwd=0.36s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=35.1s | vram=92.0/150GB host_free=183GB gpu_util=51% temp=35C
2026-05-20 03:02:12,621 INFO moe_compress.stage2_reap_ream :: DIAG layer 0 batch 3: total=0.40s fwd=0.34s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=35.5s | vram=92.1/150GB host_free=183GB gpu_util=54% temp=35C
2026-05-20 03:02:15,268 INFO moe_compress.stage2_reap_ream :: DIAG layer 0 batch 10: total=0.37s fwd=0.31s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=38.2s | vram=92.1/150GB host_free=182GB gpu_util=58% temp=35C
2026-05-20 03:02:18,906 INFO moe_compress.stage2_reap_ream :: DIAG layer 0 batch 20: total=0.36s fwd=0.30s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=41.8s | vram=92.1/150GB host_free=182GB gpu_util=60% temp=35C
2026-05-20 03:02:22,546 INFO moe_compress.stage2_reap_ream :: DIAG layer 0 batch 30: total=0.36s fwd=0.30s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=45.5s | vram=92.1/150GB host_free=181GB gpu_util=61% temp=37C
2026-05-20 03:02:33,892 INFO moe_compress.stage2_reap_ream :: DIAG layer 0: batch loop complete — 32 batches in 56.8s, now post-profile work | vram=92.1/150GB host_free=181GB gpu_util=34% temp=34C
2026-05-20 03:02:35,151 WARNING moe_compress.stage2_reap_ream :: layer 0: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 03:02:35,153 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 0: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 03:02:38,331 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 03:02:38,768 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 2/40 (idx=1) — profiling then merging to 256 experts
2026-05-20 03:02:38,774 INFO moe_compress.stage2_reap_ream :: DIAG layer 1: entering batch loop (calibration tensor + early-exit forwards) | vram=71.5/150GB host_free=181GB gpu_util=21% temp=30C
2026-05-20 03:02:39,266 INFO moe_compress.stage2_reap_ream :: DIAG layer 1 batch 1: total=0.49s fwd=0.40s hooks: input=0.02s intermed=0.05s down=0.04s (n_cb=256) | cum=0.5s | vram=92.2/150GB host_free=181GB gpu_util=53% temp=35C
2026-05-20 03:02:39,676 INFO moe_compress.stage2_reap_ream :: DIAG layer 1 batch 2: total=0.41s fwd=0.35s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=0.9s | vram=93.2/150GB host_free=181GB gpu_util=60% temp=33C
2026-05-20 03:02:40,075 INFO moe_compress.stage2_reap_ream :: DIAG layer 1 batch 3: total=0.40s fwd=0.34s hooks: input=0.02s intermed=0.04s down=0.03s (n_cb=256) | cum=1.3s | vram=93.2/150GB host_free=181GB gpu_util=61% temp=34C
2026-05-20 03:02:42,906 INFO moe_compress.stage2_reap_ream :: DIAG layer 1 batch 10: total=0.40s fwd=0.34s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.1s | vram=93.2/150GB host_free=180GB gpu_util=60% temp=36C
2026-05-20 03:02:46,952 INFO moe_compress.stage2_reap_ream :: DIAG layer 1 batch 20: total=0.40s fwd=0.34s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=8.2s | vram=93.2/150GB host_free=180GB gpu_util=59% temp=37C
2026-05-20 03:02:51,012 INFO moe_compress.stage2_reap_ream :: DIAG layer 1 batch 30: total=0.40s fwd=0.34s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=12.2s | vram=93.2/150GB host_free=179GB gpu_util=59% temp=37C
2026-05-20 03:02:51,675 INFO moe_compress.stage2_reap_ream :: DIAG layer 1: batch loop complete — 32 batches in 12.9s, now post-profile work | vram=93.2/150GB host_free=179GB gpu_util=36% temp=35C
2026-05-20 03:02:52,942 WARNING moe_compress.stage2_reap_ream :: layer 1: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 03:02:52,944 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 1: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 03:02:56,365 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 03:02:56,798 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 3/40 (idx=2) — profiling then merging to 256 experts
2026-05-20 03:02:56,803 INFO moe_compress.stage2_reap_ream :: DIAG layer 2: entering batch loop (calibration tensor + early-exit forwards) | vram=71.5/150GB host_free=177GB gpu_util=3% temp=31C
2026-05-20 03:02:57,273 INFO moe_compress.stage2_reap_ream :: DIAG layer 2 batch 1: total=0.47s fwd=0.42s hooks: input=0.02s intermed=0.05s down=0.04s (n_cb=256) | cum=0.5s | vram=92.3/150GB host_free=177GB gpu_util=54% temp=34C
2026-05-20 03:02:57,685 INFO moe_compress.stage2_reap_ream :: DIAG layer 2 batch 2: total=0.41s fwd=0.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=0.9s | vram=93.3/150GB host_free=177GB gpu_util=58% temp=36C
2026-05-20 03:02:58,092 INFO moe_compress.stage2_reap_ream :: DIAG layer 2 batch 3: total=0.41s fwd=0.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.3s | vram=93.3/150GB host_free=177GB gpu_util=60% temp=37C
2026-05-20 03:03:00,933 INFO moe_compress.stage2_reap_ream :: DIAG layer 2 batch 10: total=0.40s fwd=0.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.1s | vram=93.3/150GB host_free=177GB gpu_util=61% temp=38C
2026-05-20 03:03:04,987 INFO moe_compress.stage2_reap_ream :: DIAG layer 2 batch 20: total=0.40s fwd=0.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=8.2s | vram=93.3/150GB host_free=177GB gpu_util=59% temp=39C
2026-05-20 03:03:09,068 INFO moe_compress.stage2_reap_ream :: DIAG layer 2 batch 30: total=0.42s fwd=0.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=12.3s | vram=93.3/150GB host_free=176GB gpu_util=61% temp=39C
2026-05-20 03:03:09,766 INFO moe_compress.stage2_reap_ream :: DIAG layer 2: batch loop complete — 32 batches in 13.0s, now post-profile work | vram=93.3/150GB host_free=176GB gpu_util=64% temp=36C
2026-05-20 03:03:10,987 WARNING moe_compress.stage2_reap_ream :: layer 2: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 03:03:10,989 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 2: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 03:03:14,090 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 03:03:14,383 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 4/40 (idx=3) — profiling then merging to 256 experts
2026-05-20 03:03:14,388 INFO moe_compress.stage2_reap_ream :: DIAG layer 3: entering batch loop (calibration tensor + early-exit forwards) | vram=71.5/150GB host_free=172GB gpu_util=12% temp=31C
2026-05-20 03:03:15,707 INFO moe_compress.stage2_reap_ream :: DIAG layer 3 batch 1: total=1.32s fwd=1.26s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.3s | vram=92.4/150GB host_free=172GB gpu_util=56% temp=35C
2026-05-20 03:03:16,156 INFO moe_compress.stage2_reap_ream :: DIAG layer 3 batch 2: total=0.45s fwd=0.42s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.8s | vram=93.5/150GB host_free=172GB gpu_util=63% temp=37C
2026-05-20 03:03:16,603 INFO moe_compress.stage2_reap_ream :: DIAG layer 3 batch 3: total=0.45s fwd=0.42s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.2s | vram=93.5/150GB host_free=172GB gpu_util=59% temp=38C
2026-05-20 03:03:19,754 INFO moe_compress.stage2_reap_ream :: DIAG layer 3 batch 10: total=0.45s fwd=0.42s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.4s | vram=93.5/150GB host_free=172GB gpu_util=62% temp=39C
2026-05-20 03:03:24,270 INFO moe_compress.stage2_reap_ream :: DIAG layer 3 batch 20: total=0.45s fwd=0.42s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=9.9s | vram=93.5/150GB host_free=172GB gpu_util=59% temp=40C
2026-05-20 03:03:28,779 INFO moe_compress.stage2_reap_ream :: DIAG layer 3 batch 30: total=0.45s fwd=0.42s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.4s | vram=93.5/150GB host_free=172GB gpu_util=60% temp=39C
2026-05-20 03:03:29,508 INFO moe_compress.stage2_reap_ream :: DIAG layer 3: batch loop complete — 32 batches in 15.1s, now post-profile work | vram=93.5/150GB host_free=172GB gpu_util=54% temp=37C
2026-05-20 03:03:29,651 WARNING moe_compress.stage2_reap_ream :: layer 3: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 03:03:29,653 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 3: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 03:03:32,519 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 03:03:32,811 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 5/40 (idx=4) — profiling then merging to 229 experts
2026-05-20 03:03:32,816 INFO moe_compress.stage2_reap_ream :: DIAG layer 4: entering batch loop (calibration tensor + early-exit forwards) | vram=71.5/150GB host_free=170GB gpu_util=0% temp=32C
2026-05-20 03:03:33,366 INFO moe_compress.stage2_reap_ream :: DIAG layer 4 batch 1: total=0.55s fwd=0.50s hooks: input=0.02s intermed=0.05s down=0.04s (n_cb=256) | cum=0.6s | vram=92.4/150GB host_free=170GB gpu_util=56% temp=37C
2026-05-20 03:03:33,859 INFO moe_compress.stage2_reap_ream :: DIAG layer 4 batch 2: total=0.49s fwd=0.47s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.0s | vram=93.4/150GB host_free=170GB gpu_util=59% temp=38C
2026-05-20 03:03:34,348 INFO moe_compress.stage2_reap_ream :: DIAG layer 4 batch 3: total=0.49s fwd=0.46s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.5s | vram=93.4/150GB host_free=170GB gpu_util=60% temp=39C
2026-05-20 03:03:37,795 INFO moe_compress.stage2_reap_ream :: DIAG layer 4 batch 10: total=0.49s fwd=0.47s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.0s | vram=93.4/150GB host_free=170GB gpu_util=56% temp=39C
2026-05-20 03:03:42,724 INFO moe_compress.stage2_reap_ream :: DIAG layer 4 batch 20: total=0.49s fwd=0.47s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=9.9s | vram=93.4/150GB host_free=170GB gpu_util=59% temp=40C
2026-05-20 03:03:47,676 INFO moe_compress.stage2_reap_ream :: DIAG layer 4 batch 30: total=0.49s fwd=0.47s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.9s | vram=93.4/150GB host_free=170GB gpu_util=58% temp=41C
2026-05-20 03:03:48,442 INFO moe_compress.stage2_reap_ream :: DIAG layer 4: batch loop complete — 32 batches in 15.6s, now post-profile work | vram=93.5/150GB host_free=170GB gpu_util=34% temp=38C
2026-05-20 03:03:56,072 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 4 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:04:03,059 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 25/20000 — train_mse=7.023120e-06 holdout_mse=7.643578e-06 best=7.643578e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:06,988 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 50/20000 — train_mse=5.843554e-06 holdout_mse=6.108703e-06 best=6.108703e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:10,866 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 75/20000 — train_mse=4.954250e-06 holdout_mse=5.350021e-06 best=5.350021e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:14,740 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 100/20000 — train_mse=4.384850e-06 holdout_mse=4.893150e-06 best=4.893150e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:18,729 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 125/20000 — train_mse=4.137112e-06 holdout_mse=4.589975e-06 best=4.589975e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:22,614 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 150/20000 — train_mse=4.204379e-06 holdout_mse=4.364933e-06 best=4.364933e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:26,500 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 175/20000 — train_mse=3.598175e-06 holdout_mse=4.193877e-06 best=4.193877e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:30,405 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 200/20000 — train_mse=3.504108e-06 holdout_mse=4.049981e-06 best=4.049981e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:34,278 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 225/20000 — train_mse=4.172873e-06 holdout_mse=3.921010e-06 best=3.921010e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:38,164 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 250/20000 — train_mse=3.774909e-06 holdout_mse=3.811477e-06 best=3.811477e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:42,035 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 275/20000 — train_mse=3.139854e-06 holdout_mse=3.706675e-06 best=3.706675e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:46,119 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 300/20000 — train_mse=3.175019e-06 holdout_mse=3.623329e-06 best=3.623329e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:49,984 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 325/20000 — train_mse=2.917324e-06 holdout_mse=3.542458e-06 best=3.542458e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:53,850 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 350/20000 — train_mse=2.974231e-06 holdout_mse=3.468477e-06 best=3.468477e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:04:57,725 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 375/20000 — train_mse=2.649618e-06 holdout_mse=3.401565e-06 best=3.401565e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:01,599 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 400/20000 — train_mse=3.028493e-06 holdout_mse=3.338725e-06 best=3.338725e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:05,479 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 425/20000 — train_mse=2.732619e-06 holdout_mse=3.283738e-06 best=3.283738e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:09,347 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 450/20000 — train_mse=2.645640e-06 holdout_mse=3.232495e-06 best=3.232495e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:13,221 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 475/20000 — train_mse=2.730276e-06 holdout_mse=3.182539e-06 best=3.182539e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:17,096 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 500/20000 — train_mse=2.797955e-06 holdout_mse=3.136160e-06 best=3.136160e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:20,968 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 525/20000 — train_mse=2.690399e-06 holdout_mse=3.096637e-06 best=3.096637e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:24,856 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 550/20000 — train_mse=2.618146e-06 holdout_mse=3.054937e-06 best=3.054937e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:28,743 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 575/20000 — train_mse=2.426473e-06 holdout_mse=3.017272e-06 best=3.017272e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:32,626 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 600/20000 — train_mse=2.416957e-06 holdout_mse=2.983568e-06 best=2.983568e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:36,497 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 625/20000 — train_mse=2.260616e-06 holdout_mse=2.947873e-06 best=2.947873e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:40,592 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 650/20000 — train_mse=2.249572e-06 holdout_mse=2.928249e-06 best=2.928249e-06 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:05:44,469 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 675/20000 — train_mse=2.237479e-06 holdout_mse=2.884640e-06 best=2.884640e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:48,328 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 700/20000 — train_mse=2.214457e-06 holdout_mse=2.858520e-06 best=2.858520e-06 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:05:52,304 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 725/20000 — train_mse=2.445087e-06 holdout_mse=2.826867e-06 best=2.826867e-06 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:05:56,177 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 750/20000 — train_mse=2.612645e-06 holdout_mse=2.801718e-06 best=2.801718e-06 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:06:00,067 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 775/20000 — train_mse=2.377158e-06 holdout_mse=2.777856e-06 best=2.777856e-06 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:06:03,933 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 800/20000 — train_mse=2.247737e-06 holdout_mse=2.752019e-06 best=2.752019e-06 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:06:07,814 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 825/20000 — train_mse=2.116189e-06 holdout_mse=2.732746e-06 best=2.732746e-06 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:06:11,689 INFO moe_compress.stage2_reap_ream ::     heal layer 4: step 850/20000 — train_mse=1.975754e-06 holdout_mse=2.709279e-06 best=2.709279e-06 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:06:11,867 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 4: 850 steps (patience) — train_mse=1.975754e-06 (at_best=1.975754e-06) holdout_mse=2.709279e-06 plain_merged_holdout_mse=1.113794e-05 heal_gap=7.335248e-07 accepted=True (229 kept experts, train_router=True)
2026-05-20 03:06:12,005 WARNING moe_compress.stage2_reap_ream ::   layer 4: _remap_covariance_for_layer dropped 54 covariance entries (= 27 unique experts × ~2 matrices/expert); dropping 27 experts from covariance; keeping 229 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:06:16,877 INFO moe_compress.stage2_reap_ream ::   kept 229 / 256 experts (protected=0, ream_centroids=229) — Σ cost=14.6474, max_group=9, mean_group=0.12
2026-05-20 03:06:17,182 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 6/40 (idx=5) — profiling then merging to 128 experts
2026-05-20 03:06:17,187 INFO moe_compress.stage2_reap_ream :: DIAG layer 5: entering batch loop (calibration tensor + early-exit forwards) | vram=71.5/150GB host_free=159GB gpu_util=0% temp=31C
2026-05-20 03:06:17,809 INFO moe_compress.stage2_reap_ream :: DIAG layer 5 batch 1: total=0.62s fwd=0.57s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.6s | vram=92.3/150GB host_free=159GB gpu_util=49% temp=36C
2026-05-20 03:06:18,369 INFO moe_compress.stage2_reap_ream :: DIAG layer 5 batch 2: total=0.56s fwd=0.53s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.2s | vram=92.8/150GB host_free=159GB gpu_util=51% temp=37C
2026-05-20 03:06:18,923 INFO moe_compress.stage2_reap_ream :: DIAG layer 5 batch 3: total=0.55s fwd=0.53s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.7s | vram=93.4/150GB host_free=159GB gpu_util=52% temp=37C
2026-05-20 03:06:22,750 INFO moe_compress.stage2_reap_ream :: DIAG layer 5 batch 10: total=0.54s fwd=0.51s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.6s | vram=93.4/150GB host_free=159GB gpu_util=54% temp=39C
2026-05-20 03:06:28,119 INFO moe_compress.stage2_reap_ream :: DIAG layer 5 batch 20: total=0.54s fwd=0.51s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=10.9s | vram=93.4/150GB host_free=159GB gpu_util=54% temp=40C
2026-05-20 03:06:33,800 INFO moe_compress.stage2_reap_ream :: DIAG layer 5 batch 30: total=0.57s fwd=0.51s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=16.6s | vram=93.4/150GB host_free=158GB gpu_util=61% temp=41C
2026-05-20 03:06:34,656 INFO moe_compress.stage2_reap_ream :: DIAG layer 5: batch loop complete — 32 batches in 17.5s, now post-profile work | vram=93.4/150GB host_free=158GB gpu_util=36% temp=38C
2026-05-20 03:06:42,912 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 5 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:06:49,839 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 25/20000 — train_mse=2.480009e-05 holdout_mse=2.457240e-05 best=2.457240e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:06:52,163 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 50/20000 — train_mse=2.211227e-05 holdout_mse=2.291409e-05 best=2.291409e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:06:54,484 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 75/20000 — train_mse=2.137659e-05 holdout_mse=2.161645e-05 best=2.161645e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:06:56,808 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 100/20000 — train_mse=1.958135e-05 holdout_mse=2.051570e-05 best=2.051570e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:06:59,145 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 125/20000 — train_mse=1.939141e-05 holdout_mse=1.955827e-05 best=1.955827e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:01,476 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 150/20000 — train_mse=1.796916e-05 holdout_mse=1.877090e-05 best=1.877090e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:03,827 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 175/20000 — train_mse=1.707722e-05 holdout_mse=1.808595e-05 best=1.808595e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:06,180 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 200/20000 — train_mse=1.670271e-05 holdout_mse=1.751879e-05 best=1.751879e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:08,532 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 225/20000 — train_mse=1.675563e-05 holdout_mse=1.706252e-05 best=1.706252e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:10,884 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 250/20000 — train_mse=1.609156e-05 holdout_mse=1.666468e-05 best=1.666468e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:13,235 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 275/20000 — train_mse=1.575166e-05 holdout_mse=1.631270e-05 best=1.631270e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:15,587 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 300/20000 — train_mse=1.500531e-05 holdout_mse=1.599071e-05 best=1.599071e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:17,934 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 325/20000 — train_mse=1.563821e-05 holdout_mse=1.569588e-05 best=1.569588e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:20,299 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 350/20000 — train_mse=1.426879e-05 holdout_mse=1.542047e-05 best=1.542047e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:22,662 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 375/20000 — train_mse=1.489989e-05 holdout_mse=1.516918e-05 best=1.516918e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:25,031 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 400/20000 — train_mse=1.433629e-05 holdout_mse=1.494148e-05 best=1.494148e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:27,397 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 425/20000 — train_mse=1.383595e-05 holdout_mse=1.472476e-05 best=1.472476e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:29,748 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 450/20000 — train_mse=1.398436e-05 holdout_mse=1.451892e-05 best=1.451892e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:32,103 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 475/20000 — train_mse=1.366184e-05 holdout_mse=1.432930e-05 best=1.432930e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:34,467 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 500/20000 — train_mse=1.313318e-05 holdout_mse=1.415397e-05 best=1.415397e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:36,901 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 525/20000 — train_mse=1.308882e-05 holdout_mse=1.399076e-05 best=1.399076e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:39,349 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 550/20000 — train_mse=1.297512e-05 holdout_mse=1.383093e-05 best=1.383093e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:41,793 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 575/20000 — train_mse=1.329023e-05 holdout_mse=1.367672e-05 best=1.367672e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:44,230 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 600/20000 — train_mse=1.244191e-05 holdout_mse=1.352620e-05 best=1.352620e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:46,682 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 625/20000 — train_mse=1.230981e-05 holdout_mse=1.338881e-05 best=1.338881e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:49,124 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 650/20000 — train_mse=1.244056e-05 holdout_mse=1.325489e-05 best=1.325489e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:07:51,566 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 675/20000 — train_mse=1.240328e-05 holdout_mse=1.313074e-05 best=1.313074e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:07:54,074 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 700/20000 — train_mse=1.201728e-05 holdout_mse=1.300860e-05 best=1.300860e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:07:56,748 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 725/20000 — train_mse=1.173574e-05 holdout_mse=1.289507e-05 best=1.289507e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:07:59,189 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 750/20000 — train_mse=1.231878e-05 holdout_mse=1.278623e-05 best=1.278623e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:08:01,623 INFO moe_compress.stage2_reap_ream ::     heal layer 5: step 775/20000 — train_mse=1.149456e-05 holdout_mse=1.267872e-05 best=1.267872e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:08:01,726 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 5: 775 steps (patience) — train_mse=1.149456e-05 (at_best=1.149456e-05) holdout_mse=1.267872e-05 plain_merged_holdout_mse=2.729426e-05 heal_gap=1.184166e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:08:01,860 WARNING moe_compress.stage2_reap_ream ::   layer 5: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:08:04,374 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=65.4561, max_group=9, mean_group=1.00
2026-05-20 03:08:04,715 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 7/40 (idx=6) — profiling then merging to 128 experts
2026-05-20 03:08:04,720 INFO moe_compress.stage2_reap_ream :: DIAG layer 6: entering batch loop (calibration tensor + early-exit forwards) | vram=70.7/150GB host_free=158GB gpu_util=0% temp=32C
2026-05-20 03:08:05,378 INFO moe_compress.stage2_reap_ream :: DIAG layer 6 batch 1: total=0.66s fwd=0.61s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.7s | vram=91.6/150GB host_free=158GB gpu_util=48% temp=37C
2026-05-20 03:08:05,974 INFO moe_compress.stage2_reap_ream :: DIAG layer 6 batch 2: total=0.60s fwd=0.57s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.3s | vram=93.0/150GB host_free=158GB gpu_util=51% temp=38C
2026-05-20 03:08:06,567 INFO moe_compress.stage2_reap_ream :: DIAG layer 6 batch 3: total=0.59s fwd=0.57s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.8s | vram=93.0/150GB host_free=158GB gpu_util=52% temp=38C
2026-05-20 03:08:10,642 INFO moe_compress.stage2_reap_ream :: DIAG layer 6 batch 10: total=0.58s fwd=0.55s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.9s | vram=93.0/150GB host_free=158GB gpu_util=59% temp=40C
2026-05-20 03:08:16,372 INFO moe_compress.stage2_reap_ream :: DIAG layer 6 batch 20: total=0.57s fwd=0.55s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=11.7s | vram=93.0/150GB host_free=158GB gpu_util=54% temp=41C
2026-05-20 03:08:22,451 INFO moe_compress.stage2_reap_ream :: DIAG layer 6 batch 30: total=0.61s fwd=0.55s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=17.7s | vram=93.0/150GB host_free=158GB gpu_util=39% temp=41C
2026-05-20 03:08:23,352 INFO moe_compress.stage2_reap_ream :: DIAG layer 6: batch loop complete — 32 batches in 18.6s, now post-profile work | vram=93.0/150GB host_free=157GB gpu_util=33% temp=39C
2026-05-20 03:08:31,971 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 6 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:08:38,972 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 25/20000 — train_mse=1.809498e-05 holdout_mse=1.832489e-05 best=1.832489e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:41,387 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 50/20000 — train_mse=1.882285e-05 holdout_mse=1.737822e-05 best=1.737822e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:43,803 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 75/20000 — train_mse=1.668592e-05 holdout_mse=1.667500e-05 best=1.667500e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:46,230 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 100/20000 — train_mse=1.557450e-05 holdout_mse=1.610866e-05 best=1.610866e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:48,656 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 125/20000 — train_mse=1.564447e-05 holdout_mse=1.563752e-05 best=1.563752e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:51,080 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 150/20000 — train_mse=1.508964e-05 holdout_mse=1.523929e-05 best=1.523929e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:53,510 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 175/20000 — train_mse=1.465108e-05 holdout_mse=1.488629e-05 best=1.488629e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:55,938 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 200/20000 — train_mse=1.511506e-05 holdout_mse=1.457738e-05 best=1.457738e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:08:58,364 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 225/20000 — train_mse=1.413160e-05 holdout_mse=1.429836e-05 best=1.429836e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:00,778 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 250/20000 — train_mse=1.437835e-05 holdout_mse=1.402646e-05 best=1.402646e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:03,196 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 275/20000 — train_mse=1.355540e-05 holdout_mse=1.377391e-05 best=1.377391e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:05,618 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 300/20000 — train_mse=1.314108e-05 holdout_mse=1.356049e-05 best=1.356049e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:08,093 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 325/20000 — train_mse=1.340712e-05 holdout_mse=1.333814e-05 best=1.333814e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:10,520 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 350/20000 — train_mse=1.270476e-05 holdout_mse=1.314380e-05 best=1.314380e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:12,953 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 375/20000 — train_mse=1.241416e-05 holdout_mse=1.295539e-05 best=1.295539e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:15,388 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 400/20000 — train_mse=1.248356e-05 holdout_mse=1.279001e-05 best=1.279001e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:17,823 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 425/20000 — train_mse=1.231290e-05 holdout_mse=1.263230e-05 best=1.263230e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:20,260 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 450/20000 — train_mse=1.243013e-05 holdout_mse=1.248949e-05 best=1.248949e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:22,701 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 475/20000 — train_mse=1.196073e-05 holdout_mse=1.236149e-05 best=1.236149e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:09:25,147 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 500/20000 — train_mse=1.283964e-05 holdout_mse=1.224139e-05 best=1.224139e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:09:27,567 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 525/20000 — train_mse=1.189983e-05 holdout_mse=1.212923e-05 best=1.212923e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:09:29,987 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 550/20000 — train_mse=1.157185e-05 holdout_mse=1.201811e-05 best=1.201811e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:09:32,405 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 575/20000 — train_mse=1.149764e-05 holdout_mse=1.191789e-05 best=1.191789e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:09:34,823 INFO moe_compress.stage2_reap_ream ::     heal layer 6: step 600/20000 — train_mse=1.185624e-05 holdout_mse=1.182319e-05 best=1.182319e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:09:34,918 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 6: 600 steps (patience) — train_mse=1.185624e-05 (at_best=1.185624e-05) holdout_mse=1.182319e-05 plain_merged_holdout_mse=2.004482e-05 heal_gap=-3.305190e-08 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:09:35,052 WARNING moe_compress.stage2_reap_ream ::   layer 6: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:09:37,536 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=9, ream_centroids=119) — Σ cost=66.6281, max_group=9, mean_group=1.08
2026-05-20 03:09:37,877 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 8/40 (idx=7) — profiling then merging to 128 experts
2026-05-20 03:09:37,881 INFO moe_compress.stage2_reap_ream :: DIAG layer 7: entering batch loop (calibration tensor + early-exit forwards) | vram=69.9/150GB host_free=157GB gpu_util=20% temp=32C
2026-05-20 03:09:38,576 INFO moe_compress.stage2_reap_ream :: DIAG layer 7 batch 1: total=0.69s fwd=0.65s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.7s | vram=90.9/150GB host_free=157GB gpu_util=46% temp=37C
2026-05-20 03:09:39,209 INFO moe_compress.stage2_reap_ream :: DIAG layer 7 batch 2: total=0.63s fwd=0.61s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.3s | vram=91.5/150GB host_free=157GB gpu_util=54% temp=38C
2026-05-20 03:09:39,840 INFO moe_compress.stage2_reap_ream :: DIAG layer 7 batch 3: total=0.63s fwd=0.60s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.0s | vram=91.5/150GB host_free=157GB gpu_util=55% temp=39C
2026-05-20 03:09:44,182 INFO moe_compress.stage2_reap_ream :: DIAG layer 7 batch 10: total=0.62s fwd=0.59s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=6.3s | vram=91.5/150GB host_free=157GB gpu_util=56% temp=40C
2026-05-20 03:09:50,328 INFO moe_compress.stage2_reap_ream :: DIAG layer 7 batch 20: total=0.65s fwd=0.58s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=12.4s | vram=91.5/150GB host_free=157GB gpu_util=56% temp=41C
2026-05-20 03:09:56,777 INFO moe_compress.stage2_reap_ream :: DIAG layer 7 batch 30: total=0.64s fwd=0.58s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=18.9s | vram=91.5/150GB host_free=156GB gpu_util=57% temp=39C
2026-05-20 03:09:57,724 INFO moe_compress.stage2_reap_ream :: DIAG layer 7: batch loop complete — 32 batches in 19.8s, now post-profile work | vram=91.5/150GB host_free=156GB gpu_util=34% temp=39C
2026-05-20 03:10:06,926 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 7 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:10:14,026 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 25/20000 — train_mse=2.485239e-05 holdout_mse=2.466240e-05 best=2.466240e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:16,399 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 50/20000 — train_mse=2.223384e-05 holdout_mse=2.211516e-05 best=2.211516e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:18,822 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 75/20000 — train_mse=2.034053e-05 holdout_mse=2.054919e-05 best=2.054919e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:21,284 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 100/20000 — train_mse=1.920648e-05 holdout_mse=1.945713e-05 best=1.945713e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:23,715 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 125/20000 — train_mse=1.870105e-05 holdout_mse=1.862777e-05 best=1.862777e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:26,149 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 150/20000 — train_mse=1.750882e-05 holdout_mse=1.794361e-05 best=1.794361e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:28,592 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 175/20000 — train_mse=1.677968e-05 holdout_mse=1.736440e-05 best=1.736440e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:31,019 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 200/20000 — train_mse=1.647021e-05 holdout_mse=1.686594e-05 best=1.686594e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:33,451 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 225/20000 — train_mse=1.609919e-05 holdout_mse=1.643362e-05 best=1.643362e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:35,895 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 250/20000 — train_mse=1.500212e-05 holdout_mse=1.605725e-05 best=1.605725e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:38,336 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 275/20000 — train_mse=1.540102e-05 holdout_mse=1.571845e-05 best=1.571845e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:40,769 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 300/20000 — train_mse=1.485422e-05 holdout_mse=1.541653e-05 best=1.541653e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:43,202 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 325/20000 — train_mse=1.408286e-05 holdout_mse=1.515095e-05 best=1.515095e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:45,639 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 350/20000 — train_mse=1.444092e-05 holdout_mse=1.490428e-05 best=1.490428e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:48,078 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 375/20000 — train_mse=1.377063e-05 holdout_mse=1.468606e-05 best=1.468606e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:50,516 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 400/20000 — train_mse=1.356548e-05 holdout_mse=1.447124e-05 best=1.447124e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:52,954 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 425/20000 — train_mse=1.358183e-05 holdout_mse=1.427389e-05 best=1.427389e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:55,392 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 450/20000 — train_mse=1.318264e-05 holdout_mse=1.410428e-05 best=1.410428e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:10:57,832 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 475/20000 — train_mse=1.310851e-05 holdout_mse=1.393177e-05 best=1.393177e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:11:00,271 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 500/20000 — train_mse=1.310890e-05 holdout_mse=1.377007e-05 best=1.377007e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:11:02,717 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 525/20000 — train_mse=1.306505e-05 holdout_mse=1.361948e-05 best=1.361948e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:11:05,150 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 550/20000 — train_mse=1.246925e-05 holdout_mse=1.347532e-05 best=1.347532e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:11:07,587 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 575/20000 — train_mse=1.216996e-05 holdout_mse=1.333558e-05 best=1.333558e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:11:10,025 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 600/20000 — train_mse=1.271813e-05 holdout_mse=1.320469e-05 best=1.320469e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:11:12,463 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 625/20000 — train_mse=1.183722e-05 holdout_mse=1.308529e-05 best=1.308529e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:11:14,897 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 650/20000 — train_mse=1.178099e-05 holdout_mse=1.297206e-05 best=1.297206e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:11:17,324 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 675/20000 — train_mse=1.211408e-05 holdout_mse=1.286262e-05 best=1.286262e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:11:19,753 INFO moe_compress.stage2_reap_ream ::     heal layer 7: step 700/20000 — train_mse=1.198212e-05 holdout_mse=1.275611e-05 best=1.275611e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:11:19,853 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 7: 700 steps (patience) — train_mse=1.198212e-05 (at_best=1.198212e-05) holdout_mse=1.275611e-05 plain_merged_holdout_mse=2.914828e-05 heal_gap=7.739933e-07 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:11:19,988 WARNING moe_compress.stage2_reap_ream ::   layer 7: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:11:22,852 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=1, ream_centroids=127) — Σ cost=59.5470, max_group=9, mean_group=1.01
2026-05-20 03:11:23,216 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 9/40 (idx=8) — profiling then merging to 128 experts
2026-05-20 03:11:23,221 INFO moe_compress.stage2_reap_ream :: DIAG layer 8: entering batch loop (calibration tensor + early-exit forwards) | vram=69.1/150GB host_free=157GB gpu_util=0% temp=32C
2026-05-20 03:11:23,952 INFO moe_compress.stage2_reap_ream :: DIAG layer 8 batch 1: total=0.73s fwd=0.68s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.7s | vram=90.7/150GB host_free=157GB gpu_util=44% temp=37C
2026-05-20 03:11:24,623 INFO moe_compress.stage2_reap_ream :: DIAG layer 8 batch 2: total=0.67s fwd=0.64s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.4s | vram=90.7/150GB host_free=157GB gpu_util=54% temp=38C
2026-05-20 03:11:25,290 INFO moe_compress.stage2_reap_ream :: DIAG layer 8 batch 3: total=0.67s fwd=0.64s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.1s | vram=90.7/150GB host_free=157GB gpu_util=52% temp=39C
2026-05-20 03:11:29,896 INFO moe_compress.stage2_reap_ream :: DIAG layer 8 batch 10: total=0.65s fwd=0.63s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=6.7s | vram=90.7/150GB host_free=157GB gpu_util=54% temp=41C
2026-05-20 03:11:36,420 INFO moe_compress.stage2_reap_ream :: DIAG layer 8 batch 20: total=0.68s fwd=0.62s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=13.2s | vram=90.7/150GB host_free=156GB gpu_util=56% temp=42C
2026-05-20 03:11:43,249 INFO moe_compress.stage2_reap_ream :: DIAG layer 8 batch 30: total=0.68s fwd=0.62s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=20.0s | vram=90.7/150GB host_free=156GB gpu_util=57% temp=43C
2026-05-20 03:11:44,238 INFO moe_compress.stage2_reap_ream :: DIAG layer 8: batch loop complete — 32 batches in 21.0s, now post-profile work | vram=90.7/150GB host_free=156GB gpu_util=90% temp=40C
2026-05-20 03:11:53,289 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 8 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:12:01,028 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 25/20000 — train_mse=3.213120e-05 holdout_mse=3.279557e-05 best=3.279557e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:03,455 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 50/20000 — train_mse=2.779093e-05 holdout_mse=2.831376e-05 best=2.831376e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:05,896 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 75/20000 — train_mse=2.629373e-05 holdout_mse=2.576980e-05 best=2.576980e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:08,336 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 100/20000 — train_mse=2.255099e-05 holdout_mse=2.415171e-05 best=2.415171e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:10,773 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 125/20000 — train_mse=2.250977e-05 holdout_mse=2.299716e-05 best=2.299716e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:13,205 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 150/20000 — train_mse=2.133359e-05 holdout_mse=2.210226e-05 best=2.210226e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:15,638 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 175/20000 — train_mse=2.049030e-05 holdout_mse=2.136062e-05 best=2.136062e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:18,074 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 200/20000 — train_mse=2.084325e-05 holdout_mse=2.073388e-05 best=2.073388e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:20,505 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 225/20000 — train_mse=1.949532e-05 holdout_mse=2.019562e-05 best=2.019562e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:22,927 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 250/20000 — train_mse=1.866412e-05 holdout_mse=1.972615e-05 best=1.972615e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:25,362 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 275/20000 — train_mse=1.815938e-05 holdout_mse=1.930098e-05 best=1.930098e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:27,795 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 300/20000 — train_mse=1.762734e-05 holdout_mse=1.892662e-05 best=1.892662e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:30,260 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 325/20000 — train_mse=1.730093e-05 holdout_mse=1.857788e-05 best=1.857788e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:32,693 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 350/20000 — train_mse=1.680571e-05 holdout_mse=1.826672e-05 best=1.826672e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:35,129 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 375/20000 — train_mse=1.760783e-05 holdout_mse=1.798127e-05 best=1.798127e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:37,565 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 400/20000 — train_mse=1.724375e-05 holdout_mse=1.770588e-05 best=1.770588e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:40,015 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 425/20000 — train_mse=1.649791e-05 holdout_mse=1.745881e-05 best=1.745881e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:42,464 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 450/20000 — train_mse=1.584789e-05 holdout_mse=1.722819e-05 best=1.722819e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:44,911 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 475/20000 — train_mse=1.603988e-05 holdout_mse=1.701558e-05 best=1.701558e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:47,346 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 500/20000 — train_mse=1.548402e-05 holdout_mse=1.681079e-05 best=1.681079e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:49,825 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 525/20000 — train_mse=1.546778e-05 holdout_mse=1.661009e-05 best=1.661009e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:52,289 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 550/20000 — train_mse=1.495508e-05 holdout_mse=1.642721e-05 best=1.642721e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:54,778 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 575/20000 — train_mse=1.548754e-05 holdout_mse=1.625263e-05 best=1.625263e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:57,250 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 600/20000 — train_mse=1.466998e-05 holdout_mse=1.608143e-05 best=1.608143e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:12:59,724 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 625/20000 — train_mse=1.466756e-05 holdout_mse=1.592784e-05 best=1.592784e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:13:02,197 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 650/20000 — train_mse=1.469385e-05 holdout_mse=1.577294e-05 best=1.577294e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:13:04,658 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 675/20000 — train_mse=1.433374e-05 holdout_mse=1.563082e-05 best=1.563082e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:13:07,189 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 700/20000 — train_mse=1.393480e-05 holdout_mse=1.549429e-05 best=1.549429e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:13:09,887 INFO moe_compress.stage2_reap_ream ::     heal layer 8: step 725/20000 — train_mse=1.371541e-05 holdout_mse=1.536778e-05 best=1.536778e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:13:09,985 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 8: 725 steps (patience) — train_mse=1.371541e-05 (at_best=1.371541e-05) holdout_mse=1.536778e-05 plain_merged_holdout_mse=4.165452e-05 heal_gap=1.652369e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:13:10,119 WARNING moe_compress.stage2_reap_ream ::   layer 8: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:13:13,081 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=58.4949, max_group=9, mean_group=1.00
2026-05-20 03:13:13,436 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 10/40 (idx=9) — profiling then merging to 128 experts
2026-05-20 03:13:13,442 INFO moe_compress.stage2_reap_ream :: DIAG layer 9: entering batch loop (calibration tensor + early-exit forwards) | vram=68.4/150GB host_free=156GB gpu_util=0% temp=32C
2026-05-20 03:13:14,207 INFO moe_compress.stage2_reap_ream :: DIAG layer 9 batch 1: total=0.76s fwd=0.72s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=255) | cum=0.8s | vram=89.9/150GB host_free=156GB gpu_util=52% temp=38C
2026-05-20 03:13:14,915 INFO moe_compress.stage2_reap_ream :: DIAG layer 9 batch 2: total=0.71s fwd=0.68s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.5s | vram=90.4/150GB host_free=156GB gpu_util=71% temp=39C
2026-05-20 03:13:15,619 INFO moe_compress.stage2_reap_ream :: DIAG layer 9 batch 3: total=0.70s fwd=0.68s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.2s | vram=90.4/150GB host_free=156GB gpu_util=56% temp=39C
2026-05-20 03:13:20,475 INFO moe_compress.stage2_reap_ream :: DIAG layer 9 batch 10: total=0.68s fwd=0.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=7.0s | vram=90.4/150GB host_free=156GB gpu_util=59% temp=42C
2026-05-20 03:13:27,398 INFO moe_compress.stage2_reap_ream :: DIAG layer 9 batch 20: total=0.72s fwd=0.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.0s | vram=90.4/150GB host_free=155GB gpu_util=56% temp=43C
2026-05-20 03:13:34,595 INFO moe_compress.stage2_reap_ream :: DIAG layer 9 batch 30: total=0.72s fwd=0.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=21.2s | vram=90.4/150GB host_free=155GB gpu_util=56% temp=43C
2026-05-20 03:13:35,624 INFO moe_compress.stage2_reap_ream :: DIAG layer 9: batch loop complete — 32 batches in 22.2s, now post-profile work | vram=90.4/150GB host_free=155GB gpu_util=37% temp=42C
2026-05-20 03:13:44,719 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 9 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:13:52,582 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 25/20000 — train_mse=3.459546e-05 holdout_mse=3.500885e-05 best=3.500885e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:13:55,041 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 50/20000 — train_mse=3.136743e-05 holdout_mse=3.056268e-05 best=3.056268e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:13:57,518 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 75/20000 — train_mse=2.753111e-05 holdout_mse=2.789417e-05 best=2.789417e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:00,000 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 100/20000 — train_mse=2.618533e-05 holdout_mse=2.605533e-05 best=2.605533e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:02,480 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 125/20000 — train_mse=2.433833e-05 holdout_mse=2.466280e-05 best=2.466280e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:04,962 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 150/20000 — train_mse=2.288272e-05 holdout_mse=2.355823e-05 best=2.355823e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:07,452 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 175/20000 — train_mse=2.166548e-05 holdout_mse=2.264844e-05 best=2.264844e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:09,946 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 200/20000 — train_mse=2.150274e-05 holdout_mse=2.188634e-05 best=2.188634e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:12,445 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 225/20000 — train_mse=2.060625e-05 holdout_mse=2.122614e-05 best=2.122614e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:14,928 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 250/20000 — train_mse=2.024834e-05 holdout_mse=2.065284e-05 best=2.065284e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:17,417 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 275/20000 — train_mse=1.880663e-05 holdout_mse=2.013373e-05 best=2.013373e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:19,916 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 300/20000 — train_mse=1.940516e-05 holdout_mse=1.967308e-05 best=1.967308e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:22,410 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 325/20000 — train_mse=1.794225e-05 holdout_mse=1.925615e-05 best=1.925615e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:24,903 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 350/20000 — train_mse=1.820821e-05 holdout_mse=1.887851e-05 best=1.887851e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:27,409 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 375/20000 — train_mse=1.738880e-05 holdout_mse=1.853725e-05 best=1.853725e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:29,910 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 400/20000 — train_mse=1.781865e-05 holdout_mse=1.821369e-05 best=1.821369e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:32,402 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 425/20000 — train_mse=1.684587e-05 holdout_mse=1.792324e-05 best=1.792324e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:34,903 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 450/20000 — train_mse=1.664810e-05 holdout_mse=1.764927e-05 best=1.764927e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:37,392 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 475/20000 — train_mse=1.647645e-05 holdout_mse=1.739810e-05 best=1.739810e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:39,966 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 500/20000 — train_mse=1.623652e-05 holdout_mse=1.716163e-05 best=1.716163e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:42,457 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 525/20000 — train_mse=1.579564e-05 holdout_mse=1.694635e-05 best=1.694635e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:44,947 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 550/20000 — train_mse=1.542475e-05 holdout_mse=1.674224e-05 best=1.674224e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:47,434 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 575/20000 — train_mse=1.582351e-05 holdout_mse=1.654399e-05 best=1.654399e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:50,045 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 600/20000 — train_mse=1.531266e-05 holdout_mse=1.636050e-05 best=1.636050e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:52,536 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 625/20000 — train_mse=1.494933e-05 holdout_mse=1.619021e-05 best=1.619021e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:55,027 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 650/20000 — train_mse=1.472436e-05 holdout_mse=1.602277e-05 best=1.602277e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:14:57,522 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 675/20000 — train_mse=1.478224e-05 holdout_mse=1.586477e-05 best=1.586477e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:15:00,013 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 700/20000 — train_mse=1.444739e-05 holdout_mse=1.570892e-05 best=1.570892e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:15:02,738 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 725/20000 — train_mse=1.415159e-05 holdout_mse=1.556198e-05 best=1.556198e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:15:05,223 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 750/20000 — train_mse=1.420758e-05 holdout_mse=1.542099e-05 best=1.542099e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:15:07,708 INFO moe_compress.stage2_reap_ream ::     heal layer 9: step 775/20000 — train_mse=1.399908e-05 holdout_mse=1.529269e-05 best=1.529269e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:15:07,809 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 9: 775 steps (patience) — train_mse=1.399908e-05 (at_best=1.399908e-05) holdout_mse=1.529269e-05 plain_merged_holdout_mse=4.467380e-05 heal_gap=1.293612e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:15:07,944 WARNING moe_compress.stage2_reap_ream ::   layer 9: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:15:10,557 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=58.3308, max_group=9, mean_group=1.00
2026-05-20 03:15:10,907 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 11/40 (idx=10) — profiling then merging to 128 experts
2026-05-20 03:15:10,912 INFO moe_compress.stage2_reap_ream :: DIAG layer 10: entering batch loop (calibration tensor + early-exit forwards) | vram=67.6/150GB host_free=155GB gpu_util=0% temp=32C
2026-05-20 03:15:11,717 INFO moe_compress.stage2_reap_ream :: DIAG layer 10 batch 1: total=0.80s fwd=0.76s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.8s | vram=89.6/150GB host_free=155GB gpu_util=52% temp=37C
2026-05-20 03:15:12,463 INFO moe_compress.stage2_reap_ream :: DIAG layer 10 batch 2: total=0.74s fwd=0.72s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.6s | vram=89.6/150GB host_free=155GB gpu_util=53% temp=38C
2026-05-20 03:15:13,206 INFO moe_compress.stage2_reap_ream :: DIAG layer 10 batch 3: total=0.74s fwd=0.72s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.3s | vram=89.6/150GB host_free=155GB gpu_util=51% temp=39C
2026-05-20 03:15:18,335 INFO moe_compress.stage2_reap_ream :: DIAG layer 10 batch 10: total=0.73s fwd=0.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=7.4s | vram=89.6/150GB host_free=155GB gpu_util=58% temp=41C
2026-05-20 03:15:25,648 INFO moe_compress.stage2_reap_ream :: DIAG layer 10 batch 20: total=0.76s fwd=0.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.7s | vram=89.6/150GB host_free=154GB gpu_util=60% temp=43C
2026-05-20 03:15:33,226 INFO moe_compress.stage2_reap_ream :: DIAG layer 10 batch 30: total=0.76s fwd=0.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=22.3s | vram=89.6/150GB host_free=154GB gpu_util=57% temp=43C
2026-05-20 03:15:34,311 INFO moe_compress.stage2_reap_ream :: DIAG layer 10: batch loop complete — 32 batches in 23.4s, now post-profile work | vram=89.6/150GB host_free=154GB gpu_util=74% temp=41C
2026-05-20 03:15:43,811 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 10 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:15:51,186 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 25/20000 — train_mse=3.162956e-05 holdout_mse=3.351504e-05 best=3.351504e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:15:53,660 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 50/20000 — train_mse=2.877651e-05 holdout_mse=2.994333e-05 best=2.994333e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:15:56,143 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 75/20000 — train_mse=2.755709e-05 holdout_mse=2.769010e-05 best=2.769010e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:15:58,631 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 100/20000 — train_mse=2.498558e-05 holdout_mse=2.607460e-05 best=2.607460e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:01,116 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 125/20000 — train_mse=2.452316e-05 holdout_mse=2.484184e-05 best=2.484184e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:03,601 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 150/20000 — train_mse=2.439545e-05 holdout_mse=2.377727e-05 best=2.377727e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:06,084 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 175/20000 — train_mse=2.230853e-05 holdout_mse=2.289544e-05 best=2.289544e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:08,571 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 200/20000 — train_mse=2.170132e-05 holdout_mse=2.217765e-05 best=2.217765e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:11,047 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 225/20000 — train_mse=2.154584e-05 holdout_mse=2.156782e-05 best=2.156782e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:13,537 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 250/20000 — train_mse=2.076640e-05 holdout_mse=2.104088e-05 best=2.104088e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:16,038 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 275/20000 — train_mse=1.981227e-05 holdout_mse=2.056711e-05 best=2.056711e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:18,525 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 300/20000 — train_mse=1.974134e-05 holdout_mse=2.014203e-05 best=2.014203e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:21,019 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 325/20000 — train_mse=1.896500e-05 holdout_mse=1.975739e-05 best=1.975739e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:23,509 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 350/20000 — train_mse=1.896633e-05 holdout_mse=1.941183e-05 best=1.941183e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:26,000 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 375/20000 — train_mse=1.894782e-05 holdout_mse=1.908721e-05 best=1.908721e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:28,472 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 400/20000 — train_mse=1.730739e-05 holdout_mse=1.878557e-05 best=1.878557e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:30,940 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 425/20000 — train_mse=1.786815e-05 holdout_mse=1.851214e-05 best=1.851214e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:33,403 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 450/20000 — train_mse=1.737816e-05 holdout_mse=1.825425e-05 best=1.825425e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:35,874 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 475/20000 — train_mse=1.712104e-05 holdout_mse=1.800971e-05 best=1.800971e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:38,319 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 500/20000 — train_mse=1.670453e-05 holdout_mse=1.778085e-05 best=1.778085e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:40,770 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 525/20000 — train_mse=1.658794e-05 holdout_mse=1.756937e-05 best=1.756937e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:43,214 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 550/20000 — train_mse=1.598147e-05 holdout_mse=1.736882e-05 best=1.736882e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:45,673 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 575/20000 — train_mse=1.622696e-05 holdout_mse=1.718539e-05 best=1.718539e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:48,138 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 600/20000 — train_mse=1.563864e-05 holdout_mse=1.700408e-05 best=1.700408e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:16:50,622 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 625/20000 — train_mse=1.592033e-05 holdout_mse=1.683515e-05 best=1.683515e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:16:53,085 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 650/20000 — train_mse=1.594569e-05 holdout_mse=1.667604e-05 best=1.667604e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:16:55,557 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 675/20000 — train_mse=1.507447e-05 holdout_mse=1.653416e-05 best=1.653416e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:16:58,015 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 700/20000 — train_mse=1.524486e-05 holdout_mse=1.638842e-05 best=1.638842e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:17:00,721 INFO moe_compress.stage2_reap_ream ::     heal layer 10: step 725/20000 — train_mse=1.496127e-05 holdout_mse=1.625039e-05 best=1.625039e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:17:00,821 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 10: 725 steps (patience) — train_mse=1.496127e-05 (at_best=1.496127e-05) holdout_mse=1.625039e-05 plain_merged_holdout_mse=4.227866e-05 heal_gap=1.289129e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:17:00,955 WARNING moe_compress.stage2_reap_ream ::   layer 10: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:17:03,434 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=1, ream_centroids=127) — Σ cost=61.4967, max_group=9, mean_group=1.01
2026-05-20 03:17:03,776 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 12/40 (idx=11) — profiling then merging to 128 experts
2026-05-20 03:17:03,781 INFO moe_compress.stage2_reap_ream :: DIAG layer 11: entering batch loop (calibration tensor + early-exit forwards) | vram=66.8/150GB host_free=153GB gpu_util=0% temp=32C
2026-05-20 03:17:04,624 INFO moe_compress.stage2_reap_ream :: DIAG layer 11 batch 1: total=0.84s fwd=0.80s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.8s | vram=89.9/150GB host_free=153GB gpu_util=51% temp=37C
2026-05-20 03:17:05,402 INFO moe_compress.stage2_reap_ream :: DIAG layer 11 batch 2: total=0.78s fwd=0.75s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.6s | vram=89.9/150GB host_free=153GB gpu_util=55% temp=39C
2026-05-20 03:17:06,179 INFO moe_compress.stage2_reap_ream :: DIAG layer 11 batch 3: total=0.78s fwd=0.75s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.4s | vram=89.9/150GB host_free=153GB gpu_util=58% temp=40C
2026-05-20 03:17:11,546 INFO moe_compress.stage2_reap_ream :: DIAG layer 11 batch 10: total=0.76s fwd=0.73s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=7.8s | vram=89.9/150GB host_free=153GB gpu_util=56% temp=42C
2026-05-20 03:17:19,190 INFO moe_compress.stage2_reap_ream :: DIAG layer 11 batch 20: total=0.79s fwd=0.73s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=15.4s | vram=89.9/150GB host_free=153GB gpu_util=59% temp=43C
2026-05-20 03:17:27,125 INFO moe_compress.stage2_reap_ream :: DIAG layer 11 batch 30: total=0.79s fwd=0.73s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=23.3s | vram=89.9/150GB host_free=152GB gpu_util=58% temp=44C
2026-05-20 03:17:28,255 INFO moe_compress.stage2_reap_ream :: DIAG layer 11: batch loop complete — 32 batches in 24.5s, now post-profile work | vram=89.9/150GB host_free=152GB gpu_util=43% temp=41C
2026-05-20 03:17:37,704 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 11 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:17:45,234 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 25/20000 — train_mse=4.080535e-05 holdout_mse=4.086680e-05 best=4.086680e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:17:47,673 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 50/20000 — train_mse=3.779952e-05 holdout_mse=3.689380e-05 best=3.689380e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:17:50,123 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 75/20000 — train_mse=3.400294e-05 holdout_mse=3.421856e-05 best=3.421856e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:17:52,565 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 100/20000 — train_mse=3.220972e-05 holdout_mse=3.231039e-05 best=3.231039e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:17:55,010 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 125/20000 — train_mse=3.085608e-05 holdout_mse=3.084936e-05 best=3.084936e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:17:57,454 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 150/20000 — train_mse=2.887323e-05 holdout_mse=2.969268e-05 best=2.969268e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:17:59,900 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 175/20000 — train_mse=2.753476e-05 holdout_mse=2.872951e-05 best=2.872951e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:02,345 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 200/20000 — train_mse=2.689476e-05 holdout_mse=2.790590e-05 best=2.790590e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:04,808 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 225/20000 — train_mse=2.650257e-05 holdout_mse=2.719176e-05 best=2.719176e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:07,244 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 250/20000 — train_mse=2.505867e-05 holdout_mse=2.655597e-05 best=2.655597e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:09,680 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 275/20000 — train_mse=2.557021e-05 holdout_mse=2.597830e-05 best=2.597830e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:12,127 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 300/20000 — train_mse=2.389607e-05 holdout_mse=2.546451e-05 best=2.546451e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:14,600 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 325/20000 — train_mse=2.376930e-05 holdout_mse=2.499758e-05 best=2.499758e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:17,042 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 350/20000 — train_mse=2.302464e-05 holdout_mse=2.457193e-05 best=2.457193e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:19,480 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 375/20000 — train_mse=2.310194e-05 holdout_mse=2.417503e-05 best=2.417503e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:21,920 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 400/20000 — train_mse=2.220314e-05 holdout_mse=2.380978e-05 best=2.380978e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:24,368 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 425/20000 — train_mse=2.165038e-05 holdout_mse=2.346251e-05 best=2.346251e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:26,802 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 450/20000 — train_mse=2.155538e-05 holdout_mse=2.313821e-05 best=2.313821e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:29,249 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 475/20000 — train_mse=2.081149e-05 holdout_mse=2.283617e-05 best=2.283617e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:31,691 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 500/20000 — train_mse=2.043079e-05 holdout_mse=2.255996e-05 best=2.255996e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:34,134 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 525/20000 — train_mse=2.075921e-05 holdout_mse=2.229235e-05 best=2.229235e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:36,571 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 550/20000 — train_mse=1.986464e-05 holdout_mse=2.204213e-05 best=2.204213e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:39,035 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 575/20000 — train_mse=2.006443e-05 holdout_mse=2.181094e-05 best=2.181094e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:41,482 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 600/20000 — train_mse=1.897039e-05 holdout_mse=2.157736e-05 best=2.157736e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:43,918 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 625/20000 — train_mse=1.899189e-05 holdout_mse=2.135296e-05 best=2.135296e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:18:46,348 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 650/20000 — train_mse=1.925636e-05 holdout_mse=2.115400e-05 best=2.115400e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:18:48,786 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 675/20000 — train_mse=1.835672e-05 holdout_mse=2.096397e-05 best=2.096397e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:18:51,228 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 700/20000 — train_mse=1.875976e-05 holdout_mse=2.077456e-05 best=2.077456e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:18:53,901 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 725/20000 — train_mse=1.831821e-05 holdout_mse=2.058993e-05 best=2.058993e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:18:56,358 INFO moe_compress.stage2_reap_ream ::     heal layer 11: step 750/20000 — train_mse=1.753735e-05 holdout_mse=2.042301e-05 best=2.042301e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:18:56,452 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 11: 750 steps (patience) — train_mse=1.753735e-05 (at_best=1.753735e-05) holdout_mse=2.042301e-05 plain_merged_holdout_mse=4.802507e-05 heal_gap=2.885653e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:18:56,587 WARNING moe_compress.stage2_reap_ream ::   layer 11: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:18:59,379 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=62.5585, max_group=9, mean_group=1.00
2026-05-20 03:18:59,742 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 13/40 (idx=12) — profiling then merging to 128 experts
2026-05-20 03:18:59,747 INFO moe_compress.stage2_reap_ream :: DIAG layer 12: entering batch loop (calibration tensor + early-exit forwards) | vram=66.0/150GB host_free=152GB gpu_util=0% temp=32C
2026-05-20 03:19:00,633 INFO moe_compress.stage2_reap_ream :: DIAG layer 12 batch 1: total=0.89s fwd=0.84s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.9s | vram=89.6/150GB host_free=152GB gpu_util=42% temp=37C
2026-05-20 03:19:01,464 INFO moe_compress.stage2_reap_ream :: DIAG layer 12 batch 2: total=0.83s fwd=0.79s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.7s | vram=89.6/150GB host_free=152GB gpu_util=52% temp=39C
2026-05-20 03:19:02,279 INFO moe_compress.stage2_reap_ream :: DIAG layer 12 batch 3: total=0.81s fwd=0.79s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.5s | vram=89.6/150GB host_free=152GB gpu_util=55% temp=40C
2026-05-20 03:19:07,927 INFO moe_compress.stage2_reap_ream :: DIAG layer 12 batch 10: total=0.80s fwd=0.77s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=8.2s | vram=89.6/150GB host_free=152GB gpu_util=55% temp=43C
2026-05-20 03:19:15,978 INFO moe_compress.stage2_reap_ream :: DIAG layer 12 batch 20: total=0.83s fwd=0.77s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=16.2s | vram=89.6/150GB host_free=152GB gpu_util=56% temp=44C
2026-05-20 03:19:24,290 INFO moe_compress.stage2_reap_ream :: DIAG layer 12 batch 30: total=0.83s fwd=0.77s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=24.5s | vram=89.6/150GB host_free=151GB gpu_util=55% temp=44C
2026-05-20 03:19:25,461 INFO moe_compress.stage2_reap_ream :: DIAG layer 12: batch loop complete — 32 batches in 25.7s, now post-profile work | vram=89.6/150GB host_free=151GB gpu_util=36% temp=42C
2026-05-20 03:19:35,274 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 12 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:19:42,645 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 25/20000 — train_mse=3.763693e-05 holdout_mse=3.599242e-05 best=3.599242e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:45,104 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 50/20000 — train_mse=3.277228e-05 holdout_mse=3.239269e-05 best=3.239269e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:47,558 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 75/20000 — train_mse=3.054731e-05 holdout_mse=3.025222e-05 best=3.025222e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:50,003 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 100/20000 — train_mse=2.851644e-05 holdout_mse=2.870089e-05 best=2.870089e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:52,454 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 125/20000 — train_mse=2.672243e-05 holdout_mse=2.751041e-05 best=2.751041e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:54,895 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 150/20000 — train_mse=2.671697e-05 holdout_mse=2.652982e-05 best=2.652982e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:57,339 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 175/20000 — train_mse=2.532515e-05 holdout_mse=2.569529e-05 best=2.569529e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:19:59,797 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 200/20000 — train_mse=2.346133e-05 holdout_mse=2.497721e-05 best=2.497721e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:02,254 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 225/20000 — train_mse=2.324036e-05 holdout_mse=2.434903e-05 best=2.434903e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:04,701 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 250/20000 — train_mse=2.313730e-05 holdout_mse=2.379030e-05 best=2.379030e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:07,157 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 275/20000 — train_mse=2.250568e-05 holdout_mse=2.328450e-05 best=2.328450e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:09,605 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 300/20000 — train_mse=2.229166e-05 holdout_mse=2.283668e-05 best=2.283668e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:12,055 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 325/20000 — train_mse=2.125897e-05 holdout_mse=2.243398e-05 best=2.243398e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:14,503 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 350/20000 — train_mse=2.143791e-05 holdout_mse=2.204684e-05 best=2.204684e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:16,953 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 375/20000 — train_mse=2.074727e-05 holdout_mse=2.170119e-05 best=2.170119e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:19,405 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 400/20000 — train_mse=2.037426e-05 holdout_mse=2.138560e-05 best=2.138560e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:21,856 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 425/20000 — train_mse=1.947905e-05 holdout_mse=2.108647e-05 best=2.108647e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:24,299 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 450/20000 — train_mse=1.961072e-05 holdout_mse=2.081018e-05 best=2.081018e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:26,757 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 475/20000 — train_mse=1.892861e-05 holdout_mse=2.055751e-05 best=2.055751e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:29,200 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 500/20000 — train_mse=1.910391e-05 holdout_mse=2.030846e-05 best=2.030846e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:31,674 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 525/20000 — train_mse=1.876925e-05 holdout_mse=2.008607e-05 best=2.008607e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:34,123 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 550/20000 — train_mse=1.871755e-05 holdout_mse=1.986939e-05 best=1.986939e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:36,576 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 575/20000 — train_mse=1.831692e-05 holdout_mse=1.966157e-05 best=1.966157e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:20:39,018 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 600/20000 — train_mse=1.846775e-05 holdout_mse=1.946718e-05 best=1.946718e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:20:41,473 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 625/20000 — train_mse=1.761793e-05 holdout_mse=1.928150e-05 best=1.928150e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:20:43,924 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 650/20000 — train_mse=1.745593e-05 holdout_mse=1.910719e-05 best=1.910719e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:20:46,387 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 675/20000 — train_mse=1.671082e-05 holdout_mse=1.893945e-05 best=1.893945e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:20:48,841 INFO moe_compress.stage2_reap_ream ::     heal layer 12: step 700/20000 — train_mse=1.675443e-05 holdout_mse=1.877722e-05 best=1.877722e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:20:48,938 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 12: 700 steps (patience) — train_mse=1.675443e-05 (at_best=1.675443e-05) holdout_mse=1.877722e-05 plain_merged_holdout_mse=4.377971e-05 heal_gap=2.022792e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:20:49,074 WARNING moe_compress.stage2_reap_ream ::   layer 12: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:20:51,689 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=1, ream_centroids=127) — Σ cost=61.1365, max_group=9, mean_group=1.01
2026-05-20 03:20:52,048 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 14/40 (idx=13) — profiling then merging to 128 experts
2026-05-20 03:20:52,052 INFO moe_compress.stage2_reap_ream :: DIAG layer 13: entering batch loop (calibration tensor + early-exit forwards) | vram=65.2/150GB host_free=151GB gpu_util=0% temp=32C
2026-05-20 03:20:52,972 INFO moe_compress.stage2_reap_ream :: DIAG layer 13 batch 1: total=0.92s fwd=0.87s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=0.9s | vram=88.8/150GB host_free=151GB gpu_util=46% temp=36C
2026-05-20 03:20:53,827 INFO moe_compress.stage2_reap_ream :: DIAG layer 13 batch 2: total=0.85s fwd=0.83s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.8s | vram=88.8/150GB host_free=151GB gpu_util=53% temp=39C
2026-05-20 03:20:54,678 INFO moe_compress.stage2_reap_ream :: DIAG layer 13 batch 3: total=0.85s fwd=0.82s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.6s | vram=88.8/150GB host_free=151GB gpu_util=53% temp=39C
2026-05-20 03:21:00,581 INFO moe_compress.stage2_reap_ream :: DIAG layer 13 batch 10: total=0.84s fwd=0.81s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=8.5s | vram=88.8/150GB host_free=151GB gpu_util=57% temp=42C
2026-05-20 03:21:09,010 INFO moe_compress.stage2_reap_ream :: DIAG layer 13 batch 20: total=0.87s fwd=0.81s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=17.0s | vram=88.8/150GB host_free=151GB gpu_util=58% temp=43C
2026-05-20 03:21:17,669 INFO moe_compress.stage2_reap_ream :: DIAG layer 13 batch 30: total=0.86s fwd=0.80s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=25.6s | vram=88.8/150GB host_free=150GB gpu_util=57% temp=44C
2026-05-20 03:21:18,887 INFO moe_compress.stage2_reap_ream :: DIAG layer 13: batch loop complete — 32 batches in 26.8s, now post-profile work | vram=88.8/150GB host_free=150GB gpu_util=37% temp=41C
2026-05-20 03:21:28,669 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 13 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:21:36,375 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 25/20000 — train_mse=4.863802e-05 holdout_mse=4.750191e-05 best=4.750191e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:38,816 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 50/20000 — train_mse=4.244158e-05 holdout_mse=4.191191e-05 best=4.191191e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:41,253 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 75/20000 — train_mse=3.846418e-05 holdout_mse=3.874828e-05 best=3.874828e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:43,710 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 100/20000 — train_mse=3.631516e-05 holdout_mse=3.657722e-05 best=3.657722e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:46,160 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 125/20000 — train_mse=3.465992e-05 holdout_mse=3.493800e-05 best=3.493800e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:48,602 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 150/20000 — train_mse=3.199270e-05 holdout_mse=3.363566e-05 best=3.363566e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:51,066 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 175/20000 — train_mse=3.194742e-05 holdout_mse=3.255108e-05 best=3.255108e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:53,519 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 200/20000 — train_mse=3.073557e-05 holdout_mse=3.163644e-05 best=3.163644e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:55,969 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 225/20000 — train_mse=3.000482e-05 holdout_mse=3.084075e-05 best=3.084075e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:21:58,419 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 250/20000 — train_mse=2.880923e-05 holdout_mse=3.015301e-05 best=3.015301e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:00,873 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 275/20000 — train_mse=2.812512e-05 holdout_mse=2.951351e-05 best=2.951351e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:03,329 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 300/20000 — train_mse=2.712927e-05 holdout_mse=2.895392e-05 best=2.895392e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:05,794 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 325/20000 — train_mse=2.591230e-05 holdout_mse=2.843421e-05 best=2.843421e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:08,249 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 350/20000 — train_mse=2.616450e-05 holdout_mse=2.796687e-05 best=2.796687e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:10,718 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 375/20000 — train_mse=2.540157e-05 holdout_mse=2.753701e-05 best=2.753701e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:13,205 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 400/20000 — train_mse=2.474540e-05 holdout_mse=2.714080e-05 best=2.714080e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:15,880 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 425/20000 — train_mse=2.494317e-05 holdout_mse=2.676044e-05 best=2.676044e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:18,349 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 450/20000 — train_mse=2.478843e-05 holdout_mse=2.641107e-05 best=2.641107e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:20,812 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 475/20000 — train_mse=2.393711e-05 holdout_mse=2.608074e-05 best=2.608074e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:23,268 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 500/20000 — train_mse=2.375245e-05 holdout_mse=2.577262e-05 best=2.577262e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:25,730 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 525/20000 — train_mse=2.325640e-05 holdout_mse=2.549092e-05 best=2.549092e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:28,189 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 550/20000 — train_mse=2.293305e-05 holdout_mse=2.520267e-05 best=2.520267e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:30,647 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 575/20000 — train_mse=2.267581e-05 holdout_mse=2.494172e-05 best=2.494172e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:22:33,112 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 600/20000 — train_mse=2.239130e-05 holdout_mse=2.469267e-05 best=2.469267e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:22:35,566 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 625/20000 — train_mse=2.258537e-05 holdout_mse=2.445953e-05 best=2.445953e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:22:38,030 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 650/20000 — train_mse=2.260389e-05 holdout_mse=2.423595e-05 best=2.423595e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:22:40,479 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 675/20000 — train_mse=2.141789e-05 holdout_mse=2.402838e-05 best=2.402838e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:22:42,910 INFO moe_compress.stage2_reap_ream ::     heal layer 13: step 700/20000 — train_mse=2.110149e-05 holdout_mse=2.382901e-05 best=2.382901e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:22:43,012 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 13: 700 steps (patience) — train_mse=2.110149e-05 (at_best=2.110149e-05) holdout_mse=2.382901e-05 plain_merged_holdout_mse=6.344679e-05 heal_gap=2.727520e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:22:43,145 WARNING moe_compress.stage2_reap_ream ::   layer 13: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:22:45,965 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=56.4518, max_group=9, mean_group=1.00
2026-05-20 03:22:46,340 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 15/40 (idx=14) — profiling then merging to 128 experts
2026-05-20 03:22:46,345 INFO moe_compress.stage2_reap_ream :: DIAG layer 14: entering batch loop (calibration tensor + early-exit forwards) | vram=64.4/150GB host_free=150GB gpu_util=0% temp=32C
2026-05-20 03:22:47,300 INFO moe_compress.stage2_reap_ream :: DIAG layer 14 batch 1: total=0.95s fwd=0.91s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.0s | vram=88.0/150GB host_free=150GB gpu_util=47% temp=38C
2026-05-20 03:22:48,199 INFO moe_compress.stage2_reap_ream :: DIAG layer 14 batch 2: total=0.90s fwd=0.87s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.9s | vram=88.0/150GB host_free=150GB gpu_util=52% temp=40C
2026-05-20 03:22:49,093 INFO moe_compress.stage2_reap_ream :: DIAG layer 14 batch 3: total=0.89s fwd=0.86s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.7s | vram=88.0/150GB host_free=150GB gpu_util=63% temp=41C
2026-05-20 03:22:55,268 INFO moe_compress.stage2_reap_ream :: DIAG layer 14 batch 10: total=0.87s fwd=0.84s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=8.9s | vram=88.0/150GB host_free=150GB gpu_util=67% temp=43C
2026-05-20 03:23:04,076 INFO moe_compress.stage2_reap_ream :: DIAG layer 14 batch 20: total=0.90s fwd=0.84s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=17.7s | vram=88.0/150GB host_free=150GB gpu_util=56% temp=44C
2026-05-20 03:23:13,123 INFO moe_compress.stage2_reap_ream :: DIAG layer 14 batch 30: total=0.92s fwd=0.86s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=26.8s | vram=88.0/150GB host_free=149GB gpu_util=55% temp=44C
2026-05-20 03:23:14,376 INFO moe_compress.stage2_reap_ream :: DIAG layer 14: batch loop complete — 32 batches in 28.0s, now post-profile work | vram=88.0/150GB host_free=149GB gpu_util=66% temp=43C
2026-05-20 03:23:24,371 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 14 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:23:32,013 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 25/20000 — train_mse=4.211827e-05 holdout_mse=4.134616e-05 best=4.134616e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:34,453 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 50/20000 — train_mse=3.523568e-05 holdout_mse=3.554168e-05 best=3.554168e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:36,921 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 75/20000 — train_mse=3.209987e-05 holdout_mse=3.224521e-05 best=3.224521e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:39,392 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 100/20000 — train_mse=2.957724e-05 holdout_mse=3.015756e-05 best=3.015756e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:41,850 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 125/20000 — train_mse=2.833723e-05 holdout_mse=2.869879e-05 best=2.869879e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:44,327 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 150/20000 — train_mse=2.690687e-05 holdout_mse=2.757343e-05 best=2.757343e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:46,788 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 175/20000 — train_mse=2.561852e-05 holdout_mse=2.666004e-05 best=2.666004e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:49,240 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 200/20000 — train_mse=2.432505e-05 holdout_mse=2.588450e-05 best=2.588450e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:51,701 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 225/20000 — train_mse=2.387052e-05 holdout_mse=2.523081e-05 best=2.523081e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:54,170 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 250/20000 — train_mse=2.291601e-05 holdout_mse=2.466625e-05 best=2.466625e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:56,625 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 275/20000 — train_mse=2.268590e-05 holdout_mse=2.417711e-05 best=2.417711e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:23:59,076 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 300/20000 — train_mse=2.270925e-05 holdout_mse=2.371655e-05 best=2.371655e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:01,530 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 325/20000 — train_mse=2.176195e-05 holdout_mse=2.332363e-05 best=2.332363e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:03,992 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 350/20000 — train_mse=2.139586e-05 holdout_mse=2.295602e-05 best=2.295602e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:06,454 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 375/20000 — train_mse=2.090217e-05 holdout_mse=2.262661e-05 best=2.262661e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:08,913 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 400/20000 — train_mse=2.109371e-05 holdout_mse=2.232858e-05 best=2.232858e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:11,366 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 425/20000 — train_mse=2.047443e-05 holdout_mse=2.204970e-05 best=2.204970e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:13,823 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 450/20000 — train_mse=2.042338e-05 holdout_mse=2.179906e-05 best=2.179906e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:16,297 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 475/20000 — train_mse=1.964323e-05 holdout_mse=2.155702e-05 best=2.155702e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:18,760 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 500/20000 — train_mse=1.972958e-05 holdout_mse=2.134040e-05 best=2.134040e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:24:21,219 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 525/20000 — train_mse=1.918148e-05 holdout_mse=2.113022e-05 best=2.113022e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:24:23,683 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 550/20000 — train_mse=1.866715e-05 holdout_mse=2.093218e-05 best=2.093218e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:24:26,145 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 575/20000 — train_mse=1.877664e-05 holdout_mse=2.074237e-05 best=2.074237e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:24:28,607 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 600/20000 — train_mse=1.870654e-05 holdout_mse=2.057174e-05 best=2.057174e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:24:31,066 INFO moe_compress.stage2_reap_ream ::     heal layer 14: step 625/20000 — train_mse=1.832072e-05 holdout_mse=2.040637e-05 best=2.040637e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:24:31,162 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 14: 625 steps (patience) — train_mse=1.832072e-05 (at_best=1.832072e-05) holdout_mse=2.040637e-05 plain_merged_holdout_mse=5.495837e-05 heal_gap=2.085650e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:24:31,296 WARNING moe_compress.stage2_reap_ream ::   layer 14: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:24:34,078 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=59.6128, max_group=9, mean_group=1.00
2026-05-20 03:24:34,432 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 16/40 (idx=15) — profiling then merging to 128 experts
2026-05-20 03:24:34,437 INFO moe_compress.stage2_reap_ream :: DIAG layer 15: entering batch loop (calibration tensor + early-exit forwards) | vram=63.6/150GB host_free=149GB gpu_util=0% temp=32C
2026-05-20 03:24:35,432 INFO moe_compress.stage2_reap_ream :: DIAG layer 15 batch 1: total=1.00s fwd=0.95s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.0s | vram=87.2/150GB host_free=149GB gpu_util=47% temp=38C
2026-05-20 03:24:36,363 INFO moe_compress.stage2_reap_ream :: DIAG layer 15 batch 2: total=0.93s fwd=0.90s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=1.9s | vram=87.2/150GB host_free=149GB gpu_util=49% temp=39C
2026-05-20 03:24:37,287 INFO moe_compress.stage2_reap_ream :: DIAG layer 15 batch 3: total=0.92s fwd=0.90s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.9s | vram=87.2/150GB host_free=149GB gpu_util=57% temp=40C
2026-05-20 03:24:43,702 INFO moe_compress.stage2_reap_ream :: DIAG layer 15 batch 10: total=0.91s fwd=0.88s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=9.3s | vram=87.2/150GB host_free=149GB gpu_util=59% temp=43C
2026-05-20 03:24:52,871 INFO moe_compress.stage2_reap_ream :: DIAG layer 15 batch 20: total=0.94s fwd=0.88s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=18.4s | vram=87.2/150GB host_free=149GB gpu_util=58% temp=44C
2026-05-20 03:25:02,271 INFO moe_compress.stage2_reap_ream :: DIAG layer 15 batch 30: total=0.94s fwd=0.88s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=27.8s | vram=87.2/150GB host_free=148GB gpu_util=56% temp=42C
2026-05-20 03:25:03,580 INFO moe_compress.stage2_reap_ream :: DIAG layer 15: batch loop complete — 32 batches in 29.1s, now post-profile work | vram=87.2/150GB host_free=148GB gpu_util=37% temp=41C
2026-05-20 03:25:13,647 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 15 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:25:21,519 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 25/20000 — train_mse=4.186474e-05 holdout_mse=4.149230e-05 best=4.149230e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:23,873 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 50/20000 — train_mse=3.920395e-05 holdout_mse=3.854262e-05 best=3.854262e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:26,239 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 75/20000 — train_mse=3.684922e-05 holdout_mse=3.654848e-05 best=3.654848e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:28,579 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 100/20000 — train_mse=3.463860e-05 holdout_mse=3.504375e-05 best=3.504375e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:30,925 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 125/20000 — train_mse=3.307600e-05 holdout_mse=3.381647e-05 best=3.381647e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:33,281 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 150/20000 — train_mse=3.159491e-05 holdout_mse=3.282619e-05 best=3.282619e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:35,633 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 175/20000 — train_mse=3.178325e-05 holdout_mse=3.199678e-05 best=3.199678e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:37,982 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 200/20000 — train_mse=3.017064e-05 holdout_mse=3.129573e-05 best=3.129573e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:40,331 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 225/20000 — train_mse=2.970573e-05 holdout_mse=3.067984e-05 best=3.067984e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:42,697 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 250/20000 — train_mse=2.921881e-05 holdout_mse=3.014495e-05 best=3.014495e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:45,062 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 275/20000 — train_mse=2.858308e-05 holdout_mse=2.965724e-05 best=2.965724e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:47,430 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 300/20000 — train_mse=2.764494e-05 holdout_mse=2.921463e-05 best=2.921463e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:49,786 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 325/20000 — train_mse=2.693504e-05 holdout_mse=2.881793e-05 best=2.881793e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:52,139 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 350/20000 — train_mse=2.680135e-05 holdout_mse=2.847648e-05 best=2.847648e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:54,513 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 375/20000 — train_mse=2.688745e-05 holdout_mse=2.814561e-05 best=2.814561e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:56,878 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 400/20000 — train_mse=2.569753e-05 holdout_mse=2.783903e-05 best=2.783903e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:25:59,243 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 425/20000 — train_mse=2.548929e-05 holdout_mse=2.757058e-05 best=2.757058e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:26:01,602 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 450/20000 — train_mse=2.579319e-05 holdout_mse=2.730368e-05 best=2.730368e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:26:03,973 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 475/20000 — train_mse=2.498037e-05 holdout_mse=2.704273e-05 best=2.704273e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:26:06,337 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 500/20000 — train_mse=2.464535e-05 holdout_mse=2.680835e-05 best=2.680835e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:26:08,707 INFO moe_compress.stage2_reap_ream ::     heal layer 15: step 525/20000 — train_mse=2.472837e-05 holdout_mse=2.659085e-05 best=2.659085e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:26:08,809 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 15: 525 steps (patience) — train_mse=2.472837e-05 (at_best=2.472837e-05) holdout_mse=2.659085e-05 plain_merged_holdout_mse=5.005768e-05 heal_gap=1.862479e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:26:08,945 WARNING moe_compress.stage2_reap_ream ::   layer 15: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:26:11,577 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=60.9063, max_group=9, mean_group=1.00
2026-05-20 03:26:11,960 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 17/40 (idx=16) — profiling then merging to 128 experts
2026-05-20 03:26:11,965 INFO moe_compress.stage2_reap_ream :: DIAG layer 16: entering batch loop (calibration tensor + early-exit forwards) | vram=62.9/150GB host_free=148GB gpu_util=0% temp=32C
2026-05-20 03:26:13,006 INFO moe_compress.stage2_reap_ream :: DIAG layer 16 batch 1: total=1.04s fwd=0.98s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.0s | vram=86.5/150GB host_free=148GB gpu_util=44% temp=37C
2026-05-20 03:26:13,974 INFO moe_compress.stage2_reap_ream :: DIAG layer 16 batch 2: total=0.97s fwd=0.94s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.0s | vram=86.5/150GB host_free=148GB gpu_util=53% temp=39C
2026-05-20 03:26:14,938 INFO moe_compress.stage2_reap_ream :: DIAG layer 16 batch 3: total=0.96s fwd=0.94s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.0s | vram=86.5/150GB host_free=148GB gpu_util=54% temp=40C
2026-05-20 03:26:21,620 INFO moe_compress.stage2_reap_ream :: DIAG layer 16 batch 10: total=0.95s fwd=0.92s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=9.7s | vram=86.5/150GB host_free=148GB gpu_util=53% temp=43C
2026-05-20 03:26:31,190 INFO moe_compress.stage2_reap_ream :: DIAG layer 16 batch 20: total=0.98s fwd=0.92s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=19.2s | vram=86.5/150GB host_free=148GB gpu_util=51% temp=42C
2026-05-20 03:26:40,980 INFO moe_compress.stage2_reap_ream :: DIAG layer 16 batch 30: total=0.98s fwd=0.92s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=29.0s | vram=86.5/150GB host_free=147GB gpu_util=59% temp=45C
2026-05-20 03:26:42,344 INFO moe_compress.stage2_reap_ream :: DIAG layer 16: batch loop complete — 32 batches in 30.4s, now post-profile work | vram=86.5/150GB host_free=147GB gpu_util=100% temp=43C
2026-05-20 03:26:52,673 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 16 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:27:00,341 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 25/20000 — train_mse=5.385565e-05 holdout_mse=5.508203e-05 best=5.508203e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:02,707 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 50/20000 — train_mse=4.758460e-05 holdout_mse=4.948144e-05 best=4.948144e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:05,058 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 75/20000 — train_mse=4.421634e-05 holdout_mse=4.611696e-05 best=4.611696e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:07,413 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 100/20000 — train_mse=4.204874e-05 holdout_mse=4.337034e-05 best=4.337034e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:09,770 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 125/20000 — train_mse=3.959101e-05 holdout_mse=4.129471e-05 best=4.129471e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:12,120 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 150/20000 — train_mse=3.918459e-05 holdout_mse=3.979294e-05 best=3.979294e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:14,481 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 175/20000 — train_mse=3.741253e-05 holdout_mse=3.864166e-05 best=3.864166e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:16,842 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 200/20000 — train_mse=3.553938e-05 holdout_mse=3.767906e-05 best=3.767906e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:19,187 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 225/20000 — train_mse=3.526918e-05 holdout_mse=3.687210e-05 best=3.687210e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:21,539 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 250/20000 — train_mse=3.386985e-05 holdout_mse=3.614988e-05 best=3.614988e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:23,898 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 275/20000 — train_mse=3.313385e-05 holdout_mse=3.551757e-05 best=3.551757e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:26,254 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 300/20000 — train_mse=3.188350e-05 holdout_mse=3.495185e-05 best=3.495185e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:28,604 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 325/20000 — train_mse=3.176636e-05 holdout_mse=3.443403e-05 best=3.443403e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:30,958 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 350/20000 — train_mse=3.101908e-05 holdout_mse=3.394397e-05 best=3.394397e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:33,313 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 375/20000 — train_mse=3.095615e-05 holdout_mse=3.352595e-05 best=3.352595e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:35,697 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 400/20000 — train_mse=3.003162e-05 holdout_mse=3.312601e-05 best=3.312601e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:38,148 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 425/20000 — train_mse=3.027517e-05 holdout_mse=3.275014e-05 best=3.275014e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:40,602 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 450/20000 — train_mse=2.957521e-05 holdout_mse=3.240106e-05 best=3.240106e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:43,047 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 475/20000 — train_mse=2.903150e-05 holdout_mse=3.207210e-05 best=3.207210e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:27:45,512 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 500/20000 — train_mse=2.892571e-05 holdout_mse=3.176588e-05 best=3.176588e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:27:47,967 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 525/20000 — train_mse=2.874671e-05 holdout_mse=3.148176e-05 best=3.148176e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:27:50,423 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 550/20000 — train_mse=2.784688e-05 holdout_mse=3.120364e-05 best=3.120364e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:27:52,878 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 575/20000 — train_mse=2.787096e-05 holdout_mse=3.094992e-05 best=3.094992e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:27:55,346 INFO moe_compress.stage2_reap_ream ::     heal layer 16: step 600/20000 — train_mse=2.724665e-05 holdout_mse=3.070112e-05 best=3.070112e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:27:55,442 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 16: 600 steps (patience) — train_mse=2.724665e-05 (at_best=2.724665e-05) holdout_mse=3.070112e-05 plain_merged_holdout_mse=7.249769e-05 heal_gap=3.454470e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:27:55,578 WARNING moe_compress.stage2_reap_ream ::   layer 16: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:27:58,236 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=59.3604, max_group=9, mean_group=1.00
2026-05-20 03:27:58,606 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 18/40 (idx=17) — profiling then merging to 128 experts
2026-05-20 03:27:58,611 INFO moe_compress.stage2_reap_ream :: DIAG layer 17: entering batch loop (calibration tensor + early-exit forwards) | vram=62.1/150GB host_free=147GB gpu_util=0% temp=32C
2026-05-20 03:27:59,682 INFO moe_compress.stage2_reap_ream :: DIAG layer 17 batch 1: total=1.07s fwd=1.02s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.1s | vram=85.7/150GB host_free=147GB gpu_util=51% temp=38C
2026-05-20 03:28:00,689 INFO moe_compress.stage2_reap_ream :: DIAG layer 17 batch 2: total=1.01s fwd=0.98s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.1s | vram=85.7/150GB host_free=147GB gpu_util=55% temp=40C
2026-05-20 03:28:01,694 INFO moe_compress.stage2_reap_ream :: DIAG layer 17 batch 3: total=1.00s fwd=0.98s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.1s | vram=85.7/150GB host_free=147GB gpu_util=55% temp=40C
2026-05-20 03:28:08,638 INFO moe_compress.stage2_reap_ream :: DIAG layer 17 batch 10: total=0.99s fwd=0.96s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=10.0s | vram=85.7/150GB host_free=147GB gpu_util=55% temp=43C
2026-05-20 03:28:18,587 INFO moe_compress.stage2_reap_ream :: DIAG layer 17 batch 20: total=1.02s fwd=0.96s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=20.0s | vram=85.7/150GB host_free=146GB gpu_util=56% temp=44C
2026-05-20 03:28:28,770 INFO moe_compress.stage2_reap_ream :: DIAG layer 17 batch 30: total=1.02s fwd=0.96s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=30.2s | vram=85.7/150GB host_free=146GB gpu_util=57% temp=45C
2026-05-20 03:28:30,190 INFO moe_compress.stage2_reap_ream :: DIAG layer 17: batch loop complete — 32 batches in 31.6s, now post-profile work | vram=85.7/150GB host_free=146GB gpu_util=100% temp=43C
2026-05-20 03:28:40,668 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 17 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:28:48,433 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 25/20000 — train_mse=5.894006e-05 holdout_mse=5.734378e-05 best=5.734378e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:28:50,763 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 50/20000 — train_mse=4.970459e-05 holdout_mse=5.084579e-05 best=5.084579e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:28:53,097 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 75/20000 — train_mse=4.751887e-05 holdout_mse=4.727782e-05 best=4.727782e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:28:55,429 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 100/20000 — train_mse=4.436408e-05 holdout_mse=4.477625e-05 best=4.477625e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:28:57,766 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 125/20000 — train_mse=4.234992e-05 holdout_mse=4.286272e-05 best=4.286272e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:00,092 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 150/20000 — train_mse=3.970590e-05 holdout_mse=4.133644e-05 best=4.133644e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:02,518 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 175/20000 — train_mse=3.821067e-05 holdout_mse=4.007451e-05 best=4.007451e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:04,842 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 200/20000 — train_mse=3.829720e-05 holdout_mse=3.900766e-05 best=3.900766e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:07,176 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 225/20000 — train_mse=3.634940e-05 holdout_mse=3.808678e-05 best=3.808678e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:09,508 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 250/20000 — train_mse=3.574109e-05 holdout_mse=3.724903e-05 best=3.724903e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:11,870 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 275/20000 — train_mse=3.483357e-05 holdout_mse=3.654658e-05 best=3.654658e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:14,235 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 300/20000 — train_mse=3.496608e-05 holdout_mse=3.585090e-05 best=3.585090e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:16,579 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 325/20000 — train_mse=3.302666e-05 holdout_mse=3.527882e-05 best=3.527882e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:18,919 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 350/20000 — train_mse=3.240120e-05 holdout_mse=3.474790e-05 best=3.474790e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:21,262 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 375/20000 — train_mse=3.222106e-05 holdout_mse=3.424884e-05 best=3.424884e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:23,613 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 400/20000 — train_mse=3.116457e-05 holdout_mse=3.379147e-05 best=3.379147e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:25,969 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 425/20000 — train_mse=3.064498e-05 holdout_mse=3.340052e-05 best=3.340052e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:28,320 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 450/20000 — train_mse=3.082541e-05 holdout_mse=3.297499e-05 best=3.297499e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:30,667 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 475/20000 — train_mse=3.017025e-05 holdout_mse=3.261804e-05 best=3.261804e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:33,010 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 500/20000 — train_mse=2.876166e-05 holdout_mse=3.226133e-05 best=3.226133e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:29:35,356 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 525/20000 — train_mse=2.917111e-05 holdout_mse=3.193928e-05 best=3.193928e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:29:37,699 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 550/20000 — train_mse=2.872924e-05 holdout_mse=3.162073e-05 best=3.162073e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:29:40,044 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 575/20000 — train_mse=2.798695e-05 holdout_mse=3.135047e-05 best=3.135047e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:29:42,385 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 600/20000 — train_mse=2.765230e-05 holdout_mse=3.107580e-05 best=3.107580e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:29:44,730 INFO moe_compress.stage2_reap_ream ::     heal layer 17: step 625/20000 — train_mse=2.714359e-05 holdout_mse=3.083248e-05 best=3.083248e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:29:44,826 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 17: 625 steps (patience) — train_mse=2.714359e-05 (at_best=2.714359e-05) holdout_mse=3.083248e-05 plain_merged_holdout_mse=7.366861e-05 heal_gap=3.688893e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:29:44,961 WARNING moe_compress.stage2_reap_ream ::   layer 17: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:29:47,824 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=57.7139, max_group=9, mean_group=1.00
2026-05-20 03:29:48,207 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 19/40 (idx=18) — profiling then merging to 128 experts
2026-05-20 03:29:48,211 INFO moe_compress.stage2_reap_ream :: DIAG layer 18: entering batch loop (calibration tensor + early-exit forwards) | vram=61.3/150GB host_free=146GB gpu_util=0% temp=32C
2026-05-20 03:29:49,315 INFO moe_compress.stage2_reap_ream :: DIAG layer 18 batch 1: total=1.10s fwd=1.06s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.1s | vram=85.0/150GB host_free=146GB gpu_util=50% temp=38C
2026-05-20 03:29:50,357 INFO moe_compress.stage2_reap_ream :: DIAG layer 18 batch 2: total=1.04s fwd=1.01s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.1s | vram=85.0/150GB host_free=146GB gpu_util=52% temp=40C
2026-05-20 03:29:51,391 INFO moe_compress.stage2_reap_ream :: DIAG layer 18 batch 3: total=1.03s fwd=1.01s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.2s | vram=85.0/150GB host_free=146GB gpu_util=56% temp=41C
2026-05-20 03:29:58,582 INFO moe_compress.stage2_reap_ream :: DIAG layer 18 batch 10: total=1.02s fwd=1.00s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=10.4s | vram=85.0/150GB host_free=146GB gpu_util=59% temp=43C
2026-05-20 03:30:08,900 INFO moe_compress.stage2_reap_ream :: DIAG layer 18 batch 20: total=1.05s fwd=0.99s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=20.7s | vram=85.0/150GB host_free=145GB gpu_util=55% temp=45C
2026-05-20 03:30:19,406 INFO moe_compress.stage2_reap_ream :: DIAG layer 18 batch 30: total=1.05s fwd=0.99s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=31.2s | vram=85.0/150GB host_free=145GB gpu_util=58% temp=46C
2026-05-20 03:30:20,868 INFO moe_compress.stage2_reap_ream :: DIAG layer 18: batch loop complete — 32 batches in 32.7s, now post-profile work | vram=85.0/150GB host_free=145GB gpu_util=36% temp=42C
2026-05-20 03:30:31,443 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 18 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:30:39,483 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 25/20000 — train_mse=6.003426e-05 holdout_mse=6.057948e-05 best=6.057948e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:41,836 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 50/20000 — train_mse=5.363398e-05 holdout_mse=5.301206e-05 best=5.301206e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:44,187 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 75/20000 — train_mse=4.844803e-05 holdout_mse=4.901231e-05 best=4.901231e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:46,537 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 100/20000 — train_mse=4.661699e-05 holdout_mse=4.633500e-05 best=4.633500e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:48,884 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 125/20000 — train_mse=4.358798e-05 holdout_mse=4.432630e-05 best=4.432630e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:51,237 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 150/20000 — train_mse=4.171318e-05 holdout_mse=4.270267e-05 best=4.270267e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:53,603 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 175/20000 — train_mse=3.960878e-05 holdout_mse=4.135162e-05 best=4.135162e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:55,962 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 200/20000 — train_mse=3.841541e-05 holdout_mse=4.020655e-05 best=4.020655e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:30:58,309 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 225/20000 — train_mse=3.717003e-05 holdout_mse=3.924659e-05 best=3.924659e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:00,656 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 250/20000 — train_mse=3.639718e-05 holdout_mse=3.840792e-05 best=3.840792e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:03,006 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 275/20000 — train_mse=3.546644e-05 holdout_mse=3.765131e-05 best=3.765131e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:05,364 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 300/20000 — train_mse=3.394540e-05 holdout_mse=3.696081e-05 best=3.696081e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:07,712 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 325/20000 — train_mse=3.385087e-05 holdout_mse=3.636906e-05 best=3.636906e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:10,069 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 350/20000 — train_mse=3.340125e-05 holdout_mse=3.586350e-05 best=3.586350e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:12,431 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 375/20000 — train_mse=3.250553e-05 holdout_mse=3.530720e-05 best=3.530720e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:14,801 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 400/20000 — train_mse=3.237807e-05 holdout_mse=3.485098e-05 best=3.485098e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:17,162 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 425/20000 — train_mse=3.139489e-05 holdout_mse=3.439913e-05 best=3.439913e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:19,524 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 450/20000 — train_mse=3.076892e-05 holdout_mse=3.402031e-05 best=3.402031e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:21,881 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 475/20000 — train_mse=3.019425e-05 holdout_mse=3.362412e-05 best=3.362412e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:24,254 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 500/20000 — train_mse=2.987558e-05 holdout_mse=3.325755e-05 best=3.325755e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:26,635 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 525/20000 — train_mse=2.922654e-05 holdout_mse=3.291632e-05 best=3.291632e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:31:28,992 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 550/20000 — train_mse=2.965072e-05 holdout_mse=3.260384e-05 best=3.260384e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:31:31,355 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 575/20000 — train_mse=2.868026e-05 holdout_mse=3.228645e-05 best=3.228645e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:31:33,717 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 600/20000 — train_mse=2.823402e-05 holdout_mse=3.201999e-05 best=3.201999e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:31:36,162 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 625/20000 — train_mse=2.837609e-05 holdout_mse=3.173071e-05 best=3.173071e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:31:38,524 INFO moe_compress.stage2_reap_ream ::     heal layer 18: step 650/20000 — train_mse=2.767151e-05 holdout_mse=3.147567e-05 best=3.147567e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:31:38,623 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 18: 650 steps (patience) — train_mse=2.767151e-05 (at_best=2.767151e-05) holdout_mse=3.147567e-05 plain_merged_holdout_mse=8.337755e-05 heal_gap=3.804155e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:31:38,760 WARNING moe_compress.stage2_reap_ream ::   layer 18: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:31:41,534 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=56.5191, max_group=9, mean_group=1.00
2026-05-20 03:31:41,920 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 20/40 (idx=19) — profiling then merging to 128 experts
2026-05-20 03:31:41,925 INFO moe_compress.stage2_reap_ream :: DIAG layer 19: entering batch loop (calibration tensor + early-exit forwards) | vram=60.5/150GB host_free=145GB gpu_util=0% temp=33C
2026-05-20 03:31:43,072 INFO moe_compress.stage2_reap_ream :: DIAG layer 19 batch 1: total=1.15s fwd=1.10s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=255) | cum=1.1s | vram=84.1/150GB host_free=145GB gpu_util=42% temp=39C
2026-05-20 03:31:44,153 INFO moe_compress.stage2_reap_ream :: DIAG layer 19 batch 2: total=1.08s fwd=1.05s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.2s | vram=84.1/150GB host_free=145GB gpu_util=53% temp=41C
2026-05-20 03:31:45,230 INFO moe_compress.stage2_reap_ream :: DIAG layer 19 batch 3: total=1.08s fwd=1.05s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.3s | vram=84.1/150GB host_free=145GB gpu_util=51% temp=41C
2026-05-20 03:31:52,719 INFO moe_compress.stage2_reap_ream :: DIAG layer 19 batch 10: total=1.07s fwd=1.04s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=10.8s | vram=84.1/150GB host_free=145GB gpu_util=58% temp=44C
2026-05-20 03:32:03,464 INFO moe_compress.stage2_reap_ream :: DIAG layer 19 batch 20: total=1.09s fwd=1.03s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=255) | cum=21.5s | vram=84.1/150GB host_free=144GB gpu_util=54% temp=45C
2026-05-20 03:32:14,411 INFO moe_compress.stage2_reap_ream :: DIAG layer 19 batch 30: total=1.10s fwd=1.04s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=32.5s | vram=84.1/150GB host_free=144GB gpu_util=54% temp=45C
2026-05-20 03:32:15,916 INFO moe_compress.stage2_reap_ream :: DIAG layer 19: batch loop complete — 32 batches in 34.0s, now post-profile work | vram=84.2/150GB host_free=144GB gpu_util=35% temp=43C
2026-05-20 03:32:26,593 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 19 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:32:34,686 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 25/20000 — train_mse=7.834878e-05 holdout_mse=7.888127e-05 best=7.888127e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:37,064 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 50/20000 — train_mse=6.961464e-05 holdout_mse=6.974777e-05 best=6.974777e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:39,423 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 75/20000 — train_mse=6.386160e-05 holdout_mse=6.451331e-05 best=6.451331e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:41,796 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 100/20000 — train_mse=5.842042e-05 holdout_mse=6.088733e-05 best=6.088733e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:44,152 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 125/20000 — train_mse=5.472495e-05 holdout_mse=5.819795e-05 best=5.819795e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:46,551 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 150/20000 — train_mse=5.359107e-05 holdout_mse=5.610158e-05 best=5.610158e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:48,934 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 175/20000 — train_mse=5.121234e-05 holdout_mse=5.440816e-05 best=5.440816e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:51,318 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 200/20000 — train_mse=4.977970e-05 holdout_mse=5.300057e-05 best=5.300057e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:53,711 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 225/20000 — train_mse=4.915823e-05 holdout_mse=5.178391e-05 best=5.178391e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:56,121 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 250/20000 — train_mse=4.703630e-05 holdout_mse=5.074878e-05 best=5.074878e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:32:58,501 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 275/20000 — train_mse=4.505610e-05 holdout_mse=4.980023e-05 best=4.980023e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:00,886 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 300/20000 — train_mse=4.504906e-05 holdout_mse=4.895267e-05 best=4.895267e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:03,276 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 325/20000 — train_mse=4.525152e-05 holdout_mse=4.820925e-05 best=4.820925e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:05,693 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 350/20000 — train_mse=4.363131e-05 holdout_mse=4.751972e-05 best=4.751972e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:08,078 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 375/20000 — train_mse=4.263437e-05 holdout_mse=4.686991e-05 best=4.686991e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:10,459 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 400/20000 — train_mse=4.161717e-05 holdout_mse=4.629614e-05 best=4.629614e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:12,829 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 425/20000 — train_mse=4.131512e-05 holdout_mse=4.577554e-05 best=4.577554e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:15,201 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 450/20000 — train_mse=4.108218e-05 holdout_mse=4.528621e-05 best=4.528621e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:33:17,557 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 475/20000 — train_mse=4.009321e-05 holdout_mse=4.483965e-05 best=4.483965e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:33:19,916 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 500/20000 — train_mse=3.941896e-05 holdout_mse=4.440610e-05 best=4.440610e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:33:22,285 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 525/20000 — train_mse=3.980802e-05 holdout_mse=4.398945e-05 best=4.398945e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:33:24,671 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 550/20000 — train_mse=3.876918e-05 holdout_mse=4.362587e-05 best=4.362587e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:33:27,045 INFO moe_compress.stage2_reap_ream ::     heal layer 19: step 575/20000 — train_mse=3.712951e-05 holdout_mse=4.325827e-05 best=4.325827e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:33:27,144 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 19: 575 steps (patience) — train_mse=3.712951e-05 (at_best=3.712951e-05) holdout_mse=4.325827e-05 plain_merged_holdout_mse=1.038483e-04 heal_gap=6.128756e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:33:27,280 WARNING moe_compress.stage2_reap_ream ::   layer 19: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:33:30,088 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=56.6925, max_group=9, mean_group=1.00
2026-05-20 03:33:30,457 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 21/40 (idx=20) — profiling then merging to 128 experts
2026-05-20 03:33:30,462 INFO moe_compress.stage2_reap_ream :: DIAG layer 20: entering batch loop (calibration tensor + early-exit forwards) | vram=59.7/150GB host_free=144GB gpu_util=0% temp=32C
2026-05-20 03:33:31,639 INFO moe_compress.stage2_reap_ream :: DIAG layer 20 batch 1: total=1.18s fwd=1.13s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.2s | vram=83.3/150GB host_free=144GB gpu_util=51% temp=38C
2026-05-20 03:33:32,758 INFO moe_compress.stage2_reap_ream :: DIAG layer 20 batch 2: total=1.12s fwd=1.09s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.3s | vram=83.3/150GB host_free=144GB gpu_util=52% temp=40C
2026-05-20 03:33:33,872 INFO moe_compress.stage2_reap_ream :: DIAG layer 20 batch 3: total=1.11s fwd=1.09s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.4s | vram=83.3/150GB host_free=144GB gpu_util=52% temp=41C
2026-05-20 03:33:41,574 INFO moe_compress.stage2_reap_ream :: DIAG layer 20 batch 10: total=1.09s fwd=1.07s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=11.1s | vram=83.3/150GB host_free=144GB gpu_util=58% temp=44C
2026-05-20 03:33:52,620 INFO moe_compress.stage2_reap_ream :: DIAG layer 20 batch 20: total=1.12s fwd=1.06s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=22.2s | vram=83.3/150GB host_free=143GB gpu_util=57% temp=45C
2026-05-20 03:34:03,869 INFO moe_compress.stage2_reap_ream :: DIAG layer 20 batch 30: total=1.12s fwd=1.06s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=33.4s | vram=83.3/150GB host_free=143GB gpu_util=57% temp=46C
2026-05-20 03:34:05,405 INFO moe_compress.stage2_reap_ream :: DIAG layer 20: batch loop complete — 32 batches in 34.9s, now post-profile work | vram=83.4/150GB host_free=143GB gpu_util=36% temp=45C
2026-05-20 03:34:16,217 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 20 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:34:24,455 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 25/20000 — train_mse=6.573956e-05 holdout_mse=6.523617e-05 best=6.523617e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:26,929 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 50/20000 — train_mse=5.587843e-05 holdout_mse=5.641857e-05 best=5.641857e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:29,393 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 75/20000 — train_mse=5.010378e-05 holdout_mse=5.166200e-05 best=5.166200e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:31,859 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 100/20000 — train_mse=4.829058e-05 holdout_mse=4.850850e-05 best=4.850850e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:34,318 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 125/20000 — train_mse=4.446343e-05 holdout_mse=4.619855e-05 best=4.619855e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:36,783 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 150/20000 — train_mse=4.191125e-05 holdout_mse=4.442386e-05 best=4.442386e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:39,249 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 175/20000 — train_mse=4.134617e-05 holdout_mse=4.298597e-05 best=4.298597e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:41,713 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 200/20000 — train_mse=3.955704e-05 holdout_mse=4.178683e-05 best=4.178683e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:44,180 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 225/20000 — train_mse=3.760239e-05 holdout_mse=4.077626e-05 best=4.077626e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:46,664 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 250/20000 — train_mse=3.690221e-05 holdout_mse=3.985154e-05 best=3.985154e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:49,136 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 275/20000 — train_mse=3.615755e-05 holdout_mse=3.904969e-05 best=3.904969e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:51,601 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 300/20000 — train_mse=3.512825e-05 holdout_mse=3.830772e-05 best=3.830772e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:54,068 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 325/20000 — train_mse=3.443391e-05 holdout_mse=3.764033e-05 best=3.764033e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:56,542 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 350/20000 — train_mse=3.342970e-05 holdout_mse=3.703696e-05 best=3.703696e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:34:59,004 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 375/20000 — train_mse=3.285554e-05 holdout_mse=3.648386e-05 best=3.648386e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:01,460 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 400/20000 — train_mse=3.277772e-05 holdout_mse=3.598178e-05 best=3.598178e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:03,923 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 425/20000 — train_mse=3.189092e-05 holdout_mse=3.549983e-05 best=3.549983e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:06,386 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 450/20000 — train_mse=3.085283e-05 holdout_mse=3.508480e-05 best=3.508480e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:08,837 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 475/20000 — train_mse=3.050492e-05 holdout_mse=3.465802e-05 best=3.465802e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:11,291 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 500/20000 — train_mse=2.996518e-05 holdout_mse=3.428376e-05 best=3.428376e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:13,747 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 525/20000 — train_mse=2.952806e-05 holdout_mse=3.392731e-05 best=3.392731e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:35:16,213 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 550/20000 — train_mse=2.947739e-05 holdout_mse=3.359392e-05 best=3.359392e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:35:18,678 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 575/20000 — train_mse=2.939328e-05 holdout_mse=3.327323e-05 best=3.327323e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:35:21,149 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 600/20000 — train_mse=2.895416e-05 holdout_mse=3.296671e-05 best=3.296671e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:35:23,518 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 625/20000 — train_mse=2.780195e-05 holdout_mse=3.269676e-05 best=3.269676e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:35:25,885 INFO moe_compress.stage2_reap_ream ::     heal layer 20: step 650/20000 — train_mse=2.884693e-05 holdout_mse=3.243028e-05 best=3.243028e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:35:26,003 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 20: 650 steps (patience) — train_mse=2.884693e-05 (at_best=2.884693e-05) holdout_mse=3.243028e-05 plain_merged_holdout_mse=9.200973e-05 heal_gap=3.583357e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:35:26,139 WARNING moe_compress.stage2_reap_ream ::   layer 20: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:35:28,863 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=58.7321, max_group=9, mean_group=1.00
2026-05-20 03:35:29,227 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 22/40 (idx=21) — profiling then merging to 128 experts
2026-05-20 03:35:29,232 INFO moe_compress.stage2_reap_ream :: DIAG layer 21: entering batch loop (calibration tensor + early-exit forwards) | vram=58.9/150GB host_free=143GB gpu_util=0% temp=32C
2026-05-20 03:35:30,466 INFO moe_compress.stage2_reap_ream :: DIAG layer 21 batch 1: total=1.23s fwd=1.17s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.2s | vram=82.5/150GB host_free=143GB gpu_util=46% temp=38C
2026-05-20 03:35:31,633 INFO moe_compress.stage2_reap_ream :: DIAG layer 21 batch 2: total=1.17s fwd=1.13s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.4s | vram=82.5/150GB host_free=143GB gpu_util=54% temp=41C
2026-05-20 03:35:32,786 INFO moe_compress.stage2_reap_ream :: DIAG layer 21 batch 3: total=1.15s fwd=1.13s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.6s | vram=82.5/150GB host_free=143GB gpu_util=50% temp=41C
2026-05-20 03:35:40,779 INFO moe_compress.stage2_reap_ream :: DIAG layer 21 batch 10: total=1.13s fwd=1.11s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=11.5s | vram=82.5/150GB host_free=143GB gpu_util=55% temp=45C
2026-05-20 03:35:52,251 INFO moe_compress.stage2_reap_ream :: DIAG layer 21 batch 20: total=1.17s fwd=1.10s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=23.0s | vram=82.5/150GB host_free=142GB gpu_util=51% temp=45C
2026-05-20 03:36:03,911 INFO moe_compress.stage2_reap_ream :: DIAG layer 21 batch 30: total=1.17s fwd=1.10s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=34.7s | vram=82.5/150GB host_free=142GB gpu_util=57% temp=46C
2026-05-20 03:36:05,506 INFO moe_compress.stage2_reap_ream :: DIAG layer 21: batch loop complete — 32 batches in 36.3s, now post-profile work | vram=82.5/150GB host_free=142GB gpu_util=79% temp=43C
2026-05-20 03:36:16,615 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 21 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:36:25,126 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 25/20000 — train_mse=6.671264e-05 holdout_mse=6.642712e-05 best=6.642712e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:27,599 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 50/20000 — train_mse=5.645285e-05 holdout_mse=5.713943e-05 best=5.713943e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:30,061 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 75/20000 — train_mse=5.167733e-05 holdout_mse=5.186586e-05 best=5.186586e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:32,673 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 100/20000 — train_mse=4.736832e-05 holdout_mse=4.839606e-05 best=4.839606e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:35,135 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 125/20000 — train_mse=4.529411e-05 holdout_mse=4.585993e-05 best=4.585993e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:37,633 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 150/20000 — train_mse=4.306402e-05 holdout_mse=4.381544e-05 best=4.381544e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:40,060 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 175/20000 — train_mse=3.964981e-05 holdout_mse=4.216069e-05 best=4.216069e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:42,556 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 200/20000 — train_mse=3.924482e-05 holdout_mse=4.076355e-05 best=4.076355e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:45,035 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 225/20000 — train_mse=3.736689e-05 holdout_mse=3.963298e-05 best=3.963298e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:47,540 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 250/20000 — train_mse=3.712116e-05 holdout_mse=3.856445e-05 best=3.856445e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:50,022 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 275/20000 — train_mse=3.512015e-05 holdout_mse=3.768699e-05 best=3.768699e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:52,506 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 300/20000 — train_mse=3.445601e-05 holdout_mse=3.688288e-05 best=3.688288e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:54,997 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 325/20000 — train_mse=3.430751e-05 holdout_mse=3.615760e-05 best=3.615760e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:57,485 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 350/20000 — train_mse=3.226991e-05 holdout_mse=3.550806e-05 best=3.550806e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:36:59,976 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 375/20000 — train_mse=3.158122e-05 holdout_mse=3.493883e-05 best=3.493883e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:02,469 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 400/20000 — train_mse=3.096174e-05 holdout_mse=3.439544e-05 best=3.439544e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:04,964 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 425/20000 — train_mse=3.138622e-05 holdout_mse=3.390564e-05 best=3.390564e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:07,482 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 450/20000 — train_mse=3.056735e-05 holdout_mse=3.343263e-05 best=3.343263e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:09,972 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 475/20000 — train_mse=2.988309e-05 holdout_mse=3.301884e-05 best=3.301884e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:12,455 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 500/20000 — train_mse=2.897911e-05 holdout_mse=3.260537e-05 best=3.260537e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:14,944 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 525/20000 — train_mse=2.874113e-05 holdout_mse=3.224105e-05 best=3.224105e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:17,442 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 550/20000 — train_mse=2.882010e-05 holdout_mse=3.187869e-05 best=3.187869e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:19,920 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 575/20000 — train_mse=2.859169e-05 holdout_mse=3.155186e-05 best=3.155186e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:37:22,411 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 600/20000 — train_mse=2.762897e-05 holdout_mse=3.124514e-05 best=3.124514e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:37:24,905 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 625/20000 — train_mse=2.706191e-05 holdout_mse=3.096696e-05 best=3.096696e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:37:27,394 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 650/20000 — train_mse=2.726825e-05 holdout_mse=3.067793e-05 best=3.067793e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:37:29,881 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 675/20000 — train_mse=2.666227e-05 holdout_mse=3.040543e-05 best=3.040543e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:37:32,366 INFO moe_compress.stage2_reap_ream ::     heal layer 21: step 700/20000 — train_mse=2.642220e-05 holdout_mse=3.016469e-05 best=3.016469e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:37:32,471 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 21: 700 steps (patience) — train_mse=2.642220e-05 (at_best=2.642220e-05) holdout_mse=3.016469e-05 plain_merged_holdout_mse=8.742873e-05 heal_gap=3.742490e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:37:32,607 WARNING moe_compress.stage2_reap_ream ::   layer 21: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:37:35,229 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=1, ream_centroids=127) — Σ cost=59.9222, max_group=9, mean_group=1.01
2026-05-20 03:37:35,592 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 23/40 (idx=22) — profiling then merging to 128 experts
2026-05-20 03:37:35,597 INFO moe_compress.stage2_reap_ream :: DIAG layer 22: entering batch loop (calibration tensor + early-exit forwards) | vram=58.1/150GB host_free=141GB gpu_util=6% temp=32C
2026-05-20 03:37:36,851 INFO moe_compress.stage2_reap_ream :: DIAG layer 22 batch 1: total=1.25s fwd=1.21s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.3s | vram=81.6/150GB host_free=141GB gpu_util=48% temp=38C
2026-05-20 03:37:38,041 INFO moe_compress.stage2_reap_ream :: DIAG layer 22 batch 2: total=1.19s fwd=1.16s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.4s | vram=81.6/150GB host_free=141GB gpu_util=55% temp=40C
2026-05-20 03:37:39,223 INFO moe_compress.stage2_reap_ream :: DIAG layer 22 batch 3: total=1.18s fwd=1.15s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.6s | vram=81.6/150GB host_free=141GB gpu_util=54% temp=41C
2026-05-20 03:37:47,443 INFO moe_compress.stage2_reap_ream :: DIAG layer 22 batch 10: total=1.17s fwd=1.14s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=11.8s | vram=81.6/150GB host_free=141GB gpu_util=58% temp=45C
2026-05-20 03:37:59,260 INFO moe_compress.stage2_reap_ream :: DIAG layer 22 batch 20: total=1.20s fwd=1.14s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=23.7s | vram=81.6/150GB host_free=141GB gpu_util=62% temp=45C
2026-05-20 03:38:11,277 INFO moe_compress.stage2_reap_ream :: DIAG layer 22 batch 30: total=1.20s fwd=1.14s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=35.7s | vram=81.6/150GB host_free=140GB gpu_util=55% temp=46C
2026-05-20 03:38:12,920 INFO moe_compress.stage2_reap_ream :: DIAG layer 22: batch loop complete — 32 batches in 37.3s, now post-profile work | vram=81.7/150GB host_free=140GB gpu_util=87% temp=44C
2026-05-20 03:38:24,293 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 22 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:38:32,017 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 25/20000 — train_mse=6.905147e-05 holdout_mse=7.202899e-05 best=7.202899e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:34,353 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 50/20000 — train_mse=5.558254e-05 holdout_mse=5.448572e-05 best=5.448572e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:36,698 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 75/20000 — train_mse=4.865021e-05 holdout_mse=4.866318e-05 best=4.866318e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:39,035 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 100/20000 — train_mse=4.492072e-05 holdout_mse=4.548140e-05 best=4.548140e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:41,371 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 125/20000 — train_mse=4.166908e-05 holdout_mse=4.305480e-05 best=4.305480e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:43,711 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 150/20000 — train_mse=3.886946e-05 holdout_mse=4.121072e-05 best=4.121072e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:46,053 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 175/20000 — train_mse=3.738785e-05 holdout_mse=3.964694e-05 best=3.964694e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:48,402 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 200/20000 — train_mse=3.660892e-05 holdout_mse=3.842341e-05 best=3.842341e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:50,755 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 225/20000 — train_mse=3.560141e-05 holdout_mse=3.727764e-05 best=3.727764e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:53,096 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 250/20000 — train_mse=3.376161e-05 holdout_mse=3.636651e-05 best=3.636651e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:55,441 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 275/20000 — train_mse=3.252248e-05 holdout_mse=3.556819e-05 best=3.556819e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:38:57,790 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 300/20000 — train_mse=3.115233e-05 holdout_mse=3.483923e-05 best=3.483923e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:00,130 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 325/20000 — train_mse=3.115871e-05 holdout_mse=3.415672e-05 best=3.415672e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:02,470 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 350/20000 — train_mse=3.041540e-05 holdout_mse=3.358069e-05 best=3.358069e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:04,811 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 375/20000 — train_mse=2.926261e-05 holdout_mse=3.307163e-05 best=3.307163e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:07,175 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 400/20000 — train_mse=2.932220e-05 holdout_mse=3.257287e-05 best=3.257287e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:09,540 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 425/20000 — train_mse=2.855490e-05 holdout_mse=3.215907e-05 best=3.215907e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:11,889 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 450/20000 — train_mse=2.843993e-05 holdout_mse=3.172436e-05 best=3.172436e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:14,238 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 475/20000 — train_mse=2.761674e-05 holdout_mse=3.134333e-05 best=3.134333e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:16,579 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 500/20000 — train_mse=2.721074e-05 holdout_mse=3.096920e-05 best=3.096920e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:18,935 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 525/20000 — train_mse=2.621855e-05 holdout_mse=3.064535e-05 best=3.064535e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:21,284 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 550/20000 — train_mse=2.617332e-05 holdout_mse=3.030612e-05 best=3.030612e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:39:23,630 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 575/20000 — train_mse=2.608439e-05 holdout_mse=3.002816e-05 best=3.002816e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:39:25,967 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 600/20000 — train_mse=2.560750e-05 holdout_mse=2.975128e-05 best=2.975128e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:39:28,313 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 625/20000 — train_mse=2.532384e-05 holdout_mse=2.948679e-05 best=2.948679e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:39:30,660 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 650/20000 — train_mse=2.577556e-05 holdout_mse=2.923800e-05 best=2.923800e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:39:33,004 INFO moe_compress.stage2_reap_ream ::     heal layer 22: step 675/20000 — train_mse=2.448682e-05 holdout_mse=2.899556e-05 best=2.899556e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:39:33,106 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 22: 675 steps (patience) — train_mse=2.448682e-05 (at_best=2.448682e-05) holdout_mse=2.899556e-05 plain_merged_holdout_mse=1.036643e-04 heal_gap=4.508747e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:39:33,241 WARNING moe_compress.stage2_reap_ream ::   layer 22: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:39:35,642 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=61.1902, max_group=9, mean_group=1.00
2026-05-20 03:39:36,012 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 24/40 (idx=23) — profiling then merging to 128 experts
2026-05-20 03:39:36,017 INFO moe_compress.stage2_reap_ream :: DIAG layer 23: entering batch loop (calibration tensor + early-exit forwards) | vram=57.2/150GB host_free=140GB gpu_util=0% temp=32C
2026-05-20 03:39:37,304 INFO moe_compress.stage2_reap_ream :: DIAG layer 23 batch 1: total=1.29s fwd=1.24s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.3s | vram=80.8/150GB host_free=140GB gpu_util=47% temp=39C
2026-05-20 03:39:38,530 INFO moe_compress.stage2_reap_ream :: DIAG layer 23 batch 2: total=1.23s fwd=1.20s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.5s | vram=80.8/150GB host_free=140GB gpu_util=54% temp=41C
2026-05-20 03:39:39,751 INFO moe_compress.stage2_reap_ream :: DIAG layer 23 batch 3: total=1.22s fwd=1.19s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.7s | vram=80.8/150GB host_free=140GB gpu_util=54% temp=41C
2026-05-20 03:39:48,240 INFO moe_compress.stage2_reap_ream :: DIAG layer 23 batch 10: total=1.20s fwd=1.18s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=12.2s | vram=80.8/150GB host_free=140GB gpu_util=57% temp=44C
2026-05-20 03:40:00,407 INFO moe_compress.stage2_reap_ream :: DIAG layer 23 batch 20: total=1.24s fwd=1.18s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=24.4s | vram=80.8/150GB host_free=140GB gpu_util=60% temp=46C
2026-05-20 03:40:12,788 INFO moe_compress.stage2_reap_ream :: DIAG layer 23 batch 30: total=1.24s fwd=1.18s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=36.8s | vram=80.8/150GB host_free=139GB gpu_util=58% temp=46C
2026-05-20 03:40:14,466 INFO moe_compress.stage2_reap_ream :: DIAG layer 23: batch loop complete — 32 batches in 38.5s, now post-profile work | vram=80.9/150GB host_free=139GB gpu_util=36% temp=45C
2026-05-20 03:40:25,698 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 23 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:40:33,486 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 25/20000 — train_mse=6.769756e-05 holdout_mse=6.660392e-05 best=6.660392e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:35,817 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 50/20000 — train_mse=5.901029e-05 holdout_mse=6.023381e-05 best=6.023381e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:38,143 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 75/20000 — train_mse=5.516428e-05 holdout_mse=5.632774e-05 best=5.632774e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:40,467 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 100/20000 — train_mse=5.271713e-05 holdout_mse=5.351972e-05 best=5.351972e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:42,809 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 125/20000 — train_mse=4.933825e-05 holdout_mse=5.130651e-05 best=5.130651e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:45,139 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 150/20000 — train_mse=4.615612e-05 holdout_mse=4.951569e-05 best=4.951569e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:47,500 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 175/20000 — train_mse=4.586056e-05 holdout_mse=4.801917e-05 best=4.801917e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:49,834 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 200/20000 — train_mse=4.271469e-05 holdout_mse=4.671377e-05 best=4.671377e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:52,175 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 225/20000 — train_mse=4.166811e-05 holdout_mse=4.557731e-05 best=4.557731e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:54,539 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 250/20000 — train_mse=4.124024e-05 holdout_mse=4.456703e-05 best=4.456703e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:56,873 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 275/20000 — train_mse=4.057035e-05 holdout_mse=4.366989e-05 best=4.366989e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:40:59,214 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 300/20000 — train_mse=3.872582e-05 holdout_mse=4.287319e-05 best=4.287319e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:01,560 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 325/20000 — train_mse=3.854490e-05 holdout_mse=4.215176e-05 best=4.215176e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:03,894 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 350/20000 — train_mse=3.800608e-05 holdout_mse=4.147927e-05 best=4.147927e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:06,238 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 375/20000 — train_mse=3.678246e-05 holdout_mse=4.088471e-05 best=4.088471e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:08,591 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 400/20000 — train_mse=3.568027e-05 holdout_mse=4.033546e-05 best=4.033546e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:10,932 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 425/20000 — train_mse=3.519962e-05 holdout_mse=3.979840e-05 best=3.979840e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:13,273 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 450/20000 — train_mse=3.435821e-05 holdout_mse=3.930753e-05 best=3.930753e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:15,616 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 475/20000 — train_mse=3.420921e-05 holdout_mse=3.887039e-05 best=3.887039e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:17,975 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 500/20000 — train_mse=3.445302e-05 holdout_mse=3.845715e-05 best=3.845715e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:20,323 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 525/20000 — train_mse=3.317404e-05 holdout_mse=3.805551e-05 best=3.805551e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:41:22,661 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 550/20000 — train_mse=3.171150e-05 holdout_mse=3.768243e-05 best=3.768243e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:41:25,005 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 575/20000 — train_mse=3.247752e-05 holdout_mse=3.732710e-05 best=3.732710e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:41:27,337 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 600/20000 — train_mse=3.209012e-05 holdout_mse=3.697846e-05 best=3.697846e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:41:29,673 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 625/20000 — train_mse=3.072167e-05 holdout_mse=3.667318e-05 best=3.667318e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:41:32,003 INFO moe_compress.stage2_reap_ream ::     heal layer 23: step 650/20000 — train_mse=3.016584e-05 holdout_mse=3.638359e-05 best=3.638359e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:41:32,103 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 23: 650 steps (patience) — train_mse=3.016584e-05 (at_best=3.016584e-05) holdout_mse=3.638359e-05 plain_merged_holdout_mse=8.498277e-05 heal_gap=6.217750e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:41:32,239 WARNING moe_compress.stage2_reap_ream ::   layer 23: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:41:34,734 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=62.2373, max_group=9, mean_group=1.00
2026-05-20 03:41:35,103 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 25/40 (idx=24) — profiling then merging to 128 experts
2026-05-20 03:41:35,108 INFO moe_compress.stage2_reap_ream :: DIAG layer 24: entering batch loop (calibration tensor + early-exit forwards) | vram=56.4/150GB host_free=139GB gpu_util=0% temp=32C
2026-05-20 03:41:36,431 INFO moe_compress.stage2_reap_ream :: DIAG layer 24 batch 1: total=1.32s fwd=1.28s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.3s | vram=80.0/150GB host_free=139GB gpu_util=48% temp=38C
2026-05-20 03:41:37,693 INFO moe_compress.stage2_reap_ream :: DIAG layer 24 batch 2: total=1.26s fwd=1.23s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.6s | vram=80.0/150GB host_free=139GB gpu_util=52% temp=40C
2026-05-20 03:41:38,952 INFO moe_compress.stage2_reap_ream :: DIAG layer 24 batch 3: total=1.26s fwd=1.23s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.8s | vram=80.0/150GB host_free=139GB gpu_util=53% temp=42C
2026-05-20 03:41:47,704 INFO moe_compress.stage2_reap_ream :: DIAG layer 24 batch 10: total=1.24s fwd=1.22s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=12.6s | vram=80.0/150GB host_free=139GB gpu_util=55% temp=44C
2026-05-20 03:42:00,260 INFO moe_compress.stage2_reap_ream :: DIAG layer 24 batch 20: total=1.27s fwd=1.21s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=25.2s | vram=80.0/150GB host_free=139GB gpu_util=59% temp=46C
2026-05-20 03:42:13,015 INFO moe_compress.stage2_reap_ream :: DIAG layer 24 batch 30: total=1.28s fwd=1.22s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=37.9s | vram=80.0/150GB host_free=138GB gpu_util=55% temp=46C
2026-05-20 03:42:14,755 INFO moe_compress.stage2_reap_ream :: DIAG layer 24: batch loop complete — 32 batches in 39.6s, now post-profile work | vram=80.1/150GB host_free=138GB gpu_util=46% temp=44C
2026-05-20 03:42:26,145 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 24 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:42:33,935 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 25/20000 — train_mse=5.613586e-05 holdout_mse=5.680726e-05 best=5.680726e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:36,394 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 50/20000 — train_mse=4.966346e-05 holdout_mse=5.113759e-05 best=5.113759e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:38,864 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 75/20000 — train_mse=4.489659e-05 holdout_mse=4.767720e-05 best=4.767720e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:41,325 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 100/20000 — train_mse=4.235019e-05 holdout_mse=4.523698e-05 best=4.523698e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:43,794 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 125/20000 — train_mse=4.412766e-05 holdout_mse=4.334643e-05 best=4.334643e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:46,262 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 150/20000 — train_mse=3.985090e-05 holdout_mse=4.186569e-05 best=4.186569e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:48,723 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 175/20000 — train_mse=3.880249e-05 holdout_mse=4.061894e-05 best=4.061894e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:51,186 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 200/20000 — train_mse=3.673420e-05 holdout_mse=3.958912e-05 best=3.958912e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:53,646 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 225/20000 — train_mse=3.547299e-05 holdout_mse=3.870682e-05 best=3.870682e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:56,121 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 250/20000 — train_mse=3.472170e-05 holdout_mse=3.795640e-05 best=3.795640e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:42:58,595 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 275/20000 — train_mse=3.259275e-05 holdout_mse=3.725281e-05 best=3.725281e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:01,060 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 300/20000 — train_mse=3.297181e-05 holdout_mse=3.663834e-05 best=3.663834e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:03,518 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 325/20000 — train_mse=3.252708e-05 holdout_mse=3.607034e-05 best=3.607034e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:05,983 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 350/20000 — train_mse=3.256193e-05 holdout_mse=3.553852e-05 best=3.553852e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:08,464 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 375/20000 — train_mse=3.079874e-05 holdout_mse=3.507248e-05 best=3.507248e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:10,927 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 400/20000 — train_mse=3.093366e-05 holdout_mse=3.462541e-05 best=3.462541e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:13,395 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 425/20000 — train_mse=3.033039e-05 holdout_mse=3.421946e-05 best=3.421946e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:15,858 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 450/20000 — train_mse=2.976316e-05 holdout_mse=3.382736e-05 best=3.382736e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:18,320 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 475/20000 — train_mse=2.929642e-05 holdout_mse=3.347246e-05 best=3.347246e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:43:20,793 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 500/20000 — train_mse=2.848870e-05 holdout_mse=3.314601e-05 best=3.314601e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:43:23,262 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 525/20000 — train_mse=2.935191e-05 holdout_mse=3.282313e-05 best=3.282313e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:43:25,735 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 550/20000 — train_mse=2.801290e-05 holdout_mse=3.254364e-05 best=3.254364e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:43:28,217 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 575/20000 — train_mse=2.693223e-05 holdout_mse=3.226652e-05 best=3.226652e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:43:30,689 INFO moe_compress.stage2_reap_ream ::     heal layer 24: step 600/20000 — train_mse=2.690721e-05 holdout_mse=3.199734e-05 best=3.199734e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:43:30,787 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 24: 600 steps (patience) — train_mse=2.690721e-05 (at_best=2.690721e-05) holdout_mse=3.199734e-05 plain_merged_holdout_mse=7.430541e-05 heal_gap=5.090137e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:43:30,923 WARNING moe_compress.stage2_reap_ream ::   layer 24: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:43:33,410 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=63.2114, max_group=9, mean_group=1.00
2026-05-20 03:43:33,775 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 26/40 (idx=25) — profiling then merging to 128 experts
2026-05-20 03:43:33,780 INFO moe_compress.stage2_reap_ream :: DIAG layer 25: entering batch loop (calibration tensor + early-exit forwards) | vram=55.7/150GB host_free=138GB gpu_util=0% temp=32C
2026-05-20 03:43:35,141 INFO moe_compress.stage2_reap_ream :: DIAG layer 25 batch 1: total=1.36s fwd=1.31s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.4s | vram=79.2/150GB host_free=138GB gpu_util=44% temp=39C
2026-05-20 03:43:36,438 INFO moe_compress.stage2_reap_ream :: DIAG layer 25 batch 2: total=1.30s fwd=1.27s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.7s | vram=79.2/150GB host_free=138GB gpu_util=54% temp=40C
2026-05-20 03:43:37,736 INFO moe_compress.stage2_reap_ream :: DIAG layer 25 batch 3: total=1.30s fwd=1.27s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.0s | vram=79.2/150GB host_free=138GB gpu_util=54% temp=41C
2026-05-20 03:43:46,752 INFO moe_compress.stage2_reap_ream :: DIAG layer 25 batch 10: total=1.28s fwd=1.25s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=13.0s | vram=79.2/150GB host_free=138GB gpu_util=58% temp=44C
2026-05-20 03:43:59,671 INFO moe_compress.stage2_reap_ream :: DIAG layer 25 batch 20: total=1.31s fwd=1.25s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=25.9s | vram=79.2/150GB host_free=138GB gpu_util=55% temp=46C
2026-05-20 03:44:12,794 INFO moe_compress.stage2_reap_ream :: DIAG layer 25 batch 30: total=1.31s fwd=1.25s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=39.0s | vram=79.2/150GB host_free=137GB gpu_util=58% temp=47C
2026-05-20 03:44:14,573 INFO moe_compress.stage2_reap_ream :: DIAG layer 25: batch loop complete — 32 batches in 40.8s, now post-profile work | vram=79.3/150GB host_free=137GB gpu_util=61% temp=45C
2026-05-20 03:44:26,185 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 25 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:44:34,420 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 25/20000 — train_mse=7.589444e-05 holdout_mse=7.761883e-05 best=7.761883e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:36,887 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 50/20000 — train_mse=6.596332e-05 holdout_mse=6.982431e-05 best=6.982431e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:39,350 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 75/20000 — train_mse=6.374424e-05 holdout_mse=6.513511e-05 best=6.513511e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:41,817 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 100/20000 — train_mse=5.901771e-05 holdout_mse=6.175220e-05 best=6.175220e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:44,277 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 125/20000 — train_mse=5.643582e-05 holdout_mse=5.915355e-05 best=5.915355e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:46,748 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 150/20000 — train_mse=5.415641e-05 holdout_mse=5.706545e-05 best=5.706545e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:49,219 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 175/20000 — train_mse=5.110505e-05 holdout_mse=5.531413e-05 best=5.531413e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:51,689 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 200/20000 — train_mse=4.825779e-05 holdout_mse=5.382966e-05 best=5.382966e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:54,157 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 225/20000 — train_mse=4.830930e-05 holdout_mse=5.253841e-05 best=5.253841e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:56,627 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 250/20000 — train_mse=4.585859e-05 holdout_mse=5.140307e-05 best=5.140307e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:44:59,090 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 275/20000 — train_mse=4.598778e-05 holdout_mse=5.039605e-05 best=5.039605e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:01,566 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 300/20000 — train_mse=4.567391e-05 holdout_mse=4.949683e-05 best=4.949683e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:04,010 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 325/20000 — train_mse=4.342992e-05 holdout_mse=4.867720e-05 best=4.867720e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:06,446 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 350/20000 — train_mse=4.212364e-05 holdout_mse=4.796093e-05 best=4.796093e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:08,910 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 375/20000 — train_mse=4.167123e-05 holdout_mse=4.727738e-05 best=4.727738e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:11,350 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 400/20000 — train_mse=4.043448e-05 holdout_mse=4.665612e-05 best=4.665612e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:13,785 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 425/20000 — train_mse=3.960908e-05 holdout_mse=4.607178e-05 best=4.607178e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:16,229 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 450/20000 — train_mse=3.985332e-05 holdout_mse=4.556195e-05 best=4.556195e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:18,672 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 475/20000 — train_mse=3.803470e-05 holdout_mse=4.507703e-05 best=4.507703e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:21,126 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 500/20000 — train_mse=3.742945e-05 holdout_mse=4.458707e-05 best=4.458707e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:45:23,572 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 525/20000 — train_mse=3.834353e-05 holdout_mse=4.418469e-05 best=4.418469e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:45:26,018 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 550/20000 — train_mse=3.790704e-05 holdout_mse=4.376166e-05 best=4.376166e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:45:28,452 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 575/20000 — train_mse=3.713855e-05 holdout_mse=4.336709e-05 best=4.336709e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:45:30,900 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 600/20000 — train_mse=3.559443e-05 holdout_mse=4.298865e-05 best=4.298865e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:45:33,344 INFO moe_compress.stage2_reap_ream ::     heal layer 25: step 625/20000 — train_mse=3.532260e-05 holdout_mse=4.265428e-05 best=4.265428e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:45:33,447 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 25: 625 steps (patience) — train_mse=3.532260e-05 (at_best=3.532260e-05) holdout_mse=4.265428e-05 plain_merged_holdout_mse=1.034209e-04 heal_gap=7.331685e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:45:33,583 WARNING moe_compress.stage2_reap_ream ::   layer 25: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:45:36,083 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=59.7876, max_group=9, mean_group=1.00
2026-05-20 03:45:36,451 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 27/40 (idx=26) — profiling then merging to 128 experts
2026-05-20 03:45:36,456 INFO moe_compress.stage2_reap_ream :: DIAG layer 26: entering batch loop (calibration tensor + early-exit forwards) | vram=54.9/150GB host_free=137GB gpu_util=0% temp=32C
2026-05-20 03:45:37,852 INFO moe_compress.stage2_reap_ream :: DIAG layer 26 batch 1: total=1.40s fwd=1.35s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.4s | vram=78.5/150GB host_free=137GB gpu_util=48% temp=38C
2026-05-20 03:45:39,186 INFO moe_compress.stage2_reap_ream :: DIAG layer 26 batch 2: total=1.33s fwd=1.31s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.7s | vram=78.5/150GB host_free=137GB gpu_util=53% temp=41C
2026-05-20 03:45:40,513 INFO moe_compress.stage2_reap_ream :: DIAG layer 26 batch 3: total=1.33s fwd=1.30s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.1s | vram=78.5/150GB host_free=137GB gpu_util=54% temp=43C
2026-05-20 03:45:49,768 INFO moe_compress.stage2_reap_ream :: DIAG layer 26 batch 10: total=1.32s fwd=1.29s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=13.3s | vram=78.5/150GB host_free=137GB gpu_util=56% temp=45C
2026-05-20 03:46:03,040 INFO moe_compress.stage2_reap_ream :: DIAG layer 26 batch 20: total=1.35s fwd=1.29s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=26.6s | vram=78.5/150GB host_free=137GB gpu_util=56% temp=47C
2026-05-20 03:46:16,498 INFO moe_compress.stage2_reap_ream :: DIAG layer 26 batch 30: total=1.34s fwd=1.28s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=40.0s | vram=78.5/150GB host_free=136GB gpu_util=57% temp=45C
2026-05-20 03:46:18,320 INFO moe_compress.stage2_reap_ream :: DIAG layer 26: batch loop complete — 32 batches in 41.9s, now post-profile work | vram=78.5/150GB host_free=136GB gpu_util=90% temp=44C
2026-05-20 03:46:29,999 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 26 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:46:38,238 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 25/20000 — train_mse=7.566283e-05 holdout_mse=7.446747e-05 best=7.446747e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:40,673 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 50/20000 — train_mse=6.599084e-05 holdout_mse=6.624641e-05 best=6.624641e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:43,102 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 75/20000 — train_mse=6.200644e-05 holdout_mse=6.155091e-05 best=6.155091e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:45,534 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 100/20000 — train_mse=5.805273e-05 holdout_mse=5.824277e-05 best=5.824277e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:47,973 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 125/20000 — train_mse=5.474965e-05 holdout_mse=5.569399e-05 best=5.569399e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:50,417 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 150/20000 — train_mse=5.121613e-05 holdout_mse=5.357376e-05 best=5.357376e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:52,866 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 175/20000 — train_mse=5.000030e-05 holdout_mse=5.184368e-05 best=5.184368e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:55,305 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 200/20000 — train_mse=4.717996e-05 holdout_mse=5.042079e-05 best=5.042079e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:46:57,747 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 225/20000 — train_mse=4.680057e-05 holdout_mse=4.915755e-05 best=4.915755e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:00,197 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 250/20000 — train_mse=4.519416e-05 holdout_mse=4.809257e-05 best=4.809257e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:02,644 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 275/20000 — train_mse=4.468091e-05 holdout_mse=4.712230e-05 best=4.712230e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:05,097 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 300/20000 — train_mse=4.326990e-05 holdout_mse=4.624295e-05 best=4.624295e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:07,541 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 325/20000 — train_mse=4.200007e-05 holdout_mse=4.546693e-05 best=4.546693e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:09,989 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 350/20000 — train_mse=4.057678e-05 holdout_mse=4.475139e-05 best=4.475139e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:12,429 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 375/20000 — train_mse=4.135781e-05 holdout_mse=4.409874e-05 best=4.409874e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:14,870 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 400/20000 — train_mse=3.963085e-05 holdout_mse=4.352411e-05 best=4.352411e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:17,306 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 425/20000 — train_mse=3.787400e-05 holdout_mse=4.298886e-05 best=4.298886e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:19,770 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 450/20000 — train_mse=3.760481e-05 holdout_mse=4.247366e-05 best=4.247366e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:22,215 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 475/20000 — train_mse=3.723905e-05 holdout_mse=4.201004e-05 best=4.201004e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:24,679 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 500/20000 — train_mse=3.664310e-05 holdout_mse=4.157008e-05 best=4.157008e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:47:27,122 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 525/20000 — train_mse=3.640143e-05 holdout_mse=4.115620e-05 best=4.115620e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:47:29,579 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 550/20000 — train_mse=3.619046e-05 holdout_mse=4.075826e-05 best=4.075826e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:47:32,021 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 575/20000 — train_mse=3.529385e-05 holdout_mse=4.038339e-05 best=4.038339e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:47:34,459 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 600/20000 — train_mse=3.432110e-05 holdout_mse=4.004590e-05 best=4.004590e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:47:36,900 INFO moe_compress.stage2_reap_ream ::     heal layer 26: step 625/20000 — train_mse=3.450051e-05 holdout_mse=3.973433e-05 best=3.973433e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:47:36,997 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 26: 625 steps (patience) — train_mse=3.450051e-05 (at_best=3.450051e-05) holdout_mse=3.973433e-05 plain_merged_holdout_mse=1.006836e-04 heal_gap=5.233820e-06 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:47:37,135 WARNING moe_compress.stage2_reap_ream ::   layer 26: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:47:39,563 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=61.6315, max_group=9, mean_group=1.00
2026-05-20 03:47:39,937 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 28/40 (idx=27) — profiling then merging to 128 experts
2026-05-20 03:47:39,941 INFO moe_compress.stage2_reap_ream :: DIAG layer 27: entering batch loop (calibration tensor + early-exit forwards) | vram=54.1/150GB host_free=136GB gpu_util=0% temp=32C
2026-05-20 03:47:41,373 INFO moe_compress.stage2_reap_ream :: DIAG layer 27 batch 1: total=1.43s fwd=1.39s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.4s | vram=77.7/150GB host_free=136GB gpu_util=49% temp=39C
2026-05-20 03:47:42,745 INFO moe_compress.stage2_reap_ream :: DIAG layer 27 batch 2: total=1.37s fwd=1.34s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.8s | vram=77.7/150GB host_free=136GB gpu_util=53% temp=41C
2026-05-20 03:47:44,114 INFO moe_compress.stage2_reap_ream :: DIAG layer 27 batch 3: total=1.37s fwd=1.34s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.2s | vram=77.7/150GB host_free=136GB gpu_util=58% temp=41C
2026-05-20 03:47:53,653 INFO moe_compress.stage2_reap_ream :: DIAG layer 27 batch 10: total=1.35s fwd=1.33s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=13.7s | vram=77.7/150GB host_free=136GB gpu_util=59% temp=45C
2026-05-20 03:48:07,306 INFO moe_compress.stage2_reap_ream :: DIAG layer 27 batch 20: total=1.39s fwd=1.32s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=27.4s | vram=77.7/150GB host_free=136GB gpu_util=60% temp=46C
2026-05-20 03:48:21,173 INFO moe_compress.stage2_reap_ream :: DIAG layer 27 batch 30: total=1.38s fwd=1.33s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=41.2s | vram=77.7/150GB host_free=135GB gpu_util=57% temp=46C
2026-05-20 03:48:23,044 INFO moe_compress.stage2_reap_ream :: DIAG layer 27: batch loop complete — 32 batches in 43.1s, now post-profile work | vram=77.7/150GB host_free=135GB gpu_util=44% temp=45C
2026-05-20 03:48:34,879 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 27 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:48:43,326 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 25/20000 — train_mse=1.002853e-04 holdout_mse=1.019956e-04 best=1.019956e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:48:45,779 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 50/20000 — train_mse=9.046806e-05 holdout_mse=9.201448e-05 best=9.201448e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:48:48,225 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 75/20000 — train_mse=8.286805e-05 holdout_mse=8.579195e-05 best=8.579195e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:48:50,677 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 100/20000 — train_mse=7.771821e-05 holdout_mse=8.134305e-05 best=8.134305e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:48:53,130 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 125/20000 — train_mse=7.427535e-05 holdout_mse=7.796042e-05 best=7.796042e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:48:55,576 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 150/20000 — train_mse=7.179251e-05 holdout_mse=7.521620e-05 best=7.521620e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:48:58,015 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 175/20000 — train_mse=6.945026e-05 holdout_mse=7.301143e-05 best=7.301143e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:00,467 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 200/20000 — train_mse=6.650721e-05 holdout_mse=7.114558e-05 best=7.114558e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:02,912 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 225/20000 — train_mse=6.386101e-05 holdout_mse=6.949550e-05 best=6.949550e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:05,357 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 250/20000 — train_mse=6.246004e-05 holdout_mse=6.803824e-05 best=6.803824e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:07,808 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 275/20000 — train_mse=6.002194e-05 holdout_mse=6.675010e-05 best=6.675010e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:10,310 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 300/20000 — train_mse=6.077479e-05 holdout_mse=6.562438e-05 best=6.562438e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:12,759 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 325/20000 — train_mse=5.854445e-05 holdout_mse=6.456575e-05 best=6.456575e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:15,208 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 350/20000 — train_mse=5.740305e-05 holdout_mse=6.364774e-05 best=6.364774e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:17,658 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 375/20000 — train_mse=5.548165e-05 holdout_mse=6.280698e-05 best=6.280698e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:20,109 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 400/20000 — train_mse=5.519543e-05 holdout_mse=6.204829e-05 best=6.204829e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:22,567 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 425/20000 — train_mse=5.368798e-05 holdout_mse=6.128004e-05 best=6.128004e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:25,020 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 450/20000 — train_mse=5.307142e-05 holdout_mse=6.062304e-05 best=6.062304e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:27,470 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 475/20000 — train_mse=5.257640e-05 holdout_mse=6.002311e-05 best=6.002311e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:49:29,927 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 500/20000 — train_mse=5.178550e-05 holdout_mse=5.944521e-05 best=5.944521e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:49:32,382 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 525/20000 — train_mse=5.076496e-05 holdout_mse=5.883734e-05 best=5.883734e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:49:34,837 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 550/20000 — train_mse=4.953342e-05 holdout_mse=5.835006e-05 best=5.835006e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:49:37,289 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 575/20000 — train_mse=4.876955e-05 holdout_mse=5.784403e-05 best=5.784403e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:49:39,747 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 600/20000 — train_mse=4.728517e-05 holdout_mse=5.744296e-05 best=5.744296e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:49:42,206 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 625/20000 — train_mse=4.745862e-05 holdout_mse=5.696451e-05 best=5.696451e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:49:44,653 INFO moe_compress.stage2_reap_ream ::     heal layer 27: step 650/20000 — train_mse=4.608815e-05 holdout_mse=5.657245e-05 best=5.657245e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:49:44,753 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 27: 650 steps (patience) — train_mse=4.608815e-05 (at_best=4.608815e-05) holdout_mse=5.657245e-05 plain_merged_holdout_mse=1.292963e-04 heal_gap=1.048430e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:49:44,891 WARNING moe_compress.stage2_reap_ream ::   layer 27: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:49:47,411 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=64.7812, max_group=9, mean_group=1.00
2026-05-20 03:49:47,783 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 29/40 (idx=28) — profiling then merging to 128 experts
2026-05-20 03:49:47,788 INFO moe_compress.stage2_reap_ream :: DIAG layer 28: entering batch loop (calibration tensor + early-exit forwards) | vram=53.3/150GB host_free=135GB gpu_util=0% temp=32C
2026-05-20 03:49:49,277 INFO moe_compress.stage2_reap_ream :: DIAG layer 28 batch 1: total=1.49s fwd=1.43s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.5s | vram=76.8/150GB host_free=135GB gpu_util=47% temp=39C
2026-05-20 03:49:50,689 INFO moe_compress.stage2_reap_ream :: DIAG layer 28 batch 2: total=1.41s fwd=1.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=2.9s | vram=76.8/150GB host_free=135GB gpu_util=52% temp=41C
2026-05-20 03:49:52,096 INFO moe_compress.stage2_reap_ream :: DIAG layer 28 batch 3: total=1.41s fwd=1.38s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.3s | vram=76.8/150GB host_free=135GB gpu_util=54% temp=42C
2026-05-20 03:50:01,909 INFO moe_compress.stage2_reap_ream :: DIAG layer 28 batch 10: total=1.39s fwd=1.36s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.1s | vram=76.9/150GB host_free=135GB gpu_util=57% temp=46C
2026-05-20 03:50:16,080 INFO moe_compress.stage2_reap_ream :: DIAG layer 28 batch 20: total=1.43s fwd=1.36s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=28.3s | vram=76.9/150GB host_free=135GB gpu_util=50% temp=47C
2026-05-20 03:50:30,459 INFO moe_compress.stage2_reap_ream :: DIAG layer 28 batch 30: total=1.44s fwd=1.37s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=42.7s | vram=76.9/150GB host_free=134GB gpu_util=41% temp=46C
2026-05-20 03:50:32,410 INFO moe_compress.stage2_reap_ream :: DIAG layer 28: batch loop complete — 32 batches in 44.6s, now post-profile work | vram=76.9/150GB host_free=134GB gpu_util=75% temp=46C
2026-05-20 03:50:44,471 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 28 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:50:52,958 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 25/20000 — train_mse=1.344698e-04 holdout_mse=1.344452e-04 best=1.344452e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:50:55,305 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 50/20000 — train_mse=1.161647e-04 holdout_mse=1.144111e-04 best=1.144111e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:50:57,643 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 75/20000 — train_mse=9.986451e-05 holdout_mse=1.037117e-04 best=1.037117e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:50:59,976 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 100/20000 — train_mse=9.289684e-05 holdout_mse=9.663399e-05 best=9.663399e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:02,318 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 125/20000 — train_mse=8.783492e-05 holdout_mse=9.144534e-05 best=9.144534e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:04,649 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 150/20000 — train_mse=8.173400e-05 holdout_mse=8.748905e-05 best=8.748905e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:06,985 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 175/20000 — train_mse=7.681662e-05 holdout_mse=8.430142e-05 best=8.430142e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:09,341 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 200/20000 — train_mse=7.423353e-05 holdout_mse=8.160962e-05 best=8.160962e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:11,693 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 225/20000 — train_mse=7.367449e-05 holdout_mse=7.929239e-05 best=7.929239e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:14,024 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 250/20000 — train_mse=6.964488e-05 holdout_mse=7.732609e-05 best=7.732609e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:16,355 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 275/20000 — train_mse=6.875495e-05 holdout_mse=7.557545e-05 best=7.557545e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:18,680 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 300/20000 — train_mse=6.478922e-05 holdout_mse=7.402998e-05 best=7.402998e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:21,020 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 325/20000 — train_mse=6.378944e-05 holdout_mse=7.263025e-05 best=7.263025e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:23,352 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 350/20000 — train_mse=6.170488e-05 holdout_mse=7.138074e-05 best=7.138074e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:25,680 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 375/20000 — train_mse=6.124095e-05 holdout_mse=7.022515e-05 best=7.022515e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:28,005 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 400/20000 — train_mse=5.993784e-05 holdout_mse=6.920729e-05 best=6.920729e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:30,339 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 425/20000 — train_mse=5.842195e-05 holdout_mse=6.828072e-05 best=6.828072e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:32,667 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 450/20000 — train_mse=5.757510e-05 holdout_mse=6.742053e-05 best=6.742053e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:34,994 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 475/20000 — train_mse=5.621664e-05 holdout_mse=6.661109e-05 best=6.661109e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:37,331 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 500/20000 — train_mse=5.594962e-05 holdout_mse=6.587754e-05 best=6.587754e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:51:39,666 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 525/20000 — train_mse=5.454559e-05 holdout_mse=6.522311e-05 best=6.522311e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:51:42,009 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 550/20000 — train_mse=5.281533e-05 holdout_mse=6.458855e-05 best=6.458855e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:51:44,351 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 575/20000 — train_mse=5.149534e-05 holdout_mse=6.400763e-05 best=6.400763e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:51:46,685 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 600/20000 — train_mse=5.203331e-05 holdout_mse=6.346448e-05 best=6.346448e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:51:49,020 INFO moe_compress.stage2_reap_ream ::     heal layer 28: step 625/20000 — train_mse=5.119814e-05 holdout_mse=6.289846e-05 best=6.289846e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:51:49,124 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 28: 625 steps (patience) — train_mse=5.119814e-05 (at_best=5.119814e-05) holdout_mse=6.289846e-05 plain_merged_holdout_mse=2.079311e-04 heal_gap=1.170033e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:51:49,261 WARNING moe_compress.stage2_reap_ream ::   layer 28: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:51:51,724 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=60.2094, max_group=9, mean_group=1.00
2026-05-20 03:51:52,096 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 30/40 (idx=29) — profiling then merging to 128 experts
2026-05-20 03:51:52,101 INFO moe_compress.stage2_reap_ream :: DIAG layer 29: entering batch loop (calibration tensor + early-exit forwards) | vram=52.5/150GB host_free=134GB gpu_util=4% temp=33C
2026-05-20 03:51:53,612 INFO moe_compress.stage2_reap_ream :: DIAG layer 29 batch 1: total=1.51s fwd=1.46s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.5s | vram=76.1/150GB host_free=134GB gpu_util=56% temp=39C
2026-05-20 03:51:55,066 INFO moe_compress.stage2_reap_ream :: DIAG layer 29 batch 2: total=1.45s fwd=1.43s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.0s | vram=76.1/150GB host_free=134GB gpu_util=51% temp=40C
2026-05-20 03:51:56,515 INFO moe_compress.stage2_reap_ream :: DIAG layer 29 batch 3: total=1.45s fwd=1.42s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.4s | vram=76.1/150GB host_free=134GB gpu_util=51% temp=42C
2026-05-20 03:52:06,588 INFO moe_compress.stage2_reap_ream :: DIAG layer 29 batch 10: total=1.43s fwd=1.41s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.5s | vram=76.1/150GB host_free=134GB gpu_util=60% temp=45C
2026-05-20 03:52:21,044 INFO moe_compress.stage2_reap_ream :: DIAG layer 29 batch 20: total=1.47s fwd=1.41s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=28.9s | vram=76.1/150GB host_free=134GB gpu_util=58% temp=47C
2026-05-20 03:52:35,694 INFO moe_compress.stage2_reap_ream :: DIAG layer 29 batch 30: total=1.46s fwd=1.40s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=43.6s | vram=76.1/150GB host_free=133GB gpu_util=54% temp=45C
2026-05-20 03:52:37,668 INFO moe_compress.stage2_reap_ream :: DIAG layer 29: batch loop complete — 32 batches in 45.6s, now post-profile work | vram=76.1/150GB host_free=133GB gpu_util=100% temp=46C
2026-05-20 03:52:49,975 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 29 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:52:58,835 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 25/20000 — train_mse=1.690565e-04 holdout_mse=1.721328e-04 best=1.721328e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:01,193 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 50/20000 — train_mse=1.383564e-04 holdout_mse=1.461265e-04 best=1.461265e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:03,520 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 75/20000 — train_mse=1.207866e-04 holdout_mse=1.311882e-04 best=1.311882e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:05,848 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 100/20000 — train_mse=1.145335e-04 holdout_mse=1.211072e-04 best=1.211072e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:08,178 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 125/20000 — train_mse=1.069774e-04 holdout_mse=1.136647e-04 best=1.136647e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:10,506 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 150/20000 — train_mse=9.768023e-05 holdout_mse=1.079797e-04 best=1.079797e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:12,828 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 175/20000 — train_mse=9.257805e-05 holdout_mse=1.034236e-04 best=1.034236e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:15,151 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 200/20000 — train_mse=8.868694e-05 holdout_mse=9.960273e-05 best=9.960273e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:17,475 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 225/20000 — train_mse=8.682784e-05 holdout_mse=9.635608e-05 best=9.635608e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:19,813 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 250/20000 — train_mse=8.152709e-05 holdout_mse=9.358318e-05 best=9.358318e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:22,150 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 275/20000 — train_mse=8.062328e-05 holdout_mse=9.114757e-05 best=9.114757e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:24,486 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 300/20000 — train_mse=7.818585e-05 holdout_mse=8.897471e-05 best=8.897471e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:26,813 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 325/20000 — train_mse=7.338048e-05 holdout_mse=8.705231e-05 best=8.705231e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:29,128 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 350/20000 — train_mse=7.299394e-05 holdout_mse=8.527726e-05 best=8.527726e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:31,449 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 375/20000 — train_mse=6.986863e-05 holdout_mse=8.375103e-05 best=8.375103e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:33,772 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 400/20000 — train_mse=6.962765e-05 holdout_mse=8.230302e-05 best=8.230302e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:36,098 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 425/20000 — train_mse=6.743317e-05 holdout_mse=8.094967e-05 best=8.094967e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:38,414 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 450/20000 — train_mse=6.735829e-05 holdout_mse=7.974529e-05 best=7.974529e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:40,726 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 475/20000 — train_mse=6.411828e-05 holdout_mse=7.867478e-05 best=7.867478e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:43,057 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 500/20000 — train_mse=6.271463e-05 holdout_mse=7.764269e-05 best=7.764269e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:45,371 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 525/20000 — train_mse=6.195858e-05 holdout_mse=7.666536e-05 best=7.666536e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:47,685 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 550/20000 — train_mse=6.054166e-05 holdout_mse=7.578031e-05 best=7.578031e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:49,998 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 575/20000 — train_mse=6.078876e-05 holdout_mse=7.499534e-05 best=7.499534e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:52,317 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 600/20000 — train_mse=5.955464e-05 holdout_mse=7.415239e-05 best=7.415239e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:53:54,659 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 625/20000 — train_mse=5.871651e-05 holdout_mse=7.343186e-05 best=7.343186e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:53:56,978 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 650/20000 — train_mse=5.771108e-05 holdout_mse=7.270090e-05 best=7.270090e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:53:59,298 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 675/20000 — train_mse=5.729502e-05 holdout_mse=7.204428e-05 best=7.204428e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:54:01,637 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 700/20000 — train_mse=5.747670e-05 holdout_mse=7.144070e-05 best=7.144070e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:54:04,187 INFO moe_compress.stage2_reap_ream ::     heal layer 29: step 725/20000 — train_mse=5.544537e-05 holdout_mse=7.082525e-05 best=7.082525e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:54:04,290 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 29: 725 steps (patience) — train_mse=5.544537e-05 (at_best=5.544537e-05) holdout_mse=7.082525e-05 plain_merged_holdout_mse=2.613561e-04 heal_gap=1.537988e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:54:04,426 WARNING moe_compress.stage2_reap_ream ::   layer 29: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:54:06,936 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=58.1762, max_group=9, mean_group=1.00
2026-05-20 03:54:07,308 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 31/40 (idx=30) — profiling then merging to 128 experts
2026-05-20 03:54:07,312 INFO moe_compress.stage2_reap_ream :: DIAG layer 30: entering batch loop (calibration tensor + early-exit forwards) | vram=51.7/150GB host_free=133GB gpu_util=0% temp=32C
2026-05-20 03:54:08,858 INFO moe_compress.stage2_reap_ream :: DIAG layer 30 batch 1: total=1.55s fwd=1.50s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.5s | vram=75.3/150GB host_free=133GB gpu_util=51% temp=40C
2026-05-20 03:54:10,341 INFO moe_compress.stage2_reap_ream :: DIAG layer 30 batch 2: total=1.48s fwd=1.46s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.0s | vram=75.3/150GB host_free=133GB gpu_util=53% temp=41C
2026-05-20 03:54:11,820 INFO moe_compress.stage2_reap_ream :: DIAG layer 30 batch 3: total=1.48s fwd=1.45s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.5s | vram=75.3/150GB host_free=133GB gpu_util=54% temp=43C
2026-05-20 03:54:22,117 INFO moe_compress.stage2_reap_ream :: DIAG layer 30 batch 10: total=1.46s fwd=1.44s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=14.8s | vram=75.3/150GB host_free=133GB gpu_util=56% temp=46C
2026-05-20 03:54:36,880 INFO moe_compress.stage2_reap_ream :: DIAG layer 30 batch 20: total=1.50s fwd=1.44s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=29.6s | vram=75.3/150GB host_free=133GB gpu_util=58% temp=47C
2026-05-20 03:54:51,849 INFO moe_compress.stage2_reap_ream :: DIAG layer 30 batch 30: total=1.50s fwd=1.44s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=44.5s | vram=75.3/150GB host_free=132GB gpu_util=56% temp=47C
2026-05-20 03:54:53,859 INFO moe_compress.stage2_reap_ream :: DIAG layer 30: batch loop complete — 32 batches in 46.5s, now post-profile work | vram=75.3/150GB host_free=132GB gpu_util=55% temp=46C
2026-05-20 03:55:06,271 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 30 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:55:15,100 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 25/20000 — train_mse=1.656107e-04 holdout_mse=1.694022e-04 best=1.694022e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:17,428 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 50/20000 — train_mse=1.361586e-04 holdout_mse=1.369622e-04 best=1.369622e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:19,750 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 75/20000 — train_mse=1.144463e-04 holdout_mse=1.208895e-04 best=1.208895e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:22,084 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 100/20000 — train_mse=1.038258e-04 holdout_mse=1.110741e-04 best=1.110741e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:24,415 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 125/20000 — train_mse=9.778377e-05 holdout_mse=1.042322e-04 best=1.042322e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:26,746 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 150/20000 — train_mse=9.087743e-05 holdout_mse=9.900730e-05 best=9.900730e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:29,074 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 175/20000 — train_mse=8.618597e-05 holdout_mse=9.491802e-05 best=9.491802e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:31,396 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 200/20000 — train_mse=8.009007e-05 holdout_mse=9.158414e-05 best=9.158414e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:33,755 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 225/20000 — train_mse=8.001489e-05 holdout_mse=8.877128e-05 best=8.877128e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:36,088 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 250/20000 — train_mse=7.528211e-05 holdout_mse=8.639295e-05 best=8.639295e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:38,424 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 275/20000 — train_mse=7.277331e-05 holdout_mse=8.430832e-05 best=8.430832e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:40,774 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 300/20000 — train_mse=7.100534e-05 holdout_mse=8.249959e-05 best=8.249959e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:43,111 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 325/20000 — train_mse=6.985576e-05 holdout_mse=8.088811e-05 best=8.088811e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:45,439 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 350/20000 — train_mse=6.736208e-05 holdout_mse=7.946632e-05 best=7.946632e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:47,775 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 375/20000 — train_mse=6.579865e-05 holdout_mse=7.817863e-05 best=7.817863e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:50,105 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 400/20000 — train_mse=6.509294e-05 holdout_mse=7.699013e-05 best=7.699013e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:52,689 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 425/20000 — train_mse=6.309788e-05 holdout_mse=7.591991e-05 best=7.591991e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:55,045 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 450/20000 — train_mse=6.203476e-05 holdout_mse=7.494113e-05 best=7.494113e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:57,376 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 475/20000 — train_mse=5.972171e-05 holdout_mse=7.402085e-05 best=7.402085e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:55:59,711 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 500/20000 — train_mse=5.902095e-05 holdout_mse=7.324150e-05 best=7.324150e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:56:02,060 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 525/20000 — train_mse=5.852236e-05 holdout_mse=7.243647e-05 best=7.243647e-05 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:56:04,395 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 550/20000 — train_mse=5.622063e-05 holdout_mse=7.172357e-05 best=7.172357e-05 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:56:06,732 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 575/20000 — train_mse=5.511984e-05 holdout_mse=7.108590e-05 best=7.108590e-05 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:56:09,070 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 600/20000 — train_mse=5.691173e-05 holdout_mse=7.050746e-05 best=7.050746e-05 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:56:11,409 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 625/20000 — train_mse=5.534279e-05 holdout_mse=6.990085e-05 best=6.990085e-05 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:56:13,753 INFO moe_compress.stage2_reap_ream ::     heal layer 30: step 650/20000 — train_mse=5.291796e-05 holdout_mse=6.932432e-05 best=6.932432e-05 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:56:13,859 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 30: 650 steps (patience) — train_mse=5.291796e-05 (at_best=5.291796e-05) holdout_mse=6.932432e-05 plain_merged_holdout_mse=2.858628e-04 heal_gap=1.640636e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:56:13,996 WARNING moe_compress.stage2_reap_ream ::   layer 30: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:56:16,560 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=58.5533, max_group=9, mean_group=1.00
2026-05-20 03:56:16,938 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 32/40 (idx=31) — profiling then merging to 128 experts
2026-05-20 03:56:16,943 INFO moe_compress.stage2_reap_ream :: DIAG layer 31: entering batch loop (calibration tensor + early-exit forwards) | vram=50.9/150GB host_free=132GB gpu_util=0% temp=32C
2026-05-20 03:56:18,527 INFO moe_compress.stage2_reap_ream :: DIAG layer 31 batch 1: total=1.58s fwd=1.54s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.6s | vram=74.6/150GB host_free=132GB gpu_util=47% temp=39C
2026-05-20 03:56:20,052 INFO moe_compress.stage2_reap_ream :: DIAG layer 31 batch 2: total=1.52s fwd=1.50s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.1s | vram=74.6/150GB host_free=132GB gpu_util=55% temp=41C
2026-05-20 03:56:21,572 INFO moe_compress.stage2_reap_ream :: DIAG layer 31 batch 3: total=1.52s fwd=1.49s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.6s | vram=74.6/150GB host_free=132GB gpu_util=55% temp=42C
2026-05-20 03:56:32,180 INFO moe_compress.stage2_reap_ream :: DIAG layer 31 batch 10: total=1.51s fwd=1.48s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=15.2s | vram=74.6/150GB host_free=132GB gpu_util=58% temp=45C
2026-05-20 03:56:47,398 INFO moe_compress.stage2_reap_ream :: DIAG layer 31 batch 20: total=1.54s fwd=1.48s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=30.5s | vram=74.6/150GB host_free=132GB gpu_util=58% temp=45C
2026-05-20 03:57:02,793 INFO moe_compress.stage2_reap_ream :: DIAG layer 31 batch 30: total=1.54s fwd=1.48s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=45.9s | vram=74.6/150GB host_free=131GB gpu_util=56% temp=45C
2026-05-20 03:57:04,860 INFO moe_compress.stage2_reap_ream :: DIAG layer 31: batch loop complete — 32 batches in 47.9s, now post-profile work | vram=74.6/150GB host_free=131GB gpu_util=52% temp=45C
2026-05-20 03:57:17,480 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 31 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:57:26,950 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 25/20000 — train_mse=2.722631e-04 holdout_mse=2.693471e-04 best=2.693471e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:29,386 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 50/20000 — train_mse=2.269203e-04 holdout_mse=2.291473e-04 best=2.291473e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:31,819 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 75/20000 — train_mse=1.955663e-04 holdout_mse=2.071529e-04 best=2.071529e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:34,256 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 100/20000 — train_mse=1.813161e-04 holdout_mse=1.925719e-04 best=1.925719e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:36,706 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 125/20000 — train_mse=1.686152e-04 holdout_mse=1.818470e-04 best=1.818470e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:39,152 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 150/20000 — train_mse=1.637962e-04 holdout_mse=1.736257e-04 best=1.736257e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:41,592 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 175/20000 — train_mse=1.508764e-04 holdout_mse=1.669664e-04 best=1.669664e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:44,035 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 200/20000 — train_mse=1.468053e-04 holdout_mse=1.614752e-04 best=1.614752e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:46,481 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 225/20000 — train_mse=1.392460e-04 holdout_mse=1.569939e-04 best=1.569939e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:48,926 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 250/20000 — train_mse=1.346077e-04 holdout_mse=1.530311e-04 best=1.530311e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:51,395 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 275/20000 — train_mse=1.302204e-04 holdout_mse=1.496259e-04 best=1.496259e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:53,833 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 300/20000 — train_mse=1.243959e-04 holdout_mse=1.466697e-04 best=1.466697e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:56,280 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 325/20000 — train_mse=1.212941e-04 holdout_mse=1.439599e-04 best=1.439599e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:57:58,718 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 350/20000 — train_mse=1.187614e-04 holdout_mse=1.415550e-04 best=1.415550e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:01,164 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 375/20000 — train_mse=1.170704e-04 holdout_mse=1.394695e-04 best=1.394695e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:03,609 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 400/20000 — train_mse=1.130564e-04 holdout_mse=1.375512e-04 best=1.375512e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:06,286 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 425/20000 — train_mse=1.114368e-04 holdout_mse=1.357879e-04 best=1.357879e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:08,723 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 450/20000 — train_mse=1.048541e-04 holdout_mse=1.340902e-04 best=1.340902e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:11,173 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 475/20000 — train_mse=1.058882e-04 holdout_mse=1.326310e-04 best=1.326310e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:13,630 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 500/20000 — train_mse=1.071341e-04 holdout_mse=1.313045e-04 best=1.313045e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:58:16,064 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 525/20000 — train_mse=1.020276e-04 holdout_mse=1.300121e-04 best=1.300121e-04 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 03:58:18,492 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 550/20000 — train_mse=1.005103e-04 holdout_mse=1.288547e-04 best=1.288547e-04 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 03:58:20,929 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 575/20000 — train_mse=9.875618e-05 holdout_mse=1.278261e-04 best=1.278261e-04 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 03:58:23,367 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 600/20000 — train_mse=9.793816e-05 holdout_mse=1.267982e-04 best=1.267982e-04 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 03:58:25,810 INFO moe_compress.stage2_reap_ream ::     heal layer 31: step 625/20000 — train_mse=9.690346e-05 holdout_mse=1.259284e-04 best=1.259284e-04 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 03:58:25,913 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 31: 625 steps (patience) — train_mse=9.690346e-05 (at_best=9.690346e-05) holdout_mse=1.259284e-04 plain_merged_holdout_mse=4.125644e-04 heal_gap=2.902498e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 03:58:26,049 WARNING moe_compress.stage2_reap_ream ::   layer 31: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 03:58:28,658 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=58.7579, max_group=9, mean_group=1.00
2026-05-20 03:58:29,026 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 33/40 (idx=32) — profiling then merging to 128 experts
2026-05-20 03:58:29,032 INFO moe_compress.stage2_reap_ream :: DIAG layer 32: entering batch loop (calibration tensor + early-exit forwards) | vram=50.2/150GB host_free=131GB gpu_util=0% temp=32C
2026-05-20 03:58:30,648 INFO moe_compress.stage2_reap_ream :: DIAG layer 32 batch 1: total=1.62s fwd=1.57s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.6s | vram=73.7/150GB host_free=131GB gpu_util=52% temp=39C
2026-05-20 03:58:32,203 INFO moe_compress.stage2_reap_ream :: DIAG layer 32 batch 2: total=1.55s fwd=1.53s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.2s | vram=73.7/150GB host_free=131GB gpu_util=52% temp=41C
2026-05-20 03:58:33,757 INFO moe_compress.stage2_reap_ream :: DIAG layer 32 batch 3: total=1.55s fwd=1.53s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.7s | vram=73.7/150GB host_free=131GB gpu_util=53% temp=42C
2026-05-20 03:58:44,568 INFO moe_compress.stage2_reap_ream :: DIAG layer 32 batch 10: total=1.54s fwd=1.51s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=15.5s | vram=73.7/150GB host_free=131GB gpu_util=67% temp=46C
2026-05-20 03:59:00,168 INFO moe_compress.stage2_reap_ream :: DIAG layer 32 batch 20: total=1.57s fwd=1.51s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=31.1s | vram=73.7/150GB host_free=131GB gpu_util=54% temp=47C
2026-05-20 03:59:15,903 INFO moe_compress.stage2_reap_ream :: DIAG layer 32 batch 30: total=1.57s fwd=1.51s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=46.9s | vram=73.7/150GB host_free=130GB gpu_util=57% temp=48C
2026-05-20 03:59:18,017 INFO moe_compress.stage2_reap_ream :: DIAG layer 32: batch loop complete — 32 batches in 49.0s, now post-profile work | vram=73.8/150GB host_free=130GB gpu_util=88% temp=47C
2026-05-20 03:59:30,783 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 32 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 03:59:40,295 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 25/20000 — train_mse=3.679487e-04 holdout_mse=3.662264e-04 best=3.662264e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:42,714 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 50/20000 — train_mse=2.763272e-04 holdout_mse=2.881983e-04 best=2.881983e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:45,139 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 75/20000 — train_mse=2.342762e-04 holdout_mse=2.484441e-04 best=2.484441e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:47,567 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 100/20000 — train_mse=2.071145e-04 holdout_mse=2.241391e-04 best=2.241391e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:49,986 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 125/20000 — train_mse=1.849252e-04 holdout_mse=2.074756e-04 best=2.074756e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:52,410 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 150/20000 — train_mse=1.698330e-04 holdout_mse=1.951221e-04 best=1.951221e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:54,844 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 175/20000 — train_mse=1.573367e-04 holdout_mse=1.853737e-04 best=1.853737e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:57,257 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 200/20000 — train_mse=1.504954e-04 holdout_mse=1.775175e-04 best=1.775175e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 03:59:59,683 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 225/20000 — train_mse=1.426524e-04 holdout_mse=1.708422e-04 best=1.708422e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:02,102 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 250/20000 — train_mse=1.352525e-04 holdout_mse=1.652529e-04 best=1.652529e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:04,543 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 275/20000 — train_mse=1.299798e-04 holdout_mse=1.602687e-04 best=1.602687e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:06,987 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 300/20000 — train_mse=1.230625e-04 holdout_mse=1.561230e-04 best=1.561230e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:09,421 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 325/20000 — train_mse=1.218401e-04 holdout_mse=1.523159e-04 best=1.523159e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:11,887 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 350/20000 — train_mse=1.153937e-04 holdout_mse=1.490438e-04 best=1.490438e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:14,321 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 375/20000 — train_mse=1.115622e-04 holdout_mse=1.460804e-04 best=1.460804e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:16,765 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 400/20000 — train_mse=1.066159e-04 holdout_mse=1.434583e-04 best=1.434583e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:19,200 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 425/20000 — train_mse=1.057408e-04 holdout_mse=1.411229e-04 best=1.411229e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:21,642 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 450/20000 — train_mse=1.020221e-04 holdout_mse=1.388730e-04 best=1.388730e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:24,086 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 475/20000 — train_mse=9.728140e-05 holdout_mse=1.369510e-04 best=1.369510e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:26,517 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 500/20000 — train_mse=9.637955e-05 holdout_mse=1.351649e-04 best=1.351649e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:28,966 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 525/20000 — train_mse=9.324598e-05 holdout_mse=1.334518e-04 best=1.334518e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:31,397 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 550/20000 — train_mse=9.206191e-05 holdout_mse=1.319184e-04 best=1.319184e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:33,828 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 575/20000 — train_mse=9.082688e-05 holdout_mse=1.305786e-04 best=1.305786e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:00:36,257 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 600/20000 — train_mse=8.912691e-05 holdout_mse=1.293506e-04 best=1.293506e-04 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 04:00:38,690 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 625/20000 — train_mse=8.790039e-05 holdout_mse=1.280599e-04 best=1.280599e-04 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 04:00:41,121 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 650/20000 — train_mse=8.524819e-05 holdout_mse=1.270170e-04 best=1.270170e-04 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 04:00:43,549 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 675/20000 — train_mse=8.422566e-05 holdout_mse=1.258832e-04 best=1.258832e-04 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 04:00:45,974 INFO moe_compress.stage2_reap_ream ::     heal layer 32: step 700/20000 — train_mse=8.197415e-05 holdout_mse=1.247947e-04 best=1.247947e-04 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 04:00:46,074 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 32: 700 steps (patience) — train_mse=8.197415e-05 (at_best=8.197415e-05) holdout_mse=1.247947e-04 plain_merged_holdout_mse=6.378044e-04 heal_gap=4.282059e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 04:00:46,211 WARNING moe_compress.stage2_reap_ream ::   layer 32: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 04:00:48,698 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=61.1263, max_group=9, mean_group=1.00
2026-05-20 04:00:49,071 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 34/40 (idx=33) — profiling then merging to 128 experts
2026-05-20 04:00:49,075 INFO moe_compress.stage2_reap_ream :: DIAG layer 33: entering batch loop (calibration tensor + early-exit forwards) | vram=49.4/150GB host_free=130GB gpu_util=19% temp=32C
2026-05-20 04:00:50,731 INFO moe_compress.stage2_reap_ream :: DIAG layer 33 batch 1: total=1.66s fwd=1.61s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.7s | vram=73.0/150GB host_free=130GB gpu_util=46% temp=40C
2026-05-20 04:00:52,325 INFO moe_compress.stage2_reap_ream :: DIAG layer 33 batch 2: total=1.59s fwd=1.57s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.2s | vram=73.0/150GB host_free=130GB gpu_util=53% temp=41C
2026-05-20 04:00:53,915 INFO moe_compress.stage2_reap_ream :: DIAG layer 33 batch 3: total=1.59s fwd=1.56s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=4.8s | vram=73.0/150GB host_free=130GB gpu_util=52% temp=42C
2026-05-20 04:01:04,997 INFO moe_compress.stage2_reap_ream :: DIAG layer 33 batch 10: total=1.58s fwd=1.55s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=15.9s | vram=73.0/150GB host_free=130GB gpu_util=59% temp=45C
2026-05-20 04:01:20,992 INFO moe_compress.stage2_reap_ream :: DIAG layer 33 batch 20: total=1.61s fwd=1.55s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=31.9s | vram=73.0/150GB host_free=129GB gpu_util=62% temp=45C
2026-05-20 04:01:37,094 INFO moe_compress.stage2_reap_ream :: DIAG layer 33 batch 30: total=1.61s fwd=1.55s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=48.0s | vram=73.0/150GB host_free=129GB gpu_util=55% temp=46C
2026-05-20 04:01:39,256 INFO moe_compress.stage2_reap_ream :: DIAG layer 33: batch loop complete — 32 batches in 50.2s, now post-profile work | vram=73.0/150GB host_free=129GB gpu_util=100% temp=46C
2026-05-20 04:01:52,129 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 33 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 04:02:01,874 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 25/20000 — train_mse=3.200367e-04 holdout_mse=3.298267e-04 best=3.298267e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:04,316 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 50/20000 — train_mse=2.607223e-04 holdout_mse=2.637545e-04 best=2.637545e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:06,766 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 75/20000 — train_mse=2.155282e-04 holdout_mse=2.311396e-04 best=2.311396e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:09,207 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 100/20000 — train_mse=1.960288e-04 holdout_mse=2.110893e-04 best=2.110893e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:11,642 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 125/20000 — train_mse=1.767951e-04 holdout_mse=1.973554e-04 best=1.973554e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:14,074 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 150/20000 — train_mse=1.583058e-04 holdout_mse=1.871191e-04 best=1.871191e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:16,562 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 175/20000 — train_mse=1.564493e-04 holdout_mse=1.790508e-04 best=1.790508e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:18,993 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 200/20000 — train_mse=1.419223e-04 holdout_mse=1.726018e-04 best=1.726018e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:21,420 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 225/20000 — train_mse=1.356064e-04 holdout_mse=1.672296e-04 best=1.672296e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:23,850 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 250/20000 — train_mse=1.282316e-04 holdout_mse=1.625492e-04 best=1.625492e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:26,282 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 275/20000 — train_mse=1.273385e-04 holdout_mse=1.585326e-04 best=1.585326e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:28,708 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 300/20000 — train_mse=1.233952e-04 holdout_mse=1.551641e-04 best=1.551641e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:31,137 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 325/20000 — train_mse=1.230199e-04 holdout_mse=1.521887e-04 best=1.521887e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:33,574 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 350/20000 — train_mse=1.148154e-04 holdout_mse=1.494870e-04 best=1.494870e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:36,015 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 375/20000 — train_mse=1.147336e-04 holdout_mse=1.470384e-04 best=1.470384e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:38,459 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 400/20000 — train_mse=1.072436e-04 holdout_mse=1.449012e-04 best=1.449012e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:40,914 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 425/20000 — train_mse=1.040627e-04 holdout_mse=1.428110e-04 best=1.428110e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:43,352 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 450/20000 — train_mse=1.030599e-04 holdout_mse=1.411131e-04 best=1.411131e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:45,797 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 475/20000 — train_mse=9.847025e-05 holdout_mse=1.394490e-04 best=1.394490e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:48,233 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 500/20000 — train_mse=9.611831e-05 holdout_mse=1.379021e-04 best=1.379021e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:50,669 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 525/20000 — train_mse=9.528102e-05 holdout_mse=1.365994e-04 best=1.365994e-04 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 04:02:53,109 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 550/20000 — train_mse=9.389639e-05 holdout_mse=1.352305e-04 best=1.352305e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:02:55,571 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 575/20000 — train_mse=9.075593e-05 holdout_mse=1.341330e-04 best=1.341330e-04 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 04:02:58,034 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 600/20000 — train_mse=9.126167e-05 holdout_mse=1.329617e-04 best=1.329617e-04 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 04:03:00,494 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 625/20000 — train_mse=8.793304e-05 holdout_mse=1.321775e-04 best=1.321775e-04 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 04:03:02,954 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 650/20000 — train_mse=8.713784e-05 holdout_mse=1.310149e-04 best=1.310149e-04 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 04:03:05,419 INFO moe_compress.stage2_reap_ream ::     heal layer 33: step 675/20000 — train_mse=8.447675e-05 holdout_mse=1.301350e-04 best=1.301350e-04 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 04:03:05,519 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 33: 675 steps (patience) — train_mse=8.447675e-05 (at_best=8.447675e-05) holdout_mse=1.301350e-04 plain_merged_holdout_mse=6.000826e-04 heal_gap=4.565829e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 04:03:05,657 WARNING moe_compress.stage2_reap_ream ::   layer 33: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 04:03:08,038 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=0, ream_centroids=128) — Σ cost=59.3761, max_group=9, mean_group=1.00
2026-05-20 04:03:08,417 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 35/40 (idx=34) — profiling then merging to 128 experts
2026-05-20 04:03:08,421 INFO moe_compress.stage2_reap_ream :: DIAG layer 34: entering batch loop (calibration tensor + early-exit forwards) | vram=48.6/150GB host_free=129GB gpu_util=0% temp=32C
2026-05-20 04:03:10,112 INFO moe_compress.stage2_reap_ream :: DIAG layer 34 batch 1: total=1.69s fwd=1.65s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=255) | cum=1.7s | vram=72.2/150GB host_free=129GB gpu_util=49% temp=40C
2026-05-20 04:03:11,743 INFO moe_compress.stage2_reap_ream :: DIAG layer 34 batch 2: total=1.63s fwd=1.60s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.3s | vram=72.2/150GB host_free=129GB gpu_util=52% temp=41C
2026-05-20 04:03:13,372 INFO moe_compress.stage2_reap_ream :: DIAG layer 34 batch 3: total=1.63s fwd=1.60s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.0s | vram=72.2/150GB host_free=129GB gpu_util=53% temp=42C
2026-05-20 04:03:24,699 INFO moe_compress.stage2_reap_ream :: DIAG layer 34 batch 10: total=1.61s fwd=1.59s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=255) | cum=16.3s | vram=72.2/150GB host_free=129GB gpu_util=57% temp=45C
2026-05-20 04:03:41,085 INFO moe_compress.stage2_reap_ream :: DIAG layer 34 batch 20: total=1.65s fwd=1.59s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=32.7s | vram=72.2/150GB host_free=128GB gpu_util=59% temp=47C
2026-05-20 04:03:57,546 INFO moe_compress.stage2_reap_ream :: DIAG layer 34 batch 30: total=1.65s fwd=1.59s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=49.1s | vram=72.2/150GB host_free=128GB gpu_util=43% temp=47C
2026-05-20 04:03:59,740 INFO moe_compress.stage2_reap_ream :: DIAG layer 34: batch loop complete — 32 batches in 51.3s, now post-profile work | vram=72.2/150GB host_free=127GB gpu_util=36% temp=45C
2026-05-20 04:04:12,950 INFO moe_compress.stage2_reap_ream ::   merge-heal capture: layer 34 — 262144 (input,target) token rows captured (pool_size=262144)
2026-05-20 04:04:22,996 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 25/20000 — train_mse=3.990915e-04 holdout_mse=4.064568e-04 best=4.064568e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:25,468 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 50/20000 — train_mse=3.048446e-04 holdout_mse=3.213168e-04 best=3.213168e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:27,902 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 75/20000 — train_mse=2.672843e-04 holdout_mse=2.810692e-04 best=2.810692e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:30,334 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 100/20000 — train_mse=2.442255e-04 holdout_mse=2.566496e-04 best=2.566496e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:32,766 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 125/20000 — train_mse=2.160271e-04 holdout_mse=2.395171e-04 best=2.395171e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:35,207 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 150/20000 — train_mse=1.983392e-04 holdout_mse=2.268561e-04 best=2.268561e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:37,632 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 175/20000 — train_mse=1.836128e-04 holdout_mse=2.166813e-04 best=2.166813e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:40,069 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 200/20000 — train_mse=1.760409e-04 holdout_mse=2.086480e-04 best=2.086480e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:42,501 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 225/20000 — train_mse=1.701014e-04 holdout_mse=2.023591e-04 best=2.023591e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:44,946 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 250/20000 — train_mse=1.555594e-04 holdout_mse=1.968371e-04 best=1.968371e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:47,380 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 275/20000 — train_mse=1.509043e-04 holdout_mse=1.921575e-04 best=1.921575e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:49,806 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 300/20000 — train_mse=1.464705e-04 holdout_mse=1.879682e-04 best=1.879682e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:52,236 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 325/20000 — train_mse=1.385312e-04 holdout_mse=1.843221e-04 best=1.843221e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:54,707 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 350/20000 — train_mse=1.382393e-04 holdout_mse=1.811948e-04 best=1.811948e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:57,141 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 375/20000 — train_mse=1.318975e-04 holdout_mse=1.785656e-04 best=1.785656e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:04:59,572 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 400/20000 — train_mse=1.301404e-04 holdout_mse=1.760866e-04 best=1.760866e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:05:01,998 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 425/20000 — train_mse=1.210776e-04 holdout_mse=1.738647e-04 best=1.738647e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:05:04,422 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 450/20000 — train_mse=1.217335e-04 holdout_mse=1.719028e-04 best=1.719028e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:05:06,849 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 475/20000 — train_mse=1.206910e-04 holdout_mse=1.700516e-04 best=1.700516e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:05:09,277 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 500/20000 — train_mse=1.157037e-04 holdout_mse=1.685319e-04 best=1.685319e-04 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 04:05:11,702 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 525/20000 — train_mse=1.151185e-04 holdout_mse=1.668151e-04 best=1.668151e-04 improved (patience 0/5, min_rel_delta=0.0100)
2026-05-20 04:05:14,128 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 550/20000 — train_mse=1.130598e-04 holdout_mse=1.654264e-04 best=1.654264e-04 no-improve (patience 1/5, min_rel_delta=0.0100)
2026-05-20 04:05:16,553 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 575/20000 — train_mse=1.112948e-04 holdout_mse=1.641142e-04 best=1.641142e-04 no-improve (patience 2/5, min_rel_delta=0.0100)
2026-05-20 04:05:18,983 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 600/20000 — train_mse=1.101941e-04 holdout_mse=1.629089e-04 best=1.629089e-04 no-improve (patience 3/5, min_rel_delta=0.0100)
2026-05-20 04:05:21,421 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 625/20000 — train_mse=1.071552e-04 holdout_mse=1.618822e-04 best=1.618822e-04 no-improve (patience 4/5, min_rel_delta=0.0100)
2026-05-20 04:05:23,851 INFO moe_compress.stage2_reap_ream ::     heal layer 34: step 650/20000 — train_mse=1.022719e-04 holdout_mse=1.608589e-04 best=1.608589e-04 no-improve (patience 5/5, min_rel_delta=0.0100)
2026-05-20 04:05:23,951 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 34: 650 steps (patience) — train_mse=1.022719e-04 (at_best=1.022719e-04) holdout_mse=1.608589e-04 plain_merged_holdout_mse=7.898075e-04 heal_gap=5.858701e-05 accepted=True (128 kept experts, train_router=True)
2026-05-20 04:05:24,088 WARNING moe_compress.stage2_reap_ream ::   layer 34: _remap_covariance_for_layer dropped 256 covariance entries (= 128 unique experts × ~2 matrices/expert); dropping 128 experts from covariance; keeping 128 experts; unexpected if n_dropped_experts > (n_keys_before - n_kept).
2026-05-20 04:05:26,483 INFO moe_compress.stage2_reap_ream ::   kept 128 / 256 experts (protected=1, ream_centroids=127) — Σ cost=60.2100, max_group=9, mean_group=1.01
2026-05-20 04:05:26,864 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 36/40 (idx=35) — profiling then merging to 256 experts
2026-05-20 04:05:26,868 INFO moe_compress.stage2_reap_ream :: DIAG layer 35: entering batch loop (calibration tensor + early-exit forwards) | vram=47.8/150GB host_free=128GB gpu_util=0% temp=32C
2026-05-20 04:05:28,597 INFO moe_compress.stage2_reap_ream :: DIAG layer 35 batch 1: total=1.73s fwd=1.68s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=256) | cum=1.7s | vram=71.4/150GB host_free=128GB gpu_util=49% temp=39C
2026-05-20 04:05:30,266 INFO moe_compress.stage2_reap_ream :: DIAG layer 35 batch 2: total=1.67s fwd=1.64s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.4s | vram=71.4/150GB host_free=128GB gpu_util=54% temp=40C
2026-05-20 04:05:31,936 INFO moe_compress.stage2_reap_ream :: DIAG layer 35 batch 3: total=1.67s fwd=1.64s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.1s | vram=71.4/150GB host_free=128GB gpu_util=53% temp=42C
2026-05-20 04:05:43,547 INFO moe_compress.stage2_reap_ream :: DIAG layer 35 batch 10: total=1.65s fwd=1.63s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=16.7s | vram=71.4/150GB host_free=128GB gpu_util=56% temp=46C
2026-05-20 04:06:00,382 INFO moe_compress.stage2_reap_ream :: DIAG layer 35 batch 20: total=1.68s fwd=1.62s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=33.5s | vram=71.4/150GB host_free=127GB gpu_util=56% temp=45C
2026-05-20 04:06:17,213 INFO moe_compress.stage2_reap_ream :: DIAG layer 35 batch 30: total=1.68s fwd=1.62s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=50.3s | vram=71.4/150GB host_free=126GB gpu_util=57% temp=47C
2026-05-20 04:06:19,461 INFO moe_compress.stage2_reap_ream :: DIAG layer 35: batch loop complete — 32 batches in 52.6s, now post-profile work | vram=71.4/150GB host_free=126GB gpu_util=55% temp=46C
2026-05-20 04:06:20,087 WARNING moe_compress.stage2_reap_ream :: layer 35: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 04:06:20,089 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 35: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 04:06:22,905 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 04:06:23,292 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 37/40 (idx=36) — profiling then merging to 256 experts
2026-05-20 04:06:23,296 INFO moe_compress.stage2_reap_ream :: DIAG layer 36: entering batch loop (calibration tensor + early-exit forwards) | vram=47.8/150GB host_free=125GB gpu_util=0% temp=36C
2026-05-20 04:06:25,040 INFO moe_compress.stage2_reap_ream :: DIAG layer 36 batch 1: total=1.74s fwd=1.70s hooks: input=0.02s intermed=0.06s down=0.04s (n_cb=255) | cum=1.7s | vram=71.4/150GB host_free=125GB gpu_util=56% temp=44C
2026-05-20 04:06:26,726 INFO moe_compress.stage2_reap_ream :: DIAG layer 36 batch 2: total=1.69s fwd=1.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.4s | vram=71.4/150GB host_free=125GB gpu_util=59% temp=45C
2026-05-20 04:06:28,414 INFO moe_compress.stage2_reap_ream :: DIAG layer 36 batch 3: total=1.69s fwd=1.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.1s | vram=71.4/150GB host_free=125GB gpu_util=58% temp=45C
2026-05-20 04:06:40,257 INFO moe_compress.stage2_reap_ream :: DIAG layer 36 batch 10: total=1.72s fwd=1.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=17.0s | vram=71.4/150GB host_free=124GB gpu_util=57% temp=47C
2026-05-20 04:06:57,494 INFO moe_compress.stage2_reap_ream :: DIAG layer 36 batch 20: total=1.72s fwd=1.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=34.2s | vram=71.4/150GB host_free=124GB gpu_util=57% temp=46C
2026-05-20 04:07:14,702 INFO moe_compress.stage2_reap_ream :: DIAG layer 36 batch 30: total=1.72s fwd=1.66s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=51.4s | vram=71.4/150GB host_free=123GB gpu_util=58% temp=47C
2026-05-20 04:07:17,001 INFO moe_compress.stage2_reap_ream :: DIAG layer 36: batch loop complete — 32 batches in 53.7s, now post-profile work | vram=71.4/150GB host_free=123GB gpu_util=36% temp=45C
2026-05-20 04:07:17,140 WARNING moe_compress.stage2_reap_ream :: layer 36: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 04:07:17,142 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 36: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 04:07:20,520 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 04:07:20,912 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 38/40 (idx=37) — profiling then merging to 256 experts
2026-05-20 04:07:20,916 INFO moe_compress.stage2_reap_ream :: DIAG layer 37: entering batch loop (calibration tensor + early-exit forwards) | vram=47.8/150GB host_free=123GB gpu_util=0% temp=36C
2026-05-20 04:07:22,692 INFO moe_compress.stage2_reap_ream :: DIAG layer 37 batch 1: total=1.78s fwd=1.73s hooks: input=0.02s intermed=0.05s down=0.04s (n_cb=255) | cum=1.8s | vram=71.3/150GB host_free=123GB gpu_util=53% temp=43C
2026-05-20 04:07:24,417 INFO moe_compress.stage2_reap_ream :: DIAG layer 37 batch 2: total=1.72s fwd=1.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.5s | vram=71.3/150GB host_free=123GB gpu_util=60% temp=45C
2026-05-20 04:07:26,142 INFO moe_compress.stage2_reap_ream :: DIAG layer 37 batch 3: total=1.72s fwd=1.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.2s | vram=71.3/150GB host_free=123GB gpu_util=60% temp=46C
2026-05-20 04:07:38,286 INFO moe_compress.stage2_reap_ream :: DIAG layer 37 batch 10: total=1.76s fwd=1.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=17.4s | vram=71.3/150GB host_free=123GB gpu_util=57% temp=48C
2026-05-20 04:07:55,880 INFO moe_compress.stage2_reap_ream :: DIAG layer 37 batch 20: total=1.76s fwd=1.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=35.0s | vram=71.3/150GB host_free=122GB gpu_util=57% temp=48C
2026-05-20 04:08:13,468 INFO moe_compress.stage2_reap_ream :: DIAG layer 37 batch 30: total=1.76s fwd=1.70s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=52.6s | vram=71.3/150GB host_free=121GB gpu_util=57% temp=48C
2026-05-20 04:08:15,812 INFO moe_compress.stage2_reap_ream :: DIAG layer 37: batch loop complete — 32 batches in 54.9s, now post-profile work | vram=71.4/150GB host_free=121GB gpu_util=37% temp=46C
2026-05-20 04:08:15,950 WARNING moe_compress.stage2_reap_ream :: layer 37: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 04:08:15,952 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 37: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 04:08:19,910 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=0, ream_centroids=256) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 04:08:20,319 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 39/40 (idx=38) — profiling then merging to 256 experts
2026-05-20 04:08:20,324 INFO moe_compress.stage2_reap_ream :: DIAG layer 38: entering batch loop (calibration tensor + early-exit forwards) | vram=47.8/150GB host_free=121GB gpu_util=0% temp=36C
2026-05-20 04:08:22,137 INFO moe_compress.stage2_reap_ream :: DIAG layer 38 batch 1: total=1.81s fwd=1.77s hooks: input=0.02s intermed=0.05s down=0.04s (n_cb=256) | cum=1.8s | vram=71.4/150GB host_free=121GB gpu_util=55% temp=44C
2026-05-20 04:08:23,896 INFO moe_compress.stage2_reap_ream :: DIAG layer 38 batch 2: total=1.76s fwd=1.73s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.6s | vram=71.4/150GB host_free=121GB gpu_util=57% temp=46C
2026-05-20 04:08:25,661 INFO moe_compress.stage2_reap_ream :: DIAG layer 38 batch 3: total=1.76s fwd=1.74s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.3s | vram=71.4/150GB host_free=121GB gpu_util=58% temp=46C
2026-05-20 04:08:38,087 INFO moe_compress.stage2_reap_ream :: DIAG layer 38 batch 10: total=1.79s fwd=1.73s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=17.8s | vram=71.4/150GB host_free=121GB gpu_util=57% temp=47C
2026-05-20 04:08:56,043 INFO moe_compress.stage2_reap_ream :: DIAG layer 38 batch 20: total=1.80s fwd=1.74s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=35.7s | vram=71.4/150GB host_free=120GB gpu_util=55% temp=48C
2026-05-20 04:09:14,015 INFO moe_compress.stage2_reap_ream :: DIAG layer 38 batch 30: total=1.80s fwd=1.74s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=53.7s | vram=71.4/150GB host_free=119GB gpu_util=58% temp=48C
2026-05-20 04:09:16,403 INFO moe_compress.stage2_reap_ream :: DIAG layer 38: batch loop complete — 32 batches in 56.1s, now post-profile work | vram=71.4/150GB host_free=119GB gpu_util=83% temp=48C
2026-05-20 04:09:16,541 WARNING moe_compress.stage2_reap_ream :: layer 38: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 04:09:16,543 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 38: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 04:09:20,123 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=1, ream_centroids=255) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 04:09:20,534 INFO moe_compress.stage2_reap_ream :: Stage 2 layer 40/40 (idx=39) — profiling then merging to 256 experts
2026-05-20 04:09:20,539 INFO moe_compress.stage2_reap_ream :: DIAG layer 39: entering batch loop (calibration tensor + early-exit forwards) | vram=47.8/150GB host_free=119GB gpu_util=0% temp=36C
2026-05-20 04:09:22,398 INFO moe_compress.stage2_reap_ream :: DIAG layer 39 batch 1: total=1.86s fwd=1.81s hooks: input=0.02s intermed=0.05s down=0.04s (n_cb=256) | cum=1.9s | vram=71.3/150GB host_free=119GB gpu_util=52% temp=45C
2026-05-20 04:09:24,198 INFO moe_compress.stage2_reap_ream :: DIAG layer 39 batch 2: total=1.80s fwd=1.77s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=3.7s | vram=71.3/150GB host_free=119GB gpu_util=82% temp=46C
2026-05-20 04:09:25,994 INFO moe_compress.stage2_reap_ream :: DIAG layer 39 batch 3: total=1.80s fwd=1.77s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=5.5s | vram=71.3/150GB host_free=119GB gpu_util=57% temp=47C
2026-05-20 04:09:38,737 INFO moe_compress.stage2_reap_ream :: DIAG layer 39 batch 10: total=1.83s fwd=1.77s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=255) | cum=18.2s | vram=71.3/150GB host_free=118GB gpu_util=56% temp=48C
2026-05-20 04:09:57,101 INFO moe_compress.stage2_reap_ream :: DIAG layer 39 batch 20: total=1.84s fwd=1.78s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=36.6s | vram=71.3/150GB host_free=118GB gpu_util=57% temp=47C
2026-05-20 04:10:15,461 INFO moe_compress.stage2_reap_ream :: DIAG layer 39 batch 30: total=1.84s fwd=1.78s hooks: input=0.02s intermed=0.04s down=0.04s (n_cb=256) | cum=54.9s | vram=71.3/150GB host_free=117GB gpu_util=53% temp=45C
2026-05-20 04:10:17,906 INFO moe_compress.stage2_reap_ream :: DIAG layer 39: batch loop complete — 32 batches in 57.4s, now post-profile work | vram=71.4/150GB host_free=117GB gpu_util=34% temp=46C
2026-05-20 04:10:18,045 WARNING moe_compress.stage2_reap_ream :: layer 39: budget target (256) equals total expert count (256) — no merging will occur; check budget configuration.
2026-05-20 04:10:18,047 INFO moe_compress.stage2_reap_ream ::   merge-heal layer 39: skipped (0 merges — layer unchanged, nothing to heal)
2026-05-20 04:10:21,450 INFO moe_compress.stage2_reap_ream ::   kept 256 / 256 experts (protected=2, ream_centroids=254) — Σ cost=0.0000, max_group=1, mean_group=0.00
2026-05-20 04:11:35,771 INFO moe_compress.stage2_reap_ream :: Saved Stage 2 input covariance to /cache/ablations/SH/_stage2_input_covariance.pt
2026-05-20 04:11:35,783 INFO moe_compress.utils.model_io :: Saving compressed checkpoint to /cache/ablations/SH/stage2_pruned (stage=stage2_pruned)
2026-05-20 04:11:35,792 INFO moe_compress.utils.model_io ::   wrote compressed_metadata.json (per_layer_num_experts: 40 entries, factored_layers: 0)
2026-05-20 04:11:35,796 INFO moe_compress.utils.model_io :: Saving 9-shard safetensors (stacked MoE format) to /cache/ablations/SH/stage2_pruned
2026-05-20 04:11:45,326 INFO moe_compress.utils.model_io ::   wrote shard 1/9 (model-00001-of-00009.safetensors, 4.98 GB)
2026-05-20 04:11:54,606 INFO moe_compress.utils.model_io ::   wrote shard 2/9 (model-00002-of-00009.safetensors, 4.54 GB)
2026-05-20 04:12:04,121 INFO moe_compress.utils.model_io ::   wrote shard 3/9 (model-00003-of-00009.safetensors, 4.89 GB)
2026-05-20 04:12:14,265 INFO moe_compress.utils.model_io ::   wrote shard 4/9 (model-00004-of-00009.safetensors, 4.90 GB)
2026-05-20 04:12:23,725 INFO moe_compress.utils.model_io ::   wrote shard 5/9 (model-00005-of-00009.safetensors, 4.89 GB)
2026-05-20 04:12:33,791 INFO moe_compress.utils.model_io ::   wrote shard 6/9 (model-00006-of-00009.safetensors, 4.90 GB)
2026-05-20 04:12:43,300 INFO moe_compress.utils.model_io ::   wrote shard 7/9 (model-00007-of-00009.safetensors, 4.89 GB)
2026-05-20 04:12:53,117 INFO moe_compress.utils.model_io ::   wrote shard 8/9 (model-00008-of-00009.safetensors, 4.71 GB)
2026-05-20 04:13:01,225 INFO moe_compress.utils.model_io ::   wrote shard 9/9 (model-00009-of-00009.safetensors, 4.01 GB)
2026-05-20 04:13:01,237 INFO moe_compress.utils.model_io ::   wrote model.safetensors.index.json (1026 keys, 42.72 GB total)
2026-05-20 04:13:04,626 INFO moe_compress.stage2_reap_ream :: Stage 2 complete — pruned checkpoint at /cache/ablations/SH/stage2_pruned
2026-05-20 04:13:04,641 INFO __main__ :: Stage 2 done in 1h14m57s — durable on Hub: <not uploaded>
2026-05-20 04:13:04,860 INFO __main__ :: === Stage 2.5 — Post-Merge Router KD ===
2026-05-20 04:13:04,862 INFO moe_compress.stage5_router_kd :: Stage 5: MoE experts_implementation = 'grouped_mm' (forward dispatch via transformers.integrations.moe.ExpertsInterface)
2026-05-20 04:13:04,869 INFO moe_compress.stage5_router_kd :: Stage 5: torch.compile(student, mode='default')
2026-05-20 04:13:04,873 INFO moe_compress.utils.calibration :: Building calibration tensor: 3000 x 512 tokens, source=nvidia-cascade
2026-05-20 04:13:04,873 INFO moe_compress.utils.calibration ::   per-subset sequence counts: {'math': 630, 'science': 330, 'chat': 1680, 'instruction_following': 102, 'conversational_agent': 99, 'swe': 60, 'terminal_agent': 99}
2026-05-20 04:13:04,873 INFO moe_compress.utils.calibration :: Streaming 630 math samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=444756)
2026-05-20 04:13:22,956 INFO moe_compress.utils.calibration :: Streaming 330 science samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=772365)
2026-05-20 04:13:33,335 INFO moe_compress.utils.calibration :: Streaming 1680 chat samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=492647)
2026-05-20 04:13:49,588 INFO moe_compress.utils.calibration :: Streaming 102 instruction_following samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=307677)
2026-05-20 04:13:54,331 INFO moe_compress.utils.calibration :: Streaming 99 conversational_agent samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=234632)
2026-05-20 04:14:06,197 INFO moe_compress.utils.calibration :: Streaming 60 swe samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=976636)
2026-05-20 04:15:56,069 INFO moe_compress.utils.calibration :: Streaming 99 terminal_agent samples from nvidia/Nemotron-Cascade-2-SFT-Data (seed=435953)
2026-05-20 04:16:31,386 INFO moe_compress.utils.calibration :: Cached calibration tensor: /cache/ablations/_shared/_calibration_cache/calib_c2de01e417120090.pt (shape=(3000, 512))
2026-05-20 04:16:31,393 INFO moe_compress.stage5_router_kd :: Stage 5: 40 routers trainable; 375 steps total / 375 remaining (grad-accum=1, resume_step=0)
2026-05-20 04:16:31,393 INFO moe_compress.stage5_router_kd :: Loading teacher for KD (first live batch): Qwen/Qwen3.6-35B-A3B-FP8 (teacher_model_repo=Qwen/Qwen3.6-35B-A3B-FP8, teacher_load_in_4bit=False, device_map=auto)
2026-05-20 04:16:31,647 INFO moe_compress.utils.model_io :: Loading Qwen/Qwen3.6-35B-A3B-FP8 with AutoModelForImageTextToText (dtype=torch.bfloat16, device_map=auto)

Loading weights:   0%|          | 0/1356 [00:00<?, ?it/s]
Loading weights:   0%|          | 2/1356 [00:01<20:59,  1.08it/s]
Loading weights:   1%|          | 12/1356 [00:01<02:45,  8.10it/s]
Loading weights:   1%|▏         | 18/1356 [00:04<05:38,  3.95it/s]
Loading weights:   3%|▎         | 42/1356 [00:06<03:09,  6.92it/s]
Loading weights:   3%|▎         | 45/1356 [00:07<02:54,  7.50it/s]
Loading weights:   5%|▌         | 68/1356 [00:09<02:33,  8.38it/s]
Loading weights:   6%|▌         | 82/1356 [00:12<02:56,  7.23it/s]
Loading weights:   6%|▋         | 85/1356 [00:12<02:47,  7.59it/s]
Loading weights:   9%|▊         | 118/1356 [00:14<02:03, 10.03it/s]
Loading weights:   9%|▉         | 121/1356 [00:14<01:58, 10.44it/s]
Loading weights:  11%|█         | 144/1356 [00:17<02:03,  9.85it/s]
Loading weights:  11%|█         | 147/1356 [00:17<01:57, 10.29it/s]
Loading weights:  13%|█▎        | 170/1356 [00:20<02:00,  9.80it/s]
Loading weights:  14%|█▎        | 184/1356 [00:22<02:23,  8.14it/s]
Loading weights:  14%|█▍        | 187/1356 [00:22<02:17,  8.49it/s]
Loading weights:  16%|█▌        | 220/1356 [00:25<01:47, 10.61it/s]
Loading weights:  16%|█▋        | 223/1356 [00:25<01:42, 11.01it/s]
Loading weights:  18%|█▊        | 246/1356 [00:27<01:49, 10.17it/s]
Loading weights:  18%|█▊        | 249/1356 [00:27<01:44, 10.62it/s]
Loading weights:  20%|██        | 272/1356 [00:30<01:51,  9.76it/s]
Loading weights:  21%|██        | 286/1356 [00:33<02:13,  8.01it/s]
Loading weights:  21%|██▏       | 289/1356 [00:33<02:07,  8.37it/s]
Loading weights:  24%|██▎       | 322/1356 [00:35<01:40, 10.27it/s]
Loading weights:  24%|██▍       | 325/1356 [00:35<01:36, 10.67it/s]
Loading weights:  26%|██▌       | 348/1356 [00:38<01:42,  9.84it/s]
Loading weights:  26%|██▌       | 351/1356 [00:38<01:37, 10.28it/s]
Loading weights:  28%|██▊       | 374/1356 [00:41<01:40,  9.77it/s]
Loading weights:  29%|██▊       | 388/1356 [00:43<02:00,  8.01it/s]
Loading weights:  29%|██▉       | 391/1356 [00:43<01:54,  8.40it/s]
Loading weights:  31%|███▏      | 424/1356 [00:46<01:31, 10.23it/s]
Loading weights:  31%|███▏      | 426/1356 [00:46<01:29, 10.42it/s]
Loading weights:  33%|███▎      | 450/1356 [00:49<01:31,  9.87it/s]
Loading weights:  33%|███▎      | 453/1356 [00:49<01:27, 10.30it/s]
Loading weights:  35%|███▌      | 476/1356 [00:51<01:30,  9.76it/s]
Loading weights:  36%|███▌      | 490/1356 [00:54<01:47,  8.06it/s]
Loading weights:  36%|███▋      | 493/1356 [00:54<01:42,  8.41it/s]
Loading weights:  39%|███▉      | 526/1356 [00:57<01:18, 10.52it/s]
Loading weights:  39%|███▉      | 529/1356 [00:57<01:15, 10.95it/s]
Loading weights:  41%|████      | 552/1356 [00:59<01:19, 10.09it/s]
Loading weights:  41%|████      | 555/1356 [00:59<01:16, 10.51it/s]
Loading weights:  43%|████▎     | 578/1356 [01:02<01:19,  9.84it/s]
Loading weights:  44%|████▎     | 592/1356 [01:05<01:35,  8.01it/s]
Loading weights:  44%|████▍     | 595/1356 [01:05<01:31,  8.34it/s]
Loading weights:  46%|████▋     | 628/1356 [01:07<01:11, 10.11it/s]
Loading weights:  47%|████▋     | 631/1356 [01:08<01:08, 10.52it/s]
Loading weights:  48%|████▊     | 654/1356 [01:10<01:10,  9.92it/s]
Loading weights:  48%|████▊     | 656/1356 [01:10<01:09, 10.14it/s]
Loading weights:  50%|█████     | 680/1356 [01:13<01:09,  9.80it/s]
Loading weights:  51%|█████     | 694/1356 [01:15<01:22,  8.02it/s]
Loading weights:  51%|█████▏    | 697/1356 [01:15<01:18,  8.38it/s]
Loading weights:  54%|█████▍    | 730/1356 [01:18<01:00, 10.34it/s]
Loading weights:  54%|█████▍    | 732/1356 [01:18<00:59, 10.52it/s]
Loading weights:  56%|█████▌    | 756/1356 [01:21<00:59, 10.08it/s]
Loading weights:  56%|█████▌    | 758/1356 [01:21<00:58, 10.27it/s]
Loading weights:  58%|█████▊    | 782/1356 [01:23<00:58,  9.86it/s]
Loading weights:  59%|█████▊    | 796/1356 [01:26<01:08,  8.17it/s]
Loading weights:  59%|█████▉    | 799/1356 [01:26<01:05,  8.53it/s]
Loading weights:  61%|██████▏   | 832/1356 [01:29<00:50, 10.34it/s]
Loading weights:  62%|██████▏   | 835/1356 [01:29<00:48, 10.69it/s]
Loading weights:  63%|██████▎   | 858/1356 [01:31<00:50,  9.77it/s]
Loading weights:  63%|██████▎   | 860/1356 [01:31<00:49,  9.99it/s]
Loading weights:  65%|██████▌   | 884/1356 [01:34<00:48,  9.68it/s]
Loading weights:  66%|██████▌   | 898/1356 [01:37<00:56,  8.06it/s]
Loading weights:  66%|██████▋   | 901/1356 [01:37<00:53,  8.45it/s]
Loading weights:  69%|██████▉   | 934/1356 [01:39<00:41, 10.24it/s]
Loading weights:  69%|██████▉   | 937/1356 [01:40<00:39, 10.64it/s]
Loading weights:  71%|███████   | 960/1356 [01:42<00:39,  9.95it/s]
Loading weights:  71%|███████   | 963/1356 [01:42<00:37, 10.38it/s]
Loading weights:  73%|███████▎  | 986/1356 [01:45<00:37,  9.75it/s]
Loading weights:  74%|███████▎  | 1000/1356 [01:47<00:44,  8.01it/s]
Loading weights:  74%|███████▍  | 1003/1356 [01:47<00:42,  8.38it/s]
Loading weights:  77%|███████▋  | 1040/1356 [01:48<00:16, 19.38it/s]
Loading weights:  78%|███████▊  | 1055/1356 [01:48<00:12, 24.55it/s]
Loading weights:  79%|███████▉  | 1071/1356 [01:48<00:08, 31.90it/s]
Loading weights:  80%|████████  | 1087/1356 [01:48<00:06, 40.73it/s]
Loading weights:  81%|████████  | 1101/1356 [01:48<00:05, 49.11it/s]
Loading weights:  82%|████████▏ | 1115/1356 [01:48<00:04, 58.13it/s]
Loading weights:  83%|████████▎ | 1130/1356 [01:48<00:03, 70.78it/s]
Loading weights:  84%|████████▍ | 1144/1356 [01:48<00:02, 78.94it/s]
Loading weights:  86%|████████▌ | 1161/1356 [01:48<00:02, 93.07it/s]
Loading weights:  87%|████████▋ | 1175/1356 [01:49<00:01, 96.95it/s]
Loading weights:  88%|████████▊ | 1188/1356 [01:49<00:01, 102.75it/s]
Loading weights:  89%|████████▉ | 1205/1356 [01:49<00:01, 109.96it/s]
Loading weights:  90%|████████▉ | 1220/1356 [01:49<00:01, 112.29it/s]
Loading weights:  91%|█████████ | 1235/1356 [01:49<00:01, 113.41it/s]
Loading weights:  92%|█████████▏| 1250/1356 [01:49<00:00, 121.67it/s]
Loading weights:  93%|█████████▎| 1264/1356 [01:49<00:00, 125.25it/s]
Loading weights:  94%|█████████▍| 1279/1356 [01:49<00:00, 117.41it/s]
Loading weights:  96%|█████████▌| 1295/1356 [01:50<00:00, 122.21it/s]
Loading weights:  97%|█████████▋| 1311/1356 [01:50<00:00, 131.21it/s]
Loading weights:  98%|█████████▊| 1327/1356 [01:50<00:00, 132.76it/s]
Loading weights:  99%|█████████▉| 1341/1356 [01:50<00:00, 128.70it/s]
Loading weights: 100%|█████████▉| 1355/1356 [01:50<00:00, 98.98it/s] 
Loading weights: 100%|██████████| 1356/1356 [01:50<00:00, 12.26it/s]
2026-05-20 04:19:04,705 INFO moe_compress.utils.model_io :: Model loaded: Qwen3_5MoeForConditionalGeneration on devices=cuda:0:35.1B
2026-05-20 04:19:04,705 INFO moe_compress.stage5_router_kd :: Stage 5: MoE experts_implementation = 'grouped_mm' (forward dispatch via transformers.integrations.moe.ExpertsInterface)

Fetching ... files: 0it [00:00, ?it/s]
Fetching ... files: 1it [00:00,  2.68it/s]
Fetching ... files: 4it [00:00,  9.44it/s]
Fetching ... files: 9it [00:00, 18.78it/s]
Fetching ... files: 9it [00:00, 14.20it/s]

Fetching ... files: 0it [00:00, ?it/s]
Fetching ... files: 59it [00:00, 345.81it/s]
Fetching ... files: 94it [00:01, 43.62it/s] 
Fetching ... files: 110it [00:02, 35.46it/s]
Fetching ... files: 120it [00:03, 31.27it/s]
Fetching ... files: 127it [00:03, 30.44it/s]
Fetching ... files: 133it [00:03, 27.63it/s]
Fetching ... files: 137it [00:03, 27.38it/s]
Fetching ... files: 141it [00:04, 24.71it/s]
Fetching ... files: 144it [00:04, 24.31it/s]
Fetching ... files: 148it [00:04, 23.33it/s]
Fetching ... files: 152it [00:04, 21.07it/s]
Fetching ... files: 159it [00:04, 23.18it/s]
Fetching ... files: 162it [00:04, 23.81it/s]
Fetching ... files: 165it [00:05, 24.24it/s]
Fetching ... files: 168it [00:05, 18.61it/s]
Fetching ... files: 173it [00:05, 23.14it/s]
Fetching ... files: 176it [00:05, 17.09it/s]
Fetching ... files: 179it [00:06, 15.97it/s]
Fetching ... files: 187it [00:06, 23.60it/s]
Fetching ... files: 190it [00:06, 20.92it/s]
Fetching ... files: 195it [00:06, 21.25it/s]
Fetching ... files: 198it [00:06, 22.55it/s]
Fetching ... files: 202it [00:06, 24.37it/s]
Fetching ... files: 205it [00:07, 20.59it/s]
Fetching ... files: 210it [00:07, 22.62it/s]
Fetching ... files: 213it [00:07, 23.59it/s]
Fetching ... files: 216it [00:07, 22.16it/s]
Fetching ... files: 219it [00:07, 21.66it/s]
Fetching ... files: 222it [00:07, 19.32it/s]
Fetching ... files: 227it [00:08, 18.47it/s]
Fetching ... files: 232it [00:08, 23.36it/s]
Fetching ... files: 235it [00:08, 21.77it/s]
Fetching ... files: 238it [00:08, 20.83it/s]
Fetching ... files: 242it [00:08, 21.90it/s]
Fetching ... files: 246it [00:08, 23.67it/s]
Fetching ... files: 249it [00:09, 22.36it/s]
Fetching ... files: 252it [00:09, 21.24it/s]
Fetching ... files: 255it [00:09, 20.44it/s]
Fetching ... files: 258it [00:09, 20.93it/s]
Fetching ... files: 261it [00:09, 21.28it/s]
Fetching ... files: 264it [00:09, 20.65it/s]
Fetching ... files: 267it [00:10, 19.30it/s]
Fetching ... files: 271it [00:10, 22.63it/s]
Fetching ... files: 274it [00:10, 20.87it/s]
Fetching ... files: 277it [00:10, 14.85it/s]
Fetching ... files: 284it [00:10, 22.64it/s]
Fetching ... files: 287it [00:10, 21.90it/s]
Fetching ... files: 291it [00:11, 24.08it/s]
Fetching ... files: 294it [00:11, 16.07it/s]
Fetching ... files: 301it [00:11, 20.61it/s]
Fetching ... files: 304it [00:11, 18.81it/s]
Fetching ... files: 309it [00:12, 21.06it/s]
Fetching ... files: 312it [00:12, 21.07it/s]
Fetching ... files: 317it [00:12, 22.85it/s]
Fetching ... files: 320it [00:12, 21.67it/s]
Fetching ... files: 323it [00:12, 22.83it/s]
Fetching ... files: 326it [00:12, 23.12it/s]
Fetching ... files: 329it [00:12, 22.01it/s]
Fetching ... files: 332it [00:13, 19.18it/s]
Fetching ... files: 337it [00:13, 24.19it/s]
Fetching ... files: 340it [00:13, 18.81it/s]
Fetching ... files: 344it [00:13, 22.53it/s]
Fetching ... files: 347it [00:13, 21.68it/s]
Fetching ... files: 350it [00:13, 20.60it/s]
Fetching ... files: 354it [00:14, 23.59it/s]
Fetching ... files: 357it [00:14, 18.24it/s]
Fetching ... files: 362it [00:14, 21.30it/s]
Fetching ... files: 365it [00:14, 19.83it/s]
Fetching ... files: 370it [00:14, 21.68it/s]
Fetching ... files: 373it [00:15, 21.34it/s]
Fetching ... files: 377it [00:15, 23.57it/s]
Fetching ... files: 380it [00:15, 22.07it/s]
Fetching ... files: 383it [00:15, 22.43it/s]
Fetching ... files: 386it [00:15, 18.18it/s]
Fetching ... files: 389it [00:15, 20.40it/s]
Fetching ... files: 394it [00:16, 21.69it/s]
Fetching ... files: 397it [00:16, 22.45it/s]
Fetching ... files: 400it [00:16, 22.00it/s]
Fetching ... files: 403it [00:16, 21.47it/s]
Fetching ... files: 406it [00:16, 21.15it/s]
Fetching ... files: 410it [00:16, 19.61it/s]
Fetching ... files: 414it [00:16, 21.53it/s]
Fetching ... files: 418it [00:17, 21.15it/s]
Fetching ... files: 421it [00:17, 21.88it/s]
Fetching ... files: 424it [00:17, 21.93it/s]
Fetching ... files: 427it [00:17, 20.34it/s]
Fetching ... files: 430it [00:17, 20.03it/s]
Fetching ... files: 434it [00:17, 22.57it/s]
Fetching ... files: 437it [00:18, 20.41it/s]
Fetching ... files: 440it [00:18, 19.96it/s]
Fetching ... files: 444it [00:18, 21.82it/s]
Fetching ... files: 447it [00:18, 20.87it/s]
Fetching ... files: 451it [00:18, 24.53it/s]
Fetching ... files: 454it [00:18, 18.74it/s]
Fetching ... files: 460it [00:19, 21.33it/s]
Fetching ... files: 463it [00:19, 20.10it/s]
Fetching ... files: 468it [00:19, 21.40it/s]
Fetching ... files: 471it [00:19, 19.71it/s]
Fetching ... files: 476it [00:19, 21.49it/s]
Fetching ... files: 479it [00:20, 22.18it/s]
Fetching ... files: 482it [00:20, 19.95it/s]
Fetching ... files: 485it [00:20, 20.98it/s]
Fetching ... files: 489it [00:20, 20.09it/s]
Fetching ... files: 493it [00:20, 22.77it/s]
Fetching ... files: 496it [00:20, 23.76it/s]
Fetching ... files: 499it [00:20, 23.91it/s]
Fetching ... files: 502it [00:21, 22.69it/s]
Fetching ... files: 505it [00:21, 23.89it/s]
Fetching ... files: 508it [00:21, 20.26it/s]
Fetching ... files: 511it [00:21, 19.73it/s]
Fetching ... files: 515it [00:21, 19.31it/s]
Fetching ... files: 519it [00:21, 22.37it/s]
Fetching ... files: 523it [00:22, 20.10it/s]
Fetching ... files: 527it [00:22, 21.24it/s]
Fetching ... files: 531it [00:22, 20.15it/s]
Fetching ... files: 535it [00:22, 22.74it/s]
Fetching ... files: 539it [00:22, 21.65it/s]
Fetching ... files: 543it [00:22, 23.98it/s]
Fetching ... files: 547it [00:23, 20.23it/s]
Fetching ... files: 551it [00:23, 23.39it/s]
Fetching ... files: 554it [00:23, 23.38it/s]
Fetching ... files: 557it [00:23, 21.13it/s]
Fetching ... files: 560it [00:23, 19.94it/s]
Fetching ... files: 563it [00:23, 20.48it/s]
Fetching ... files: 567it [00:24, 23.06it/s]
Fetching ... files: 570it [00:24, 19.17it/s]
Fetching ... files: 575it [00:24, 22.77it/s]
Fetching ... files: 578it [00:24, 19.26it/s]
Fetching ... files: 583it [00:24, 24.36it/s]
Fetching ... files: 586it [00:25, 18.53it/s]
Fetching ... files: 591it [00:25, 23.35it/s]
Fetching ... files: 594it [00:25, 19.59it/s]
Fetching ... files: 597it [00:25, 19.64it/s]
Fetching ... files: 601it [00:25, 22.18it/s]
Fetching ... files: 604it [00:25, 22.98it/s]
Fetching ... files: 607it [00:26, 18.10it/s]
Fetching ... files: 610it [00:26, 19.52it/s]
Fetching ... files: 613it [00:26, 18.19it/s]
Fetching ... files: 618it [00:26, 22.32it/s]
Fetching ... files: 621it [00:26, 21.12it/s]
Fetching ... files: 624it [00:26, 22.55it/s]
Fetching ... files: 627it [00:27, 22.02it/s]
Fetching ... files: 630it [00:27, 18.76it/s]
Fetching ... files: 633it [00:27, 19.74it/s]
Fetching ... files: 637it [00:27, 22.28it/s]
Fetching ... files: 640it [00:27, 19.56it/s]
Fetching ... files: 644it [00:27, 23.23it/s]
Fetching ... files: 647it [00:28, 17.86it/s]
Fetching ... files: 652it [00:28, 21.72it/s]
Fetching ... files: 655it [00:28, 19.00it/s]
Fetching ... files: 660it [00:28, 20.39it/s]
Fetching ... files: 663it [00:28, 18.17it/s]
Fetching ... files: 666it [00:29, 16.27it/s]
Fetching ... files: 673it [00:29, 22.08it/s]
Fetching ... files: 676it [00:29, 17.02it/s]HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm.h
2026-05-20 04:19:46,048 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,048 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_array.h
2026-05-20 04:19:46,113 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_array.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,113 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_batched.h
2026-05-20 04:19:46,181 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_batched.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,181 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_complex.h
2026-05-20 04:19:46,274 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_complex.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,274 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_grouped.h
2026-05-20 04:19:46,290 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_grouped.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,290 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_layernorm_mainloop_fusion.h
2026-05-20 04:19:46,300 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_layernorm_mainloop_fusion.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,300 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_sparse.h
2026-05-20 04:19:46,371 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_sparse.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,372 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_sparse_universal.h
2026-05-20 04:19:46,377 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/include/cutlass/gemm/device/gemm_sparse_universal.h
Rate limited. Waiting 282.0s before retry [Retry 1/5].
2026-05-20 04:19:46,377 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 282.0s before retry [Retry 1/5].

Fetching ... files: 676it [00:48, 17.02it/s]
Fetching ... files: 681it [05:12, 19.48s/it]
Fetching ... files: 682it [05:12, 17.75s/it]
Fetching ... files: 685it [05:12, 12.84s/it]
Fetching ... files: 690it [05:12,  7.76s/it]
Fetching ... files: 693it [05:12,  5.80s/it]
Fetching ... files: 698it [05:12,  3.62s/it]
Fetching ... files: 702it [05:13,  2.55s/it]
Fetching ... files: 706it [05:13,  1.80s/it]
Fetching ... files: 711it [05:13,  1.19s/it]
Fetching ... files: 717it [05:13,  1.33it/s]
Fetching ... files: 720it [05:14,  1.64it/s]
Fetching ... files: 725it [05:14,  2.36it/s]
Fetching ... files: 728it [05:14,  2.95it/s]
Fetching ... files: 732it [05:14,  4.07it/s]
Fetching ... files: 735it [05:14,  4.90it/s]
Fetching ... files: 738it [05:14,  6.20it/s]
Fetching ... files: 742it [05:15,  8.01it/s]
Fetching ... files: 745it [05:15,  7.98it/s]
Fetching ... files: 751it [05:15, 11.58it/s]
Fetching ... files: 756it [05:15, 13.94it/s]
Fetching ... files: 761it [05:16, 16.44it/s]
Fetching ... files: 764it [05:16, 15.64it/s]
Fetching ... files: 769it [05:16, 18.80it/s]
Fetching ... files: 772it [05:16, 16.32it/s]
Fetching ... files: 776it [05:16, 19.39it/s]
Fetching ... files: 779it [05:17, 19.70it/s]
Fetching ... files: 782it [05:17, 21.05it/s]
Fetching ... files: 786it [05:17, 18.95it/s]
Fetching ... files: 789it [05:17, 19.85it/s]
Fetching ... files: 794it [05:17, 17.40it/s]
Fetching ... files: 798it [05:17, 20.72it/s]
Fetching ... files: 801it [05:18, 17.61it/s]
Fetching ... files: 806it [05:18, 21.39it/s]
Fetching ... files: 809it [05:18, 17.64it/s]
Fetching ... files: 812it [05:18, 19.40it/s]
Fetching ... files: 817it [05:19, 18.95it/s]
Fetching ... files: 822it [05:19, 23.50it/s]
Fetching ... files: 825it [05:19, 18.78it/s]
Fetching ... files: 828it [05:19, 20.64it/s]
Fetching ... files: 833it [05:19, 20.29it/s]
Fetching ... files: 838it [05:19, 22.55it/s]
Fetching ... files: 841it [05:20, 19.46it/s]
Fetching ... files: 846it [05:20, 22.94it/s]
Fetching ... files: 849it [05:20, 24.11it/s]
Fetching ... files: 852it [05:20, 17.20it/s]
Fetching ... files: 858it [05:20, 20.20it/s]
Fetching ... files: 863it [05:21, 22.86it/s]
Fetching ... files: 866it [05:21, 23.04it/s]
Fetching ... files: 869it [05:21, 20.16it/s]
Fetching ... files: 873it [05:21, 20.95it/s]
Fetching ... files: 877it [05:21, 20.60it/s]
Fetching ... files: 881it [05:21, 23.67it/s]
Fetching ... files: 884it [05:22, 22.23it/s]
Fetching ... files: 887it [05:22, 19.98it/s]
Fetching ... files: 890it [05:22, 21.75it/s]
Fetching ... files: 893it [05:22, 19.43it/s]
Fetching ... files: 896it [05:22, 19.94it/s]
Fetching ... files: 899it [05:23, 14.25it/s]
Fetching ... files: 906it [05:23, 23.21it/s]
Fetching ... files: 910it [05:23, 22.09it/s]
Fetching ... files: 913it [05:23, 21.27it/s]
Fetching ... files: 916it [05:23, 20.51it/s]
Fetching ... files: 919it [05:23, 17.47it/s]
Fetching ... files: 925it [05:24, 23.44it/s]
Fetching ... files: 928it [05:24, 19.52it/s]
Fetching ... files: 933it [05:24, 22.59it/s]
Fetching ... files: 936it [05:24, 18.29it/s]
Fetching ... files: 940it [05:24, 20.90it/s]
Fetching ... files: 943it [05:25, 19.46it/s]
Fetching ... files: 948it [05:25, 23.24it/s]
Fetching ... files: 951it [05:25, 18.54it/s]
Fetching ... files: 956it [05:25, 22.32it/s]
Fetching ... files: 959it [05:25, 18.31it/s]
Fetching ... files: 965it [05:26, 23.14it/s]
Fetching ... files: 968it [05:26, 21.10it/s]
Fetching ... files: 972it [05:26, 21.92it/s]
Fetching ... files: 975it [05:26, 17.10it/s]
Fetching ... files: 980it [05:27, 16.49it/s]
Fetching ... files: 988it [05:27, 24.07it/s]
Fetching ... files: 991it [05:27, 20.32it/s]
Fetching ... files: 995it [05:27, 23.04it/s]
Fetching ... files: 998it [05:27, 18.90it/s]
Fetching ... files: 1001it [05:27, 19.48it/s]
Fetching ... files: 1005it [05:28, 20.38it/s]
Fetching ... files: 1008it [05:28, 22.05it/s]
Fetching ... files: 1011it [05:28, 23.66it/s]
Fetching ... files: 1014it [05:28, 18.85it/s]
Fetching ... files: 1019it [05:28, 22.89it/s]
Fetching ... files: 1022it [05:28, 18.74it/s]
Fetching ... files: 1025it [05:29, 19.31it/s]
Fetching ... files: 1029it [05:29, 20.13it/s]
Fetching ... files: 1032it [05:29, 21.55it/s]
Fetching ... files: 1035it [05:29, 20.40it/s]
Fetching ... files: 1038it [05:29, 20.52it/s]
Fetching ... files: 1041it [05:29, 20.64it/s]
Fetching ... files: 1044it [05:29, 21.39it/s]
Fetching ... files: 1047it [05:30, 17.43it/s]
Fetching ... files: 1050it [05:30, 19.60it/s]
Fetching ... files: 1053it [05:30, 18.78it/s]
Fetching ... files: 1056it [05:30, 18.51it/s]
Fetching ... files: 1061it [05:30, 21.15it/s]
Fetching ... files: 1064it [05:30, 22.07it/s]
Fetching ... files: 1067it [05:31, 21.81it/s]
Fetching ... files: 1070it [05:31, 23.55it/s]
Fetching ... files: 1073it [05:31, 21.53it/s]
Fetching ... files: 1076it [05:31, 20.85it/s]
Fetching ... files: 1079it [05:31, 19.80it/s]HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/__init__.py
2026-05-20 04:24:48,104 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/__init__.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,105 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/emitter_base.py
2026-05-20 04:24:48,204 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/emitter_base.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,204 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm100_emitter.py
2026-05-20 04:24:48,220 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm100_emitter.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,220 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm100_nodes.py
2026-05-20 04:24:48,270 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm100_nodes.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,270 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm80_emitter.py
2026-05-20 04:24:48,323 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm80_emitter.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,323 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm80_nodes.py
2026-05-20 04:24:48,352 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm80_nodes.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,352 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm90_emitter.py
2026-05-20 04:24:48,429 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm90_emitter.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,429 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].
HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm90_nodes.py
2026-05-20 04:24:48,497 WARNING huggingface_hub.utils._http :: HTTP Error 429 thrown while requesting HEAD https://huggingface.co/kernels/kernels-community/deep-gemm/resolve/12130d957e9ba821b042f0fbca6a0992ee96e234/build/torch211-cxx11-cu130-x86_64-linux/include/third-party/cutlass/python/cutlass_cppgen/backend/evt/backend/sm90_nodes.py
Rate limited. Waiting 280.0s before retry [Retry 1/5].
2026-05-20 04:24:48,497 WARNING huggingface_hub.utils._http :: Rate limited. Waiting 280.0s before retry [Retry 1/5].

Fetching ... files: 1079it [05:48, 19.80it/s]
Fetching ... files: 1081it [10:12, 30.63s/it]
Fetching ... files: 1084it [10:12, 20.89s/it]
Fetching ... files: 1086it [10:12, 16.05s/it]
Fetching ... files: 1091it [10:12,  8.78s/it]
Fetching ... files: 1094it [10:12,  6.33s/it]
Fetching ... files: 1097it [10:12,  4.53s/it]
Fetching ... files: 1100it [10:13,  3.24s/it]
Fetching ... files: 1103it [10:13,  2.30s/it]
Fetching ... files: 1106it [10:13,  1.63s/it]
Fetching ... files: 1109it [10:13,  1.17s/it]
Fetching ... files: 1113it [10:13,  1.32it/s]
Fetching ... files: 1116it [10:14,  1.72it/s]
Fetching ... files: 1124it [10:14,  3.31it/s]
Fetching ... files: 1127it [10:14,  4.10it/s]
Fetching ... files: 1132it [10:14,  5.56it/s]
Fetching ... files: 1138it [10:14,  8.28it/s]
Fetching ... files: 1142it [10:15,  9.22it/s]
Fetching ... files: 1147it [10:15, 12.23it/s]
Fetching ... files: 1151it [10:15, 12.82it/s]
Fetching ... files: 1154it [10:15, 14.01it/s]
Fetching ... files: 1157it [10:15, 14.01it/s]
Fetching ... files: 1160it [10:16, 15.24it/s]
Fetching ... files: 1163it [10:16, 16.12it/s]
Fetching ... files: 1166it [10:16, 16.17it/s]
Fetching ... files: 1170it [10:16, 20.32it/s]
Fetching ... files: 1173it [10:16, 19.48it/s]
Fetching ... files: 1176it [10:16, 19.68it/s]
Fetching ... files: 1180it [10:17, 20.85it/s]
Fetching ... files: 1183it [10:17, 15.65it/s]
Fetching ... files: 1189it [10:17, 21.91it/s]
Fetching ... files: 1192it [10:17, 21.05it/s]
Fetching ... files: 1196it [10:17, 22.85it/s]
Fetching ... files: 1199it [10:17, 20.91it/s]
Fetching ... files: 1202it [10:18, 20.35it/s]
Fetching ... files: 1205it [10:18, 21.40it/s]
Fetching ... files: 1209it [10:18, 20.48it/s]
Fetching ... files: 1213it [10:18, 20.09it/s]
Fetching ... files: 1218it [10:18, 19.85it/s]
Fetching ... files: 1221it [10:19, 15.79it/s]
Fetching ... files: 1228it [10:19, 21.17it/s]
Fetching ... files: 1233it [10:19, 22.35it/s]
Fetching ... files: 1236it [10:19, 22.14it/s]
Fetching ... files: 1239it [10:20, 12.94it/s]
Fetching ... files: 1249it [10:20, 23.71it/s]
Fetching ... files: 1254it [10:20, 22.85it/s]
Fetching ... files: 1258it [10:20, 23.11it/s]
Fetching ... files: 1262it [10:21, 21.62it/s]
Fetching ... files: 1265it [10:21, 22.50it/s]
Fetching ... files: 1268it [10:21, 21.72it/s]
Fetching ... files: 1271it [10:21, 20.90it/s]
Fetching ... files: 1275it [10:21, 23.25it/s]
Fetching ... files: 1278it [10:21, 18.90it/s]
Fetching ... files: 1282it [10:22, 19.63it/s]
Fetching ... files: 1285it [10:22, 18.71it/s]
Fetching ... files: 1289it [10:22, 21.66it/s]
Fetching ... files: 1292it [10:22, 20.57it/s]
Fetching ... files: 1295it [10:22, 19.35it/s]
Fetching ... files: 1299it [10:22, 20.18it/s]
Fetching ... files: 1302it [10:22, 21.99it/s]
Fetching ... files: 1305it [10:23, 23.73it/s]
Fetching ... files: 1308it [10:23, 20.88it/s]
Fetching ... files: 1312it [10:23, 20.82it/s]
Fetching ... files: 1315it [10:23, 20.02it/s]
Fetching ... files: 1320it [10:23, 23.06it/s]
Fetching ... files: 1323it [10:23, 21.88it/s]
Fetching ... files: 1327it [10:24, 23.75it/s]
Fetching ... files: 1330it [10:24, 20.50it/s]
Fetching ... files: 1334it [10:24, 21.92it/s]
Fetching ... files: 1337it [10:24, 18.77it/s]
Fetching ... files: 1339it [10:24, 18.15it/s]
Fetching ... files: 1345it [10:24, 21.89it/s]
Fetching ... files: 1348it [10:25, 21.03it/s]
Fetching ... files: 1351it [10:25, 22.54it/s]
Fetching ... files: 1354it [10:25, 20.94it/s]
Fetching ... files: 1358it [10:25, 23.88it/s]
Fetching ... files: 1361it [10:25, 20.16it/s]
Fetching ... files: 1364it [10:25, 19.05it/s]
Fetching ... files: 1368it [10:26, 22.78it/s]
Fetching ... files: 1371it [10:26, 23.43it/s]
Fetching ... files: 1374it [10:26, 20.82it/s]
Fetching ... files: 1377it [10:26, 21.63it/s]
Fetching ... files: 1380it [10:26, 21.59it/s]
Fetching ... files: 1383it [10:26, 19.84it/s]
Fetching ... files: 1386it [10:26, 21.14it/s]
Fetching ... files: 1389it [10:27, 18.70it/s]
Fetching ... files: 1392it [10:27, 20.05it/s]
Fetching ... files: 1396it [10:27, 19.71it/s]
Fetching ... files: 1399it [10:27, 18.16it/s]
Fetching ... files: 1404it [10:27, 22.13it/s]
Fetching ... files: 1407it [10:28, 17.21it/s]
Fetching ... files: 1412it [10:28, 21.96it/s]
Fetching ... files: 1412it [10:28,  2.25it/s]
/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py:2202: UserWarning: Dynamo detected a call to a `functools.lru_cache`-wrapped function at 'fused_norm_gate.py:565'. Dynamo ignores the cache wrapper and directly traces the wrapped function. Silent incorrectness is only a *potential* risk, not something we have observed. Enable TORCH_LOGS=+dynamo for a DEBUG stack trace.

This call originates from:
  File "/usr/local/lib/python3.12/dist-packages/fla/modules/fused_norm_gate.py", line 565, in layer_norm_gated_bwd
    NS = min(get_multiprocessor_count(x.device.index), T)

  torch._dynamo.utils.warn_once(msg)
/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py:2082: UserWarning: Dynamo does not know how to trace the builtin `cuda_utils.get_device_properties.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).
If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.
If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.
  torch._dynamo.utils.warn_once(explanation + "\n" + "\n".join(hints))
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8] torch._dynamo hit config.recompile_limit (8)
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py:425)
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8]    last reason: 13/7: cache_params.layers[9].has_previous_state == False       # if use_precomputed_states:  # transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py:443 in forward
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8] User stack trace:
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8]   File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py", line 443, in forward
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8]     if use_precomputed_states:
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
W0520 04:31:46.868000 205 torch/_dynamo/convert_frame.py:1743] [13/8] To diagnose recompilation issues, see https://docs.pytorch.org/docs/main/user_guide/torch_compiler/compile/programming_model.recompilation.html
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8] torch._dynamo hit config.recompile_limit (8)
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8]    function: '__call__' (/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py:59)
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8]    last reason: 11/1: kwargs['attention_mask'] is None                         # return super().__call__(*args, **kwargs)  # transformers/modeling_layers.py:93 in __call__
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8] User stack trace:
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8]   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 93, in __call__
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8]     return super().__call__(*args, **kwargs)
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
W0520 04:31:55.072000 205 torch/_dynamo/convert_frame.py:1743] [11/8] To diagnose recompilation issues, see https://docs.pytorch.org/docs/main/user_guide/torch_compiler/compile/programming_model.recompilation.html
2026-05-20 04:31:55,622 INFO moe_compress.stage5_router_kd :: Stage 5 first-batch sanity: teacher shape=(8, 511, 248320) dtype=torch.float32 finite=True abs_max=3.750e+01 mean=-1.697e+00 std=2.242e+00 ; student shape=(8, 511, 248320) dtype=torch.float32 finite=True abs_max=3.350e+01 mean=-1.586e+00 std=2.229e+00 ; initial_loss=1.568642e-01 finite=True
2026-05-20 04:32:28  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:133): TileLang begins to compile kernel `kernel` with `out_idx=None`
2026-05-20 04:32:38  [TileLang:tilelang.jit.kernel:INFO] (kernel.py:141): TileLang completes to compile kernel `kernel`
2026-05-20 04:32:40,284 INFO fla.ops.backends :: [FLA Backend] common.chunk_bwd_dqkwg -> tilelang
2026-05-20 04:33:49,938 INFO moe_compress.stage5_router_kd ::   epoch=0 step=50 window_loss=0.342537 raw_kl=0.342537 ema=0.342537 best_ema=0.342537@50 lr=4.911e-05 T=1.000 grad_norm=2.3281 | vram=134.3/150GB host_free=62GB gpu_util=98% temp=50C
2026-05-20 04:34:13,034 INFO moe_compress.stage5_router_kd ::   epoch=0 step=100 window_loss=0.426340 raw_kl=0.426340 ema=0.359298 best_ema=0.342537@50 lr=4.439e-05 T=1.000 grad_norm=2.8125 | vram=134.3/150GB host_free=62GB gpu_util=98% temp=50C
2026-05-20 04:34:13,249 INFO moe_compress.stage5_router_kd :: Stage 5: checkpoint saved at step 100 (epoch 0, batch 99)
2026-05-20 04:34:36,438 INFO moe_compress.stage5_router_kd ::   epoch=0 step=150 window_loss=0.224294 raw_kl=0.224294 ema=0.332297 best_ema=0.332297@150 lr=3.645e-05 T=1.000 grad_norm=0.5156 | vram=134.3/150GB host_free=62GB gpu_util=100% temp=52C
2026-05-20 04:34:59,634 INFO moe_compress.stage5_router_kd ::   epoch=0 step=200 window_loss=0.269996 raw_kl=0.269996 ema=0.319837 best_ema=0.319837@200 lr=2.681e-05 T=1.000 grad_norm=0.5508 | vram=134.3/150GB host_free=62GB gpu_util=100% temp=53C
2026-05-20 04:34:59,808 INFO moe_compress.stage5_router_kd :: Stage 5: checkpoint saved at step 200 (epoch 0, batch 199)
2026-05-20 04:35:22,958 INFO moe_compress.stage5_router_kd ::   epoch=0 step=250 window_loss=0.345850 raw_kl=0.345850 ema=0.325039 best_ema=0.319837@200 lr=1.729e-05 T=1.000 grad_norm=2.6250 | vram=134.3/150GB host_free=63GB gpu_util=100% temp=49C
2026-05-20 04:35:46,155 INFO moe_compress.stage5_router_kd ::   epoch=0 step=300 window_loss=0.215397 raw_kl=0.215397 ema=0.303111 best_ema=0.303111@300 lr=9.725e-06 T=1.000 grad_norm=0.4297 | vram=134.3/150GB host_free=63GB gpu_util=98% temp=53C
2026-05-20 04:35:46,334 INFO moe_compress.stage5_router_kd :: Stage 5: checkpoint saved at step 300 (epoch 0, batch 299)
2026-05-20 04:36:09,959 INFO moe_compress.stage5_router_kd ::   epoch=0 step=350 window_loss=0.174018 raw_kl=0.174018 ema=0.277292 best_ema=0.277292@350 lr=5.542e-06 T=1.000 grad_norm=0.7617 | vram=134.3/150GB host_free=63GB gpu_util=100% temp=53C
2026-05-20 04:36:21,533 INFO moe_compress.stage5_router_kd :: Stage stage2p5: reloaded best router state from step=350 (raw_kl_ema=0.277292); missing=986 (expected — non-router params not in best), unexpected=0
2026-05-20 04:36:21,533 INFO moe_compress.utils.model_io :: Saving compressed checkpoint to /cache/ablations/SH/stage2p5_final (stage=stage2p5_final)
2026-05-20 04:36:21,541 INFO moe_compress.utils.model_io ::   wrote compressed_metadata.json (per_layer_num_experts: 40 entries, factored_layers: 0)
2026-05-20 04:36:21,546 INFO moe_compress.utils.model_io :: Saving 9-shard safetensors (stacked MoE format) to /cache/ablations/SH/stage2p5_final
2026-05-20 04:36:30,272 INFO moe_compress.utils.model_io ::   wrote shard 1/9 (model-00001-of-00009.safetensors, 4.98 GB)
2026-05-20 04:36:39,775 INFO moe_compress.utils.model_io ::   wrote shard 2/9 (model-00002-of-00009.safetensors, 4.54 GB)
2026-05-20 04:36:48,465 INFO moe_compress.utils.model_io ::   wrote shard 3/9 (model-00003-of-00009.safetensors, 4.89 GB)
2026-05-20 04:36:57,970 INFO moe_compress.utils.model_io ::   wrote shard 4/9 (model-00004-of-00009.safetensors, 4.90 GB)
2026-05-20 04:37:06,782 INFO moe_compress.utils.model_io ::   wrote shard 5/9 (model-00005-of-00009.safetensors, 4.89 GB)
2026-05-20 04:37:16,005 INFO moe_compress.utils.model_io ::   wrote shard 6/9 (model-00006-of-00009.safetensors, 4.90 GB)
2026-05-20 04:37:24,771 INFO moe_compress.utils.model_io ::   wrote shard 7/9 (model-00007-of-00009.safetensors, 4.89 GB)
2026-05-20 04:37:34,418 INFO moe_compress.utils.model_io ::   wrote shard 8/9 (model-00008-of-00009.safetensors, 4.71 GB)
2026-05-20 04:37:42,380 INFO moe_compress.utils.model_io ::   wrote shard 9/9 (model-00009-of-00009.safetensors, 4.01 GB)
2026-05-20 04:37:42,390 INFO moe_compress.utils.model_io ::   wrote model.safetensors.index.json (1026 keys, 42.72 GB total)
2026-05-20 04:37:42,589 INFO moe_compress.stage5_router_kd :: Stage stage2p5 complete → /cache/ablations/SH/stage2p5_final
2026-05-20 04:37:42,597 INFO __main__ :: Stage 2p5 done in 0h24m37s — durable on Hub: <not uploaded>
2026-05-20 04:37:42,599 INFO __main__ :: Stopping after stage 2 as requested.
2026-05-20 04:37:56,358 INFO __main__ :: Artifacts directory: /cache/ablations/SH
2026-05-20 04:37:56,358 INFO __main__ :: Pipeline target: 35.0% total parameter reduction
2026-05-20 04:37:56,358 INFO __main__ :: Stage 6 will run (stop_after_stage=6): overriding model.attn_implementation 'sdpa' -> 'eager' for student load to satisfy Spec §9 lines 821, 838 (eager attn for Stage 6 gate).
2026-05-20 04:37:56,358 INFO __main__ :: Loading stage 6 input from /cache/ablations/SH/stage2p5_final — stage5_final/ not found (typical ablation harness path: Stage 2 → 2.5 → 6, skipping 3-5).
2026-05-20 04:38:00,885 INFO moe_compress.utils.model_io :: Building skeleton AutoModelForImageTextToText from config on CPU (no_init_weights — random init skipped)
[WARNING] Field "span" in type "ir.TupleType" duplicates an ancestor field in "ir.Type". Child types should not re-register inherited fields.
[WARNING] Field "span" in type "ir.FuncType" duplicates an ancestor field in "ir.Type". Child types should not re-register inherited fields.
[WARNING] Field "span" in type "ir.TensorMapType" duplicates an ancestor field in "ir.Type". Child types should not re-register inherited fields.
[WARNING] Field "binding_blocks" in type "script.ir_builder.relax.FunctionFrame" duplicates an ancestor field in "script.ir_builder.relax.SeqExprFrame". Child types should not re-register inherited fields.
[WARNING] Field "output" in type "script.ir_builder.relax.FunctionFrame" duplicates an ancestor field in "script.ir_builder.relax.SeqExprFrame". Child types should not re-register inherited fields.
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'span' in 'ir.TupleType' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'span' in 'ir.FuncType' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'span' in 'ir.TensorMapType' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'name' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'tag' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'attrs' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'shape' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'dtype' in 'relax.TEPlaceholderOp' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
[WARNING] Field "policy_type" in type "tl.GemmSPWarpPolicy" duplicates an ancestor field in "tl.GemmWarpPolicy". Child types should not re-register inherited fields.
[WARNING] Field "m_warp" in type "tl.GemmSPWarpPolicy" duplicates an ancestor field in "tl.GemmWarpPolicy". Child types should not re-register inherited fields.
[WARNING] Field "n_warp" in type "tl.GemmSPWarpPolicy" duplicates an ancestor field in "tl.GemmWarpPolicy". Child types should not re-register inherited fields.
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'policy_type' in 'tl.GemmSPWarpPolicy' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'm_warp' in 'tl.GemmSPWarpPolicy' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
/usr/local/lib/python3.12/dist-packages/tvm_ffi/registry.py:85: UserWarning: Field 'n_warp' in 'tl.GemmSPWarpPolicy' duplicates an ancestor field. Child types should not re-register inherited fields.
  info = core._register_object_by_index(type_index, cls)
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
2026-05-20 04:38:04,183 INFO moe_compress.utils.model_io :: Streaming state_dict from /cache/ablations/SH/stage2p5_final (9 shards) → cuda
2026-05-20 04:38:22,167 INFO moe_compress.utils.model_io ::   shard 1/9: model-00001-of-00009.safetensors (380 loaded, 0 unexpected)
2026-05-20 04:38:36,736 INFO moe_compress.utils.model_io ::   shard 2/9: model-00002-of-00009.safetensors (51 loaded, 0 unexpected)
2026-05-20 04:38:51,954 INFO moe_compress.utils.model_io ::   shard 3/9: model-00003-of-00009.safetensors (102 loaded, 0 unexpected)
2026-05-20 04:39:07,783 INFO moe_compress.utils.model_io ::   shard 4/9: model-00004-of-00009.safetensors (105 loaded, 0 unexpected)
2026-05-20 04:39:22,426 INFO moe_compress.utils.model_io ::   shard 5/9: model-00005-of-00009.safetensors (102 loaded, 0 unexpected)
2026-05-20 04:39:32,696 INFO moe_compress.utils.model_io ::   shard 6/9: model-00006-of-00009.safetensors (105 loaded, 0 unexpected)
2026-05-20 04:39:33,624 INFO moe_compress.utils.model_io ::   shard 7/9: model-00007-of-00009.safetensors (102 loaded, 0 unexpected)
2026-05-20 04:39:34,543 INFO moe_compress.utils.model_io ::   shard 8/9: model-00008-of-00009.safetensors (54 loaded, 0 unexpected)
2026-05-20 04:39:35,338 INFO moe_compress.utils.model_io ::   shard 9/9: model-00009-of-00009.safetensors (25 loaded, 0 unexpected)
2026-05-20 04:39:35,523 INFO moe_compress.utils.model_io :: Largest single tensor: model.language_model.layers.0.mlp.experts.gate_up_proj (1.07 GB)
2026-05-20 04:39:35,523 INFO moe_compress.utils.model_io ::   load done: missing=0 unexpected=0
2026-05-20 04:39:36,754 WARNING moe_compress.utils.trackio_log :: trackio.log failed; subsequent log failures will not emit a warning
Traceback (most recent call last):
  File "/cache/code/moe_compress/max_quality/src/moe_compress/utils/trackio_log.py", line 190, in trackio_log
    mod.log(metrics)
  File "/usr/local/lib/python3.12/dist-packages/trackio/__init__.py", line 700, in log
    raise RuntimeError("Call trackio.init() before trackio.log().")
RuntimeError: Call trackio.init() before trackio.log().
2026-05-20 04:39:36,757 INFO moe_compress.stage6alt_thermometer :: === Stage 6alt — Thermometer ===
2026-05-20 04:39:36,757 INFO moe_compress.stage6_validate :: Stage 6: MoE experts_implementation = 'grouped_mm'
2026-05-20 04:39:36,759 INFO moe_compress.stage6_validate :: Stage 6 [student]: torch._dynamo.disable on 30 linear-attention sublayer(s)
2026-05-20 04:39:36,762 INFO moe_compress.stage6_validate :: Stage 6 [student]: forced torch-native fallback for chunk_gated_delta_rule + causal_conv1d on 30 Qwen3_5MoeGatedDeltaNet block(s)
2026-05-20 04:39:36,762 INFO moe_compress.stage6_validate :: Stage 6 [student]: replaced fla FusedRMSNormGated with torch-native Qwen3_5MoeRMSNormGated on 30 GDN block(s)

Generating test split:   0%|          | 0/4358 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 4358/4358 [00:00<00:00, 73197.38 examples/s]

Generating train split:   0%|          | 0/36718 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 36718/36718 [00:00<00:00, 446575.44 examples/s]

Generating validation split:   0%|          | 0/3760 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 3760/3760 [00:00<00:00, 427734.83 examples/s]
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (297193 > 262144). Running this sequence through the model will result in indexing errors
2026-05-20 04:39:46,702 INFO moe_compress.stage6alt_thermometer :: Stage 6alt corpus: wikitext (wikitext/wikitext-2-raw-v1:test) 64 x 2048
2026-05-20 04:39:46,702 INFO moe_compress.stage6alt_thermometer :: Stage 6alt: scoring student
2026-05-20 04:39:46,705 INFO moe_compress.stage6alt_thermometer :: Stage 6alt BPT: 64 sequences x len=2048, batch_size=8
2026-05-20 04:39:56,190 INFO moe_compress.stage6alt_thermometer ::   BPT forward 8/8 batches
2026-05-20 04:39:56,304 WARNING lm_eval.models.huggingface :: `pretrained` model kwarg is not of type `str`. Many other model arguments may be ignored. Please do not launch via accelerate or use `parallelize=True` if passing an existing model this way.
2026-05-20 04:39:56,307 WARNING lm_eval.models.huggingface :: Passed an already-initialized model through `pretrained`, assuming single-process call to evaluate() or custom distributed integration
2026-05-20 04:39:56,321 INFO lm_eval.evaluator :: Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-05-20 04:39:56,321 INFO lm_eval.evaluator :: Using pre-initialized model

Generating train split:   0%|          | 0/2251 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 2251/2251 [00:00<00:00, 204447.34 examples/s]

Generating test split:   0%|          | 0/2376 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 2376/2376 [00:00<00:00, 304881.64 examples/s]

Generating validation split:   0%|          | 0/570 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 570/570 [00:00<00:00, 129467.85 examples/s]
2026-05-20 04:40:14,838 INFO lm_eval.evaluator_utils :: Selected tasks:
2026-05-20 04:40:14,838 INFO lm_eval.evaluator_utils :: Task: arc_easy (arc/arc_easy.yaml)
2026-05-20 04:40:14,838 WARNING lm_eval.evaluator :: Overwriting default num_fewshot of arc_easy from None to 0
2026-05-20 04:40:14,838 INFO lm_eval.api.task :: Building contexts for arc_easy on rank 0...

  0%|          | 0/100 [00:00<?, ?it/s]
 82%|████████▏ | 82/100 [00:00<00:00, 818.62it/s]
100%|██████████| 100/100 [00:00<00:00, 818.70it/s]
2026-05-20 04:40:14,964 INFO lm_eval.evaluator :: Running loglikelihood requests

Tokenizing inputs:   0%|          | 0/399 [00:00<?, ?it/s]
Tokenizing inputs: 100%|██████████| 399/399 [00:00<00:00, 7382.71it/s]

Running loglikelihood requests:   0%|          | 0/399 [00:00<?, ?it/s]Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64
Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/399 [00:05<34:05,  5.14s/it]
Running loglikelihood requests:  18%|█▊        | 70/399 [00:05<00:18, 18.09it/s]
Running loglikelihood requests:  35%|███▍      | 139/399 [00:05<00:06, 40.07it/s]
Running loglikelihood requests:  53%|█████▎    | 212/399 [00:05<00:02, 67.85it/s]
Running loglikelihood requests:  72%|███████▏  | 287/399 [00:06<00:01, 100.89it/s]
Running loglikelihood requests:  92%|█████████▏| 367/399 [00:06<00:00, 141.91it/s]
Running loglikelihood requests: 100%|██████████| 399/399 [00:06<00:00, 63.46it/s] 
2026-05-20 04:40:22,127 WARNING lm_eval.models.huggingface :: `pretrained` model kwarg is not of type `str`. Many other model arguments may be ignored. Please do not launch via accelerate or use `parallelize=True` if passing an existing model this way.
2026-05-20 04:40:22,131 WARNING lm_eval.models.huggingface :: Passed an already-initialized model through `pretrained`, assuming single-process call to evaluate() or custom distributed integration
2026-05-20 04:40:22,131 INFO lm_eval.evaluator :: Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-05-20 04:40:22,131 INFO lm_eval.evaluator :: Using pre-initialized model

Generating train split:   0%|          | 0/39905 [00:00<?, ? examples/s]
Generating train split:  50%|█████     | 20000/39905 [00:00<00:00, 189855.81 examples/s]
Generating train split:  98%|█████████▊| 39000/39905 [00:00<00:00, 180246.37 examples/s]
Generating train split: 100%|██████████| 39905/39905 [00:00<00:00, 182705.43 examples/s]

Generating test split:   0%|          | 0/10003 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 10003/10003 [00:00<00:00, 190484.86 examples/s]

Generating validation split:   0%|          | 0/10042 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 10042/10042 [00:00<00:00, 177991.52 examples/s]

Map:   0%|          | 0/39905 [00:00<?, ? examples/s]
Map:   4%|▍         | 1514/39905 [00:00<00:02, 14967.79 examples/s]
Map:   8%|▊         | 3153/39905 [00:00<00:02, 15798.64 examples/s]
Map:  13%|█▎        | 5000/39905 [00:00<00:02, 16344.61 examples/s]
Map:  18%|█▊        | 7177/39905 [00:00<00:04, 8121.66 examples/s] 
Map:  23%|██▎       | 9000/39905 [00:00<00:03, 9883.09 examples/s]
Map:  28%|██▊       | 11000/39905 [00:00<00:02, 11545.03 examples/s]
Map:  33%|███▎      | 13000/39905 [00:01<00:02, 12871.33 examples/s]
Map:  38%|███▊      | 14985/39905 [00:01<00:01, 14183.87 examples/s]
Map:  43%|████▎     | 17087/39905 [00:01<00:01, 14124.57 examples/s]
Map:  47%|████▋     | 18936/39905 [00:01<00:01, 14751.01 examples/s]
Map:  53%|█████▎    | 21033/39905 [00:01<00:01, 14491.64 examples/s]
Map:  57%|█████▋    | 22671/39905 [00:01<00:01, 14939.80 examples/s]
Map:  63%|██████▎   | 25000/39905 [00:01<00:01, 14748.29 examples/s]
Map:  67%|██████▋   | 26623/39905 [00:01<00:00, 15100.94 examples/s]
Map:  73%|███████▎  | 28946/39905 [00:02<00:00, 15228.06 examples/s]
Map:  78%|███████▊  | 31052/39905 [00:02<00:00, 14837.58 examples/s]
Map:  82%|████████▏ | 32616/39905 [00:02<00:00, 8908.03 examples/s] 
Map:  85%|████████▌ | 34007/39905 [00:02<00:00, 9744.30 examples/s]
Map:  89%|████████▉ | 35633/39905 [00:02<00:00, 10999.86 examples/s]
Map:  93%|█████████▎| 37040/39905 [00:02<00:00, 11661.62 examples/s]
Map:  97%|█████████▋| 38672/39905 [00:03<00:00, 12754.32 examples/s]
Map: 100%|██████████| 39905/39905 [00:03<00:00, 12222.95 examples/s]

Map:   0%|          | 0/10042 [00:00<?, ? examples/s]
Map:  18%|█▊        | 1788/10042 [00:00<00:00, 17759.05 examples/s]
Map:  39%|███▉      | 3958/10042 [00:00<00:00, 6388.36 examples/s] 
Map:  54%|█████▎    | 5395/10042 [00:00<00:00, 8068.63 examples/s]
Map:  70%|██████▉   | 7000/10042 [00:00<00:00, 9769.14 examples/s]
Map:  87%|████████▋ | 8691/10042 [00:00<00:00, 11521.23 examples/s]
Map: 100%|██████████| 10042/10042 [00:00<00:00, 10155.40 examples/s]
2026-05-20 04:40:39,389 INFO lm_eval.evaluator_utils :: Selected tasks:
2026-05-20 04:40:39,390 INFO lm_eval.evaluator_utils :: Task: hellaswag (hellaswag/hellaswag.yaml)
2026-05-20 04:40:39,390 WARNING lm_eval.evaluator :: Overwriting default num_fewshot of hellaswag from None to 0
2026-05-20 04:40:39,390 INFO lm_eval.api.task :: Building contexts for hellaswag on rank 0...

  0%|          | 0/200 [00:00<?, ?it/s]
100%|██████████| 200/200 [00:00<00:00, 2214.58it/s]
2026-05-20 04:40:39,495 INFO lm_eval.evaluator :: Running loglikelihood requests

Tokenizing inputs:   0%|          | 0/800 [00:00<?, ?it/s]
Tokenizing inputs:  83%|████████▎ | 667/800 [00:00<00:00, 6664.46it/s]
Tokenizing inputs: 100%|██████████| 800/800 [00:00<00:00, 6639.62it/s]

Running loglikelihood requests:   0%|          | 0/800 [00:00<?, ?it/s]Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/800 [00:03<46:18,  3.48s/it]Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64

Running loglikelihood requests:   8%|▊         | 65/800 [00:05<00:55, 13.29it/s]
Running loglikelihood requests:  16%|█▌        | 129/800 [00:06<00:22, 29.58it/s]
Running loglikelihood requests:  24%|██▍       | 193/800 [00:06<00:12, 48.99it/s]
Running loglikelihood requests:  32%|███▏      | 257/800 [00:06<00:07, 71.33it/s]
Running loglikelihood requests:  40%|████      | 321/800 [00:07<00:05, 95.74it/s]
Running loglikelihood requests:  48%|████▊     | 385/800 [00:07<00:03, 121.04it/s]
Running loglikelihood requests:  56%|█████▌    | 449/800 [00:07<00:02, 146.24it/s]
Running loglikelihood requests:  64%|██████▍   | 513/800 [00:07<00:01, 169.57it/s]
Running loglikelihood requests:  72%|███████▏  | 577/800 [00:08<00:01, 191.11it/s]
Running loglikelihood requests:  80%|████████  | 641/800 [00:08<00:00, 210.28it/s]
Running loglikelihood requests:  88%|████████▊ | 705/800 [00:08<00:00, 228.56it/s]
Running loglikelihood requests:  96%|█████████▌| 769/800 [00:08<00:00, 254.17it/s]
Running loglikelihood requests: 100%|██████████| 800/800 [00:08<00:00, 92.39it/s] 
2026-05-20 04:40:48,983 INFO moe_compress.stage6alt_thermometer :: stage6alt: teacher cache key mismatch; recomputing
2026-05-20 04:40:48,984 INFO moe_compress.stage6alt_thermometer :: Stage 6alt: teacher BPT cache miss — loading teacher
2026-05-20 04:41:14,347 INFO moe_compress.utils.model_io :: Loading Qwen/Qwen3.6-35B-A3B with AutoModelForImageTextToText (dtype=torch.bfloat16, device_map=auto)
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!

Loading weights:   0%|          | 0/1026 [00:00<?, ?it/s]
Loading weights:   0%|          | 1/1026 [00:02<45:45,  2.68s/it]
Loading weights:   0%|          | 2/1026 [00:05<49:16,  2.89s/it]
Loading weights:   1%|          | 7/1026 [00:05<09:43,  1.75it/s]
Loading weights:   1%|          | 12/1026 [00:05<04:40,  3.61it/s]
Loading weights:   2%|▏         | 16/1026 [00:10<10:17,  1.64it/s]
Loading weights:   2%|▏         | 25/1026 [00:10<04:42,  3.54it/s]
Loading weights:   3%|▎         | 30/1026 [00:12<04:56,  3.36it/s]
Loading weights:   3%|▎         | 33/1026 [00:15<07:27,  2.22it/s]
Loading weights:   4%|▍         | 43/1026 [00:15<03:51,  4.24it/s]
Loading weights:   5%|▍         | 48/1026 [00:17<04:13,  3.85it/s]
Loading weights:   5%|▍         | 51/1026 [00:20<06:34,  2.47it/s]
Loading weights:   6%|▌         | 58/1026 [00:24<07:58,  2.02it/s]
Loading weights:   7%|▋         | 67/1026 [00:25<04:47,  3.34it/s]
Loading weights:   8%|▊         | 77/1026 [00:25<02:57,  5.35it/s]
Loading weights:   8%|▊         | 82/1026 [00:29<05:33,  2.83it/s]
Loading weights:   9%|▉         | 95/1026 [00:30<03:06,  4.98it/s]
Loading weights:  10%|▉         | 100/1026 [00:34<05:13,  2.95it/s]
Loading weights:  11%|█         | 113/1026 [00:34<03:03,  4.98it/s]
Loading weights:  12%|█▏        | 118/1026 [00:39<05:13,  2.90it/s]
Loading weights:  12%|█▏        | 127/1026 [00:44<06:03,  2.47it/s]
Loading weights:  14%|█▎        | 139/1026 [00:44<03:46,  3.92it/s]
Loading weights:  14%|█▍        | 147/1026 [00:44<02:48,  5.21it/s]
Loading weights:  15%|█▍        | 152/1026 [00:49<04:44,  3.07it/s]
Loading weights:  16%|█▌        | 163/1026 [00:49<03:00,  4.79it/s]
Loading weights:  16%|█▋        | 169/1026 [00:53<04:35,  3.11it/s]
Loading weights:  18%|█▊        | 181/1026 [00:53<02:47,  5.03it/s]
Loading weights:  18%|█▊        | 187/1026 [00:55<03:00,  4.65it/s]
Loading weights:  19%|█▊        | 191/1026 [00:58<04:25,  3.14it/s]
Loading weights:  19%|█▉        | 196/1026 [00:59<04:17,  3.22it/s]
Loading weights:  19%|█▉        | 198/1026 [01:03<06:21,  2.17it/s]
Loading weights:  20%|██        | 208/1026 [01:03<03:30,  3.89it/s]
Loading weights:  21%|██        | 213/1026 [01:03<02:42,  5.02it/s]
Loading weights:  21%|██▏       | 219/1026 [01:03<01:57,  6.89it/s]
Loading weights:  22%|██▏       | 224/1026 [01:08<04:52,  2.74it/s]
Loading weights:  23%|██▎       | 234/1026 [01:08<02:49,  4.68it/s]
Loading weights:  23%|██▎       | 239/1026 [01:13<05:14,  2.50it/s]
Loading weights:  25%|██▍       | 252/1026 [01:13<02:50,  4.54it/s]
Loading weights:  25%|██▌       | 257/1026 [01:18<04:52,  2.63it/s]
Loading weights:  26%|██▌       | 265/1026 [01:23<05:31,  2.30it/s]
Loading weights:  27%|██▋       | 276/1026 [01:23<03:24,  3.66it/s]
Loading weights:  28%|██▊       | 285/1026 [01:23<02:22,  5.21it/s]
Loading weights:  28%|██▊       | 291/1026 [01:28<03:58,  3.08it/s]
Loading weights:  30%|██▉       | 303/1026 [01:28<02:24,  5.00it/s]
Loading weights:  30%|███       | 310/1026 [01:32<03:38,  3.27it/s]
Loading weights:  31%|███▏      | 321/1026 [01:32<02:21,  4.99it/s]
Loading weights:  32%|███▏      | 327/1026 [01:37<03:42,  3.14it/s]
Loading weights:  33%|███▎      | 334/1026 [01:41<04:46,  2.42it/s]
Loading weights:  34%|███▎      | 346/1026 [01:42<02:53,  3.91it/s]
Loading weights:  35%|███▍      | 355/1026 [01:42<02:03,  5.41it/s]
Loading weights:  35%|███▌      | 360/1026 [01:46<03:37,  3.06it/s]
Loading weights:  36%|███▋      | 372/1026 [01:47<02:13,  4.88it/s]
Loading weights:  37%|███▋      | 377/1026 [01:51<03:37,  2.99it/s]
Loading weights:  38%|███▊      | 389/1026 [01:51<02:12,  4.82it/s]
Loading weights:  38%|███▊      | 394/1026 [01:53<02:25,  4.34it/s]
Loading weights:  39%|███▉      | 398/1026 [01:56<03:17,  3.18it/s]
Loading weights:  39%|███▉      | 403/1026 [02:00<04:50,  2.15it/s]
Loading weights:  40%|████      | 415/1026 [02:00<02:38,  3.84it/s]
Loading weights:  41%|████      | 423/1026 [02:01<01:53,  5.29it/s]
Loading weights:  42%|████▏     | 427/1026 [02:02<02:09,  4.62it/s]
Loading weights:  42%|████▏     | 430/1026 [02:05<03:27,  2.87it/s]
Loading weights:  43%|████▎     | 441/1026 [02:05<01:55,  5.04it/s]
Loading weights:  43%|████▎     | 445/1026 [02:10<03:44,  2.59it/s]
Loading weights:  45%|████▍     | 459/1026 [02:10<01:54,  4.96it/s]
Loading weights:  45%|████▌     | 464/1026 [02:15<03:16,  2.86it/s]
Loading weights:  46%|████▌     | 472/1026 [02:20<03:56,  2.34it/s]
Loading weights:  47%|████▋     | 481/1026 [02:20<02:38,  3.44it/s]
Loading weights:  48%|████▊     | 493/1026 [02:20<01:37,  5.49it/s]
Loading weights:  49%|████▊     | 498/1026 [02:25<02:49,  3.11it/s]
Loading weights:  50%|████▉     | 510/1026 [02:25<01:44,  4.94it/s]
Loading weights:  50%|█████     | 515/1026 [02:30<02:56,  2.90it/s]
Loading weights:  51%|█████▏    | 526/1026 [02:30<01:50,  4.54it/s]
Loading weights:  52%|█████▏    | 531/1026 [02:31<01:58,  4.19it/s]
Loading weights:  52%|█████▏    | 535/1026 [02:34<02:40,  3.07it/s]
Loading weights:  53%|█████▎    | 541/1026 [02:36<02:25,  3.33it/s]
Loading weights:  53%|█████▎    | 543/1026 [02:39<03:35,  2.24it/s]
Loading weights:  54%|█████▍    | 553/1026 [02:39<01:58,  4.00it/s]
Loading weights:  55%|█████▍    | 561/1026 [02:39<01:18,  5.92it/s]
Loading weights:  55%|█████▌    | 566/1026 [02:43<02:37,  2.92it/s]
Loading weights:  56%|█████▌    | 576/1026 [02:44<01:34,  4.77it/s]
Loading weights:  57%|█████▋    | 583/1026 [02:45<01:36,  4.60it/s]
Loading weights:  57%|█████▋    | 587/1026 [02:48<02:18,  3.18it/s]
Loading weights:  58%|█████▊    | 597/1026 [02:48<01:23,  5.11it/s]
Loading weights:  59%|█████▊    | 601/1026 [02:50<01:36,  4.41it/s]
Loading weights:  59%|█████▉    | 604/1026 [02:53<02:33,  2.75it/s]
Loading weights:  59%|█████▉    | 610/1026 [02:58<03:28,  1.99it/s]
Loading weights:  61%|██████    | 621/1026 [02:58<01:52,  3.58it/s]
Loading weights:  61%|██████    | 628/1026 [02:58<01:21,  4.91it/s]
Loading weights:  62%|██████▏   | 634/1026 [03:03<02:20,  2.80it/s]
Loading weights:  63%|██████▎   | 648/1026 [03:03<01:13,  5.12it/s]
Loading weights:  64%|██████▎   | 653/1026 [03:08<02:07,  2.93it/s]
Loading weights:  65%|██████▍   | 663/1026 [03:08<01:21,  4.45it/s]
Loading weights:  65%|██████▌   | 668/1026 [03:09<01:28,  4.05it/s]
Loading weights:  65%|██████▌   | 672/1026 [03:13<02:03,  2.87it/s]
Loading weights:  66%|██████▌   | 679/1026 [03:14<01:46,  3.26it/s]
Loading weights:  66%|██████▋   | 681/1026 [03:17<02:35,  2.22it/s]
Loading weights:  67%|██████▋   | 688/1026 [03:17<01:39,  3.39it/s]
Loading weights:  69%|██████▉   | 706/1026 [03:18<00:41,  7.74it/s]
Loading weights:  70%|███████   | 719/1026 [03:18<00:25, 11.84it/s]
Loading weights:  72%|███████▏  | 735/1026 [03:18<00:15, 18.46it/s]
Loading weights:  73%|███████▎  | 749/1026 [03:18<00:10, 25.56it/s]
Loading weights:  75%|███████▍  | 766/1026 [03:18<00:07, 36.99it/s]
Loading weights:  76%|███████▌  | 780/1026 [03:18<00:05, 46.34it/s]
Loading weights:  77%|███████▋  | 793/1026 [03:18<00:04, 55.88it/s]
Loading weights:  79%|███████▉  | 809/1026 [03:18<00:03, 69.61it/s]
Loading weights:  80%|████████  | 822/1026 [03:18<00:02, 79.92it/s]
Loading weights:  82%|████████▏ | 838/1026 [03:19<00:01, 94.54it/s]
Loading weights:  83%|████████▎ | 852/1026 [03:19<00:01, 101.67it/s]
Loading weights:  85%|████████▍ | 869/1026 [03:19<00:01, 113.62it/s]
Loading weights:  86%|████████▋ | 887/1026 [03:19<00:01, 123.57it/s]
Loading weights:  88%|████████▊ | 903/1026 [03:19<00:00, 127.83it/s]
Loading weights:  89%|████████▉ | 917/1026 [03:19<00:00, 126.13it/s]
Loading weights:  91%|█████████ | 934/1026 [03:19<00:00, 127.72it/s]
Loading weights:  92%|█████████▏| 949/1026 [03:19<00:00, 128.95it/s]
Loading weights:  94%|█████████▍| 965/1026 [03:20<00:00, 130.15it/s]
Loading weights:  96%|█████████▌| 983/1026 [03:20<00:00, 141.24it/s]
Loading weights:  97%|█████████▋| 998/1026 [03:20<00:00, 137.65it/s]
Loading weights:  99%|█████████▊| 1012/1026 [03:20<00:00, 126.71it/s]
Loading weights: 100%|█████████▉| 1025/1026 [03:20<00:00, 106.68it/s]
Loading weights: 100%|██████████| 1026/1026 [03:20<00:00,  5.12it/s] 
2026-05-20 04:44:39,921 INFO moe_compress.utils.model_io :: Model loaded: Qwen3_5MoeForConditionalGeneration on devices=cuda:0:35.1B
2026-05-20 04:44:39,924 INFO moe_compress.stage6_validate :: Stage 6: MoE experts_implementation = 'grouped_mm'
2026-05-20 04:44:39,925 INFO moe_compress.stage6_validate :: Stage 6 [teacher]: torch._dynamo.disable on 30 linear-attention sublayer(s)
2026-05-20 04:44:39,928 INFO moe_compress.stage6_validate :: Stage 6 [teacher]: forced torch-native fallback for chunk_gated_delta_rule + causal_conv1d on 30 Qwen3_5MoeGatedDeltaNet block(s)
2026-05-20 04:44:39,928 INFO moe_compress.stage6_validate :: Stage 6 [teacher]: replaced fla FusedRMSNormGated with torch-native Qwen3_5MoeRMSNormGated on 30 GDN block(s)
2026-05-20 04:44:39,930 INFO moe_compress.stage6alt_thermometer :: Stage 6alt BPT: 64 sequences x len=2048, batch_size=8
2026-05-20 04:44:49,019 INFO moe_compress.stage6alt_thermometer ::   BPT forward 8/8 batches
2026-05-20 04:44:49,019 WARNING lm_eval.models.huggingface :: `pretrained` model kwarg is not of type `str`. Many other model arguments may be ignored. Please do not launch via accelerate or use `parallelize=True` if passing an existing model this way.
2026-05-20 04:44:49,023 WARNING lm_eval.models.huggingface :: Passed an already-initialized model through `pretrained`, assuming single-process call to evaluate() or custom distributed integration
2026-05-20 04:44:49,023 INFO lm_eval.evaluator :: Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-05-20 04:44:49,023 INFO lm_eval.evaluator :: Using pre-initialized model
2026-05-20 04:45:04,495 INFO lm_eval.evaluator_utils :: Selected tasks:
2026-05-20 04:45:04,496 INFO lm_eval.evaluator_utils :: Task: arc_easy (arc/arc_easy.yaml)
2026-05-20 04:45:04,496 WARNING lm_eval.evaluator :: Overwriting default num_fewshot of arc_easy from None to 0
2026-05-20 04:45:04,496 INFO lm_eval.api.task :: Building contexts for arc_easy on rank 0...

  0%|          | 0/100 [00:00<?, ?it/s]
 84%|████████▍ | 84/100 [00:00<00:00, 836.69it/s]
100%|██████████| 100/100 [00:00<00:00, 837.06it/s]
2026-05-20 04:45:04,618 INFO lm_eval.evaluator :: Running loglikelihood requests

Tokenizing inputs:   0%|          | 0/399 [00:00<?, ?it/s]
Tokenizing inputs: 100%|██████████| 399/399 [00:00<00:00, 7407.87it/s]

Running loglikelihood requests:   0%|          | 0/399 [00:00<?, ?it/s]Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64
Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/399 [00:05<34:24,  5.19s/it]
Running loglikelihood requests:  18%|█▊        | 70/399 [00:05<00:18, 17.92it/s]
Running loglikelihood requests:  35%|███▍      | 139/399 [00:05<00:06, 39.66it/s]
Running loglikelihood requests:  53%|█████▎    | 212/399 [00:05<00:02, 67.30it/s]
Running loglikelihood requests:  72%|███████▏  | 287/399 [00:06<00:01, 100.25it/s]
Running loglikelihood requests:  92%|█████████▏| 367/399 [00:06<00:00, 142.86it/s]
Running loglikelihood requests: 100%|██████████| 399/399 [00:06<00:00, 63.12it/s] 
2026-05-20 04:45:12,001 WARNING lm_eval.models.huggingface :: `pretrained` model kwarg is not of type `str`. Many other model arguments may be ignored. Please do not launch via accelerate or use `parallelize=True` if passing an existing model this way.
2026-05-20 04:45:12,004 WARNING lm_eval.models.huggingface :: Passed an already-initialized model through `pretrained`, assuming single-process call to evaluate() or custom distributed integration
2026-05-20 04:45:12,005 INFO lm_eval.evaluator :: Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-05-20 04:45:12,005 INFO lm_eval.evaluator :: Using pre-initialized model
2026-05-20 04:45:22,109 INFO lm_eval.evaluator_utils :: Selected tasks:
2026-05-20 04:45:22,110 INFO lm_eval.evaluator_utils :: Task: hellaswag (hellaswag/hellaswag.yaml)
2026-05-20 04:45:22,110 WARNING lm_eval.evaluator :: Overwriting default num_fewshot of hellaswag from None to 0
2026-05-20 04:45:22,110 INFO lm_eval.api.task :: Building contexts for hellaswag on rank 0...

  0%|          | 0/200 [00:00<?, ?it/s]
100%|██████████| 200/200 [00:00<00:00, 2294.34it/s]
2026-05-20 04:45:22,211 INFO lm_eval.evaluator :: Running loglikelihood requests

Tokenizing inputs:   0%|          | 0/800 [00:00<?, ?it/s]
Tokenizing inputs:  84%|████████▍ | 673/800 [00:00<00:00, 6722.54it/s]
Tokenizing inputs: 100%|██████████| 800/800 [00:00<00:00, 6688.88it/s]

Running loglikelihood requests:   0%|          | 0/800 [00:00<?, ?it/s]Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64

Running loglikelihood requests:   0%|          | 1/800 [00:03<47:11,  3.54s/it]Passed argument batch_size = auto:8.0. Detecting largest batch size
Determined largest batch size: 64

Running loglikelihood requests:   8%|▊         | 65/800 [00:06<00:56, 13.05it/s]
Running loglikelihood requests:  16%|█▌        | 129/800 [00:06<00:23, 29.02it/s]
Running loglikelihood requests:  24%|██▍       | 193/800 [00:06<00:12, 48.08it/s]
Running loglikelihood requests:  32%|███▏      | 257/800 [00:06<00:07, 69.94it/s]
Running loglikelihood requests:  40%|████      | 321/800 [00:07<00:05, 93.84it/s]
Running loglikelihood requests:  48%|████▊     | 385/800 [00:07<00:03, 118.54it/s]
Running loglikelihood requests:  56%|█████▌    | 449/800 [00:07<00:02, 143.16it/s]
Running loglikelihood requests:  64%|██████▍   | 513/800 [00:07<00:01, 166.30it/s]
Running loglikelihood requests:  72%|███████▏  | 577/800 [00:08<00:01, 187.28it/s]
Running loglikelihood requests:  80%|████████  | 641/800 [00:08<00:00, 206.82it/s]
Running loglikelihood requests:  88%|████████▊ | 705/800 [00:08<00:00, 225.05it/s]
Running loglikelihood requests:  96%|█████████▌| 769/800 [00:08<00:00, 253.42it/s]
Running loglikelihood requests: 100%|██████████| 800/800 [00:08<00:00, 90.78it/s] 
2026-05-20 04:45:31,982 INFO moe_compress.stage6alt_thermometer :: stage6alt: teacher BPT cache saved -> /cache/ablations/_shared/thermometer_teacher_cache.json (key=1de59f44ecc5909fe27f32a459674bc96f4fa318564914a33e5c3b9a5f500830)
2026-05-20 04:45:37,088 INFO moe_compress.stage6alt_thermometer :: Stage 6alt complete: corpus=wikitext student_bpt=3.3188 teacher_bpt=2.7242 bpt_gap=0.5946460742381849 top1_agreement=0.7320850491523743 -> /cache/ablations/SH/stage6alt_eval.json
2026-05-20 04:45:37,144 INFO __main__ :: Pipeline complete.
2026-05-20 04:45:38,689 INFO __main__ :: [SH] complete in 108.0 min
* Run finished. Uploading logs to the remote Trackio server (please wait...)
2026-05-20 04:45:38,703 INFO __main__ :: [SH] uploading _shared/ to bucket pirola/moe-strategy-35pct

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            
Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            


  ...e/ablations/_summary.json: 100%|██████████| 1.35kB / 1.35kB            



  ...ablations/_leaderboard.md: 100%|██████████|   643B /   643B            


  ...e/ablations/_summary.json: 100%|██████████| 1.35kB / 1.35kB            



  ...ablations/_leaderboard.md: 100%|██████████|   643B /   643B            
Processing Files (2 / 2)      : 100%|██████████| 1.99kB / 1.99kB,   195B/s  

New Data Upload               : 100%|██████████| 1.99kB / 1.99kB,   195B/s  


  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json:  93%|█████████▎| 1.64MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            


  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json:  93%|█████████▎| 1.64MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            
Processing Files (6 / 10)     :  10%|▉         | 5.00MB / 50.6MB,  488kB/s  









  ...calib_0c65ad6ae83119b9.pt:   7%|▋         | 1.08MB / 16.4MB            










  ...calib_c2de01e417120090.pt:   7%|▋         |  811kB / 12.3MB            











  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            


  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json:  93%|█████████▎| 1.65MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            









  ...calib_0c65ad6ae83119b9.pt:   7%|▋         | 1.08MB / 16.4MB            










  ...calib_c2de01e417120090.pt:   7%|▋         |  811kB / 12.3MB            











  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            
Processing Files (7 / 10)     :  47%|████▋     | 23.7MB / 50.6MB, 2.29MB/s  

New Data Upload               :   7%|▋         | 1.89MB / 28.7MB,  184kB/s  
Processing Files (2 / 2)      : 100%|██████████| 1.99kB / 1.99kB,   195B/s  

New Data Upload               : 100%|██████████| 1.99kB / 1.99kB,   195B/s  

  ...e/ablations/_summary.json: 100%|██████████| 1.35kB / 1.35kB            

  ...ablations/_leaderboard.md: 100%|██████████|   643B /   643B            
2026-05-20 04:45:40,672 INFO __main__ :: Waiting for 1 background upload(s) to finish…



  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json:  95%|█████████▌| 1.69MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            









  ...calib_0c65ad6ae83119b9.pt:  33%|███▎      | 5.40MB / 16.4MB            










  ...calib_c2de01e417120090.pt:  33%|███▎      | 4.05MB / 12.3MB            











  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            
Processing Files (7 / 10)     :  62%|██████▏   | 31.3MB / 50.6MB, 2.91MB/s  

New Data Upload               :  33%|███▎      | 9.46MB / 28.7MB,  899kB/s  


  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json:  97%|█████████▋| 1.71MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            









  ...calib_0c65ad6ae83119b9.pt:  53%|█████▎    | 8.65MB / 16.4MB            










  ...calib_c2de01e417120090.pt:  53%|█████▎    | 6.49MB / 12.3MB            











  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            
Processing Files (7 / 10)     :  73%|███████▎  | 37.0MB / 50.6MB, 3.38MB/s  

New Data Upload               :  53%|█████▎    | 15.1MB / 28.7MB, 1.42MB/s  


  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json: 100%|█████████▉| 1.77MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            









  ...calib_0c65ad6ae83119b9.pt:  99%|█████████▉| 16.2MB / 16.4MB            










  ...calib_c2de01e417120090.pt:  99%|█████████▉| 12.2MB / 12.3MB            











  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            
Processing Files (7 / 10)     :  99%|█████████▉| 50.3MB / 50.6MB, 4.56MB/s  

New Data Upload               :  99%|█████████▉| 28.4MB / 28.7MB, 2.64MB/s  


  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            



  ...ometer_teacher_cache.json: 100%|██████████| 1.77MB / 1.77MB            




  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            





  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            






  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            







  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            








  ...budget_decomposition.json: 100%|██████████|   762B /   762B            









  ...calib_0c65ad6ae83119b9.pt: 100%|██████████| 16.4MB / 16.4MB            










  ...calib_c2de01e417120090.pt: 100%|██████████| 12.3MB / 12.3MB            











  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            
Processing Files (10 / 10)    : 100%|██████████| 50.6MB / 50.6MB, 4.37MB/s  

New Data Upload               : 100%|██████████| 28.7MB / 28.7MB, 2.54MB/s  
Processing Files (10 / 10)    : 100%|██████████| 50.6MB / 50.6MB, 4.37MB/s  

New Data Upload               : 100%|██████████| 28.7MB / 28.7MB, 2.54MB/s  

  ...red/stage1_blacklist.json: 100%|██████████| 1.68MB / 1.68MB            

  ...ometer_teacher_cache.json: 100%|██████████| 1.77MB / 1.77MB            

  ...calib_177709b63084cd14.pt: 100%|██████████| 1.64MB / 1.64MB            

  ...age1_ablation_filter.json: 100%|██████████| 24.2kB / 24.2kB            

  ...hared/stage1_budgets.json: 100%|██████████| 2.81kB / 2.81kB            

  ...ared/ablation_config.yaml: 100%|██████████| 5.63kB / 5.63kB            

  ...budget_decomposition.json: 100%|██████████|   762B /   762B            

  ...calib_0c65ad6ae83119b9.pt: 100%|██████████| 16.4MB / 16.4MB            

  ...calib_c2de01e417120090.pt: 100%|██████████| 12.3MB / 12.3MB            

  ...calib_2665e9aa8c2907ca.pt: 100%|██████████| 16.8MB / 16.8MB            
2026-05-20 04:45:42,613 INFO __main__ :: [SH] uploading stage6alt_eval.json to bucket pirola/moe-strategy-35pct

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            


  ...ns/SH/stage6alt_eval.json: 100%|██████████|   897B /   897B            


  ...ns/SH/stage6alt_eval.json: 100%|██████████|   897B /   897B            
Processing Files (1 / 1)      : 100%|██████████|   897B /   897B,   ???B/s  

New Data Upload               : 100%|██████████|   897B /   897B,   ???B/s  
Processing Files (1 / 1)      : 100%|██████████|   897B /   897B,   ???B/s  

New Data Upload               : 100%|██████████|   897B /   897B,   ???B/s  

  ...ns/SH/stage6alt_eval.json: 100%|██████████|   897B /   897B            
2026-05-20 04:45:43,706 INFO __main__ :: [SH] upload complete → pirola/moe-strategy-35pct/SH/stage6alt_eval.json

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            


  ...e/ablations/_summary.json: 100%|██████████| 1.35kB / 1.35kB            



  ...ablations/_leaderboard.md: 100%|██████████|   643B /   643B            


  ...e/ablations/_summary.json: 100%|██████████| 1.35kB / 1.35kB            



  ...ablations/_leaderboard.md: 100%|██████████|   643B /   643B            
Processing Files (2 / 2)      : 100%|██████████| 1.99kB / 1.99kB,   ???B/s  

New Data Upload               : |          |  0.00B /  0.00B,   ???B/s  

  ...e/ablations/_summary.json: 100%|██████████| 1.35kB / 1.35kB            

  ...ablations/_leaderboard.md: 100%|██████████|   643B /   643B            
2026-05-20 04:45:44,504 INFO __main__ :: Wrote summary + leaderboard to /cache/ablations
2026-05-20 04:45:44,504 INFO __main__ :: ============================================================
2026-05-20 04:45:44,504 INFO __main__ :: Ablation summary (completed=1, failed=0):
2026-05-20 04:45:44,504 INFO __main__ ::   SH : bpt_gap=0.5946 top1_agree=0.7321
2026-05-20 04:45:44,504 INFO __main__ :: ============================================================
2026-05-20 04:45:44,504 INFO __main__ :: Leaderboard: /cache/ablations/_leaderboard.md
* Run finished. Uploading logs to the remote Trackio server (please wait...)
/dev/vda4       484G  276G  159G  64% /cache
[mergeheal] 04:45:44 :: STEP 3 — save everything to bucket pirola/moe-strategy-35pct
Hint: A new version of huggingface_hub (1.15.0) is available! You are using version 1.14.0.
To update, run: hf update
Sync plan: /cache/ablations/ -> hf://buckets/pirola/moe-strategy-35pct/
  Uploads: 36
  Downloads: 0
  Deletes: 0
  Skips: 3
Syncing...

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B