lilcheaty commited on
Commit
8c5abfe
·
verified ·
1 Parent(s): e3b498c

model card: single-pass build, retract double-quant caveat

Browse files
Files changed (1) hide show
  1. README.md +35 -21
README.md CHANGED
@@ -108,32 +108,42 @@ offloaded to CPU after encoding — it runs once per prompt, not once per sampli
108
 
109
  ## How `pruned_nvfp4` was built
110
 
111
- The obvious tool does **not** work. The StarNodes model converter passes non-floating-point
112
- tensors through untouched:
 
113
 
114
  ```python
115
- if not tensor.dtype.is_floating_point:
116
- return tensor # already-int8 weights are copied verbatim
117
  ```
118
 
119
- so running it on an int8 checkpoint silently produces a byte-identical file. A real
120
- dequantize -> requantize is required. `comfy-kitchen` exposes both halves:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
 
122
  ```python
123
- params = ckt.TensorWiseINT8Layout.Params(
124
- scale=weight_scale, orig_dtype=torch.bfloat16, orig_shape=qdata.shape,
125
- is_weight=True, convrot=True, convrot_groupsize=256)
126
- deq = ckt.TensorWiseINT8Layout.dequantize(qdata, params) # -> bf16
127
- nq, nparams = ckt.TensorCoreNVFP4Layout.quantize(deq.contiguous()) # -> NVFP4
128
- tensors = ckt.TensorCoreNVFP4Layout.state_dict_tensors(nq, nparams)
129
  ```
130
 
131
- Per-layer config lives in a `comfy_quant` uint8 tensor holding JSON, e.g.
132
- `{"format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256}`, rewritten to
133
- `{"format": "nvfp4"}` on output. Note `quantize()` requires bf16/fp16 float32 raises
134
- `Unsupported dtype code`.
135
-
136
- Full script: `pruned_to_nvfp4.py` in this repo. 200 layers, ~6 seconds on one GPU.
137
 
138
  ## Prompting: H3 wants a structured IR, not prose
139
 
@@ -217,6 +227,10 @@ frame `length` must satisfy 17n+5.
217
  VRAM (~20 GB vs ~12 GB) and can spend ~14% more time per step, Comfy-Org's
218
  `pruned_int8_convrot` may be the better choice. This repo's value is smallest-footprint
219
  and fastest, not highest-fidelity.
 
 
 
 
220
  - That comparison is **not controlled**: the two runs also differed in text encoder
221
  (nvfp4_awq vs int8_convrot), it was n=1, and matched seeds do not produce matched
222
  trajectories across different quantization. Treat it as a lead, not a result.
@@ -226,9 +240,9 @@ frame `length` must satisfy 17n+5.
226
  - 15s (362 frames) generates fine; see the VRAM note below.
227
  - The **`fl2va` file is unbenchmarked** — it was produced by the same script over the same
228
  layer set as `ref2va`, but has not been run end-to-end.
229
- - `pruned_nvfp4` is **doubly quantized** (bf16 -> int8_convrot by Comfy-Org -> NVFP4 here).
230
- Error from both passes compounds. It held up in testing, but that is a real caveat.
231
- - Benchmarks are single-GPU, one card, one resolution.
232
 
233
  Failure cases are welcome in the discussions tab — concrete artifacts beat aggregate scores.
234
 
 
108
 
109
  ## How `pruned_nvfp4` was built
110
 
111
+ **Single pass from `pruned_bf16`.** Comfy-Org's `minimax_h3_{ref2va,fl2va}_pruned_bf16`
112
+ (40.2 GB) is full-precision weights with AdaLN already refactored into the timestep table, so
113
+ the 200 attn/mlp layers are quantized once, straight from bf16:
114
 
115
  ```python
116
+ nq, nparams = ckt.TensorCoreNVFP4Layout.quantize(w.contiguous()) # bf16 -> NVFP4
117
+ tensors = ckt.TensorCoreNVFP4Layout.state_dict_tensors(nq, nparams)
118
  ```
119
 
120
+ The layer set is `blocks.<N>.{attn.qkv_proj, attn.out_proj, mlp.fc1, mlp.fc2}.weight` 50
121
+ blocks x 4 = 200, verified set-identical to the layers Comfy-Org quantize in their
122
+ `pruned_int8_convrot` release. Everything else (AdaLN, norms, embedders, token_refiner, all
123
+ biases) is copied verbatim at source precision: 332 tensors, 1132 in the output.
124
+
125
+ Per-layer config lives in a `comfy_quant` uint8 tensor holding JSON, written as
126
+ `{"format": "nvfp4"}`; there is no file-level `__metadata__`.
127
+
128
+ Full script: `bf16_to_nvfp4.py` in this repo. 200 layers in ~5 seconds on one GPU.
129
+
130
+ <details>
131
+ <summary>Earlier builds requantized from int8 (superseded)</summary>
132
+
133
+ Before Comfy-Org published a pruned bf16 checkpoint, the only pruned source was
134
+ `pruned_int8_convrot`, so these files were built bf16 -> int8_convrot (Comfy-Org) -> NVFP4 —
135
+ two lossy passes whose error compounds. That path needed a real dequantize -> requantize,
136
+ because the StarNodes model converter passes non-floating-point tensors through untouched:
137
 
138
  ```python
139
+ if not tensor.dtype.is_floating_point:
140
+ return tensor # already-int8 weights are copied verbatim
 
 
 
 
141
  ```
142
 
143
+ so running it on an int8 checkpoint silently produced a byte-identical file. The old script
144
+ (`pruned_to_nvfp4.py`) is kept for reference. Both files were rebuilt single-pass on
145
+ 2026-08-05; the output key set and dtype census are unchanged, only the error path is.
146
+ </details>
 
 
147
 
148
  ## Prompting: H3 wants a structured IR, not prose
149
 
 
227
  VRAM (~20 GB vs ~12 GB) and can spend ~14% more time per step, Comfy-Org's
228
  `pruned_int8_convrot` may be the better choice. This repo's value is smallest-footprint
229
  and fastest, not highest-fidelity.
230
+ - **That observation predates the single-pass rebuild and has not been repeated since.** It
231
+ was made against the doubly-quantized files, so some of the smearing may have been
232
+ compounding error rather than 4-bit weights. Treat the int8-is-better lead as untested
233
+ against the current files.
234
  - That comparison is **not controlled**: the two runs also differed in text encoder
235
  (nvfp4_awq vs int8_convrot), it was n=1, and matched seeds do not produce matched
236
  trajectories across different quantization. Treat it as a lead, not a result.
 
240
  - 15s (362 frames) generates fine; see the VRAM note below.
241
  - The **`fl2va` file is unbenchmarked** — it was produced by the same script over the same
242
  layer set as `ref2va`, but has not been run end-to-end.
243
+ - Benchmarks are single-GPU, one card, one resolution, and were measured on the older
244
+ doubly-quantized files. Size and layout are unchanged by the rebuild, so s/it and VRAM
245
+ should carry over, but they have not been re-measured.
246
 
247
  Failure cases are welcome in the discussions tab — concrete artifacts beat aggregate scores.
248