mrjackspade/Ideogram4-Natural-Language-Text-Encoder

Ideogram 4 Natural-Language Text Encoder — S020 v1

This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON. It operates as a direct checkpoint replacement through the stock ComfyUI loader.

This repository publishes the strength-0.2 training lineage at 500-step intervals through optimizer step 5000. Each milestone contains a merged scaled-FP8 text encoder and its matching rank-64 LoRA. A milestone resolves to the first complete checkpoint in its 500-step band, so the initial 500 milestone resolved to optimizer step 510. See MILESTONES.md for the live artifact index.

Files

Path	Purpose
text_encoders/*_step_XXXXXXXX_fp8_scaled.safetensors	Standalone drop-in scaled-FP8 text encoders
loras/*_step_XXXXXXXX_r64_a64.safetensors	Matching unmerged research LoRAs
manifests/checkpoint_step_XXXXXXXX.json	Training checkpoint and controller metadata
manifests/merge_step_XXXXXXXX.json	Merge inputs, output hash, and per-projection quantization metrics
MILESTONES.md	Human-readable live release index
MILESTONES.json	Machine-readable live release index
SHA256SUMS	SHA-256 inventory for every merged encoder and LoRA
COMPATIBILITY.md	Component boundary, provenance, and third-party compatibility notice

The LoRA uses this project's native tensor names (layers.N.<projection>.lora_a and lora_b). The merged text encoder is the recommended artifact for inference; the LoRA is provided for research, inspection, and reproducibility.

Installation

Download the merged file into the normal ComfyUI text-encoder directory:

ComfyUI/models/text_encoders/

Select it in the same Ideogram 4 text-encoder loader used for the stock qwen3vl_8b_fp8_scaled.safetensors.

What was trained

Task definition

Observed stock behavior motivated the following target:

short natural language       -> weak/broken stock conditioning
detailed natural language    -> weak/broken stock conditioning
structured Magic-Prompt JSON -> functional stock conditioning

Each natural-language prompt therefore has a structured teacher target. Because the natural-language and JSON sequences differ in length, training matches their effect at Ideogram's image-side response:

c_T = stock_text_encoder(teacher_json)
c_S = adapted_text_encoder(natural_language)

v_T = ideogram_conditional(z_t, t, c_T)
v_S = ideogram_conditional(z_t, t, c_S)

loss = mean((v_S - v_T) ** 2)

Ideogram parameters remain frozen while gradients pass through the student conditional forward to the text-encoder adapters. The objective is entirely focused on matching the structured teacher response.

Stock text-encoder path

The stock Ideogram path uses Qwen3-VL 8B as a contextual feature extractor. Hidden states from 13 taps (0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 35) are concatenated to 53,248 features per text token, then projected to the 4,608-wide Ideogram DiT. The student preserves this architecture and tensor layout.

Dataset and teacher

• 4,000 source prompts from public Civitai generation metadata.
• Exactly 1,000 prompts from each source rating group: SFW, R, X, and XXX.
• Exact MaxMin diversity selection in Ideogram's own conditioning space, after literal cleanup, provenance filtering, and exclusion of edit/image-input records.
• 3,900 training prompts and 100 deterministic owner-reviewed holdouts (25 per rating group).
• Eight cached behavior states per training prompt: 31,200 training examples.
• Teacher JSON generated by accounts/fireworks/models/nemotron-3-ultra-nvfp4 through Fireworks, temperature 0, high reasoning effort, using the project's Magic-Prompt system instructions.
• Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress projection, stored as float16.

This release contains the trained weights, configuration, checkpoint metadata, and merge provenance.

Eight-state trajectory supervision

Teacher trajectories use the stock V4_DEFAULT_20 20-step schedule at 512×512. Eight deterministic pre-step states are captured at loop indices:

[19, 16, 14, 11, 8, 5, 3, 0]

Each cache entry stores the pre-step latent and exact positive conditional teacher velocity. All eight states are independent velocity-matching examples during training. The unconditional branch is a separate text-free model and is identical for teacher and student, so conditional velocity matching is sufficient for the text-encoder objective.

Strength-0.2 first-step steering target

The teacher trajectory includes one first-conditional-pass spatial correction derived from the authoritative Ideogram 4 debanner direction:

• correction tensor SHA-256: 5ce873adae5701e9d5f05ebfa8f8b923a1622745c6e9a2bcb3e22fd090ed30c3;
• Ideogram blocks: 25, 26, 27, and 28;
• distillation strength: 0.2;
• first denoising step and first positive conditional pass only;
• image tokens only;
• subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's original L2 norm;
• the correction is confined to the positive conditional pass.

Only the first cached target velocity is directly corrected. The later seven target velocities are uncorrected but evaluated on the trajectory produced by the corrected first update. The student always uses an unpatched stock Ideogram conditional model; the behavior is distilled into the text encoder.

The cache contract fingerprints the correction tensor, metadata, strength, block set, schedule, resolution, conditional checkpoint, conditioning, and all eight target velocities. Incompatible caches are rejected before model loading.

Adapter geometry

This checkpoint uses the following adapter geometry:

• rank: 64;
• alpha: 64;
• blocks: all 36 Qwen text blocks (0–35);
• projections per block: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj;
• adapted projections: 252;
• trainable parameters: 174,587,904;
• LoRA tensors: 504.

Per block, the adapter contains 4,849,664 parameters. Adapting all 36 blocks gives every tapped depth a trainable path for reshaping the conditioning features consumed by Ideogram.

Optimization and numerics

• Optimizer: AdamW, FP32 adapter parameters and optimizer state.
• Weight decay: 0.01.
• Physical batch size: 2.
• Gradient accumulation: 8.
• Effective optimizer batch: 16 behavior examples.
• Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
• Gradient clipping: global norm 1.0.
• Optimizer state is held on CPU between updates.
• The training graph contains the Qwen text transformer with all 36 text blocks.
• Initial learning rate: 3.2e-4.

Training is quantization-aware for the final deployment format. For each adapted projection, the forward path forms the dequantized stock weight plus the LoRA delta, then requantizes using the stock per-tensor scale to float8_e4m3fn. A straight-through estimator supplies LoRA gradients without constructing a dense base-weight gradient.

Learning rate is managed by a transactional loss-curve controller. It evaluates fixed 50-step windows every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed window for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at the rollback boundary, and continues at half the LR. A rollback requires both deterministic held-out loss and interval-averaged training loss to fail their descent tests. Reaching the positive LR floor allows training to continue.

Step-510 checkpoint metrics

Metric	Value
Optimizer step	510
Training velocity MSE	0.0179036569
Held-out velocity MSE	0.0171197626
Learning rate	8e-5
LR reductions completed	2
Gradient norm	0.0149142
Step time on RTX 5090	12.94 s
Peak reserved VRAM	24.463 GiB
Behavior microbatches	8 batch-2, 0 singleton

Evaluation combines velocity MSE with fixed-seed stock/teacher/checkpoint image comparisons and owner visual review. Standardized perceptual, OCR, and broad safety benchmarking remain future work.

Merge and deployment format

The standalone file was merged on CPU into the exact stock Comfy scaled-FP8 checkpoint layout:

• stock base SHA-256: 4ba424cf62e51392e4d1a39933e803706f4e823c1065f36aaf149c6453f66bcd;
• adapter SHA-256: 85359811e619276f85742072c2b3327915bbb56bcefb874f2f3101a5d1216eef;
• output SHA-256: f0d1009af7407b60bda169967bb5e4c3e8ea264042b70f3d7b667bb0c2404ece;
• output tensors: 1,254;
• merged projections: 252;
• stock per-tensor FP8 scales preserved;
• tensor inventory remains compatible with the stock text-encoder checkpoint layout.

The base file is byte-identical to the text encoder published in Comfy-Org/Qwen3-VL and Comfy-Org/Ideogram-4 under the base SHA above.

Limitations

• Step 510 is the first public research milestone from a longer training lineage.
• The learned conditioning targets Ideogram 4's frozen conditional model and Comfy text-encoder ingress path; its intended use is image-conditioning inference through that interface.
• Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at other resolutions, schedules, samplers, and seeds requires independent evaluation.
• Teacher targets can be imperfect or semantically inconsistent despite validation.
• The corpus deliberately spans adult source-rating groups. Deployment requires an application-level safety policy appropriate to the intended use.
• Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
• FP8 requantization rounds low-magnitude LoRA deltas to the available scaled-FP8 grid; per-projection retention metrics are included in the merge manifest.

Component and licensing boundary

The release artifacts consist exclusively of modified Qwen3-VL text-encoder parameters. They start from the independently published Comfy-Org/Qwen3-VL scaled-FP8 checkpoint and are released under its Apache-2.0 license.

During training, a separately obtained, frozen Ideogram 4 model served as a differentiable downstream evaluator of the conditioning interface. Optimization updated the Qwen adapter parameters, and the release files contain the resulting Qwen text-encoder weights.

Ideogram 4 is acquired separately under its own license and usage policy. Users combining the two components are responsible for the terms applicable to each. This repository is independently produced and has no affiliation with Ideogram, Inc.