noemacee/gr00t-n17-g1-sonic-ablate-sonic2

GR00T N1.7 G1 sonic — data ablation, sonic2 arm

Private research checkpoint produced by MicroAGI from NVIDIA GR00T N1.7 and an internal Unitree G1 teleoperation dataset.

One arm of a three-way data ablation over two capture sessions that differ in camera geometry. This arm trains on g1-demoday-teleop-sonic-2 only (640x480).

Published checkpoint: step 53,000 of 60,000, selected on flow loss (also best action MSE on disk).

Lowest eval_flow_loss of the run (0.123631) and also the lowest eval_action_mse among the checkpoints that were saved. The global action-MSE minimum was step 58000 (0.140586, 0.35% better), but 58000 is not in the checkpoint schedule so no weights exist for it.

Not evaluated on hardware. Offline metrics only. No physical rollouts have been run on this checkpoint.

Task

Put all the objects on the table into the gray tote, then carry the tote to the blue roller conveyor and place it on the conveyor.

Single task, single embodiment, single camera (ego_view).

Input geometry — read this before deploying

Stream the camera at its native training resolution, RGB, 50 fps, no pre-scaling: 640x480. The first preprocessing op is an area-average downscale to a 256 px short edge; feeding it the native frame is what makes that anti-aliasing correct.

640x480 (native)
  1. SmallestMaxSize(256, INTER_AREA)   -> 341x256
  2. FractionalCenterCrop(0.95)         -> 323x243
  3. SmallestMaxSize(256, INTER_AREA)   -> 340x256
  4. Qwen2VL smart_resize(factor=32)    -> 352x256
                                          = 88 vision tokens

• cv2.INTER_AREA is not optional — bilinear or Lanczos at the same output size changes the pixel statistics the model was fit on.
• The crop is linear (int(h*0.95) x int(w*0.95)), not area-0.95. Aspect ratio is preserved end to end, which is why the two arms differ in token count.
• Normalization is mean/std = 0.5 per channel, applied by the VLM processor. Do not pre-normalize.
• Single frame, no history. video.delta_indices = [0]. No ring buffer needed.
• Color jitter is train-only; the eval path is deterministic.

No letterboxing — do not pad to square

The training config set GR00T_LETTER_BOX=1 and the launcher logged letter_box_transform=True, but letterboxing was never actually applied. Gr00tN1d7Processor.from_pretrained applies overrides through an allowlist that does not contain letter_box_transform, so the flag was silently dropped and the processor was built from the base nvidia/GR00T-N1.7-3B config. config.json and processor_config.json in this repo both correctly record letter_box_transform: false, and that is what the weights were trained under.

Padding frames to square at inference would be a mismatch, not a fix. Match the chain above instead.

Encoding budget: the training video averaged 6.07 Mb/s AV1 (yuv420p), roughly 0.38-0.40 bits/pixel. Compression artifacts at about that level are in-distribution; going much below it puts artifacts into the frame that survive the downscale.

Metrics at the published checkpoint

Metric	Value
eval_action_mse	0.141081
eval_action_mae	0.255489
eval_flow_loss	0.123631

eval_action_mse/mae are a full denoising rollout against ground truth in train-normalized units — what the policy is graded on. eval_flow_loss is the flow-matching objective (MSE on predicted velocity at one random timestep) — what is optimized. Evaluation is a paired comparison: the flow timestep and the rollout's initial noise are pinned per batch, so successive evals see identical samples, timesteps and noise, and step-to-step differences reflect the model rather than sampling noise.

Held-out val for this arm: 14 episodes / 40,888 frames, disjoint at episode granularity. Evaluated every 1,000 steps on the first 4,000 samples. Full per-step history is in observations/eval_history.csv.

The two arms' metrics are not comparable to each other. Each arm owns its own train/val partition drawn from its own capture session, so eval_* numbers are computed on different data. A lower number in one arm does not mean a better model. These are ranked by deploying them on the robot.

Training

Field	Value
Base checkpoint	nvidia/GR00T-N1.7-3B
Dataset	g1-demoday-teleop-sonic-2 (internal, LeRobot v2.1)
Robot type	unitree_g1_omnihand_sonic
Embodiment tag	new_embodiment
Camera preset	640x480, AV1, yuv420p, 50 fps
Train episodes / frames	95 / 280,366
Val episodes / frames	14 / 40,888
Hardware	2x NVIDIA RTX PRO 6000 Blackwell (96 GB), DeepSpeed ZeRO-2
Effective batch	48 = 12 per device x 2 ranks x accumulation 2
Steps	60,000 total; this checkpoint is step 53,000
Sample draws	2,880,000 over a 280,366-frame corpus (~10.3 epochs)
Learning rate	2e-4, cosine_with_min_lr, min 5e-5, warmup ratio 0.05
Weight decay / grad clip	1e-5 / 1.0
Optimizer	adamw_torch
Attention	SDPA (no flash-attn wheel for sm_120)
Augmentation	color jitter — brightness 0.3, contrast 0.4, saturation 0.5, hue 0.08
Precision	bf16 compute, tf32 matmul; FP8 ZeRO-2 gradient transport
Action horizon	40 (0.8 s at 50 fps), 4 inference timesteps
Final train loss	0.106262
Run wall-clock	9 h 41 m for 60,000 steps (1.72 it/s, 82.6 samples/s)
W&B run	noemace/sonic_groot/ablate-sonic2-2gpu-60k-r1

Val is normalized with train statistics; statistics.json here has been verified to hash-match this arm's train split.

Contents

Inference-ready. Two bf16 safetensor shards (1,031 tensors, 6.91 GB) plus config.json, and the processor flattened to the repo root (processor_config.json, statistics.json, embodiment_id.json) so from_pretrained works directly against this repo.

Also kept: experiment_cfg/ (full resolved run config and dataset statistics), training_args.bin, and wandb_config.json. Note the run was trained with save_only_model: true, so there is no trainer_state.json and these weights cannot be resumed from.

observations/ carries the run record: eval_history.csv (60 rows, every 1,000 steps), train_history.csv, the gzipped training log, and the raw W&B run directory.

from transformers import AutoModel, AutoProcessor
import gr00t.model.gr00t_n1d7.processing_gr00t_n1d7  # registers Gr00tN1d7

repo = "MicroAGI-Labs/gr00t-n17-g1-sonic-ablate-sonic2"
model = AutoModel.from_pretrained(repo, trust_remote_code=True)
proc  = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
assert proc.letter_box_transform is False   # see input-geometry note above

wandb_config.json records project finetune-gr00t-n1d7, but the run actually logged to sonic_groot. The W&B row above is correct.

Sibling arms

Repository	Arm / camera	Step	eval_action_mse	eval_flow_loss
gr00t-n17-g1-sonic-ablate-sonic1-mse	sonic1 / 720x406	60,000	0.133321	0.130176
gr00t-n17-g1-sonic-ablate-sonic1-flow	sonic1 / 720x406	57,000	0.135011	0.119489

The two arms' metrics are not comparable to each other. Each arm owns its own train/val partition drawn from its own capture session, so eval_* numbers are computed on different data. A lower number in one arm does not mean a better model. These are ranked by deploying them on the robot.