Training a coding agent using the OpenCode harness in remote HF sandboxes with TRL and OpenEnv

Sergio Paniego

sergiopaniego

Follow

TRL recently shipped support for training a coding agent natively with OpenEnv, via the OpenCode harness (see the announcement).

In this post we walk through a concrete example, end to end. We take a real, off-the-shelf coding agent, let it run its own loop against real coding problems, and train it with AsyncGRPO on the exact tokens it produced. Then we go one step further and run every rollout in its own remote Hugging Face sandbox, so the rollouts scale out beyond a single node.

We use OpenCode as the concrete agent, but the architecture is agent-agnostic in principle: any agent that runs in a sandbox and talks to the model over an API whose calls can be captured (today, OpenAI-compatible chat completions) can be trained the same way.

We will cover the architecture, how each rollout runs in a remote sandbox, how to run the whole thing on Hugging Face Jobs, and whether it actually learns.

Reproduce it. The full training script is opencode_hf_sandbox.py. Its docstring has the exact commands to serve vLLM, expose it to the sandboxes with a tunnel, and run the trainer. To do all three in a single Hugging Face Jobs job, use the launcher:

hf jobs uv run --flavor h200x2 --secrets HF_TOKEN --timeout 7200s launcher.py

More on the setup below.

Loop-owning: the agent runs, the trainer learns

TRL can now train an agent harness directly, through its OpenEnv integration. The usual way to RL-train an agent is for the trainer to drive the loop itself: sample a turn, parse the tool calls, run them, feed the results back. That trains a copy of the loop, not the harness you actually run.

Loop-owning flips it. The harness (here, OpenCode) runs its own loop, exactly as it ships. TRL never drives the turns, it reads back what the harness did and trains on that.

The agent runs to completion on its own, and TRL trains on the exact tokens the harness produced.

The architecture, piece by piece

Four components, wired together by TRL and OpenEnv:

• The harness, in a sandbox. Each rollout gets an isolated OpenEnv session: a container with its own filesystem and processes, started from a Docker image that carries the harness. Inside, it runs its full tool loop against a real workspace.
• A transparent proxy that captures tokens. It sits between the harness and your vLLM server and records every model call with its token IDs and logprobs, per turn. This is the key trick: nothing is re-tokenized or guessed, you train on the real tokens the policy emitted.
• A hidden-test verifier for the reward. You define a verify() that inspects the final workspace and returns a reward. This part is entirely task-specific. In our example the problems come from agentica-org/DeepCoder-Preview-Dataset: each problem statement is the prompt the agent works on, and the same problem's hidden tests (never shown to the agent) are the reward. The agent writes solution.py and is rewarded on the fraction of those tests it passes.
• The trainer. When the harness finishes, TRL reconstructs the training samples from the recorded turns and runs GRPO. The reward propagates to every trained token through the group-relative advantage.

The whole thing is wired with one worker. Setting harness_adapter=None selects loop-owning mode:

from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer
from trl.experimental.async_grpo.openenv_harness import HarnessRolloutWorker, has_tool_call

worker = HarnessRolloutWorker(
    harness_session_factory=build_factory(...),  # an OpenEnv ResourceSessionFactory
    harness_adapter=None,                         # loop-owning: the agent runs its own loop
    rollout_reward_fn=opencode_reward,            # dense verifier + penalties for bad behavior
    train_turn_fn=has_tool_call,                  # reinforce action turns, not prose
    ...
)

One remote sandbox per rollout

Running the harness on the same node works (that is the local-subprocess example), but it caps you at one machine. To scale the rollouts out, the sandbox becomes a remote Hugging Face sandbox, one per rollout, chosen by a single backend:

from opencode_env.sandbox import HFSandboxBackend

factory = OpenCodeSessionFactory(
    config=config,
    sandbox_backend=HFSandboxBackend(image="ghcr.io/huggingface/openenv-opencode-sandbox:latest"),
    mode="transparent_proxy",   # the in-sandbox proxy captures token ids + logprobs
    verifier=DeepCoderStdinVerifier(tests_by_id),
)

That image is pre-baked: the harness and the proxy are already inside, so there is no cold install per rollout. Every rollout gets a clean, isolated sandbox, and they run concurrently.

Same training logic. The rollouts just run in parallel across the cloud, one isolated sandbox each.

The one thing to get right: two vLLM URLs

There is exactly one catch, and it is inherent to remote rollouts. The trainer and vLLM live on the same node and sync weights over NCCL, so that link stays on localhost. But the remote sandboxes cannot see localhost, they need to reach the same vLLM from the outside.

So there are two URLs:

The sandbox URL is not tied to any tunnel provider. Point it at any vLLM the sandboxes can reach.

Running it on Hugging Face Jobs

On Hugging Face Jobs you submit one script, so a small launcher wraps the three pieces (serve vLLM, expose it with a tunnel, run the trainer) into a single job:

hf jobs uv run --flavor h200x2 --secrets HF_TOKEN --timeout 7200s launcher.py

Inside the job, the launcher wires up the two URLs and runs the trainer:

python opencode_hf_sandbox.py --model Qwen/Qwen3-8B \
    --vllm-url http://localhost:8000 --sandbox-vllm-url <tunnel-url> \
    --n-prompts 32 --max-steps 10

The cost detail. Every remote sandbox runs on the cpu-basic flavor, which is almost free (about $0.01/hour per instance), so thousands of rollouts stay cheap. The real cost is the single GPU job doing vLLM + training.

Two things to watch out for

Remote rollouts add two rough edges that are worth calling out honestly.

Remote sandboxes need operational care. Running each rollout in its own remote sandbox (any sandboxes, not just HF) adds moving parts a local subprocess does not have: a sandbox can be slow to start, error out mid-run, or be left running after a run ends. So plan for it, keep an eye on stray sandboxes and close them properly, and handle the ones that fail.

Exposing vLLM to the sandboxes. The launcher opens a quick public tunnel to vLLM so the remote sandboxes can reach it. That is convenient for a demo, but it is a plain, unauthenticated endpoint. For anything beyond a throwaway run, point sandbox-vllm-url at a properly reachable and access-controlled vLLM instead.

Does it learn?

A short run to see the loop close end to end in remote sandboxes: Qwen/Qwen3-8B, 10 steps, over 32 problems. The reward moves up from about 0.27 to 0.71 over the run (noisy, but upward), with reward_std around 0.4 to 0.5, so GRPO has a real within-group signal to learn from.

• reward: ~0.27 to ~0.71 over 10 steps (noisy, upward)
• trackio: sergiopaniego/opencode-hf-sandbox
• model: Qwen3-8B-opencode-deepcoder-grpo

reward over our 10-step run (live trackio dashboard)

Ten steps is a short, noisy run, but it is enough to show the pipeline closes and the policy learns from the exact tokens the agent produced.

What we tried, and what worked

The reference example trains Qwen3-4B-Instruct-2507 locally and learns well (reward ~0.4 to ~0.6 over 110 steps). We started from that same model but running the rollouts in remote sandboxes, and it did not take: the reward climbed for a few steps and then collapsed, with the agent spamming tool calls without ever solving.

We are not fully sure why. Our best guess is a mix of things: we only ran a handful of steps, running rollouts remotely and asynchronously adds lag and the odd failed sandbox, and a 4B has a harder time getting a foothold on these problems in the first place. Moving to Qwen3-8B was what made it click, it solved enough for the reward to have something to climb, and that is the run this post is built on (reward ~0.27 to ~0.71).

A first version, and what is coming

This is the first version of the integration, built on both TRL and OpenEnv, and we are actively evolving it. The setup in this post is the manual version: you pick a sandbox backend, wire the two vLLM URLs, and point the trainer at the harness. It works, and it is the right way to understand what is actually happening end to end.

The next step is a restructuring around Harbor: one integration that makes a whole matrix of coding agents (Claude Code, Pi, Codex, OpenCode, Cursor, ...) and sandbox backends trainable through the same path, instead of wiring each one by hand.

The loop-owning training path (AsyncGRPO + the harness rollout worker) keeps evolving, and the OpenEnv integration guide tracks the current state. So treat this post as the ground-truth version: the moving parts are all here, in the open, and the tooling around them is only getting simpler.

Resources

• Training stack: TRL + OpenEnv
• Full training script: opencode_hf_sandbox.py
• HF Jobs launcher: launcher.py (gist)
• Sandbox image: ghcr.io/huggingface/openenv-opencode-sandbox
• Dataset: agentica-org/DeepCoder-Preview-Dataset
• Trained model: Qwen3-8B-opencode-deepcoder-grpo
• Dashboard: sergiopaniego/opencode-hf-sandbox