Use ORA

Connect a training loop

ORA runs a language model as a robot policy and keeps a full record of every episode. A reinforcement-learning or fine-tuning loop needs nothing else from ORA's side: it serves the policy it trains as a model endpoint, launches episodes against it, and reads the verdict and the model calls back from the evidence. ORA has no training code of its own.

ora help rl-interface prints a short version of this page.

Serve the policy as an endpoint

ORA talks to a policy through a provider profile: an OpenAI-compatible (chat completions) or Anthropic Messages endpoint, a model id and a key variable. A checkpoint you are training is one more endpoint. HTTPS endpoints are admitted, and so is plain HTTP on this machine's loopback address, for a server such as vLLM or SGLang running next to ORA.

Add the endpoint as a profile of your own. ora providers add asks for the key without echoing it and probes the endpoint with it:

./scripts/ora providers add my-policy --protocol openai_compatible \
    --endpoint http://127.0.0.1:8000/v1 --model my-checkpoint --images

It prints what it recorded, including the context window (131072 tokens unless --context-window says otherwise). --images declares that the model reads images, which the robot environments' observations carry; ./scripts/ora providers test my-policy probes the endpoint again later. Model APIs lists every option.

A new checkpoint is a new profile, or the same profile once the server serves the new weights. The plan of every run records the profile id and endpoint, never the key, so runs of different checkpoints stay apart.

Run episodes against it

ora run prepares one task and opens it in the terminal UI; the profile replaces the default model:

./scripts/ora run --env robotwin --task adjust_bottle --provider-profile my-policy

For unattended batches, compose RunSpecs with the environment's composer and launch the path it prints, one RunSpec per task and seed (ora spec robotwin compose --help lists the composer's inputs):

./scripts/ora spec robotwin compose adjust_bottle --seed SEED --qualification MANIFEST \
    --experiment my-training --resource-profile SLOT --service-report REPORT \
    --cohort round-1 --provider-profile my-policy
./scripts/ora launch outputs/my-training/specs/CONFIG.json --yes

Each launch holds its plan's execution-resource lease, so episodes on the qualified simulator slots of one host profile run side by side and a busy slot refuses a second launch. ora suite run queues one attempt per cell across fixed lanes (evaluation protocol). Launching still passes the normal preflight; see run and inspect.

Identify each model call

Each episode's agent trace, agent-events.jsonl (ora.agent-event.v1), records every model call as a provider_request and provider_response pair. Every event carries run_id, policy_session_id, observation_step and model_call_id, and is hash-chained to the previous one. The exact request bytes are a content-addressed artifact that the request event names by its sha256; after sealing they are packed losslessly into native/provider-requests.tar.zst, and ORA's readers return the original bytes either way.

The wire request carries no ORA identity by default. A catalog profile that declares session_affinity: x_session_id_run_v1 sends the header X-Session-ID: ora:<run_id>:native with every policy request, so a server or router can group requests by run.

Read the reward and export the calls

The reward is the environment's official verdict, official_success in the run's result.json (RoboTwin check_success, LIBERO termination, RoboDojo's task verdict). A model's own claim of success never counts, and Synthetic and free play record null. evaluation.json says whether the run may be used for training (learning_eligibility).

Check the trace, then export the model calls of an evaluated run (a sealed run's files are first checked against its seal):

./scripts/ora trace-validate RUN_ID
./scripts/ora export-sft RUN_ID --destination sft/RUN_ID

The destination directory holds:

File Contents
conversations.openai-tools.jsonl One record per accepted model call: the request's messages and tools, then the model's reply
attribution.jsonl What each reply decided
loss-mask.jsonl Which message is the target (only the final reply)
tokens.jsonl Each record's model call and its token usage as the provider reported it
manifest.json The run id, the source commit, official_success, the termination, the sha256 of every exact request and the usage totals

The export refuses a run whose trace does not verify, whose evaluation is missing or invalid, or that is not learning-eligible: an incomplete episode, a run with operator messages, free play, or a recorded runtime fault.

The usage in tokens.jsonl is exact for what the serving model processed in each call. It is not the length of the exported record: the record leaves out the earlier reasoning the call's history carried. Count a record's tokens with the tokenizer and chat template you train with.

Export every call as it was sent

The training records above are a projection. To keep a run's model calls exactly as the provider received them, with every image sent to a model and a proof that the export rebuilds them, use the wire view. It records learning eligibility in its manifest instead of requiring it: learning_eligible is true only for a sealed run whose evaluation is valid and found it eligible. Pass a run id, or a run directory together with --destination:

./scripts/ora export-sft RUN_ID --view wire --destination wire/RUN_ID
File Contents
requests.jsonl One line per model call, rejected replies included: the request body the provider received (OpenAI Chat Completions: messages with earlier reasoning, tools, sampling and template options), images replaced by images/<sha256>.<ext> references
responses.jsonl ORA's record of each reply (accepted or rejected, its output rejection, usage) and provider_reply, the reply rebuilt from the archived stream: content, reasoning, tool calls including a truncated fragment, the provider's own finish reason
streams.jsonl The provider's response exactly as archived (the streamed chunks)
compactor.jsonl Each context compactor call: its request, ORA's record of the reply, the stream and the provider reply
images/, images.jsonl Each image once, with its media type, size and the calls that sent it
epochs.jsonl The messages once for each run of calls whose requests extend one another; each request line names its epoch, its prefix length and its last message when that differs (a rolling camera tail)
tokens.jsonl Each policy call's prompt and sampled token ids where the run or the serving engine has them, checked against the call's usage (token ids)
manifest.json Run identity, verdict, learning eligibility, counts and the replay check

Before it moves the export into place, ORA rebuilds every request twice, from its own line and from its epoch, puts the images back, serializes it the way the run archived it, and requires the archived sha256. Compactor requests, every stream, every provider reply and the image index (each file's sha256 and the calls that sent it) are checked as well. A mismatch writes nothing; the export directory gets the group permissions your umask allows. To check an export again later: this needs the manifest and every file it lists, checks their hashes, and derives every row and the manifest's description again from the run, so an edit anywhere in the export is found:

./scripts/ora export-sft RUN_ID --view wire --destination wire/RUN_ID --check

It prints the counts and exits 2 when anything differs. Tool results, script frames and receipts need no separate file: they are messages and images inside the requests. Images the run recorded but no request carried (for example script frames a model never saw) are counted in the manifest, not exported.

The export refuses a run whose seal no longer matches, or whose trace does not verify. Two exceptions are labelled in manifest.trace.status: incomplete, a run whose last request got no reply, exports every call and marks that request answered: false; and sealed_valid_unverified_here, a sealed run that its own evaluation found valid but whose trace this checkout's verifier reads differently (an older or newer trace shape), exports when every request still rebuilds to its sealed sha256. For a provider that speaks another protocol (Anthropic Messages), request is the canonical OpenAI-format request and wire_request the body actually sent; both are checked.

Token ids

A vLLM-served profile can ask for them on every call with request_options: {return_token_ids: true}: each archived stream then carries the prompt's ids and every sampled id, and tokens.jsonl reads both from it. For any other run, give the export the serving engine's tokenize endpoint and the variable that holds its key:

./scripts/ora export-sft RUN_ID --view wire --destination wire/RUN_ID \
  --tokenize-url https://HOST:PORT/v1/tokenize --tokenize-key-env SERVE_API_KEY

vLLM serves it at <root>/tokenize, sglang at <root>/v1/tokenize. ORA sends each call's messages, tools and chat_template_kwargs, with its reasoning_effort merged into the template kwargs as vLLM's chat endpoint does: a tokenize request has no effort field, and the template would otherwise render its default effort, a prompt of the same length with one token changed. Use the engine that served the run (same model, tokenizer and template): every call is checked against the usage it reported, so another one shows up as length mismatches, but a same-length difference only shows where the run also streamed its ids (tokenize_equals_stream). sglang tokenizes before it expands images, so its ids hold one placeholder token per image (image_form: placeholder, which the check counts as the usage's image_tokens); vLLM's are expanded. Sampled ids exist only where the stream carried them: sglang refuses token ids while streaming, so its replies stay a count. Each prompt is stored as the prefix it shares with the previous call plus its own suffix, with the sha256 of the whole sequence; extends_previous says whether it starts with the previous prompt followed by that call's sampled ids, which is where one epoch can train as one token sequence. --check cannot ask the engine again: it rebuilds every row from the recorded ids and the run's streams and usage.

Share a run as an OpenAI package

To hand a run to someone who has neither ORA nor the run directory, export it as a package of standard OpenAI Chat Completions data:

./scripts/ora export-sft RUN_ID --view openai --destination pkg/RUN_ID
./scripts/ora export-sft RUN_ID --view openai --destination pkg/RUN_ID --check
Path Contents
conversations.jsonl One conversation per epoch, every image inline as a data URL, the epoch's final reply last: {run_id, epoch, model_call_ids, messages, tools}
calls.jsonl One line per model call: the request as sent (images as wire/images/...), the reply, usage and token counts
wire/ The wire export above, with its token ids and replay check
replay-program/ The run's actions as code, when the run has one
ora_package.py A standard-library tool: verify, calls OUT (one OpenAI sample per call, images inline), images DIR, speculator OUT [--per-epoch] ({input_ids, loss_mask} rows)
README.md, package.json What the run was and how to use it; the sha256 of every file outside wire/

--tokenize-url and --tokenize-key-env work as for the wire view. Whoever receives the package checks it with python3 ora_package.py verify: every request rebuilt to its archived sha256, every stream, image and token id, and the conversations and calls derived again from wire/; with --tokenize-url URL --key-env NAME a serving engine of the same model also tokenizes every request again. --check here does the same and checks wire/ against the run itself. The messages keep two conventions of the serving stack: each assistant turn's reasoning is in reasoning_content, and each tool call's arguments are the JSON the engine's tool-call parser produced, which ORA sends back in the history unchanged. vLLM's parser coerces values to the tool schema (a number the model wrote as 0.0 or -0.0 comes back as 0), so where it did, the sampled token ids, not the messages, are what the model wrote.

ora report RUN ... summarizes stop reasons, validity, tokens and cost over many runs. Configuration and evidence describes the rest of the run directory.