ORA runs a language model as a robot policy. This page explains the parts a run passes through and who holds authority at each step. The runtime lifecycle and public interfaces pages hold the exact contracts.
Two loops
Three parts share a run, and each decides one thing.
| Part | Decides | Cannot |
|---|---|---|
| The agent (the policy) | What to look at and what to try next | Change simulator state directly, read privileged task state, or set the result |
| The Supervisor (the trusted runtime) | Whether a proposed action is admitted, and how it executes | Choose the agent's next step |
| The environment's official boundary | Whether the task succeeded | Be overruled by the agent |
The agent loop is a model calling tools. The runtime loop is the Supervisor
checking each action proposal, dispatching it to an isolated environment
worker and publishing the measured result. Success comes only from the
benchmark's own check: RoboTwin check_success, LIBERO termination or
RoboDojo's task verdict. A model's finish_episode call or a written claim of
success does not change that verdict. Synthetic, the simulator-free test
world, reports official_success: null.
In the source, the agent's tools, composition and context rules are
ora/agent/ and its model loop is the TypeScript core in ora-pi/; the
Supervisor and the rest of a run's orchestration are ora/orchestrator/
(the episode loop in ora/orchestrator/episode/); the environment worker and
the simulator adapters are ora/environment/. The code map
links each package's README.
One turn
Before a run starts, ORA resolves a RunSpec (task, provider, resources and budgets) into a content-addressed RunPlan, checks it in preflight and claims a resource lease. The Supervisor then opens one policy session and one environment session and runs turns until the episode ends.
flowchart TD
A["Model input: prompt layers, task, tool schemas, current observation"] --> B["Model call"]
B --> C{"Tool call"}
C -->|read-only tool| D["Observation, perception or geometry result"]
D --> A
C -->|state-changing tool| E["Versioned action proposal"]
E --> F{"Supervisor validation"}
F -->|refused| G["Refusal with public constraints"]
G --> A
F -->|admitted| H["Isolated environment worker executes"]
H --> I["Measured receipt and fresh observation"]
I --> A
H -->|episode boundary reached| J["Official verdict"]
J --> K["Evaluate and seal evidence"]
Model input. The request is built from the prompt bundle's ordered layers: a system layer, project instructions (the action loop, the environment's boundary and the embodiment manual with the robot's frames, units and control semantics), the task in the first user message, the ToolSurface's schemas in the provider's tools field, and the current public state in the latest user message. That state carries the camera images, proprioception, the previous action's receipt and the remaining budget. Context assembly describes the layers and how tool history is kept; frames and units describes the robot facts the manual publishes.
Model call. The agent core sends the request to an OpenAI-compatible or Anthropic Messages endpoint and receives text and tool calls.
Tool call. Every tool is defined by the run's ToolSurface, which fixes its name, schema and declared effect. Tools fall into two groups:
- Read-only tools (
observation.dump_state, perception, geometry,compute) run against the current observation and return evidence. They cannot change robot state. - State-changing tools submit a versioned action proposal. In the shared
profiles this is
robot.submit_action; a surface may name its motion, gripper or skill tools differently, but every one of them goes to the Supervisor.
Supervisor validation. The Supervisor checks the proposal against the action contract, the observation it was made from, the remaining budget and the environment's limits. A refusal names the failed stage, a stable code and the public constraint that was violated, and the refused action is not executed. One action runs per observation step.
Execution. An admitted proposal goes to the environment worker, which
runs in the simulator's own interpreter over local_rpc (Synthetic runs in
process). Motion planning and simulator stepping happen behind this boundary.
The worker returns a measured receipt and a fresh observation, and the next
turn starts from them. Contact can change the measured state; the agent sees
that change as evidence.
End of the episode. An episode ends when the environment reaches its boundary, a budget runs out, the agent calls a lifecycle tool, or an operator stops it. On landing the Supervisor derives metrics, evaluates the trace, binds the official verdict and attempts to seal the run: a hash-bound, immutable inventory of plan, requests, responses, tool calls, proposals, receipts and result. An invalid trace gets a recorded seal refusal instead of a seal. See configuration and evidence and the evaluation protocol.
The agent core
The agent core runs the model loop: provider calls, request assembly, context
management and the agent-side evidence. By default it is a TypeScript core in
ora-pi/, built on pinned
Pi packages
and run by Node.js under Node's permission model.
Robot semantics stay in Python. The Supervisor starts the core as a child
process and talks to it over a JSON-lines socket. The core calls back into
Python (ora.agent.pi) for every tool, observation and static context slot,
receives images as sha256 references, and may write only the run's agent
evidence. Tools, the ToolSurface, the action contracts and all authority over
the robot live on the Python side. If the core process exits, the policy
session restarts it a bounded number of times and resumes from the Pi session
file without running any tool twice.
In a benchmark episode the core applies ORA's context composition and retention policy, and Pi commands that would change the model, reasoning effort, compaction or session are refused. In free play the agent is Pi itself as a coding agent, with Pi's own context, compaction and coding tools, plus ORA's robot tools.
ORA's earlier Python Native core ran the same lifecycle in process. It was removed from ORA and remains only in the robot-harness repository; Pi reproduced its requests byte for byte. A plan resolved for it does not start, while its sealed runs still evaluate, open in the viewer and replay. See agent core.
Every interface uses the same run path
The terminal UI, the browser page, scripts and remote sessions are clients of a run. None of them has its own execution path: a launch from any of them resolves the plan, runs preflight, takes the lease and starts a Supervisor, and every robot action still crosses that Supervisor. Typed messages enter the conversation before the next model call; they cannot alter a request in flight, execute a tool or override admission. A run that received an operator message keeps its official verdict but is marked human-assisted and excluded from autonomous success rates.
| Interface | What it is for | How it connects | Who drives it | Read next |
|---|---|---|---|---|
ora run |
One task: the Synthetic first episode, or one LIBERO, RoboTwin or RoboDojo task on a single-GPU host | Starts a Supervisor; a simulator task first opens the task picker | The model | First episode, one GPU |
ora play |
Free play: a qualified scene with no task, never scored | The scene runs in the background; your terminal attaches to it | You give instructions; the model carries them out | Free play |
Terminal UI, ora attach |
Watch and steer a running run, detach and come back | Attaches to the run's agent; detaching leaves the run going | The model; you steer or stop | Interactive sessions |
ora watch |
Follow a live run or replay a finished one | Reads the recorded evidence only | No one; read-only | Run and inspect |
ora web |
The conversation and cameras of a live run in the browser | A local web server connected to the run, opened with a one-time token | The model; you send messages or stop | Web view |
ora-remote |
Use a run on a GPU server from your own computer | Runs any of the commands above on the server over one SSH connection | Whoever drives the wrapped command | Remote GPU server |
Codemode, ora-robot |
Let the agent script the robot in free play | Each script call goes to the Supervisor like any other action; codemode runs in a sandbox | A script the agent writes | Scripting the robot |
| JSON events and RPC | Drive or observe a run from another program | One controlling client and any number of observers; ora watch --raw prints the event stream |
Your program | Public interfaces |
| Python API | Load, resolve and validate plans; compose benchmark cells; call run_episode |
In your own Python process; launches still go through the CLI | Your code | Python API |
What the policy can and cannot see
The policy sees public observations, the task instruction, the robot manual and the advertised tools. Perception tools compute their results from the public camera data.
The policy never receives object ground truth, simulator segmentation or contact state, the task source, expert code or the private success predicate. It cannot set the official verdict. Tool refusals carry public constraints only, not private paths, object state or stack traces. In free play the scene's task language and success predicate are withheld as well.
One caution for free play: the agent's shell runs as you, with credential variables removed, and ORA adds no sandbox around it, so the shell can read any file you can read. Codemode is the sandboxed way to script.
See safety for the operator-facing rules and policy inputs and outputs for the exact contract.
Read next
- Architecture for the components and trust boundary.
- Runtime lifecycle for plan, admission, execution, landing and sealing.
- Glossary for RunSpec, RunPlan, ToolSurface, receipt and seal.
- Code map for where each part lives in the source.