How ORA decides, counts and reports a benchmark outcome. Where ORA has no rule yet, the page says "not specified"; a reporting convention we recommend but no tool computes yet is marked recommended. For how a run executes end to end, see How ORA works; for what the policy sees and sends, see policy inputs and outputs.
What decides success
Only the selected environment's official boundary supplies the boolean verdict. The model's self-assessment never does.
| Environment | Official verdict | Recorded as |
|---|---|---|
| RoboTwin | The simulator's check_success, latched as eval_success |
official_success in result.json (per round in a session) |
| LIBERO / LIBERO-Pro | LIBERO termination; a truncated episode reports false |
official_success |
| RoboDojo | The task's official result, latched inside the worker bridge | official_success |
| RoboCasa365 | RoboCasa's _check_success, latched at the first control step it holds |
official_success |
| RoboLab | RoboLab's success termination; the episode freezes at success | official_success |
| Synthetic | None | official_success: null |
Free play (ora play) |
Withheld by design | official_success: null |
official_success always means a success inside the benchmark's official limit.
Fresh RoboCasa365 and RoboLab runs continue to three times that limit, and fresh
RoboDojo runs until a tool budget ends them; a success there is
success_after_official_limit, recorded with the native actions it took and
never counted as an official success. See
the extended budget.
A policy's finish_episode call, or any claim in its text, cannot override
official evidence. RoboDojo's default surface offers no check_task and no
success outcome: the simulator's official predicate ends a successful episode
by itself, and the model can only give up. On surfaces that advertise
check_task, the Supervisor
writes policy-task-check.json at every published observation from the
trusted environment boundary, and check_task returns only the boolean
task_success from it. The policy learns whether the task currently holds,
never the predicate, its source or object state. With the faithful_check_task
capability, finish_episode refuses outcome: success unless check_task is
true at the current boundary, and once success is confirmed it admits only
success. These refusals are policy-side task state, not infrastructure
faults. The file is per round and archived at each round reset.
Units of evaluation
| Unit | Meaning |
|---|---|
| Task | A benchmark task identity (a RoboTwin task, a LIBERO suite and task index, a RoboDojo, RoboCasa365 or RoboLab task). |
| Case (cell) | One task on one seed, initial state or layout, with its scene configuration. A cell is the admission unit: launch_admission.cell_id inside a cohort_id. |
| Run | One launched execution, keyed by its exact run_id, under outputs/<experiment>/runs/<run_id>/. |
| Episode / round | One reset-to-end interaction with one official verdict. A single-episode run is its own round. |
| Session | A persistent run of K rounds (session.cadence: persistent_agent_k_rounds_v1, rounds 1 to 32, optional round_seeds). Each round has its own verdict; the session-level official_success stays null. Sessions are explicit inputs, never implied by task identity. |
| Attempt | The run report's unit: one round of a run. A session that stopped early keeps the round in flight as one more attempt. |
| Cohort / group | A cohort is the set of single-attempt cells; a queue group is one launch intent over a cell list under outputs/<experiment>/groups/<group>/. |
Three counters measure effort inside an episode and must not be confused:
| Counter | What one unit is |
|---|---|
| Robot decision | One executed robot action proposal. A refusal whose receipt certifies zero native execution and state_changed: false spends no decision; it is counted as refused_proposals. |
| Model call | One provider request. |
| Simulator step | Backend-specific: one LIBERO control step (env.step, 20 Hz); one RoboTwin action count (take_action_cnt); one RoboDojo physics step (0.004 s, ten per native action); one RoboCasa365 control step (20 Hz, base moves included); one RoboLab control step (15 Hz, eight physics steps); one Synthetic step. |
Budgets and stop conditions
A RunSpec must give all four budgets as positive integers. They are ceilings,
not expected durations. In a session, budgets apply per round
(budget_scope: per_round).
| Budget | Benchmark builders (LIBERO-Pro, RoboTwin, RoboCasa365, RoboLab) | RoboDojo builder | Synthetic quickstart |
|---|---|---|---|
max_robot_decisions |
80 | 80 | 1 |
max_model_calls |
200 | 200 | 6 |
max_sim_steps |
10,000 | Task ceiling (below) | 1 |
wall_time_s |
21,600 (6 h) | 21,600 (6 h) | 120 |
The official suite uses the benchmark-builder values for both arms.
Environment step limits apply on top of the budgets:
| Environment | Native limit |
|---|---|
| LIBERO / LIBERO-Pro | max_sim_steps is passed to the environment as its horizon and shown to the policy as its native-action allowance, one per 50 ms control tick (a strict-limit run sets the suite's protocol limit: 220, 280, 300 and 520 for spatial, object, goal and libero_10). |
| RoboTwin | The task's own step_lim; take_action_cnt at or above it ends the episode (robotwin_step_limit_exhausted), as does reaching max_sim_steps. |
| RoboDojo | The task's official native-action limit (step_lim, 200 to 1,900 across the 54 admitted tasks; each *_random variant has its base task's). max_sim_steps must lie in [10, limit x 10] on a ten-step boundary and defaults to limit x 10. |
| RoboCasa365 | The task's evaluation horizon from RoboCasa's dataset registry (750, 750 and 450 control steps), passed to the environment and shown to the policy as its native-action allowance. |
| RoboLab | The upstream episode length at 15 Hz (750 or 900 control steps), shown to the policy as its native-action allowance. |
Only these principals may end a round: the environment, the operator, an
experiment budget, the policy's lifecycle request, or infrastructure. A round
stopped by a budget or by finish_episode still carries the standing official
predicate (normally false). Only an infrastructure fault, or a backend that
produced no verdict, leaves the boolean null.
Seeds and case lists
| Benchmark | Case list | Seed or state selection |
|---|---|---|
| LIBERO | Spatial, Object, Goal and LIBERO-10 suites from the pinned code checkout | Fixed init-state files per task |
| LIBERO-Pro official core | 16 suites (Object, Goal, Spatial, LIBERO-10 x _task, _swap, _lan, _object, reported as Task, Pos, Sem, Obj) x 10 tasks x init states 0 to 49: 8,000 cells at the pinned code and data revisions |
Init state seed % 50. Env perturbations are excluded because their states are generated at run time. |
| RoboTwin official | 50 tasks, each under demo_clean and demo_randomized |
Per task and configuration, the first 100 expert-accepted seeds in ascending order from requested seed 100000 x (1 + outer_seed), unseen backgrounds. A seed is accepted when the expert plans and check_success holds after replay; expert rejections and infrastructure failures do not count as policy trials. Accepted seeds live in a sealed seed manifest from ora qualify robotwin prepare / qualify. Development (practice) seeds lie below 100000, scanned with prepare --practice-start (commands). |
| RoboDojo | The 54 settings at RoboDojo f57e163 and assets 1127bda: the 42 official tasks and the 12 *_random variants of the generalization tasks (task-contract registry-v10); each task's contract record is reviewed on its own layout 0, a variant's on <task>_random_0.json. swap_blocks, press_by_number, organize_table and stack_blocks_by_language results from the earlier pin (RoboDojo ee67a146) are not pooled with later ones (pinned revision) |
Layout 0 of evaluation seed 0 by default. One episode can run on another official layout of evaluation seed 0 that the reviewed layout-portability table (ora/policy/environments/robodojo/task-contracts/layout-portability-v5.json) lists for the task: layouts 0 to 9 for every admitted task except sweep_blocks, whose contract holds on layouts 0, 3, 4 and 7, and hang_mugs_random, whose contract holds on layouts 0 to 4 and 6 to 9. A *_random variant's layout k is its own <task>_random_<k>.json. A session can use a reviewed layout pool (evaluation seed 0). The default K-round schedule is the first K pool entries, and each round's environment seed equals its layout ordinal. A RunSpec that names its own task file (policy.task_file, task files) delivers no reviewed contract and may run any official layout of evaluation seeds 0 to 2 (EVAL_SEEDS), the seeds the pinned Assets hold; its instruction is read back from the reset. |
| RoboCasa365 | 3 admitted tasks of the target split (target object instances, kitchen layout and style pairs 1 to 10) | The seed samples the kitchen, the target and the instruction. An official RoboCasa365 suite is not specified. |
| RoboLab | 3 admitted tasks | The seed is the environment seed. An official RoboLab suite is not specified. |
| Synthetic | One built-in world | Not applicable |
Cases are selected by explicit cell ids, never at random. The official-suite
example names them lp-<suite>-t<task>-s<state> and
rt-<clean|random>-<task>-s<seed>; run.py cells lists a selection and
--accepted-limit truncates each RoboTwin manifest. See
examples/official_suite and
benchmark setup.
An external case list, such as the RoboTwin leaderboard's 250 C2R pairs, is
not shipped. Such a list can name seeds the expert never solves. To keep the
same denominator, ORA runs those cells under a labelled
robotwin_expert_unsolved_seed_v1 qualification exception that cites seed
manifests showing the expert failed. The exception stays in the plan, so
readers can separate those cells from admitted ones. A RoboDojo official suite
is not specified.
Validity
Evaluation writes evaluation.json with one of three statuses:
| Status | Meaning | Sealable |
|---|---|---|
valid |
Every evidence check passed. | Yes |
incomplete |
The run stopped early for an expected reason only (trusted_fault, causal_count_mismatch), including an operator stop. |
Yes |
invalid |
Any other evidence violation. | No: the seal is refused and recorded as a refusal. |
Each round also records termination_authority and behavioral_sample_valid.
The latter is false when infrastructure or the operator ended the round, or
when an operator message reached the policy (human_intervention: true).
Infrastructure faults vs task failures. A fault carries a layer (provider,
context, tool, protocol, environment, evidence, operator, orchestrator). A
trusted fault ends the session (or round) under infrastructure authority, as
does a policy abort the Harness admits on the round's recorded infrastructure
evidence; neither records a verdict. A task failure is an official false
reached through the environment, a budget or the policy's own finish. Physical
contact that changes measured state is not, by itself, an infrastructure fault.
Retry and replacement. Benchmark cells are single_attempt_cell
admissions with retry_allowed: false; launch consumes the claim. ora retry
applies only to an unsealed, incomplete attempt that dispatched no robot
decision and whose plan permits retry. It creates a new run and never resumes
or overwrites the old directory. A replacement for a consumed cell needs a new
cohort or cell id; a replacement for an externally interrupted cell declares a
successor generation (experiment identity).
The original attempt stays in the evidence and is not pooled with its
replacement.
Sealing. seal.json binds the file inventory, the evaluation status and
the sealed source commit, and makes the files read-only. Later revalidation
(ora revalidate) writes separate reports and never changes the sealed run.
The run report also reads operator corrections under
<namespace>/diagnostics/corrections/. A correction overrides validity and
learning eligibility while the sealed evaluation is kept, and a hash mismatch
is flagged.
Learning eligibility. evaluation.json records
learning_eligibility.eligible with reasons. A run is ineligible when it did
not complete, had human intervention, had a trusted fault, is a free-play
session, has a receipt its backend labelled an execution fault, fails the
RoboDojo evidence checks, or, for LIBERO-Pro, lacks the qualified controller
evidence on every physical receipt. A behaviorally valid run can still be
ineligible.
How to report results
The run report defines a valid attempt as one whose run validity is
valid, whose round has an official boolean and whose behavioral sample is
not marked invalid. Its success_rate is successes over valid attempts, and
it lists attempts, valid attempts, excluded attempts by reason, and every
ending class. Report the following with every result:
- every rate with its numerator and denominator;
- the observation/action abstraction, the task and seed protocol, attempts and budgets, and differences from the leaderboard protocol, as the setup guide requires;
- results per benchmark row, without pooling across renderers, treatments or families;
- the result tier, never pooled across tiers (below).
Result tiers
Every plan records result_tier, what the policy carried into the episode
beyond its prompt and tools. The run report groups results by tier first and
counts each tier's check_task calls.
| Tier | Meaning | Produced here |
|---|---|---|
zero_shot |
The default: no experience from the benchmark, no per-task recipe in the prompt, check_task advertised where the surface offers it |
Yes, the default |
zero_shot_no_check |
zero_shot with check_task removed from a surface that offers it: measures reliance on the official Boolean, which a real robot does not have. A surface with no check_task, RoboDojo's default, stays zero_shot |
Yes, set result_tier in the RunSpec |
with_experience |
Experience gathered on the same benchmark's practice partition | No: refused until an experience library binding exists |
frozen_program |
A program compiled after practice and run without a model | No: refused; such results come from outside the Runtime |
Plans recorded before the field carry no tier and are reported as
unlabelled, apart from zero_shot. Report a with_experience result only
in its own row, with the practice episodes, tokens and GPU hours beside the
evaluation cost.
Practice and evaluation partitions
Every plan records its partition: evaluation (the default) or practice.
A practice plan may not use an episode from any benchmark's evaluation set;
resolution refuses it, using the list in
ora/environment/benchmarks/evaluation-sets-v1.json (see
practice initial states and evaluation sets).
Practice runs are never evaluation results. The run report's first table
groups runs by partition and gives each group's runs, valid attempts,
successes, model calls, input and output tokens, API-equivalent cost and wall
hours, so the practice budget is reported beside evaluation; plans recorded
before the field are unlabelled.
Recommended, not yet computed by ora report:
- Show two rates side by side: successes over all selected cases, counting a case without a valid outcome as a failure, and the run report's rate over valid attempts.
- Report
success_after_official_limitapart from both groups below. - Split failures by ending class: task failure (
failure_latch,episode_end,environment_end,policy_finish); timeout or budget (native_action_limit,decision_budget,model_call_budget,wall_time_budget,simulation_step_cap); infrastructure (infrastructure); and others (operator_interrupt,in_progress,other) listed separately. - With repeats, give the mean and a 95% confidence interval and name the method. The run report gives means and percentiles for effort, not intervals.
Checking that a suite run is complete
./scripts/ora suite status --experiment NS --group Gprintsqueue_status(completeorpaused), counts per cell state and the first error. A cell settles assealed_terminal,trusted_fault_recordedorunsealed_recorded; a group is complete only when every cell settled.ora report --namespace outputs/NS --cohort IDreports attempts, validity and endings per result tier, per cohort and per (model, task). Compare its attempt count with the selected cell list (run.py cells), and list cells with no run.- Rerun missing cells with
queue --resumein a new group, which skips cells already sealed or claimed in the cohort. ora statussummarizes active leases and the indexed local run registry;ora workspaceshows which evidence roots are indexed.
Auditing one case
Open the run directory and read, in order: plan.json (task, seed, budgets,
source), result.json (termination, authority, counters, verdict, rounds),
evaluation.json (status, reasons, learning eligibility), seal.json or the
seal refusal, then the canonical events, receipts and provider evidence.
ora watch RUN_ID replays the transcript with per-round verdicts. File names
and schemas are described in
configuration and evidence
and run and inspect.