Namespace official-suite. This example composes the two benchmarks' official
evaluations on the default baseline: a RunSpec names no tool surface, prompt
bundle or Context, so the default resolver composes the default catalog's
LIBERO-Pro and RoboTwin instances. It selects only the cell, the Provider
profile, budgets, the resident-service binding and the qualification evidence.
What a cell is
One episode on one seed or initial state, single-attempt admission, no persistent session:
- LIBERO-Pro:
benchmarks/libero_pro.official_core_evaluation()-- the sixteen core suites (Obj/Pos/Sem/Task x four families) x ten tasks x initial states 0-49, 8,000 cells at the pinned code and data revisions. The instruction is the perturbation's BDDL language from the pinned task catalog. - RoboTwin: per task and scene configuration, every expert-accepted seed a
sealed seed manifest lists (
ora qualify robotwin prepare --accepted-per-task 100 --task-config demo_clean|demo_randomized, thenora qualify robotwin qualify). The instruction is the public per-task instruction inora/environment/benchmarks/robotwin-instructions-v1.json;place_a2b_leftandplace_a2b_rightname the objects the seed spawns and take theirs from--episode-names FILE({"episodes": {"<task>:s<seed>": {"instruction": ...}}}), without which those cells are refused.
Budgets are 80 decisions / 200 model calls / 10,000 native steps / 6 h per
episode. The LIBERO arm uses the default
experiment Provider (Low); the RoboTwin arm uses High. --provider-profile overrides either, for example with a profile
for a self-hosted model.
build_spec writes runtime.task for a RoboTwin cell, which binds the
reviewed completion contract; without it the plan records
task_contract.delivered: false. Single cells and other cell lists compose
through the maintained ora/orchestrator/spec/benchmark.py
(RunSpec inputs).
Commands
uv run --locked python examples/official_suite/run.py cells --arm libero_pro
uv run --locked python examples/official_suite/run.py cells --arm robotwin \
--task-config demo_clean --seed-manifests <qualify-output>/seed-manifests
uv run --locked python examples/official_suite/run.py services qualify
uv run --locked python examples/official_suite/run.py compose --arm libero_pro \
--config lp-libero_object_swap-t00-s0 --lane <resource-profile> \
--service-report <report.json>
uv run --locked python examples/official_suite/run.py queue --arm robotwin \
--task-config demo_randomized --seed-manifests <dir> --accepted-limit 10 \
--lanes <profile>... --group-id pilot-1 --service-report <report.json>
cells lists a selection; compose writes one RunSpec; queue launches a
selection across lanes, one attempt per cell, and stops at the first refusal.
--resume skips cells earlier groups of this cohort already sealed. Every
launch still passes the shared preflight: host marker, services, Provider
probe and the seed manifest (RoboTwin).
Evidence boundary
Runs land under outputs/official-suite/. A result is reported per
benchmark row with the differences from the leaderboards named (no
demonstrations, RGB-D plus tools, pose targets, Harness budgets).
No result exists yet.