ORA admits all 120 tasks of RoboLab-120 at the pinned revision ad45d4f9
(ora/environment/benchmarks/robolab.py, decided 2026-11-05). Admission and a
delivered completion contract are separate states:
- Admitted. A RunSpec may name the task. The instruction is the task's
upstream default instruction; the official limit is its
episode_length_sat 15 Hz. Every seed resets to the same authored layout: the environment's only reset event isreset_scene_to_defaultand no task adds one, so a seed varies the physics, not the scene. - Reviewed contract. The registry's
taskshold the public completion contracts the resolver delivers: all 120 tasks. Three were reviewed by hand on 2026-10-18; 97 generated contracts were promoted on 2026-11-05 after their seed-0 witness matched every case, and the other 20 on 2026-11-16, once their reviews' witness plans stated a placement or a grasp the generated plan could not reach (each review'switness_plan.plan_fixsays why). - Pending review. A generated contract whose witness has a case that did
not match stays in the registry's
pending_review(none at present). Its review keepsstatus: needs_reviewwith those witness checksfailed. The resolver withholds such a contract (task_contract.reason: contract_review_pending) and launch preflight fails therobolab_task_contractcheck, so a pending task composes but does not launch.
How a contract is made
ora dev robolab-tasks generate --source-root $ROBOLAB_ROOT reads every task
class by AST (no simulator import), checks it against
robolab/tasks/_metadata/task_metadata.json, and writes the roster block, one
<Task>-completion-v1.md and one reviews/<Task>-review-v1.json per pending
task, the registry's pending_review and the table below; --check reports
drift instead. RoboLab's verdict is the termination manager's terminated
flag, the OR of every non-timeout termination term, so the contract restates
those terms and never the partial subtasks scores; each review lists where
the two differ (termination_vs_subtasks).
The predicate families and what their public clauses say:
| Family | Termination | Public clauses |
|---|---|---|
in_container |
object_in_container (all, any or exactly K), object_inside |
the objects inside the container (on a shelf frame or a closed box: on it), touching it where required; released |
groups_in_containers |
object_groups_in_containers |
each group in its own container; released |
on_surface |
object_on_top |
resting on the surface, supported by it with its center over it; released |
stacked |
stacked |
one object on top of the other, touching, in the stated or any order (the term requires no release; that gap stays private) |
out_of_container, out_onto_surface |
object_outside_of, object_outside_of_and_on_surface |
out of the container (and resting directly on the table); released |
picked_up |
object_picked_up, or several of them OR-ed |
held in the gripper and lifted clear above the table; the task completes while it is held |
upright |
object_upright |
standing upright; released |
ordered_in_container |
objects_placed_in_container_in_order |
both in the container; the first placement order is latched and cannot be undone |
relative_position |
object_left_of and its three siblings |
left of, right of, in front of or behind the reference as seen from the robot; released |
centered |
object_on_center |
standing at the table's center, touching it; released |
Clauses name objects in natural language, never by identifier. An
instruction that leaves the policy to infer which objects it means, by
category, attribute, comparison or property ("the black items", "the larger
object", "the green spoons", "the serving spoons with no holes"), keeps that
descriptor in its contract: naming the objects would answer the task, so the
names stay in the private review (object_identity, withheld) and the release
clause gives no count. task_wording.refers_by_descriptor selects these 39
tasks from the instruction text and INSTRUCTION_DESCRIPTORS holds each
descriptor in the instruction's own words; the generator refuses a task the
two disagree on. An instruction that names its objects concretely ("the
banana", "the yellow block", "the red mug") keeps the names.
A review fills every check it can verify from source: predicate and helpers,
reset and randomization (with the reset relations a contract must mention,
such as a banana that starts in the crate), assets and frames, success timing,
no expert strategy and privacy. success_witness, counterexamples,
public_observability and, for upright, assets_and_frames wait for the
seed-0 witness. Each review also carries its compatibility (ready, ready with
note or blocked, with reasons) and the witness_plan the witness runs.
Witness and promotion
ora qualify robolab contract-witness --task T ... --gpu N --evidence-root DIR
runs each task's witness plan in the RoboLab worker, one Kit process at a time:
every case starts from a fresh reset of seed 0, moves objects by direct state
edits (and, for the pick-up family, closes the gripper on the object), then
holds the robot and reads RoboLab's own success termination. A placement turns
the object to fit the opening (a quarter turn, or into a container on its long
axis), takes the free spot nearest the middle and sets it low inside the walls
or on the surface, piling only when nothing is free; a placement beside an
object on a table with no free stretch parks the unnamed clutter of the nearest
stretch off the table and records it as the case's parked. The pick-up grasp
waits for the jaw to start closing before it calls the jaw stalled. An
exact-count negative sets K+1 objects down at once, so its verdict counts from
tick 15 (ignored_ticks), after they land; all_but_one keeps the n-1 objects
nearest the container inside (count_in). Where the generated placement cannot
reach the state a case needs, a review's plan may state it instead: put sets
an object down at a stated spot (env-frame center, or a reference object's
centroid moved by dx/dy), with its lowest point at table, top:<object>,
floor:<object>[+dz] or a height, turned own, yaw:D, flat:D (longest axis
level along yaw D), long_up or tilt:T:A; park moves named clutter off the
table; and grasp may name the object's axes for the jaw line, the approach and
across the pads (order), how deep along the approach it sits (depth), flip
it end for end and shift it along the jaw line (a pitcher held by its rim).
Such a plan carries a plan_fix note that says why. RoboLab latches success at
its first true step, so an exact-count negative whose objects land one at a time
can pass through K and be latched despite the landing window; such a plan sets
all K+1 down in one step (put with settle: false). It also projects
every named object into the two public cameras at reset. It writes
DIR/outputs/ora-backends/task-contract-witness/robolab/<Task>-s0-witness.json
(ora.robolab-task-contract-witness.v2) and fills the review's evidence and
witness checks with that file's sha256; a later witness of the task replaces an
earlier run's witness_worker_failed or solved_at_reset blocker. A RoboLab
worker ends its process with os._exit instead of closing Kit
(RoboLab). ora dev robolab-tasks promote then
moves every review whose nine checks passed, and which is not blocked, into the
registry's tasks; run ./scripts/ora lock and update
task_contracts.robolab.task_count and pending_review_count in
tests/data/default-expectations.json in the same change.
The roster
ora spec robolab tasks prints the same state as JSON lines. Compatibility
codes: long_horizon (official limit at least twice the 900 native actions of
the longest task admitted before), longer_horizon (above 900),
many_objects (five or more objects to move; the default 80 robot decisions
may bind first), exact_count (exactly K, one more fails), wide_object (no
face fits the open 85 mm gripper), reorientation, order_latched,
floor_rack (the storage rack stands on the floor beside the table),
remote_scene_geometry (the scene loads the purple crate from the Omniverse
content server), scoring_gap (the termination is weaker than the
instruction; recorded in the private review) and reset_relation.
| Task | Family | Episode (s) | Official limit | Contract | Compatibility | Notes |
|---|---|---|---|---|---|---|
AnimalsInBinTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon, wide_object |
AppleAndYogurtInBowlTask |
in_container | 120 | 1800 | reviewed | ready with note | long_horizon |
BBQSauceInBinTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon |
BagelsOnPlateTask |
on_surface | 60 | 900 | reviewed | ready | - |
BananaInBowlTask |
in_container | 50 | 750 | reviewed | ready | - |
BananaOnPlateTask |
on_surface | 40 | 600 | reviewed | ready | - |
BananaThenRubiksCubeTask |
ordered_in_container | 60 | 900 | reviewed | ready with note | order_latched |
BananasInBinOneMoreTask |
in_container | 60 | 900 | reviewed | ready with note | exact_count, scoring_gap, reset_relation |
BananasInBinThreeTotalTask |
in_container | 60 | 900 | reviewed | ready with note | exact_count, scoring_gap, reset_relation |
BananasInCrateTask |
in_container | 60 | 900 | reviewed | ready with note | remote_scene_geometry, exact_count, reset_relation |
BananasOutOfBinTask |
out_of_container | 90 | 1350 | reviewed | ready with note | longer_horizon, reset_relation |
BigPumpkinInBinTask |
in_container | 60 | 900 | reviewed | ready with note | wide_object |
BlackItemsInBinTask |
in_container | 120 | 1800 | reviewed | ready with note | long_horizon, many_objects, reset_relation |
BlockStackingOrderAgnosticTask |
stacked | 90 | 1350 | reviewed | ready with note | longer_horizon, scoring_gap |
BlockStackingSpecifiedOrderTask |
stacked | 90 | 1350 | reviewed | ready with note | longer_horizon, scoring_gap |
BlocksInBinTask |
in_container | 150 | 2250 | reviewed | ready with note | long_horizon |
BowlInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
BowlStackingLeftOnRightTask |
in_container | 20 | 300 | reviewed | ready | - |
BowlStackingRightOnLeftTask |
in_container | 20 | 300 | reviewed | ready | - |
ButterAboveRaisinTask |
on_surface | 40 | 600 | reviewed | ready | - |
CannedFoodInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
ClampInRightBinTask |
in_container | 60 | 900 | reviewed | ready | - |
CleanUpToysTask |
in_container | 300 | 4500 | reviewed | ready with note | long_horizon, many_objects, wide_object, scoring_gap |
ClearOrganicObjectsTask |
in_container | 240 | 3600 | reviewed | ready with note | long_horizon, many_objects, wide_object |
ClutterPlasticTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon, wide_object |
ClutterPumpkinTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon, wide_object |
CoffeePotInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
CondimentsInBinTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
CookingClearPlateTask |
out_of_container | 180 | 2700 | reviewed | ready with note | long_horizon, reset_relation |
CookingPickPastaToolTask |
in_container | 60 | 900 | reviewed | ready | reset_relation |
CubesAndBlocksInBinTask |
in_container | 240 | 3600 | reviewed | ready with note | long_horizon, many_objects |
DishesInBinTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon, wide_object |
ElectronicsInBinTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon, reset_relation |
FoodPacking1BoxesTask |
in_container | 60 | 900 | reviewed | ready | - |
FoodPacking1CansTask |
in_container | 60 | 900 | reviewed | ready | - |
FoodPacking2BoxesTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
FoodPacking2CansTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
FoodPacking3BoxesTask |
in_container | 240 | 3600 | reviewed | ready with note | long_horizon |
FoodPacking3CansTask |
in_container | 240 | 3600 | reviewed | ready with note | long_horizon |
FoodPackingByColorTask |
groups_in_containers | 120 | 1800 | reviewed | ready with note | long_horizon, wide_object |
FruitsGreenLimesOnPlateTask |
on_surface | 90 | 1350 | reviewed | ready with note | longer_horizon |
FruitsMovingOrangeOrLimeTask |
in_container | 60 | 900 | reviewed | ready | - |
FruitsMovingTask |
in_container | 60 | 900 | reviewed | ready | - |
FruitsOnPlate3Task |
on_surface | 200 | 3000 | reviewed | ready with note | long_horizon, exact_count, wide_object, reset_relation |
FruitsOnPlateTask |
on_surface | 300 | 4500 | reviewed | ready with note | long_horizon, many_objects, wide_object, reset_relation |
FruitsOnionTask |
in_container | 60 | 900 | reviewed | ready | - |
FruitsOnionToPlateTask |
on_surface | 60 | 900 | reviewed | ready | - |
FruitsOrangesOnPlateTask |
on_surface | 90 | 1350 | reviewed | ready with note | longer_horizon, reset_relation |
GrabABagelTask |
picked_up | 30 | 450 | reviewed | ready | - |
GrabAFruitTask |
picked_up | 30 | 450 | reviewed | ready | - |
GreenSpoonsInPotTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
HammersInLeftBinTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
JugsOnShelfTask |
in_container | 120 | 1800 | reviewed | ready with note | long_horizon, floor_rack, wide_object |
KeyboardOutOfBinTask |
out_onto_surface | 60 | 900 | reviewed | ready | reset_relation |
LargerObjectRaisinBoxInBinTask |
in_container | 30 | 450 | reviewed | ready | - |
MarkerInMugTask |
in_container | 40 | 600 | reviewed | ready | - |
MouseOnKeyboardTask |
on_surface | 60 | 900 | reviewed | ready | - |
MoveBananaToBagelPlateTask |
on_surface | 90 | 1350 | reviewed | ready with note | longer_horizon |
MustardAboveRaisinTask |
on_surface | 40 | 600 | reviewed | ready | - |
MustardInLeftBinTask |
in_container | 30 | 450 | reviewed | ready | - |
MustardInRightBinTask |
in_container | 30 | 450 | reviewed | ready | - |
NonHammerToolsInRightBinTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
OneBottleInSquarePailTask |
in_container | 60 | 900 | reviewed | ready | - |
OneBottleOnShelfTask |
in_container | 60 | 900 | reviewed | ready with note | floor_rack |
PhoneOrRemoteInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
PickDrillTask |
picked_up | 40 | 600 | reviewed | ready | - |
PickGlassesTask |
picked_up | 30 | 450 | reviewed | ready | - |
PickOrangeObjectTask |
picked_up | 60 | 900 | reviewed | ready | - |
PickUpBluePitcherTask |
picked_up | 30 | 450 | reviewed | ready with note | wide_object |
PickUpGreenObjectTask |
picked_up | 30 | 450 | reviewed | ready | - |
PinkSpoonInPotTask |
in_container | 60 | 900 | reviewed | ready | - |
PlasticBottlesInSquarePailTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
PutBowlOnShelfTopTask |
on_surface | 60 | 900 | reviewed | ready | - |
PutMugsOnShelfTask |
on_surface | 180 | 2700 | reviewed | ready with note | long_horizon, wide_object |
PutTwoMugsOnShelfTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon, exact_count, wide_object |
RecycleCartonTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon |
RecycleCartonsOnBoxTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon |
RecycleCartonsVerticalCrateTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon |
RedDishesInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
RedItemsInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
ReorientAllMugsTask |
upright | 90 | 1350 | reviewed | ready with note | longer_horizon, reorientation, wide_object, reset_relation |
ReorientJugTask |
upright | 60 | 900 | reviewed | ready with note | reorientation, wide_object |
ReorientRedMugTask |
upright | 60 | 900 | reviewed | ready with note | reorientation |
ReorientWhiteMugsTask |
upright | 60 | 900 | reviewed | ready with note | reorientation, wide_object, reset_relation |
RubiksCubeAndBananaTask |
in_container | 60 | 900 | reviewed | ready | - |
RubiksCubeBehindBowlTask |
relative_position | 30 | 450 | reviewed | ready | - |
RubiksCubeInFrontOfBowlTask |
relative_position | 30 | 450 | reviewed | ready | - |
RubiksCubeLeftOfBowlTask |
relative_position | 30 | 450 | reviewed | ready | - |
RubiksCubeOrBananaTask |
in_container | 30 | 450 | reviewed | ready | - |
RubiksCubeRightOfBowlTask |
relative_position | 30 | 450 | reviewed | ready | - |
RubiksCubeTask |
in_container | 40 | 600 | reviewed | ready | - |
RubiksCubeThenBananaTask |
ordered_in_container | 60 | 900 | reviewed | ready with note | order_latched |
RubiksCubesInBinTask |
in_container | 120 | 1800 | reviewed | ready with note | long_horizon |
SauceBottlesCrateTask |
in_container | 40 | 600 | reviewed | ready with note | remote_scene_geometry |
SmallPumpkinInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
SmallerObjectButterInBinTask |
in_container | 30 | 450 | reviewed | ready | - |
SmartphoneInBinTask |
in_container | 60 | 900 | reviewed | ready | - |
SpoonInMugTask |
in_container | 60 | 900 | reviewed | ready | reset_relation |
SpoonsInPotTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
Stack3RubiksCubeTask |
stacked | 60 | 900 | reviewed | ready with note | scoring_gap |
StackWhiteMugsTask |
stacked | 60 | 900 | reviewed | ready with note | wide_object, scoring_gap |
StackYellowOnRedTask |
stacked | 60 | 900 | reviewed | ready with note | scoring_gap |
TakeMeasuringSpoonOutTask |
out_of_container | 40 | 600 | reviewed | ready with note | scoring_gap, reset_relation |
TakeMugsOffOfShelfTask |
out_onto_surface | 180 | 2700 | reviewed | ready with note | long_horizon, wide_object, reset_relation |
TakeSpatulaOffShelfTask |
out_onto_surface | 60 | 900 | reviewed | ready | reset_relation |
ThrowAwayAppleTask |
in_container | 60 | 900 | reviewed | ready | - |
ThrowAwaySnacksTask |
in_container | 120 | 1800 | reviewed | ready with note | long_horizon |
ToolOrganizationBothTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon, scoring_gap |
ToolOrganizationTask |
in_container | 180 | 2700 | reviewed | ready with note | long_horizon |
ToolsPickingAllHammersTask |
out_onto_surface | 240 | 3600 | reviewed | ready with note | long_horizon, reset_relation |
ToolsPickingDrillTask |
out_onto_surface | 60 | 900 | reviewed | ready with note | scoring_gap, reset_relation |
ToolsPickingHammerTask |
out_onto_surface | 60 | 900 | reviewed | ready | reset_relation |
ToyInBinTask |
in_container | 60 | 900 | reviewed | ready with note | wide_object |
UnstackRubiksCubeTask |
out_onto_surface | 90 | 1350 | reviewed | ready with note | longer_horizon, reset_relation |
UtensilsInMugTask |
in_container | 90 | 1350 | reviewed | ready with note | longer_horizon |
WhiteMugInCenterOfTableTask |
centered | 30 | 450 | reviewed | ready with note | wide_object |
WhiteMugsInBinTask |
in_container | 60 | 900 | reviewed | ready with note | wide_object |
WoodSpatulaToBowlTask |
in_container | 60 | 900 | reviewed | ready | - |
YellowAndWhiteObjectsInBinTask |
in_container | 60 | 900 | reviewed | ready with note | wide_object |
YogurtInBowlTask |
in_container | 40 | 600 | reviewed | ready | - |