Python API

Plans and benchmark cells

These functions compose or inspect configuration. Resolving a plan does not launch an episode. Source, host, controller, provider, and qualification gates still apply at launch.

from ora.orchestrator.spec import (
    load_run_spec,
    resolve_run_spec,
    validate_run_plan,
)

load_run_spec

load_run_spec(path: Path) -> dict[str, Any]

resolve_run_spec

resolve_run_spec(
    spec: Mapping[str, Any],
    *,
    root: Path = PROJECT_ROOT,
) -> dict[str, Any]

Validate one RunSpec and bind it into a content-addressed RunPlan.

Sections resolve in this fixed order, each raising the first error it finds, so a RunSpec with several invalid fields always reports the same one. The policy and runtime sections default and normalize shallow copies of spec["policy"] and spec["runtime"].

validate_run_plan

validate_run_plan(plan: Mapping[str, Any]) -> dict[str, Any]

Verify one persisted RunPlan's complete content-addressed identity.

Benchmark builders

from ora.orchestrator.spec.benchmark import (
    build_robotwin_spec,
    build_libero_spec,
    build_libero_pro_spec,
)

build_robotwin_spec

build_robotwin_spec(
    task: str,
    *,
    seed: int,
    experiment: str,
    resource_profile: str,
    seed_manifest: Path | None = None,
    expert_unsolved: Sequence[Path] | None = None,
    expert_unsolved_reason: str | None = None,
    task_config: str = 'demo_randomized',
    episode_names: Mapping[str, Any] | None = None,
    host: str | None = None,
    config: str | None = None,
    provider_profile: str | None = None,
    provider_catalog: Path | None = None,
    budgets: Mapping[str, int] | None = None,
    service_report: Path | None = None,
    cohort_id: str | None = None,
    milestone: str | None = None,
    evidence_tier: str = 'behavioral',
    rounds: int | None = None,
    baseline: str | None = None,
    root: Path = PROJECT_ROOT,
) -> dict[str, Any]

One RoboTwin cell (task, seed, scene configuration) on the default baseline.

seed_manifest must admit the seed (admitted_seed_manifest); the spec names its content-addressed import path and compose imports it. Instead, expert_unsolved names official scan manifests that show the expert failed this seed, with expert_unsolved_reason: the cell then runs under a labelled qualification.exception (backends.robotwin.qualification_exception), for an external case list scored on its own denominator. Exactly one of the two is given.

build_libero_spec

build_libero_spec(
    suite: str,
    task_id: int,
    *,
    seed: int,
    experiment: str,
    resource_profile: str,
    host: str | None = None,
    config: str | None = None,
    provider_profile: str | None = None,
    provider_catalog: Path | None = None,
    budgets: Mapping[str, int] | None = None,
    service_report: Path | None = None,
    cohort_id: str | None = None,
    milestone: str | None = None,
    evidence_tier: str = 'behavioral',
    rounds: int | None = None,
    baseline: str | None = None,
    root: Path = PROJECT_ROOT,
) -> dict[str, Any]

One original LIBERO cell (base suite, task, initial state seed) on the LIBERO-Pro backend.

The four source-paired base suites (spatial, object, goal, 10) run on the same pinned LIBERO-Pro code and data as the core suites; LIBERO-90 is not an official evaluation suite and has no pinned language here. The policy receives the official instruction (base_task_language); when the BDDL file words the same goal differently, the plan records both texts verbatim so launch preflight admits exactly that divergence.

build_libero_pro_spec

build_libero_pro_spec(
    suite: str,
    task_id: int,
    *,
    seed: int,
    experiment: str,
    resource_profile: str,
    host: str | None = None,
    config: str | None = None,
    provider_profile: str | None = None,
    provider_catalog: Path | None = None,
    budgets: Mapping[str, int] | None = None,
    service_report: Path | None = None,
    cohort_id: str | None = None,
    milestone: str | None = None,
    evidence_tier: str = 'behavioral',
    rounds: int | None = None,
    baseline: str | None = None,
    root: Path = PROJECT_ROOT,
) -> dict[str, Any]

One LIBERO-Pro core cell (suite, task, initial state seed) on the default baseline.

The official core evaluation runs initial states 0-49 of each task (libero_pro.official_core_evaluation); the Runtime selects state seed % 50, so a seed outside that range would repeat one.

Source: docs/api/plans.md