Contribute

Testing

tests/README.md is the reference for the suite: what each directory of the mirrored tree covers, the tests/support/ and conftest rules, the marker lanes, the commands, and what a run needs from its environment. This page is about choosing and changing checks.

Choose a check

Start with the changed contract, then run the modules that own it: every test module states the contract it protects in its docstring, and tests/ora/<package path>/ covers ora/<package path>/.

Change Check
One contract uv run --locked pytest -q tests/ora/<package path>/test_<name>.py
One package uv run --locked pytest -q tests/ora/<package path>/
Anything in Python uv run --locked pytest -q -n auto -m "not agent_core" (CI's tests job, scripts/dev/check lane A)
The agent core, its Python bridge (ora/agent/pi/) or anything that reaches the model-visible request Also uv run --locked pytest -q -m agent_core and uv run --locked node --test ora-pi/test/*.test.ts ora-pi/test/*.test.mjs (CI's agent-core job, lanes B and C)
Context projection, request assembly or provider wire bytes Also the opt-in replays of the sealed acceptance runs (ORA_TEST_SEALED_*); the Synthetic checks do not exercise those environments' tool surfaces
Before a pull request scripts/dev/check: every lane in parallel, then the documentation build and the no-write check

Inspect CPU, memory, existing jobs and simulator slots before scheduling checks, and use the available CPU capacity for independent offline cases; the root conftest.py runs BLAS and OpenMP with one thread per test process, so pytest-xdist workers do not oversubscribe the host. Fixture-based robot checks do not certify simulator physics or live service readiness.

Simulator-only code

The suite reaches a simulator backend's host side without the simulator: RunSpec builders compose and resolve against a scripted host profile, preflight checks run against scripted probes, and the scripted qualification tools (ora qualify robocasa|robolab motion) drive a scripted worker session (tests/support/orchestrator/sparse_motion.py). Code that runs only inside a simulator interpreter is not reached in process: the RoboCasa365 session and scene construction (robosuite), the RoboLab bridge and session (Isaac Sim and Kit), the RoboTwin play scenes and robot asset geometry (SAPIEN), the RoboDojo robot asset geometry (Omniverse), and the LIBERO probe and preview workers. Qualify a change there on the simulator host (docs/developer/authoring/qualification.md); a host-safe test of its pure parts belongs beside the backend's other tests.

Families split by protected contract

Family Directory Base contract Children
test_control_plane tests/ora/orchestrator/ Planning, leases and sealing preflight in tests/ora/orchestrator/preflight/, workspace beside it, viewer in tests/viewer/; the former cli child is tests/ora/app/test_main.py
test_runtime_v1 tests/ora/orchestrator/episode/ Lifecycle and session/resource contracts state, tools, libero beside it; metrics in tests/ora/orchestrator/evaluation/
test_provider_adapters tests/ora/agent/providers/ Provider adapters, profiles and the shared Synthetic fixtures test_anthropic_provider, test_openai_provider; test_sft_export in tests/ora/agent/evidence/
test_lossless_context_runtime tests/ora/agent/memory/ Recoverable projection cost in tests/ora/agent/context/lossless/
test_robotwin_motion_v2 tests/ora/environment/backends/robotwin/ Motion execution grasp, probe, profiles, sparse, surface

Each child states its own contract in its docstring. Shared helpers and constants keep a single owner, a module in tests/support/<area>/ that the base module and its children import, and a fixture they share is registered in the narrowest conftest.py above them. Do not duplicate constants into children: an asset update must not leave independent copies of its expected hash behind.

For another split or move, compare the complete test-function/decorator AST set and the per-module pytest collection before and after, and the fixtures each test sets up (pytest --setup-plan). Exercise cross-module helper consumers as well as the moved tests. A collection count alone cannot detect replacement of one test by another.

Timing and parallel execution

Splitting improves navigation and gives file-based schedulers smaller work units. It does not itself demonstrate faster wall time: uneven file costs can still put most work in one shard.

When diagnosing a slow or failing check, inspect subprocess startup, installed client versions, shard imbalance and the host's load before changing the test. The tests that wait on real time (tests/README.md lists them) fail first on a loaded host or a stalling file system, so reproduce such a failure on a quiet host before treating it as a regression. Report the worker count, selected cases, elapsed time and failures.

Pinned assertions and readability

The module-size, parameter-delivery and ownership tests are ratchets. A failure requires identifying which contract changed before editing an expected value. Frozen profile hashes and tool inventories are deliberate boundaries. Fix a stale probe when its implementation no longer checks the contract; fix product code when it violates the existing contract.

tests/repository/test_module_size_ratchet.py governs product Python modules, not test-file length. Names and docstrings describe the executable test; the module docstring states the module contract. Prefer behavior checks over source-layout assertions unless the source itself is the artifact under test, such as delivered prompt text.