Appendix

Evaluation details, failure audit and provenance for ORA: Open Robot-Use Agents.

Protocol

RoboTwinAll 250 cases of the published C2R list, run directly on the randomized scenes with no clean-scene pass.
LIBERO-ProAll 800 Task and Swap cases, run with the current default harness.
LIBEROThe same evaluation run as LIBERO-Pro, all 400 cases, with the official base-suite instructions.
Model and budgetSelf-hosted DeepSeek-V4.1-Flash, low reasoning effort, task memory off. One scored outcome per case; at most 80 robot decisions, 200 model calls, 10,000 simulator steps and six hours. Five LIBERO and LIBERO-Pro cases had an infrastructure failure without a verdict and were rerun once as explicit -r1 replacements. The original attempts remain in the evidence; scores use their replacements.
TermsTask: a benchmark task. Case: a task with one seed from the published list. Episode: one attempt at a case. Round: one attempt within a session of several rounds on the same scene. Decision: one call that moves the robot.
Success checkThe model may call check_task, which returns the official success result for the current state without ending the episode. It never sees the check's code. We have not confirmed whether RPent's[1] agents have an equivalent query.
SuccessThe environment's own verdict, from a sealed, valid evaluation. A case without a valid outcome counts as a failure in the charts; the exact tables also give the rate over valid outcomes.
Versus RPentSame cases, different conditions: RGB-D images, geometry and grasp tools, ORA's task text (RPent's seed-specific wording for nine place_a2b cases), different action and time budgets. RPent's C2R protocol explores the clean scenes first and carries task memory to the randomized ones, with LingBot-VLA[2] as its primitive.
Harness versionRoboTwin results come from the earlier evaluation run. LIBERO and LIBERO-Pro come from a full rerun after the overhead-footprint change, with official base-suite instructions and restored observation rules. Several changes separate the rerun from the earlier tables, so it is not an ablation of overhead localization.
Leaderboard notesRPent lists GR00T-N1.7[3], π0.5[4] and StarVLA[5] without stating how they were run. RPent / GPT-5.5's 62.4% is the repository's update; the paper's Codex row reports 58.0%.

Table A1. Evaluation protocol for the results on the main page.

What changed in the full rerun

The LIBERO and LIBERO-Pro results come from a full rerun that replaced an earlier evaluation of the same cases. Paired case by case, the rerun turned 36 LIBERO failures into successes and 34 successes into failures (399 pairs, exact McNemar p = 0.91); on LIBERO-Pro the counts were 67 and 52 (796 pairs, p = 0.20). The small overall gains are therefore not a reliable improvement. Several harness changes went into the rerun, so it does not isolate the effect of any one of them.

Sources

RPent percentages and available denominators, including contributor corrections and the updated GPT-5.5 Long results, come from the pinned leaderboard page and its versioned result data, checked 2026-09-22. ORA counts come from the run records of 2026-10-12 11:25 PDT; we checked the plan, result, evaluation and seal hashes of every counted run. The page source includes the data and export scripts.

Exact values by benchmark and suite

RoboTwin C2R

MethodSuccessCount
ORA + DeepSeek-V4.1-Flash69.2%173/250 cases; 0 without a valid outcome; 69.2% of valid outcomes
RPent + GPT-5.5 (xhigh)62.4%156/250
RPent + Opus-4.7 (max)58.4%250 cases
LingBot-VLA[2]50.4%—
π0.5[4]47.9%—
GR00T-N1.7[3]20.7%—
StarVLA[5]10.6%—

LIBERO-Pro

MethodSuccessCount
ORA + DeepSeek-V4.1-Flash81.4%651/800 cases; 0 without a valid outcome; 81.4% of valid outcomes
RPent + GPT-6 Astra (low) [a]92.63%741/800
RPent + Opus-4.7 (max)82.4%800 cases
RPent + GPT-5.5 (xhigh)75.13%800 cases
RPent Flash Mode / Molmo2-8B [b][6]72.63%581/800
RPent + Qwen3.6 27B (no reasoning)[7]70.63%565/800
ASPIRE [c][8]61.36%—
π_RLinf[9]50.0%—
π0.511.0%—
AtomVLA[10]6.3%—
X-VLA[11]3.8%—
MolmoAct[12]1.5%—
π0[4]0.3%—
By suiteSpatial TaskSpatial SwapObject TaskObject SwapGoal TaskGoal SwapLong TaskLong Swap
ORA + DeepSeek-V4.1-Flash98/10091/10093/10098/10077/10078/10052/10064/100
RPent + GPT-6 Astra (low) [a]100%98%100%99%88%99%85%72%
RPent + GPT-5.5 (xhigh)81.0%69.0%94.0%91.0%75.0%66.0%70.00%55.00%
RPent Flash Mode / Molmo2-8B [b]79.00%70.00%86.00%93.00%74.00%65.00%60.00%54.00%
RPent + Qwen3.6 27B (no reasoning)82.00%78.00%83.00%84.00%68.00%68.00%61.00%41.00%
ASPIRE [c]60.0%51.0%95.0%98.0%45.0%81.0%38.3%22.6%
π_RLinf42.0%59.0%71.0%78.0%45.0%42.0%49.0%14.0%
π0.51.0%20.0%1.0%17.0%2.0%38.0%1.0%8.0%
AtomVLA1.0%16.0%0.0%10.0%11.0%2.0%9.0%1.0%
X-VLA0.0%0.0%8.0%2.0%9.0%1.0%10.0%0.0%
MolmoAct0.0%0.0%0.0%6.0%0.0%0.0%6.0%0.0%
π00.0%0.0%0.0%2.0%0.0%0.0%0.0%0.0%
Cap-X[13]14.0%12.0%18.0%22.0%17.0%26.0%——
RPent / GPT-6 Motor Only (low)——————38.0%—
RATS[14]31.0%29.0%63.0%61.0%36.0%43.0%——

LIBERO

MethodSuccessCount
ORA + DeepSeek-V4.1-Flash83.5%334/400 cases; 0 without a valid outcome; 83.5% of valid outcomes
AtomVLA97.0%—
RPent + Opus-4.7 (max)96.0%384/400
π_RLinf95.3%—
π094.2%—
NORA[15]79.5%—
OpenVLA[16]76.5%—
By suiteSpatialObjectGoalLong
ORA + DeepSeek-V4.1-Flash95/10092/10084/10063/100
AtomVLA96.4%99.6%97.6%94.4%
RPent + Opus-4.7 (max)97.0%100.0%94.0%93.0%
π_RLinf99.0%96.0%97.0%89.0%
π096.8%98.8%95.8%85.2%
NORA85.6%89.4%80.0%63.0%
OpenVLA84.7%88.4%79.2%53.7%

Ours: rate over all cases, a case without a valid outcome counting as a failure; suite cells are successes / valid outcomes. — not reported; missing suites are not inferred. [a] Astra uses separately frozen exploration-memory snapshots for Long (157/200) and the other six suites (584/600). [b] Flash Mode uses released GPT-5.5 exploration memory and reports the best-performing seed from s0–s9. [c] ASPIRE's 61.36% is the board's mean of eight reported percentages; its Long results use zero-shot transfer from a LIBERO-90 skill library[8]. The suite table also keeps RPent's motor-only Astra result (38% on Long Task only), which uses neither VLA primitives nor task memory.

Success by task family, our runs only
LIBEROPro · TaskPro · Swap
Spatial95/10098/10091/100
Object92/10093/10098/100
Goal84/10077/10078/100
Long63/10052/10064/100

Successes / valid official outcomes. Long is the weakest family in all three settings.

Cost

Cost covers the evaluation runs on the main page and the RoboDojo scan. Tokens are priced at DeepSeek's off-peak USD rates for DeepSeek-V4.1-Flash: $0.003, $0.15 and $0.60 per million cached input, uncached input and output tokens[17]. The self-hosted model does not report cache hits, so the cost script reconstructs them from request lengths: a call that contains the whole previous request and response counts that prefix as cached; any other call counts only the fixed prompt. The resulting hit rates are estimates, not measured cache hits. Simulator and GPU costs are excluded. Script: page/cost_usd.py.

BenchmarkEpisodesMedian episodeMean episodeSpend per solved case
RoboTwin250$0.12$0.24$0.35
LIBERO-Pro800$0.10$0.28$0.35
LIBERO405$0.09$0.25$0.30
RoboDojo126$0.35$0.39$2.73

Table A2. API-equivalent model-token cost per episode and per solved case.

Failure audit and time

Figure A1. Main cause of each failed RoboDojo round in the 42-task scan, one auditor judgement per round; 27 of the 64 motion cases are low-confidence. No round exhausted its decision budget. Most failed rounds ended when the model stopped immediately after a failed motion. The auditor judged recovery still possible in only three rounds. In two earlier scans, 91% of 746 failed inverse-kinematics attempts found no solution from 32 seeds although the targets were inside the workspace bounds.

Figure A2. Where episode time goes. Time shares come from a separate timing sample of 79 episodes, 12,206 model calls and 91.7 episode-hours, using paired trace timestamps. The episode medians come from the later evaluation runs on the main page: RoboTwin 25.0 min, LIBERO-Pro 15.9 min, LIBERO 13.8 min. A paragraph-level classifier labelled 58% of reasoning paragraphs as concerning geometry, a broader category and a different unit from responses that compute geometry by hand. The audit estimated that batching independent read-only calls could remove at most 7% of model calls in this sample.

Media and case provenance

ItemEpisode
Figure 1 (overview video)ORA's own runs only: RoboDojo episodes in the terminal UI and the 40 solved rounds across LIBERO, LIBERO-Pro, RoboTwin and RoboDojo
Figure 3 (terminal replay)RoboDojo stack blocks by language, round 2
Figure 14, Learning across roundsTop: RoboDojo stack blocks by language, rounds 1–2 of one session from the 42-task scan. Bottom: RoboTwin open laptop, rounds 1–2 of one trial session outside the reported evaluation. All panes at one simulated-time speed
Figure 15, task wordingLIBERO-Spatial task 4, seeds 1 and 2; left from the earlier evaluation with our paraphrased instruction, right from the later evaluation with the benchmark's instruction
Figure 16, completion contractRoboTwin adjust bottle and move playing card away, one randomized seed each; left from the run set without the contract (a delivery bug), right from the evaluation with it
Figure 17, text groundingTop: LIBERO-Pro Object Swap task 4, seed 4. Bottom: RoboTwin stack bowls two, one randomized seed. Left panes are the returned tool images
Figure 18, placement centreLIBERO-Long task 6 seed 5 and LIBERO-Pro Long Swap task 4 seed 2; same-commit A/B, left with the camera-view mask centre, right with the centre seen from above
Figure 19, refused movesTop: RoboTwin blocks ranking size, one randomized seed. Bottom: LIBERO-Pro Spatial Swap task 1, seed 4. Not matched comparisons
Figures 4 and 5Slide photo (X post); Robocurve video still (X post)

Table A3. Source run of every video and image on the main page.

All videos use original captured frames, not simulator reconstructions, shown at simulation time with model and tool waiting removed; text panels show only visible model text, tool calls, results and receipts. Each montage tile runs from reset to the official success at native resolution or above. RoboTwin tiles are clean scenes from warm-up sessions (not evaluation runs), LIBERO and LIBERO-Pro tiles come from the earlier evaluation, and RoboDojo tiles from the first scans, frame-interpolated from 10 Hz. The 40 rounds cost $5.40 in model tokens at the prices above ($0.13 each), not counting earlier failed rounds in the same sessions.

CaseSessionRounds
Bread to skilletOctober 5, 19:31 PDT23 decisions, failed → 8, succeeded
Open laptopOctober 5, 19:28 PDT23 decisions, failed → 6, succeeded

Both are two-round sessions on a repeated seed, task memory off, chosen after inspecting outcomes rather than sampled. The laptop quotes are the model's visible text.

Evidence for the findings

FindingEvidenceKind
Overhead footprint localizationOverhead footprint centre vs mask median, LIBERO libero_10 t04/t06: 29 → 42 of 59 paired states, McNemar p = 0.011; 30 states from other families 25 → 21, p = 0.13.paired A/B
Observation shortening and restored rulesDelivery trim v1 vs untrimmed, RoboDojo, same commit: 10 vs 17 of 27 rounds, 24% more model calls. Trim v2 (six rules restored): 15 vs 13 of 24 complete pairs.same-commit A/B
Missing constants get guessed28 of 46 runs guessed the controlled-point-to-fingertip offset (0–0.103 m); in one LIBERO episode the value was in context and was never used. 21 of 25 RoboTwin runs stated the ALOHA jaw opening 117 times with 26 values from 50 to 100 mm; the true 94.9 mm appeared in none of their inputs. Matched pair: the same RoboTwin task and seed, failed without the number and succeeded with it in the tool help; the second run also had the completion contract and a newer tool interface.run audits, one matched pair
The benchmark's own instructionLIBERO-Spatial task 4 with a paraphrased instruction (“top layer” for “top drawer”): 2 of 10 seeds; with the official instruction: 10 of 10. The run sets also differ in harness version.before/after, ten seeds
Write down what counts as done206 RoboTwin cases run twice with the same model, effort and seeds; a delivery bug left the completion contract out of the first set: 110 → 132 successes (53.4% → 64.1%), 43 against 21 discordant cases, exact McNemar p ≈ 0.008. The tool interface also changed between the sets. A later rewording of nine contracts: 56% → 53%, p = 1.0.before/after, confounded
The robot card is what changes between robotsThe same model, planner, evidence rules, task template and tool-schema delivery on all three benchmarks; per robot: the card, the benchmark boundary, the tool profile, the controller and the simulator worker.configuration
Reachability feedback after refusalsEarlier RoboDojo runs abandoned refused poses as unreachable. Offline sweeps found tilted alternatives; in a replay of 77 refusals, nearest reachable orientations resolved 69. Bare argument errors: 14 identical GPT-6 Astra calls in five minutes.run audit, offline replay
Geometry calculations in model responsesA corpus audit labelled 12.2% of 35,270 responses as manual geometry derivations. A separate prompt-only test on 11 completed pairs of previously failed RoboTwin cases found arithmetic in 76.2% of baseline reasoning calls and 76.3% with calculator-routing instructions. These are different populations and measures. Three earlier RoboDojo runs wrote an upward quaternion for “approach from above”; poses given as world axes were added afterwards.corpus count, run audit

Table A4. The evidence behind each finding on the main page, and how strong it is.

Two controlled tests did not establish an improvement. With the RoboDojo motion-safety fixes, 12 of 102 rounds succeeded, against 11 of 102 without them (same code otherwise). A closed grasp skill on LIBERO succeeded in 35 of 60 paired states, against 41 of 60 without the skill (p = 0.18). Rerun variability is substantial: 15 of 30 failed LIBERO control cases succeeded on a plain rerun.

Still open

Success on LIBERO-Long and the two LIBERO-Pro long suites ranges from 52 to 64%, against 91 to 98% on Spatial and Object. On October 6, an earlier harness solved 18 of 126 rounds in a first scan of 42 two-arm RoboDojo tasks[18], completing 9 tasks at least once. In 64 of the 108 failed rounds the target was reasonable and the motion went wrong (27 of these judgements are low-confidence). The benchmark results compare complete systems in simulation with one language model; the component tests cover single changes under narrower conditions.

What goes wrongHarness change it points to
Free-space moves and the return home ignore objects on the table and allow joint jumps of about 3 rad; an arm can sweep away a block it just placed.Check the path, not only the end point.
A grasp slips in transport and the receipt does not say so; the model places with an empty gripper.Report a dropped object.
Many requested orientations have no inverse-kinematics solution across 32 solver seeds, leading to refused requests.Answer with the nearest reachable pose.
In the timing sample, a classifier labelled 58% of reasoning paragraphs as geometry-related.Hand the model derived geometry.

Table A5. Failure patterns that remain open and the harness change each one points to.

References

  1. 1Yixian Zhang et al. (2026). Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents. arXiv:2607.08448. Code and leaderboard: RPent (RLinf).
  2. 2Wei Wu et al. (2026). A Pragmatic VLA Foundation Model. arXiv:2601.18692.
  3. 3NVIDIA (2025). GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734.
  4. 4Physical Intelligence (2025). openpi. Software.
  5. 5StarVLA Community (2026). StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing. arXiv:2604.05014.
  6. 6Christopher Clark et al. (2026). Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding. arXiv:2601.10611.
  7. 7Qwen Team (2026). Qwen3.6-27B. Model card.
  8. 8Runyu Lu et al. (2026). ASPIRE: Agentic Skills Discovery for Robotics. arXiv:2607.00272.
  9. 9Kang Chen et al. (2025). πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889.
  10. 10Xiaoquan Sun et al. (2026). AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models. arXiv:2603.08519.
  11. 11Zheng et al. (2026). X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model. ICLR.
  12. 12Lee et al. (2025). MolmoAct: Action Reasoning Models that can Reason in Space. arXiv 2508.07917.
  13. 13Letian Fu et al. (2026). CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. ICML.
  14. 14Junyi Zhang et al. (2026). Playful Agentic Robot Learning. arXiv:2606.19419.
  15. 15Chia-Yu Hung et al. (2025). NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Tasks. arXiv:2504.19854.
  16. 16Moo Jin Kim et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model. CoRL.
  17. 17DeepSeek (n.d.). Models & Pricing. Documentation.
  18. 18Tianxing Chen et al. (2026). RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. arXiv:2607.04434.