Protocol
Table A1. Evaluation protocol for the results on the main page.
What changed in the full rerun
The LIBERO and LIBERO-Pro results come from a full rerun that replaced an earlier evaluation of the same cases. Paired case by case, the rerun turned 36 LIBERO failures into successes and 34 successes into failures (399 pairs, exact McNemar p = 0.91); on LIBERO-Pro the counts were 67 and 52 (796 pairs, p = 0.20). The small overall gains are therefore not a reliable improvement. Several harness changes went into the rerun, so it does not isolate the effect of any one of them.
Sources
RPent percentages and available denominators, including contributor corrections and the updated GPT-5.5 Long results, come from the pinned leaderboard page and its versioned result data, checked 2026-09-22. ORA counts come from the run records of 2026-10-12 11:25 PDT; we checked the plan, result, evaluation and seal hashes of every counted run. The page source includes the data and export scripts.
Exact values by benchmark and suite
RoboTwin C2R
LIBERO-Pro
LIBERO
Ours: rate over all cases, a case without a valid outcome counting as a failure; suite cells are successes / valid outcomes. — not reported; missing suites are not inferred. [a] Astra uses separately frozen exploration-memory snapshots for Long (157/200) and the other six suites (584/600). [b] Flash Mode uses released GPT-5.5 exploration memory and reports the best-performing seed from s0–s9. [c] ASPIRE's 61.36% is the board's mean of eight reported percentages; its Long results use zero-shot transfer from a LIBERO-90 skill library[8]. The suite table also keeps RPent's motor-only Astra result (38% on Long Task only), which uses neither VLA primitives nor task memory.
Success by task family, our runs onlySuccesses / valid official outcomes. Long is the weakest family in all three settings.
Cost
Cost covers the evaluation runs on the main page and the RoboDojo scan. Tokens are priced at DeepSeek's off-peak USD rates for DeepSeek-V4.1-Flash: $0.003, $0.15 and $0.60 per million cached input, uncached input and output tokens[17]. The self-hosted model does not report cache hits, so the cost script reconstructs them from request lengths: a call that contains the whole previous request and response counts that prefix as cached; any other call counts only the fixed prompt. The resulting hit rates are estimates, not measured cache hits. Simulator and GPU costs are excluded. Script: page/cost_usd.py.
Table A2. API-equivalent model-token cost per episode and per solved case.
Failure audit and time
Motion execution on a reasonable target64rounds
Strategy, order or tool use29rounds
Perception or localization7rounds
Simulator anomaly3rounds
Gave up while recovery was still possible3rounds
Undetermined2rounds
Figure A1. Main cause of each failed RoboDojo round in the 42-task scan, one auditor judgement per round; 27 of the 64 motion cases are low-confidence. No round exhausted its decision budget. Most failed rounds ended when the model stopped immediately after a failed motion. The auditor judged recovery still possible in only three rounds. In two earlier scans, 91% of 746 failed inverse-kinematics attempts found no solution from 32 seeds although the targets were inside the workspace bounds.
Waiting for the model63%58.0 h
Executing an action until the next observation25%22.7 h
Running tools, including closed-loop grasps7%6.6 h
Everything else5%4.4 h
Figure A2. Where episode time goes. Time shares come from a separate timing sample of 79 episodes, 12,206 model calls and 91.7 episode-hours, using paired trace timestamps. The episode medians come from the later evaluation runs on the main page: RoboTwin 25.0 min, LIBERO-Pro 15.9 min, LIBERO 13.8 min. A paragraph-level classifier labelled 58% of reasoning paragraphs as concerning geometry, a broader category and a different unit from responses that compute geometry by hand. The audit estimated that batching independent read-only calls could remove at most 7% of model calls in this sample.
Table A3. Source run of every video and image on the main page.
All videos use original captured frames, not simulator reconstructions, shown at simulation time with model and tool waiting removed; text panels show only visible model text, tool calls, results and receipts. Each montage tile runs from reset to the official success at native resolution or above. RoboTwin tiles are clean scenes from warm-up sessions (not evaluation runs), LIBERO and LIBERO-Pro tiles come from the earlier evaluation, and RoboDojo tiles from the first scans, frame-interpolated from 10 Hz. The 40 rounds cost $5.40 in model tokens at the prices above ($0.13 each), not counting earlier failed rounds in the same sessions.
Both are two-round sessions on a repeated seed, task memory off, chosen after inspecting outcomes rather than sampled. The laptop quotes are the model's visible text.