The post in 50 seconds. Opening footage: Anthropic; NVIDIA, ASPIRE.

TL;DR

  • Robot-use agents are still far too slow for the real world. Benchmarks like RoboDojo don't show it, because their simulators wait for the model before stepping.
  • The agent's time goes mostly to per-request overhead, time to first token (TTFT) above all, which it pays on every tool-call round trip. So we cut the number of requests, kept the context short and made the tools faster.
  • We then pipelined thinking and motion: the robot carries out the agent's actions while the model plans the next ones, which hides most of the model's time behind motion.
  • Finished towel-folding runs went from a median of about 18 minutes to about 6, and on a cup task the robot moved 84% of the time. What remains is latency: the robot acts on what it saw about 5 seconds earlier.

Code: github.com/fast-rua/making-rua-fast, the robot API, the robot server and the agent harness we used, with a small simulator to try them in.

Over the past year, robot-use agents (RUAs) have become a serious alternative to end-to-end robot policies. GPT-6 Astra, driving a robot through code, scored above all 40 public policies on RoboDojo[1]. Uni-robot-agent put open-weight models such as Qwen3.8-Flash-Next on the same benchmark, well ahead of VLA and world-action-model baselines like π0.5 and OpenWAM-α[2]. Anthropic reports 64.9% on RoboDojo Sim for Fable 5.2[3].

On a real robot, though, these agents are slow. In Fable 5.2's launch video the robot solves a puzzle cube[3], but even with the pauses cut out, the solve takes about an hour.

Benchmarks like RoboDojo don't capture this. Their evaluation loop is synchronous: the simulator blocks until the policy returns its next actions, and episodes are limited by the number of steps, not by time[4]. In this setting a long TTFT or slow decoding costs nothing. JustRUA, a controlled study of GPT-6 Astra agents on RoboDojo, says as much: it reports what an episode costs, not how long it takes, because in simulation time can be hidden by running episodes in parallel[5]. On a real robot the clock keeps running while the model thinks, and the robot sits idle until the next command arrives.

Benchmark simulator the world waits for the agent think act think act think act sim clock time that counts:only the acts Real robot the clock never stops wall clock time that counts:all of it the robot idles while the model thinks
Figure 1. A simulator that waits for the model only counts the actions. On a real robot, every model step is time the robot spends standing still.

In this post we describe our first attempts at closing this gap on a real two-arm robot. We didn't change the model. Everything we did was in the harness around it: the tools the agent calls, what goes into its context, and how its actions are scheduled.

Setup

Our cell has two YAM arms and four cameras: one overhead, one in front and one on each wrist. The agent is GPT-6 Astra running in Codex at its ultrafast service tier and medium reasoning effort. It is given a task description and a Python robot API, with no scripted skills or demonstrations.

The two YAM arms at the back of a wooden table, with a blue towel laid out in front of them The four camera views tiled in one image: top, front, left wrist, right wrist
Figure 2. Our cell. Left: the two YAM arms, with a towel laid out for folding. Right: the four camera views tiled into one image, as the agent receives them.

Like PyRUA-Lean[6], we use code as the action interface: the robot is a Python object, and the agent acts by writing short programs against it. We also designed the tools as a white box. The robot API is ordinary Python source that the agent can read, and in practice it starts most tasks by reading it and then writing its own helper functions, such as one that lifts a particular cup. A separate robot server checks every command against a workspace fence and a minimum distance between the arms, which lets the agent run code it wrote itself without anyone reviewing it first.

Sandbox (bubblewrap) No internet: only the model API, the robot server and the cameras. Model GPT-6 Astra OpenAI API Codex its tools include exec_command write_stdin view_image no web search or browser Persistent Python kernel imports once; variables persist kernel.queue([a, b, c]) runs while the model thinks checks run after every motion a report as each action ends white box Robot API source read-only; the agent reads it robot/ MANUAL.md /workspace (read-write) the agent's own helpers def lift_cup(): ... Robot server fence + arm distance checked on every command IK, control 2 arms, 4 cameras calls results code reports commands results frames reads imports
Figure 3. The harness. Codex and everything the agent runs sit in a sandbox with no internet access. The agent reads the robot API's source, writes its own helpers, and runs its code in a long-lived Python process; the robot server checks every command, and results, images and reports flow back.

Where the time goes

Our early towel-folding runs on the real robot took between 13 and 37 minutes. Figure 4 profiles one of the longest. The model accounted for 78% of the wall time and tool calls for nearly all the rest. The robot moved for only a quarter of the run, and much of that motion happened inside blocking calls the agent was waiting on.

Figure 4. Where the time went in an early towel-folding run. The summary hold is the API holding each response until its reasoning summary was written; we turned summaries off for later runs. TTFT is estimated as model time minus decoding, which includes the hidden reasoning tokens.

Very little of the model's time went to producing output. About two thirds of it was TTFT. Measured from the harness, it covers prefill, the network and any other processing on the server; through the API we can't tell how it splits. A response with fewer than 100 output tokens still had a median TTFT of about two seconds, and an agent pays it on every tool-call round trip. Many of these requests did very little: the agent would spend a whole round trip viewing an image, checking on a program it had started, or asking where an arm had ended up. On average it made several requests per robot action, and the robot was idle through all of them.

Agent workloads are input-heavy, especially robot-use agents with their frequent image reads: the context grows with every step, while each output is a short tool call. So TTFT, paid on every request, outweighs decoding. To go faster, we needed the agent to make fewer requests and make each one cheaper, and the requests that remained had to happen while the robot was moving rather than before it.

1. Faster decoding

In short timed requests, Astra's ultrafast tier generated tokens about four times faster than the Fast (priority) tier, with the same time to first output. In our towel runs the median model response fell from 2.9 s to 1.9 s, a third shorter. These were different episodes, so the comparison is rough, but it fits the profile above: decoding was the smaller part of each response, and a faster decoder leaves TTFT where it was.

2. Fewer round trips

Since every request has a fixed cost, we looked for requests that were predictable, meaning the agent almost always made them around a certain kind of action, and merged them into the request that writes the action. Two patterns stood out (Figure 5). Before an action, the agent often reviewed its plan: it drew a ghost of the planned grasp into the camera images, looked at it, and then ran the action in a second request. After the action, it spent several more round trips looking: it captured the cameras, read each image in its own request, and asked for the robot's state. Neither involves a new decision.

Figure 5. One action and the requests around it (illustrative). Before: the agent writes the action and a review of its plan, looks at the review, and runs the action in a second request; when the motion ends, it captures the cameras and reads each image in its own request. After: the action starts as soon as it is written, the robot server checks each command as it runs, and the result comes back with the state, the camera views and the output of the agent's own checks.

The look comes back with the action. A program that renders images now returns them in its own tool result, in the same step, so the model sees them without a separate request. Every action ends by printing the robot's state (where each arm is, whether the grippers are holding something, and any faults) and attaching a camera view. The agent can add its own outputs with hooks: functions it registers once, which run at the end of every motion and add what they print and draw to the same result. In the crumpled-towel run, one hook found the towel in the front camera and listed a dozen points on it. When an action ends, all of this comes back together as one message, the action's report (Figure 6).

Two smaller changes removed follow-up requests. Error messages say which limit was hit and by how much, so the agent can usually fix the problem on its next try. And checking whether a set of candidate poses is reachable takes one call for the whole list rather than one call per pose.

Code runs as soon as it is written. The reviews changed the agent's plan in only about 4 of 99 cases, yet each one cost a full round trip per action. Now the agent starts the action in the same request that writes it, and looks at the plan while the arm is already moving. If something is wrong, it stops the action and the arms hold. Checks that need no judgment happen on the robot server, which checks every command as it is sent (reach, the fence, the distance between the arms) and refuses a bad one before the arm moves. Section 5 takes this further with pipelining: the agent writes the next actions while one runs.

1. Written and run in one request the model's code, sent to its Python kernel its own check def smooth_far(r): r.move(above, ee=arms) r.move(press, line=True) r.move(drag, line=True) r.move(lift, line=True) kernel.start(smooth_far) def towel_points(r, frames): viz.marks(frames["front"], ...) kernel.after_motion(towel_points) It starts as soon as the request ends: no separate request to run it, and no review before it starts. 2. While it runs arm moving: smooth_far check The robot server checks each command as it is sent (reach, fence, arm distance) and refuses a bad one before the arm moves. Meanwhile the model keeps working: it can look at the plan or the cameras. kernel.action().stop() stops it at once, and the arms hold. 3. When it ends, a report arrives a message to the model, images attached [action smooth_far] done in 15.1 s [state] left xyz=[0.55,0.29,0.08] ... [view] after-smooth_far every camera in one image [marks towel-points] 1:(0.69,-0.30,0.03) ... (12 points) output of the agent's own check No extra requests for images or state.
Figure 6. One action from start to end, adapted from the crumpled-towel run in Video 3 (code shortened, shown as a single action). The model writes the action and its own check (a hook) in one request, and both start at once. While the arm moves, the robot server checks every command, and the model keeps working and can stop the action. When the action ends, its report arrives as a message with the result, the arms' state, every camera in one image and the output of the hook.

3. Keep the context short

A robot agent looks at a lot of images, and as they pile up in the context, TTFT grows on every request, even though the earlier context is cached. We handled this in two ways. First, we send the four camera views as a single tiled image instead of four separate ones, which keeps TTFT noticeably lower even at the same resolution (Figure 8).

Second, once about 20 images have built up, the harness moves the conversation to a fresh thread in which older images are replaced by short text notes and only the newest few are kept (Figure 7). The first request after a prune can't use the prompt cache, so we prune between turns.

Before prompt + API After prompt + API Turns and text stay. Older images become notes like [image omitted: views/after-grasp.png] The newest images stay as they are.
imagetext turntext note for a removed image
Figure 7. Pruning old images. Every image still in the context adds to TTFT on every later request, so when about 20 have built up, the harness keeps the text of the conversation and the newest few images, and replaces the rest with short notes naming their files.
Figure 8. TTFT after 45 four-camera looks in one conversation. Tiling the four views into one image helps, and a smaller grid helps more.

4. Faster tools

Once most of the extra requests were gone, the time spent inside tools became noticeable, at 6 to 10% of the run. Most of it sat between a look and the next action, where nothing else could happen. The largest improvement came from running the agent's code in a persistent Python process. Imports are paid for once, Python starts a call in tens of milliseconds instead of a few hundred, and variables and helper functions survive from one call to the next (Figure 9). We also made camera capture faster.

1. One call defines far def far(x, y, z): return robot.pose([x, y, z], ...) F = {a: far(.58, y, .022) ...} FE = {...} for a in F: print(r.reachable([F[a],FE[a]],ee=a)) viz.Overlay(r, "front").ghost(...) left [(True, None), (True, None)] [view] far-strokes 2. 5 s later, the next uses it floor = GripperFloor(m, ...) for a in ["left", "right"]: floor.lowest(a, far(.58, .15, .022)) left ... far low 0.0159 right ... far low 0.0159 3. 18 s later, floor is still there for ang in [0, .15, .3]: T = scoop(.32, .25, .07, ang) print(floor.lowest("left", T), r.reachable([T], ee="left")) scoop 0.15 low 0.010 [(False, ...)] scoop 0.3 low 0.021 [(True, None)] defined in this call used again in a later call robot, np and viz were imported once, when the kernel started. Before each call's code runs (illustrative scale): a new Python process a few hundred ms to start and import; every name is gone when it exits the kernel tens of ms; names, helpers and imports stay
Figure 9. Three calls from the crumpled-towel run in Video 3, all sent to the same Python process (code and output shortened). The second call uses the helper far defined in the first, and the third uses floor from the second. None of them imports robot, np or viz. A new Python process for each call would take a few hundred milliseconds to start and import, and would lose every name when it exits.

5. Pipelining: hiding time behind motion

Even with all of the above, the agent still makes a few requests for each action, and as long as the next action can't start until the model has written it, the robot waits through every one of them. The last change was to let the model think while the robot moves.

The agent writes each action as a Python function and passes it to the kernel. The robot runs the actions in order, in the background, while the model plans what comes next, and the model can change the plan at any point: it can replace everything after a given action, or stop the one that is running.

kernel.queue([lift_left, lift_right])             # run in order, in the background
kernel.queue(new_plan, replace_from="lift_right") # swap out lift_right and what follows, while lift_left runs
kernel.action().stop()                            # stop now; the arms hold

The timing of feedback turned out to matter (Figures 10 and 11). The report for one action arrives as the next one starts, so the model reads how the last action went while the robot is already busy, and still has time to revise the action after that. If the model finishes its turn while the robot is still moving, the harness sends it a fresh camera view so that it gets started on the next step instead of sitting idle. And because each action starts from wherever the previous one left off, the agent writes actions that check their own preconditions, for instance that a cup is still in the gripper before carrying it anywhere.

Robot runs each in turn action 1 action 2 action 3′ Model plans ahead write 2 and 3 read report 1, replace 3 read report 2, write 4 report 1 + camera grid report 2 replace_from=3 The report for action 1 arrives as action 2 starts. The model reads it while the arm moves, and can still change action 3 before it begins.
Figure 10. How pipelining overlaps thinking with motion. Each report lands while the next action is already running, so the model's request time is hidden, as long as it writes the next action before the robot runs out of them.
Robot waiting its turn Model smooth_first smooth_far smooth_outer smooth_far waits smooth_outer waits report report 1 2 3 ghost of the strokes 3D view of the next crease 5 s 1 It adds the next stroke def smooth_far(r): r.move(F, ee=list(F), line=True) ... kernel.queue(smooth_far) [action smooth_far] queued behind smooth_first as #3 2 A report as smooth_far starts a message to the model [action smooth_first] done in 15.1 s [queue] started: smooth_far (#3; it had waited 7.9 s) [queue] empty behind it: as #3 ends, the arms stop until the next action is queued [state] left xyz=[0.405,0.247,...] [view] after-smooth_first [marks towel-points] 1:(0.345,...) 3 2.8 s later, the next one def smooth_outer(r): r.move(start, ...) ... kernel.queue(smooth_outer) [action smooth_outer] queued behind smooth_far as #4 smooth_outer starts the moment smooth_far ends: the arms never stop.
robot movingwaiting its turnmodel workingtool
Figure 11. 36 seconds of the crumpled-towel run in Video 3 (code and messages shortened). (1) While smooth_first runs, the model adds the next stroke, which waits its turn. In between, it checks its next plans: a ghost of the strokes on the front camera, then a 3D view of the next crease. (2) The report for smooth_first arrives as smooth_far starts, and says that nothing is waiting behind it. (3) The model adds the next stroke 2.8 s later, so it starts the moment smooth_far ends.
TTFTgeneratingtool / image readrobot moving
Figure 12. The same cup task in three harnesses, on one time axis. Top: a vanilla agent loop. Middle: sections 1–4, without pipelining. Bottom: with pipelining (the real run). Model time barely changes from middle to bottom; most of it now happens while the arm moves. The top two rows are replays of the real run, explained under Results. Excludes the agent's initial reading of the robot API code.

Uni-robot-agent[2] uses the same idea. Its tool calls can run in the background, so the model keeps reasoning while the robot executes, and in simulation this cut episode times substantially. Our version differs mainly in how results come back: the agent doesn't ask for them. Each action's report reaches it as a message as the next action starts, while there is still time to change what comes after.

Results

We first compare real runs. Figure 13 shows every towel-folding run that finished the task or ran for at least a few minutes, grouped by the three versions of the harness we ran on the robot: the original, one with fewer round trips, a shorter context and faster tools but no pipelining, and the full harness with pipelining. Run times vary a lot, because they depend on how the towel starts out and on the choices the agent makes, and many runs were stopped before the end. Alongside time to finish, we therefore show two measures that don't need a finished run: how much of the time the robot was moving, and how much of the model's time overlapped motion.

Cutting round trips raised the robot's moving share from a median of 29% to 52%, because the robot spent less time waiting between actions. It hid none of the model's time, though, since every action still blocked. Pipelining is what hid it: the median share of model time that overlapped motion went from almost none to about half. Runs that finished took a median of about 6 minutes with the full harness, against 16 to 18 minutes with the earlier versions.

Figure 13. Real towel-folding runs, grouped by harness version. Filled dots finished the task; hollow dots were stopped early. Time to finish counts only finished runs. Vertical bars are medians. Excludes the agent's initial reading of the robot API code.

Because real runs vary this much, they can't tell us what each design contributes on its own. For that we took one real run of the cup task, made with the full harness, and replayed it without each design in turn. The replay keeps the same model responses, tool calls and actions in the same order, and adds back the requests each design removes, at the step and tool costs we measured on the robot. Only the last bar in Figure 14 is the real run.

Figure 14. Wall time for the same cup task as each design is added. Model requests fall from 164 to 59. Pipelining removes no model time; it overlaps it with motion. Excludes the agent's initial reading of the robot API code.

Together, the changes reduce the wall time of this task from 667 s to 204 s, a 3.3× speedup. Before pipelining, almost all of the gain comes from cutting the number of model requests. Pipelining doesn't reduce the model's time at all, but it moves 137 of the 182 seconds of model time into periods when the robot is moving. From the first motion to the end, the robot goes from moving 24% of the time to 84%.

The table below lists the three runs from our demo videos, all made with the full harness. The last column is the median time from the latest observation the model had received (a report or a camera view) to the moment it submitted its next action.

Task (real runs)Time to finishActionsRobot moving*Model time hiddenObservation → action
Three cups hide a pear, a ball and nothing; put the pear under the empty cup204 s1484%75%5.7 s
Spread a partly folded towel and fold it in half143 s1167%49%4.7 s
The same, from a crumpled towel484 s3278%74%6.0 s

* From the first motion to the end of the task. Times and videos exclude the agent's initial reading of the robot API code.

JustRUA[5] reports two more things for each of its simulated episodes: what the episode consumed, and what the agent was doing in it. We looked at the same two for these three runs. Unlike the table above, the numbers below cover the whole episode, including the agent's initial reading of the robot API code.

Run (whole episode)RequestsInput tokensCachedOutput tokensContext*
Cups605.8 M95%14.0 k80 k
Partly folded towel447.4 M96%10.0 k156 k
Crumpled towel11026.3 M98%36.7 k195 k

* Median input tokens per request.

Like theirs, our episodes are almost all input: a run reads between 5.8 and 26.3 million input tokens and writes between 10 and 37 thousand, a ratio of 400 to 700 to one. Nearly all of that input had been sent before and came from the cache, so it is cheap to pay for, but each request still carries a context of 80 to 195 thousand tokens, which is why TTFT and not decoding sets the pace.

The second table splits the model's time in the three runs by what each request was for, and shows for how much of that time the robot was idle.

Request (three runs)RequestsModel timeShareRobot idle
Queue or run a motion57252 s32%30%
Capture or render camera views62250 s32%36%
Look up 3D points or IK1890 s11%6%
Read files3053 s7%85%
Wait on a running command1534 s4%19%
Message only1125 s3%22%
Read the robot's state26 s1%0%
Other1981 s10%36%
All214792 s100%32%

Motions and camera views take about a third of the model's time each, and for both the robot was idle for about a third of it. Geometry lookups are almost fully hidden, because the agent makes them while an earlier action is still running. Reading files is the opposite: most of it happens before the first motion, when there is nothing to hide it behind.

Video 1. The cup task at 2.5×. Below the cameras: the robot lane (each action), the agent lane (dark grey while the model works, light grey for tools) and the number of actions waiting.

TaskThree cups hide a pear, a ball and nothing; put the pear under the empty cup.

1×, no cuts
  1. I’ll inspect the cups, identify the pear and the empty cup, then move the pear and verify its placement.
  2. I’ll lift the cups vertically to reveal their contents. I’m checking each grasp for clearance so the fingers don’t push a cup into the object underneath.
  3. The left cup contains the ball. I’ll replace it and continue checking the other two cups.
  4. The right cup has revealed the pear. I’m checking that the middle cup is empty and clearing both cups so I can grasp the pear without catching either rim.
  5. The pear is under the right cup. I’m checking the empty cup and preparing the transfer.
  6. The middle cup is empty. I’ll move the pear to that cup’s original position, then lower the empty cup over it.
  7. The empty cup is set aside, and the gripper is closing around the pear. I’ll verify the lift before moving it across the table.
  8. The pear is securely lifted. I’m moving it to the middle position, then I’ll check its location before covering it with the empty cup.
  9. The pear is being placed at the middle position. The empty cup will follow once placement is verified.
  10. The pear is on the table at the middle position. I’m checking the cup’s alignment before lowering it.
  11. The cup now covers the pear. I’ll release it, withdraw the gripper, and check that it settles on the table.
  12. The previously empty cup is resting over the pear, and both grippers are clear. I’m marking the task complete.
  1. lift_left_cup
  2. lift_right_cup
  3. replace_ball_cup
  4. lift_center_cup
  5. park_right_cup
  6. park_center_cup
  7. view_pear
  8. grasp_pear
  9. pick_empty_cup
  10. transfer_pear
  11. hover_empty_cup
  12. lower_cup_check
  13. seat_cup
  14. release_cup
The same run at 1×, with nothing cut or sped up: the front camera for its 204 seconds. On the picture: what the model said as it worked, and the action under way, both from the run's record.
Video 2. Spreading a partly folded towel and folding it in half, at 2.5×.
Video 3. The same task from a crumpled towel, at 3×. The first fold comes out crinkled, and the agent lifts it until the cloth hangs taut, then lays it back down.

Limitations

Hiding the model's time behind motion is not the same as making the agent fast. Even when pipelining hides nearly all of it, every action is still based on an observation that is about 5 seconds old by the time the action is submitted (Figure 15). The robot moves most of the time, but it reacts like a player in a game with very high ping.

World keeps changing: the towel settles, a held object slips observation Model TTFT reason, write the action action starts from a 5-second-old view median 4.7–6.0 s
Figure 15. The latency pipelining can't hide: from the latest observation the model received to the action it submitted in response. Median per run, for the three runs in the table.

This is also why pipelining helps some tasks much more than others. Lifting three cups in turn can be planned from a single look, so several actions wait their turn and the robot rarely stops. Cloth needs a look after almost every grasp, so at most one action waits behind the running one, and every few actions the robot waits while the model looks and thinks (Figure 16). Tasks that need fine, live correction are the worst case. In our ring-stacking runs, where the agent had to line a ring up over a peg, the robot was moving only 28 to 36% of the time even with pipelining, because each small correction depended on seeing the result of the last one.

Figure 16. Part of the crumpled-towel run where pipelining didn't help. The robot waits while the agent looks before folding, and again while it re-plans after a sweep went wrong.

Closing this gap needs a shorter loop from observation to action. That means lower TTFT and less reasoning on routine steps, but it may also mean splitting the work, so that a faster, smaller model or a reactive controller handles the corrections that can't wait several seconds.

What's next

Throughout this work we treated Codex and OpenAI's serving as a black box. We could change only what the harness sends and when, and we could not even see how the time before the first output splits between prefill, queueing and the rest of the server. Much of the remaining latency sits on that side. With a model we serve ourselves, the harness and the server can be designed together, and we see many openings there: processing the next camera view while the arm is still moving, before the model is asked about it; dropping old images from the cache without paying for a full prefill; and scheduling a robot's requests so that a correction is not stuck in a queue. We think co-designing serving with the harness can take this a good deal further.

Acknowledgement

We thank Quanquan Peng and Xiaolong Wang for providing the YAM robot arms and for their guidance on using them.

Contributors

Shaoxiong Duan, Yang Yuanbo, Junda Chen, Quanquan Peng, Hao Zhang

References

  1. 1An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo. arXiv:2609.24170
  2. 2Democratizing Robot-Use Agents: A Unified Harness with Open Models (uni-robot-agent). arXiv:2611.01560
  3. 3Anthropic, Claude Fable and Mythos 5.2. anthropic.com
  4. 4RoboDojo. arXiv:2607.04434; evaluation loop in github.com/robodojo-benchmark/RoboDojo (src/eval_client/eval_env.py).
  5. 5JustRUA: A Controlled Study of GPT-6 Robot-Use Agents. arXiv:2611.16009
  6. 6Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens (PyRUA-Lean). arXiv:2610.01939

Citation

@misc{realtimerua2026,
  title        = {Getting Robot-Use Agents to Run in Real Time},
  author       = {Duan, Shaoxiong and Yang, Yuanbo and Chen, Junda and Peng, Quanquan and Zhang, Hao},
  year         = {2026},
  month        = nov,
  howpublished = {FastRUA Blog},
  url          = {https://fastrua.ai/blog/2026-11-realtime-rua/}
}