Why robot-use agents
LLM agents now handle much digital work, including coding[11] and computer use, as computer-use agents (CUAs)[12, 13]. However, most existing robot policies, such as vision-language-action models (VLAs)[14, 15] and world action models (WAMs)[16], still rely on robot-specific training data, collected by teleoperation[17], with handheld grippers such as UMI[18] or from egocentric videos[19]. An alternative is to build on the reasoning and tool calling of existing LLM agents. As a recent talk asked: could a strong enough agent operate a robot from its manual alone (Figure 4)[20]?
Figure 4. From Jianyuan Wang's talk on general robotics at IMW 2026; credit[20].
Figure 5. Zero-shot direct robot control with GPT-6 Astra; credit[21].
This suggests bringing the agent loop to the physical world, as in a recent robot-use agent[22]: a tool call moves the robot, the model reasons about what it sees, and the next camera frame starts the next turn. Meanwhile, frontier models such as GPT-6 Astra have shown strong zero-shot direct robot control (Figure 5)[21], and several recent demos operate robots this way[23, 24, 25].
Open-weight models, however, have barely been tested on this, yet robot use needs them for two reasons. First, to understand how well language models manipulate objects and how to improve them, we need large amounts of reproducible data, which means a model we can serve ourselves and experiment with at an acceptable cost, rather than paying frontier-model API prices. Second, a deployed robot system built on an LLM must be fast and efficient enough to run on local hardware. That needs an open model small enough to deploy, and task data that only an open model makes affordable to collect.
Where ORA fits
Methods that control robots with learned models fall into four families. They differ in the role of the language model and in how often it makes a decision (Table 1).
Vision-language-action (VLA) models[14, 3, 15] and, more recently, world action models that also predict future video[16, 26] map camera images and an instruction to actions at every control step and are trained on the largest robot datasets. Because the vision-language backbone is fine-tuned into a visuomotor policy that outputs actions, these systems use little of its ability to reason and act as an agent, for example to measure before moving, plan several steps ahead or change approach after a failure.
A common response is to add a language model as a planner on top of a learned policy, the System 2 / System 1 design[27, 28, 29, 5]. The language model splits the task into steps and the policy carries them out. In RPent[1], a language model directs a VLA policy using task-specific memory. The model can also command some motions itself, but most of the manipulation is still done by the VLA, which is trained on robot demonstrations.
A third family has the language model write the policy as code. Code as Policies[30] turns the scene and the task into a program that calls perception and control primitives. CaP-Agent0[31], ASPIRE[32] and RATs[33] add an agent that runs the program, reads the outcome and revises it; ASPIRE and RATs also keep the skills that work. These methods need no robot demonstrations, and they are economical to run: the model is called once or a few times per task, and the program then runs without it. The price is that the program executes as one or a few open-loop segments. Getting it right often takes trial runs (ASPIRE, for example, learns its skills on separate seeds before it is tested), and between revisions no model looks at the scene: the program reacts only through the checks written into it, which limits how well it carries over to new scenes.
The fourth family runs the agent loop directly, in the ReAct style[34]: the model reasons, calls a tool, reads the result and decides again. Frontier models have controlled robots this way[21, 23], and Guava[35], OpenRUA[36], Agent as Policy[37] and Robo-Harness K1[38] are harnesses for such agents. When the model issued every low-level command itself, the best model in one study reached 5.5% on LIBERO[23]. ORA is in this family. Each tool call covers one bounded motion, which a controller executes, and the model sees the result before its next decision. It costs a model call at every step, so at today's latencies and prices this family is slower and costlier to run than a program written once (see Cost and Limitations).
Table 1. Four families of methods that control robots with learned models, by the role of the language model and how often it decides. Some systems combine several families.