Agentic robot learning
An Agentic Real2Sim2Real System
for Task-Specific Robot Data Generation
One image. One instruction.
Simulation demonstrations for real-world robot learning.
Learned policies on the real robot
Selected real-robot executions. Playback speeds are shown for each clip.
The idea
Simulation can supply robot training data at scale. Preparing the scene, designing expert behavior, and configuring collection still require substantial work for each new task.
ARSTAG brings these stages into a coordinated agent workflow. Given a single RGB image and a natural-language instruction, specialized agents reconstruct the relevant scene, generate robot-feasible demonstrations, and train a visuomotor policy for real-world deployment.
Method
A coordinator delegates work and uses tool feedback to guide revisions across stages.
Vision-language reasoning selects objects and their supports in a scene graph. Caption-guided segmentation and identity binding link graph objects to image masks. Object reconstruction and support repair produce a usable simulation scene.
The collection agent generates an expert that uses ground-truth simulation states and state-based skills. Feasibility screening and task-consistent randomization produce demonstrations, including recovery from disturbed states.
The learning agent configures policy training and closed-loop simulation evaluation. The resulting visuomotor policies use visual observations and proprioception for real-robot execution.
Demonstration library
Browse more simulation and real-robot recordings, with paired comparisons of the reconstruction and data-generation stages.
The same task is executed across sampled object placements, robot configurations, and visual backgrounds. All 24 successful demonstrations play automatically in four consecutive groups of six. Each group finishes before the next begins.
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Screw-to-basket placement · 1× playback
Group 1 of 4 · Scenes 1–6
We also test perturbations during real-world execution. The learned policy uses updated visual observations to adjust its motion and recover.
2.28× playback · operator-induced perturbation
4.53× playback · operator-induced perturbation
2.28× playback · operator-induced perturbation
4.53× playback · operator-induced perturbation
During collection, we perturb the robot trajectory or induce a mis-grasp. Only the expert continuation from the disturbed state enters the training dataset.
The arm is displaced by 14 cm. The expert continues from the disturbed state. 2.1× playback.
After an empty grasp, the object is relocated by 38 cm. The expert re-grasps it. 3.3× playback.
Appearance and scene context link each selected scene-graph object to the intended mask and supporting surface. Each comparison includes its scene description or instruction.
Each side-by-side pair shows the same scene without repair on the left and with repair on the right. Repair aligns object-support contacts before simulation.
Left: without repair · Right: with repair · 0.44× slow motion
Left: without repair · Right: with repair · 0.44× slow motion
Left: without repair · Right: with repair · 0.09× slow motion
Left: without repair · Right: with repair · 0.09× slow motion
Left: without repair · Right: with repair · 0.09× slow motion
Left: without repair · Right: with repair · 0.30× slow motion
Real-world evaluation
Seven tasks cover general grasping, key-region grasping, placement, and stacking. All three policy architectures use the same generated demonstrations for each task.
| Task | ACT | Diffusion Policy | π0.5 |
|---|---|---|---|
| Cone grasping | 55.0 | 77.5 | 65.0 |
| Brick grasping | 25.0 | 57.5 | 92.5 |
| Teapot grasping | 40.0 | 67.5 | 72.5 |
| Screw grasping | 62.5 | 72.5 | 90.0 |
| Screw to basket | 42.5 | 62.5 | 60.0 |
| Nut to plate | 35.0 | 62.5 | 67.5 |
| Bowl stacking | 45.0 | 67.5 | 75.0 |
| Average across seven tasks | 43.6 | 66.8 | 74.6 |
40 real-robot trials per task and policy. Bold values mark the highest success rate in each row.
Randomization ablation
Leave-one-out ablations evaluate screw grasping under out-of-distribution conditions. Each cell reports success over 40 held-out simulation episodes.
| Tested family | Full recipe | Without family |
|---|---|---|
| Execution | 75.0% | 40.0% |
| Geometry | 62.5% | 40.0% |
| Background | 70.0% | 0.0% |
Demonstration scale
With 100,000 training steps held fixed, Diffusion Policy improves as the screw-grasping dataset grows. Each policy is evaluated over 40 held-out simulation episodes.
| Demonstrations | Success |
|---|---|
| 50 | 0.0% |
| 100 | 7.5% |
| 200 | 27.5% |
| 500 | 52.5% |
| 1,000 | 62.5% |
Selected out-of-distribution screw-grasping trials. The top recording uses the full randomization recipe; the bottom removes the named family.
Forced mis-grasp followed by target relocation. Same initial layout.
An out-of-distribution table and torso configuration. Same initial layout.
An unseen HDRI environment. Same initial layout and background.
The background video is a qualitative comparison: the full model uses a 200,000-step checkpoint and the ablated model a 100,000-step checkpoint. It does not isolate randomization under a matched training budget.
The full source animations show pipeline milestones and how tool feedback leads to scene revisions.
Agent activity, elapsed time, and token use through the workflow. The animation includes accelerated time and labeled milestones.
The same task, tools, and collection/training budget. Tool feedback prompts the agent to revise the scene when the fixed pipeline stalls.