> ## Documentation Index
> Fetch the complete documentation index at: https://abstract-pbc.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Robot models

> Cameras, frames, native state, and action shapes for each served robot model.

Each model below has its own robot, cameras, frame counts, state fields and coordinate frame.
Build every request from the table of the model you call.
Every model answers on [Get action](https://abstract-pbc.mintlify.site/robots/get-action.md); [Robot episodes](https://abstract-pbc.mintlify.site/robots/chat-completions.md) shows the loop.

`GET /v1/environments/models` lists the robot models your account can call, including your own
fine-tunes. Use the IDs it returns; a model missing from that list is unavailable.

The YAM models were trained on two I2RT YAM arms, with one overhead camera and one camera on each
wrist, and take and answer end-effector poses. Xiaomi uses the native ARX X5 contract; FLUX uses
native Franka joints with DROID's camera layout. Both are API-only. `verne/v0.5` takes the YAM
models' poses and cameras but was not trained on the YAM; it is API-only too. `runway/praxis-1`
was trained on the YAM and takes the same poses, with its own frames and rate; it is API-only.

Rules for the YAM models below:

* Send frames as JPEG, camera-major: every frame of the first camera, then every frame of the next camera.
* Within one camera, send frames oldest first. The last frame is the latest.
* Send `robot_state` as the state at the time of the latest frame.
* Their packed `robot_state` has exactly two fields: `cartesian_position` and `gripper_position`. On the public API, send the six `left_`/`right_` pose and gripper fields in [`observation.state`](https://abstract-pbc.mintlify.site/robots/get-action). Joint angles are refused for these YAM models.
* Action row `k` is meant for time `t + k * dt`, where `t` is when you captured the latest frame.
* A model that is not stateful takes one frame per camera on every request.
* A stateful model keeps the episode on the server. It takes one frame per camera on the first request of a session, then more on every later request.

## Poses

A pose is 9 numbers: the grasp-site position (x, y, z, in metres), then a rotation6d.
The grasp site is the point between the closed fingertips.
A rotation6d is the first column of the rotation matrix, then the second column.
The two columns must be nonzero and not parallel. The server makes them orthonormal before use.

`cartesian_position` is 18 numbers: the left pose, then the right pose.
Each action row is 20 numbers: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
The action space is `ee_6d_position`.

Your robot turns each pose into its own joint angles with its own inverse kinematics.

Every YAM model below except `microsoft/rho` was trained in YAM joint space.
For these models, each pose is in that arm's own base frame: x forward, y left, z up.
The server solves each pose to YAM joint angles, runs the model, and turns its answer back into poses through the same YAM kinematics.
A pose that a YAM arm cannot reach returns `400 pose_unreachable`. Send a `session_id` on these models even where it is optional: the gateway then starts each solve from the session's previous one, so successive chunks stay on one arm configuration.
`microsoft/rho` takes and answers poses in its checkpoint's own frame, with no inverse kinematics on the server.

> [!WARNING]
>
> These checkpoints were trained with YAM arms in view of their wrist cameras and are untested on
> other arms or camera layouts. A well-formed chunk is not proof of task success or safe motion.
> Check every action against your robot's limits before you execute it.

## allenai/molmoact2

Ai2's MolmoAct2, trained on the bimanual YAM.
It reads one frame per camera and keeps no state between requests.

| Fact | Value |
| - | - |
| Model id | `allenai/molmoact2` |
| Cameras, in order | `top`, `left`, `right` |
| Image size | 640x360 JPEG. The server resizes other sizes to 640x360. |
| Frames per camera, first request | 1 |
| Frames per camera, later requests | 1 |
| `cartesian_position` | 18 numbers: the left grasp-site pose, then the right. Each arm's own base frame. |
| `gripper_position` | 2 numbers: left, then right. 0 is closed, 1 is open. |
| Action rows per chunk | 30 |
| Values per action row | 20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. |
| Action space | `ee_6d_position` |
| Control rate | 30 Hz (`dt` = 1/30 s) |
| Stateful | No |
| `session_id` | Optional |

```json robot_state theme={"theme":"css-variables"}
{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}
```

## microsoft/rho

Microsoft's Rho-YAM-Box.
It was trained on end-effector poses, so the server runs no inverse kinematics for it.

| Fact | Value |
| - | - |
| Model id | `microsoft/rho` |
| Cameras, in order | `scene`, `left_wrist`, `right_wrist` |
| Image size | Any size JPEG. The model keeps the aspect ratio and pads, so do not crop or stretch frames. |
| Frames per camera, first request | 1 |
| Frames per camera, later requests | 1 |
| `cartesian_position` | 18 numbers: the left pose, then the right. The checkpoint's frame. |
| `gripper_position` | 2 numbers: left, then right, in the checkpoint's native gripper units. These are not MolmoAct2's 0 to 1 units. |
| Action rows per chunk | 25 |
| Values per action row | 20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. |
| Action space | `ee_6d_position` |
| Control rate | 15 Hz (`dt` = 1/15 s) |
| Stateful | No |
| `session_id` | Optional |

Positions are in metres, in the checkpoint's coordinate frame, not in each arm's base frame.
The model computes 50 rows and returns the first 25.

```json robot_state theme={"theme":"css-variables"}
{
	"gripper_position": [0.0, 0.0],
	"cartesian_position": [
		0.3, 0.2, 0.3, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.3, -0.2, 0.3, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0
	]
}
```

## perceptron/isaac-0.5

Perceptron's Isaac 0.5, trained on the bimanual YAM.
It reads one frame per camera and keeps no state between requests.
It was trained on Ai2's MolmoAct2-BimanualYAM recordings, so it expects the same overhead and wrist views as MolmoAct2.

| Fact | Value |
| - | - |
| Model id | `perceptron/isaac-0.5` |
| Cameras, in order | `top`, `left`, `right` |
| Image size | 640x360 JPEG. The server resizes other sizes to 640x360. |
| Frames per camera, first request | 1 |
| Frames per camera, later requests | 1 |
| `cartesian_position` | 18 numbers: the left grasp-site pose, then the right. Each arm's own base frame. |
| `gripper_position` | 2 numbers: left, then right. 0 is closed, 1 is open. |
| Action rows per chunk | 30 |
| Values per action row | 20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. |
| Action space | `ee_6d_position` |
| Control rate | 30 Hz (`dt` = 1/30 s) |
| Stateful | No |
| `session_id` | Optional |

The model computes 50 rows and returns the first 30.
Its state scaling comes from one recorded session of that dataset.
Expect worse motion on a rig whose cameras or start pose differ from those recordings.

```json robot_state theme={"theme":"css-variables"}
{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}
```

## physical-intelligence/pi0.5

π0.5 fine-tuned on the bimanual YAM.
It reads one frame per camera and keeps no state between requests.
It was trained on Ai2's MolmoAct2-BimanualYAM recordings, so it expects the same overhead and wrist views as MolmoAct2.

| Fact | Value |
| - | - |
| Model id | `physical-intelligence/pi0.5` |
| Cameras, in order | `top`, `left`, `right` |
| Image size | 640x360 JPEG. The server resizes other sizes to 640x360. |
| Frames per camera, first request | 1 |
| Frames per camera, later requests | 1 |
| `cartesian_position` | 18 numbers: the left grasp-site pose, then the right. Each arm's own base frame. |
| `gripper_position` | 2 numbers: left, then right. 0 is closed, 1 is open. |
| Action rows per chunk | 30 |
| Values per action row | 20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. |
| Action space | `ee_6d_position` |
| Control rate | 30 Hz (`dt` = 1/30 s) |
| Stateful | No |
| `session_id` | Optional |

The weights are a community fine-tune of π0.5, not a release by Physical Intelligence.
They have not been evaluated on a robot.
Expect worse motion on a rig whose cameras or start pose differ from those recordings.

```json robot_state theme={"theme":"css-variables"}
{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}
```

## xiaomi/xiaomi-robotics-1

Xiaomi Robotics 1's [RoboDojo ARX X5 benchmark fine-tune](https://huggingface.co/datasets/RoboDojo-Benchmark/RoboDojo/tree/63fcc045f2e974902eb7800d29de48004cf86016/ckpt/RoboDojo/Xiaomi_Robotics_1/RoboDojo-sim-arx_x5-ee-0),
served under Apache-2.0. This is a benchmark-specific checkpoint for two ARX X5 arms.
Abstract has tested inference and the native request format, not task success or operation on
a physical robot. It is callable when `GET /v1/environments/models` lists it.

| Fact | Value |
| - | - |
| Model id | `xiaomi/xiaomi-robotics-1` |
| Robot | Two ARX X5 arms, six measured joints per arm |
| Cameras | `overhead`, `wrist_l`, `wrist_r` on the public API; mapped to `ego`, `left_wrist`, `right_wrist` |
| Frames per camera | One current JPEG on every call |
| Image processing | Aspect ratio at most 200; resized on a 32-pixel grid with at most 160,000 pixels per view |
| Pose state | `left_eef_xyz` \[3], `left_eef_rot6` \[6], `right_eef_xyz` \[3], `right_eef_rot6` \[6] |
| Joint state | `left_joint_position` \[6] and `right_joint_position` \[6], measured radians |
| Gripper state | `left_grip` \[1] and `right_grip` \[1], normalized 0 closed to 1 open |
| Pose frame | RoboDojo/world frame for the observed grasp-site poses and returned targets |
| Action rows | 30 absolute end-effector targets per arm, ten values per row |
| Action space | `ee_6d_position` |
| Control rate | 25 Hz (`dt` = 0.04 s) |
| Stateful | No; `session_id` is optional |
| Input token limit | 8,192 processed tokens, including images; public 512-character prompts fit |

Use the [native request fields](https://abstract-pbc.mintlify.site/robots/get-action.md#xiaomi-arx-x5); joints condition the model
and the observed poses restore its relative predictions. The output keeps the world frame,
with normalized grippers bounded to \[0, 1]. No YAM inverse kinematics or frame conversion is
applied. The model is API-only, and it is not supported by the physical YAM CLI.

## black-forest-labs/flux-3-action

Black Forest Labs' [FLUX 3 Action DROID checkpoint](https://huggingface.co/black-forest-labs/flux-3-action-droid/tree/d84c80adc0721780245aabb75e45d4164ead7b97),
for a Franka Panda with a Robotiq gripper. Its weights use the
[FLUX Kommunity 1.0 license](https://huggingface.co/black-forest-labs/flux-3-action-droid/blob/d84c80adc0721780245aabb75e45d4164ead7b97/LICENSE.md).
Abstract has tested native inference with synthetic observations, not task success or
operation on a physical robot. It is callable when `GET /v1/environments/models` lists it.

| Fact | Value |
| - | - |
| Model id | `black-forest-labs/flux-3-action` |
| Robot | Franka Panda with a Robotiq gripper, DROID embodiment |
| Cameras, in order | `exterior_image_0_left`, `exterior_image_1_left`, `wrist_image_left` |
| Image size | 640x360 JPEG |
| Frames per camera | One current frame on every call |
| Joint state | `joint_position`: seven measured joint angles in radians |
| Gripper state | `gripper_position`: one closed fraction, 0 open to 1 closed |
| Action rows | 32 absolute joint targets, eight values per row: seven joint radians, then gripper closed fraction |
| Action space | `joint_position`, returned in `actions.joint_chunk` |
| Control rate | 15 Hz (`dt` = 1/15 s) |
| Stateful | No; `session_id` is optional |

The state contains exactly those two fields, captured with the three camera frames. No
end-effector fields or inverse kinematics are involved. Input grippers must be in \[0, 1];
finite output gripper predictions are not clamped to that interval.

Use the [DROID request example](https://abstract-pbc.mintlify.site/robots/get-action.md#flux-droid). Each call resets the policy's
inference state and reports zero cached tokens; [usage](https://abstract-pbc.mintlify.site/robots/usage.md#flux-estimate) includes
the actual padded positive and negative text contexts. This model is API-only. It is not
offered in the physical YAM CLI.

FLUX applies prohibited-use prompt filters and allows only numeric native actions with
supported metadata in its replies. It refuses an entire chunk if any joint target exceeds
the published Panda/FER position bounds, without clamping values. These filters are mandatory. They do not assess scene
hazards, determine copyright or legal status, or enforce physical motion limits. Your
controller must enforce joint velocity, acceleration, force and workspace limits and provide a stop
mechanism. See [FLUX content filters](https://abstract-pbc.mintlify.site/robots/get-action.md#flux-content-filters).

## verne/v0.5

[Verne V0.5](https://github.com/vernerobotics/v0.5)'s pretrained base checkpoint, a world action
model. It keeps a memory of the episode on the server. Built with Verne V0.5.

> [!WARNING]
>
> This checkpoint is not tuned to any robot or task, and [Verne's model
> card](https://huggingface.co/vernerobotics/v0.5-base) lists deployment without fine-tuning as out
> of scope. It has not been evaluated on a robot. Its published LIBERO and RoboTwin scores belong to
> other checkpoints.

| Fact | Value |
| - | - |
| Model id | `verne/v0.5` |
| Cameras, in order | `top`, `left`, `right` |
| Image size | Any size. The model stretches each view to 256x256, so send the full field of view. |
| Frames per camera, first request | 1 |
| Frames per camera, later requests | 4 |
| `cartesian_position` | 18 numbers: the left grasp-site pose, then the right. Each arm's own base frame. |
| `gripper_position` | 2 numbers: left, then right. 0 is closed, 1 is open. |
| Action rows per chunk | 18 |
| Values per action row | 20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. |
| Action space | `ee_6d_position` |
| Control rate | 30 Hz (`dt` = 1/30 s) |
| Stateful | Yes |
| `session_id` | Required |

```json robot_state theme={"theme":"css-variables"}
{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}
```

Run the chunk, holding the arms still while the next one is planned, then send the next request.
The memory counts time from the rows it served: four frames evenly across those 18 rows, the newest
now. It reads the two arms relative to each other as if their bases stand 0.48 m apart facing the
same way. The model's progress estimate and its base targets are not returned. This model is
API-only. It is not offered in the physical YAM CLI.

Your use of this model must follow Sections 4 to 6 of the [Verne V0.5 Model License](https://github.com/vernerobotics/v0.5/blob/91ed7ed6a6c3635469809a34ba24be12933953be/LICENSE):
export and sanctions limits, banned uses, and physical safety.

## runway/praxis-1

Runway's [Praxis-1](https://github.com/runwayml/praxis-1-oss), a video-action policy, in the
[checkpoint](https://huggingface.co/runwayml/praxis-1-yam) Runway fine-tuned for the bimanual YAM.
On each call it predicts the next stretch of video and the next actions together, from the frames
your robot saw while it ran the last chunk.

> [!WARNING]
>
> Abstract has not evaluated this checkpoint on a robot. It was trained on one YAM rig, with its
> cameras and its arms 0.61 m apart.

| Fact | Value |
| - | - |
| Model id | `runway/praxis-1` |
| Cameras, in order | `top`, `left`, `right` |
| Image size | Any size from 416x260 up. 16:10 is used whole; another shape is cut about its centre. |
| Frames per camera, first request | 1 |
| Frames per camera, later requests | 21 |
| `cartesian_position` | 18 numbers: the left grasp-site pose, then the right. Each arm's own base frame. |
| `gripper_position` | 2 numbers: left, then right. 0 is closed, 1 is open. |
| Action rows per chunk | 21 |
| Values per action row | 20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. |
| Action space | `ee_6d_position` |
| Control rate | 15 Hz (`dt` = 1/15 s) |
| Stateful | Yes |
| `session_id` | Required |

```json robot_state theme={"theme":"css-variables"}
{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}
```

Run all 21 rows, capturing one frame per camera at the end of each row, then send those 21 frames
with the next request: 1/15 s apart, the last one now. Hold the arms still while the next chunk is
planned, and do not count that pause in the frames. A frame narrower than 416 pixels or shorter
than 260 after the cut returns `400 invalid_observation`; frames near 416x260 keep requests small.

Write the task as the checkpoint's training captions were written: "Pick up the pen from the
center of the table, then place the pen in the bag", with no closing period. An instruction that
encodes to more than 512 text tokens is refused.

The model reads the two arms as if their bases stand 0.61 m apart, as on the rig it was trained
on. Its predicted video is not returned. This model is API-only. It is not offered in the
physical YAM CLI.

## What the server checks

| Check | Error code |
| - | - |
| Frame count per camera does not match the table | `image_count_mismatch` |
| A state array has the wrong length, or a value is not a finite number | `state_shape_mismatch` |
| The state carries a field outside the selected model's declared contract | `state_shape_mismatch` |
| A gripper value is outside 0 to 1 on `allenai/molmoact2`, `perceptron/isaac-0.5` or `physical-intelligence/pi0.5` | `invalid_image` |
| An input gripper value is outside 0 to 1 on `xiaomi/xiaomi-robotics-1` or `black-forest-labs/flux-3-action` | `state_shape_mismatch` |
| A rotation6d is degenerate | `invalid_image` |
| A pose is out of a YAM arm's reach, on the YAM models above except `microsoft/rho` | `pose_unreachable` |
| A frame is not a decodable JPEG, or has an edge over 4096 pixels | `invalid_image` |
| A FLUX instruction matches the prohibited-use prompt filter | `flux_content_blocked` |

`invalid_image` covers any frame or state value the model cannot use.
Its message names the problem.
See [Errors and limits](https://abstract-pbc.mintlify.site/errors.md) for every code.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.