Robot models

Each model below has its own robot, cameras, frame counts, state fields and coordinate frame. Build every request from the table of the model you call. Every model answers on Get action; Robot episodes shows the loop. GET /v1/environments/models lists the robot models your account can call, including your own fine-tunes. Use the IDs it returns; a model missing from that list is unavailable. The YAM models were trained on two I2RT YAM arms, with one overhead camera and one camera on each wrist, and take and answer end-effector poses. Xiaomi uses the native ARX X5 contract; FLUX uses native Franka joints with DROID’s camera layout. Both are API-only. verne/v0.5 takes the YAM models’ poses and cameras but was not trained on the YAM; it is API-only too. runway/praxis-1 was trained on the YAM and takes the same poses, with its own frames and rate; it is API-only. Rules for the YAM models below:

Send frames as JPEG, camera-major: every frame of the first camera, then every frame of the next camera.
Within one camera, send frames oldest first. The last frame is the latest.
Send robot_state as the state at the time of the latest frame.
Their packed robot_state has exactly two fields: cartesian_position and gripper_position. On the public API, send the six left_/right_ pose and gripper fields in observation.state. Joint angles are refused for these YAM models.
Action row k is meant for time t + k * dt, where t is when you captured the latest frame.
A model that is not stateful takes one frame per camera on every request.
A stateful model keeps the episode on the server. It takes one frame per camera on the first request of a session, then more on every later request.

Poses

A pose is 9 numbers: the grasp-site position (x, y, z, in metres), then a rotation6d. The grasp site is the point between the closed fingertips. A rotation6d is the first column of the rotation matrix, then the second column. The two columns must be nonzero and not parallel. The server makes them orthonormal before use. cartesian_position is 18 numbers: the left pose, then the right pose. Each action row is 20 numbers: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper. The action space is ee_6d_position. Your robot turns each pose into its own joint angles with its own inverse kinematics. Every YAM model below except microsoft/rho was trained in YAM joint space. For these models, each pose is in that arm’s own base frame: x forward, y left, z up. The server solves each pose to YAM joint angles, runs the model, and turns its answer back into poses through the same YAM kinematics. A pose that a YAM arm cannot reach returns 400 pose_unreachable. Send a session_id on these models even where it is optional: the gateway then starts each solve from the session’s previous one, so successive chunks stay on one arm configuration. microsoft/rho takes and answers poses in its checkpoint’s own frame, with no inverse kinematics on the server.

These checkpoints were trained with YAM arms in view of their wrist cameras and are untested on other arms or camera layouts. A well-formed chunk is not proof of task success or safe motion. Check every action against your robot’s limits before you execute it.

allenai/molmoact2

Ai2’s MolmoAct2, trained on the bimanual YAM. It reads one frame per camera and keeps no state between requests.

Fact	Value
Model id	`allenai/molmoact2`
Cameras, in order	`top`, `left`, `right`
Image size	640x360 JPEG. The server resizes other sizes to 640x360.
Frames per camera, first request	1
Frames per camera, later requests	1
`cartesian_position`	18 numbers: the left grasp-site pose, then the right. Each arm’s own base frame.
`gripper_position`	2 numbers: left, then right. 0 is closed, 1 is open.
Action rows per chunk	30
Values per action row	20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
Action space	`ee_6d_position`
Control rate	30 Hz (`dt` = 1/30 s)
Stateful	No
`session_id`	Optional

robot_state

{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}

microsoft/rho

Microsoft’s Rho-YAM-Box. It was trained on end-effector poses, so the server runs no inverse kinematics for it.

Fact	Value
Model id	`microsoft/rho`
Cameras, in order	`scene`, `left_wrist`, `right_wrist`
Image size	Any size JPEG. The model keeps the aspect ratio and pads, so do not crop or stretch frames.
Frames per camera, first request	1
Frames per camera, later requests	1
`cartesian_position`	18 numbers: the left pose, then the right. The checkpoint’s frame.
`gripper_position`	2 numbers: left, then right, in the checkpoint’s native gripper units. These are not MolmoAct2’s 0 to 1 units.
Action rows per chunk	25
Values per action row	20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
Action space	`ee_6d_position`
Control rate	15 Hz (`dt` = 1/15 s)
Stateful	No
`session_id`	Optional

Positions are in metres, in the checkpoint’s coordinate frame, not in each arm’s base frame. The model computes 50 rows and returns the first 25.

robot_state

{
	"gripper_position": [0.0, 0.0],
	"cartesian_position": [
		0.3, 0.2, 0.3, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.3, -0.2, 0.3, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0
	]
}

perceptron/isaac-0.5

Perceptron’s Isaac 0.5, trained on the bimanual YAM. It reads one frame per camera and keeps no state between requests. It was trained on Ai2’s MolmoAct2-BimanualYAM recordings, so it expects the same overhead and wrist views as MolmoAct2.

Fact	Value
Model id	`perceptron/isaac-0.5`
Cameras, in order	`top`, `left`, `right`
Image size	640x360 JPEG. The server resizes other sizes to 640x360.
Frames per camera, first request	1
Frames per camera, later requests	1
`cartesian_position`	18 numbers: the left grasp-site pose, then the right. Each arm’s own base frame.
`gripper_position`	2 numbers: left, then right. 0 is closed, 1 is open.
Action rows per chunk	30
Values per action row	20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
Action space	`ee_6d_position`
Control rate	30 Hz (`dt` = 1/30 s)
Stateful	No
`session_id`	Optional

The model computes 50 rows and returns the first 30. Its state scaling comes from one recorded session of that dataset. Expect worse motion on a rig whose cameras or start pose differ from those recordings.

robot_state

{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}

physical-intelligence/pi0.5

π0.5 fine-tuned on the bimanual YAM. It reads one frame per camera and keeps no state between requests. It was trained on Ai2’s MolmoAct2-BimanualYAM recordings, so it expects the same overhead and wrist views as MolmoAct2.

Fact	Value
Model id	`physical-intelligence/pi0.5`
Cameras, in order	`top`, `left`, `right`
Image size	640x360 JPEG. The server resizes other sizes to 640x360.
Frames per camera, first request	1
Frames per camera, later requests	1
`cartesian_position`	18 numbers: the left grasp-site pose, then the right. Each arm’s own base frame.
`gripper_position`	2 numbers: left, then right. 0 is closed, 1 is open.
Action rows per chunk	30
Values per action row	20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
Action space	`ee_6d_position`
Control rate	30 Hz (`dt` = 1/30 s)
Stateful	No
`session_id`	Optional

The weights are a community fine-tune of π0.5, not a release by Physical Intelligence. They have not been evaluated on a robot. Expect worse motion on a rig whose cameras or start pose differ from those recordings.

robot_state

{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}

xiaomi/xiaomi-robotics-1

Xiaomi Robotics 1’s RoboDojo ARX X5 benchmark fine-tune, served under Apache-2.0. This is a benchmark-specific checkpoint for two ARX X5 arms. Abstract has tested inference and the native request format, not task success or operation on a physical robot. It is callable when GET /v1/environments/models lists it.

Fact	Value
Model id	`xiaomi/xiaomi-robotics-1`
Robot	Two ARX X5 arms, six measured joints per arm
Cameras	`overhead`, `wrist_l`, `wrist_r` on the public API; mapped to `ego`, `left_wrist`, `right_wrist`
Frames per camera	One current JPEG on every call
Image processing	Aspect ratio at most 200; resized on a 32-pixel grid with at most 160,000 pixels per view
Pose state	`left_eef_xyz` [3], `left_eef_rot6` [6], `right_eef_xyz` [3], `right_eef_rot6` [6]
Joint state	`left_joint_position` [6] and `right_joint_position` [6], measured radians
Gripper state	`left_grip` [1] and `right_grip` [1], normalized 0 closed to 1 open
Pose frame	RoboDojo/world frame for the observed grasp-site poses and returned targets
Action rows	30 absolute end-effector targets per arm, ten values per row
Action space	`ee_6d_position`
Control rate	25 Hz (`dt` = 0.04 s)
Stateful	No; `session_id` is optional
Input token limit	8,192 processed tokens, including images; public 512-character prompts fit

Use the native request fields; joints condition the model and the observed poses restore its relative predictions. The output keeps the world frame, with normalized grippers bounded to [0, 1]. No YAM inverse kinematics or frame conversion is applied. The model is API-only, and it is not supported by the physical YAM CLI.

black-forest-labs/flux-3-action

Black Forest Labs’ FLUX 3 Action DROID checkpoint, for a Franka Panda with a Robotiq gripper. Its weights use the FLUX Kommunity 1.0 license. Abstract has tested native inference with synthetic observations, not task success or operation on a physical robot. It is callable when GET /v1/environments/models lists it.

Fact	Value
Model id	`black-forest-labs/flux-3-action`
Robot	Franka Panda with a Robotiq gripper, DROID embodiment
Cameras, in order	`exterior_image_0_left`, `exterior_image_1_left`, `wrist_image_left`
Image size	640x360 JPEG
Frames per camera	One current frame on every call
Joint state	`joint_position`: seven measured joint angles in radians
Gripper state	`gripper_position`: one closed fraction, 0 open to 1 closed
Action rows	32 absolute joint targets, eight values per row: seven joint radians, then gripper closed fraction
Action space	`joint_position`, returned in `actions.joint_chunk`
Control rate	15 Hz (`dt` = 1/15 s)
Stateful	No; `session_id` is optional

The state contains exactly those two fields, captured with the three camera frames. No end-effector fields or inverse kinematics are involved. Input grippers must be in [0, 1]; finite output gripper predictions are not clamped to that interval. Use the DROID request example. Each call resets the policy’s inference state and reports zero cached tokens; usage includes the actual padded positive and negative text contexts. This model is API-only. It is not offered in the physical YAM CLI. FLUX applies prohibited-use prompt filters and allows only numeric native actions with supported metadata in its replies. It refuses an entire chunk if any joint target exceeds the published Panda/FER position bounds, without clamping values. These filters are mandatory. They do not assess scene hazards, determine copyright or legal status, or enforce physical motion limits. Your controller must enforce joint velocity, acceleration, force and workspace limits and provide a stop mechanism. See FLUX content filters.

verne/v0.5

Verne V0.5’s pretrained base checkpoint, a world action model. It keeps a memory of the episode on the server. Built with Verne V0.5.

This checkpoint is not tuned to any robot or task, and Verne’s model card lists deployment without fine-tuning as out of scope. It has not been evaluated on a robot. Its published LIBERO and RoboTwin scores belong to other checkpoints.

Fact	Value
Model id	`verne/v0.5`
Cameras, in order	`top`, `left`, `right`
Image size	Any size. The model stretches each view to 256x256, so send the full field of view.
Frames per camera, first request	1
Frames per camera, later requests	4
`cartesian_position`	18 numbers: the left grasp-site pose, then the right. Each arm’s own base frame.
`gripper_position`	2 numbers: left, then right. 0 is closed, 1 is open.
Action rows per chunk	18
Values per action row	20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
Action space	`ee_6d_position`
Control rate	30 Hz (`dt` = 1/30 s)
Stateful	Yes
`session_id`	Required

robot_state

{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}

Run the chunk, holding the arms still while the next one is planned, then send the next request. The memory counts time from the rows it served: four frames evenly across those 18 rows, the newest now. It reads the two arms relative to each other as if their bases stand 0.48 m apart facing the same way. The model’s progress estimate and its base targets are not returned. This model is API-only. It is not offered in the physical YAM CLI. Your use of this model must follow Sections 4 to 6 of the Verne V0.5 Model License: export and sanctions limits, banned uses, and physical safety.

runway/praxis-1

Runway’s Praxis-1, a video-action policy, in the checkpoint Runway fine-tuned for the bimanual YAM. On each call it predicts the next stretch of video and the next actions together, from the frames your robot saw while it ran the last chunk.

Abstract has not evaluated this checkpoint on a robot. It was trained on one YAM rig, with its cameras and its arms 0.61 m apart.

Fact	Value
Model id	`runway/praxis-1`
Cameras, in order	`top`, `left`, `right`
Image size	Any size from 416x260 up. 16:10 is used whole; another shape is cut about its centre.
Frames per camera, first request	1
Frames per camera, later requests	21
`cartesian_position`	18 numbers: the left grasp-site pose, then the right. Each arm’s own base frame.
`gripper_position`	2 numbers: left, then right. 0 is closed, 1 is open.
Action rows per chunk	21
Values per action row	20: left xyz, left rotation6d, left gripper, then right xyz, right rotation6d, right gripper.
Action space	`ee_6d_position`
Control rate	15 Hz (`dt` = 1/15 s)
Stateful	Yes
`session_id`	Required

robot_state

{
	"cartesian_position": [
		0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0, 0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0
	],
	"gripper_position": [1.0, 0.5]
}

Run all 21 rows, capturing one frame per camera at the end of each row, then send those 21 frames with the next request: 1/15 s apart, the last one now. Hold the arms still while the next chunk is planned, and do not count that pause in the frames. A frame narrower than 416 pixels or shorter than 260 after the cut returns 400 invalid_observation; frames near 416x260 keep requests small. Write the task as the checkpoint’s training captions were written: “Pick up the pen from the center of the table, then place the pen in the bag”, with no closing period. An instruction that encodes to more than 512 text tokens is refused. The model reads the two arms as if their bases stand 0.61 m apart, as on the rig it was trained on. Its predicted video is not returned. This model is API-only. It is not offered in the physical YAM CLI.

What the server checks

Check	Error code
Frame count per camera does not match the table	`image_count_mismatch`
A state array has the wrong length, or a value is not a finite number	`state_shape_mismatch`
The state carries a field outside the selected model’s declared contract	`state_shape_mismatch`
A gripper value is outside 0 to 1 on `allenai/molmoact2`, `perceptron/isaac-0.5` or `physical-intelligence/pi0.5`	`invalid_image`
An input gripper value is outside 0 to 1 on `xiaomi/xiaomi-robotics-1` or `black-forest-labs/flux-3-action`	`state_shape_mismatch`
A rotation6d is degenerate	`invalid_image`
A pose is out of a YAM arm’s reach, on the YAM models above except `microsoft/rho`	`pose_unreachable`
A frame is not a decodable JPEG, or has an edge over 4096 pixels	`invalid_image`
A FLUX instruction matches the prohibited-use prompt filter	`flux_content_blocked`

invalid_image covers any frame or state value the model cannot use. Its message names the problem. See Errors and limits for every code.

Get started

LLM models

Reference

Robot models

Poses

allenai/molmoact2

microsoft/rho

perceptron/isaac-0.5

physical-intelligence/pi0.5

xiaomi/xiaomi-robotics-1

black-forest-labs/flux-3-action

verne/v0.5

runway/praxis-1

What the server checks