Robot episodes

Robots use POST /v1/get_action, not chat completions. An episode is a loop:

Send one frame per camera, the task, and the state fields the selected model requires.
Check the returned action space and execute its rows at dt seconds per row, capturing frames as you go.
Send those frames and the latest state with the same session_id and task.
Call POST /v1/end_session when done.

After the first call, send as many frames per camera as the last answer’s next_frames_per_camera, oldest first. An idle session expires after 60 seconds. 410 session_expired means start a new episode with one frame per camera.

Example

This example uses MolmoAct2’s YAM end-effector contract. Replace the camera, state and execution functions with your drivers. The example only reads files; it moves nothing.

episode.py

import base64
import os
import uuid

import requests

API = "https://api.abstract.inc/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['ABSTRACT_API_KEY']}"}
MODEL = "allenai/molmoact2"
CAMERAS = {"overhead": "top", "wrist_l": "left", "wrist_r": "right"}
POSE = [0.3, 0.0, 0.2, -0.5, 0.0, -0.866, 0.0, 1.0, 0.0]


def capture(count):
    return {role: [base64.b64encode(open(f"{camera}.jpg", "rb").read()).decode()
                   for _ in range(count)] for role, camera in CAMERAS.items()}


def read_state():
    return {f"{side}_{field}": values for side in ("left", "right")
            for field, values in (("eef_xyz", POSE[:3]), ("eef_rot6", POSE[3:]), ("grip", [1.0]))}


def execute(actions, dt):
    pass  # Your controller checks and solves the poses, then plays one row every dt seconds.


session_id, frames = str(uuid.uuid4()), capture(1)
try:
    for step in range(3):
        response = requests.post(f"{API}/get_action", headers=HEADERS, json={
            "model": MODEL, "session_id": session_id,
            "observation_id": f"{session_id}.{step}", "expected_step": step,
            "observation": {"prompt": "fold the towel", "state": read_state(), "video": frames},
        }, timeout=300)
        if not response.ok:
            raise SystemExit(f"HTTP {response.status_code}: {response.text}")
        chunk = response.json()
        actions = [left + right for left, right in zip(
            chunk["actions"]["left_eef_chunk"], chunk["actions"]["right_eef_chunk"], strict=True)]
        execute(actions, chunk["dt"])
        frames = capture(chunk["next_frames_per_camera"])  # In production, capture during execution.
finally:
    requests.post(f"{API}/end_session", headers=HEADERS,
                  json={"model": MODEL, "session_id": session_id}, timeout=30)

Retrying an observation

Choose a session_id before the first call. Send a unique observation_id and the next expected_step (starting at zero) with each observation. Send only one call at a time. If an answer is lost, retry the identical body under the same id. A replay does not run or bill another chunk. While it runs, 409 observation_in_progress carries Retry-After: wait, then retry the same id and body. observation_lost means that id will never run again: do not execute a missing chunk, and send the next observation under a new id. observation_outcome_unknown, from a legacy gateway, means retry only that id. Never send a new observation until its outcome is known. Idempotency-Key can replace observation_id; if both are sent they must match. The example stops on errors rather than retrying. See Errors for retry advice.

Get started

Robot models

LLM models

Reference

Example

Retrying an observation