Playground API

Everything the browser editor does is available over HTTP with an API key — this is what the Unity client uses. Base URL: https://arx.dev/api/v1/playground. Send Authorization: Bearer <key> (get a key). Full schema: OpenAPI (tag Playground).

Scenes

KEY=...; API=https://arx.dev/api/v1/playground

curl -s -H "Authorization: Bearer $KEY" $API/scenes                      # list
curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     -d '{"name":"Forest"}' $API/scenes                                  # create → whole scene
curl -s -H "Authorization: Bearer $KEY" $API/scenes/SCENE_ID             # whole scene

A scene:

{
  "id": "ldwxk0g5", "name": "Forest",
  "objects": [
    { "id": "o_ab12cd34", "type": "asset", "asset_id": "3ma8qmgw", "name": "SM_Env_Tree_01",
      "position": [3.2, 0, -4.1], "rotation": [0, 45, 0], "scale": [1.1, 1.1, 1.1],
      "size_m": { "x": 2.3, "y": 4.69, "z": 2.26 }, "group": "Nature/Trees", "static": true, "grounded": true },
    { "id": "o_ef56gh78", "type": "primitive", "shape": "cylinder", "size": [6, 0.1, 6],
      "color": "#2c6e9b", "position": [0, 0, 0], "rotation": [0, 0, 0], "scale": [1, 1, 1], "material": "m_x1y2z3w4" },
    { "id": "o_ij90kl12", "type": "light", "color": "#ffb15a", "intensity": 4, "range": 12, "position": [2, 1.2, 1], "rotation": [0, 0, 0], "scale": [1, 1, 1] },
    { "id": "o_mn34op56", "type": "camera", "name": "Overlook", "fov": 55, "position": [22, 14, 22], "rotation": [-25, 45, 0], "scale": [1, 1, 1] }
  ],
  "environment": { "preset": "night", "ground": "grass", "ground_size": 120, "fog": 0.15, "ground_material": "m_x1y2z3w4" },
  "materials": { "m_x1y2z3w4": { "id": "m_x1y2z3w4", "name": "Pond", "color": "#2c6e9b", "roughness": 0.1, "opacity": 0.85,
                                 "pattern": "none", "tile_size": 2, "tint": false } },
  "settings": { "asset_lock": "@synty", "groups": { "Nature": { "locked": true } } },
  "terrain": { "enabled": true, "style": "hills", "seed": 7, "height": 14, "scale": 90, "roughness": 0.5, "water_level": -2, "flat_radius": 0 },
  "terrain_chunks": [ { "cx": 0, "cz": 0, "h": "<base64 raw-deflate int16 cm>", "p": "<base64 raw-deflate uint8 or null>" } ],
  "assets": { "3ma8qmgw": { "...": "render info, see below" } },
  "turns": [ { "id": "k2j3h4g5", "goal": "build me a forest", "status": "done", "steps": [], "reply": "Forest program complete…" } ]
}

Conventions

Metres; y up; the ground is y = 0; the default camera looks at the origin from +z.
position is the object’s base (its bottom), for every type — and for assets, the centre of their footprint: clients anchor an asset at the bottom centre of its bounds, not at the model’s pivot, so size_m × scale is exactly the box it fills around position.
rotation is degrees, Euler XYZ. Cameras store [pitch, yaw, 0], applied yaw first, looking down −z.
group is a folder path (Nature/Trees); static marks scenery (clients may instance it and skip it for clicks); grounded means y follows the terrain (set it when placing on the ground).
interaction (assets and primitives) makes a door: { kind: hinge | slide | lift, part, hinge: left | right, angle, direction: left | right, open, auto }. part names the prefab node(s) that move (comma-separated; none = the whole object). Open / close it with an update whose changes are { interaction: { open: true } } (partial hashes merge); interaction: null removes it. Interactive objects are never static.
Coordinates are three.js (right-handed). For Unity (left-handed): position (-x, y, z); rotation: build the quaternion, then use (qx, -qy, -qz, qw); cameras additionally turn 180° about y (a three.js camera looks down −z, a Unity camera down +z). The Unity client does this for you.

Changing a scene: ops

POST /scenes/:id/ops with { "ops": [...], "origin": "<your client id>" }. The response has the applied ops (ids and defaults filled in) and render info for any asset you added.

op	fields
`add`	`object` (as above; `id` optional)
`update`	`id`, `changes` — any of position, rotation, scale, name, color, size, intensity, range, fov, emissive, material (null = original look), group (null = no folder), static, grounded
`remove`	`ids`
`env`	`environment` — preset (holodeck, day, sunset, night, overcast, space), ground (grid, grass, sand, snow, dirt, water, stone, asphalt, none), ground_size, fog 0–1, sky_color, ground_color, ground_material
`clear`	— removes every object
`material`	`material` — upsert: name, color, roughness, metalness, emissive, emissive_intensity, opacity, pattern, pattern_color, tile_size, tint
`material_remove`	`id`
`settings`	`settings` — `asset_lock`: `@scope`, `@scope/Pack` or null
`group`	`path`, `changes` — any of hidden, locked, static, rename (new path), offset `[x,y,z]`, delete — on the folder and its subfolders; comes back as plain updates / removes / settings
`terrain`	`terrain` — enabled, style (hills, mountains, plains, islands, dunes, flat), seed, height, scale, roughness, water_level (null = none), flat_radius, layers (`{ "path": { "color": "#…", "material": "m_…" } }`, null resets a layer)
`terrain_brush`	`mode` (raise, lower, flatten, smooth, noise, terrace, paint), `x`, `z`, `radius` (≤ 80 m), `strength` (metres for raise/lower; 0–1 otherwise), `height` (flatten target / terrace step), `hardness` 0–1, `layer` (paint) — comes back as `terrain_chunks`
`terrain_path`	`points` `[{x,z}…]`, `width`, `mode` (flatten = road, lower = river, raise = ridge), `depth`, `layer` (painted down the middle), `strength`
`terrain_paint_rule`	`layer`, `area` (circle or rectangle), `min_height`, `max_height`, `min_slope`, `max_slope`
`terrain_undo`	— puts back what the last terrain edit changed (repeatable, last 30); chunks that hadn’t existed come back as `{ cx, cz, deleted: true }`
`terrain_reset`	— drops every sculpt and paint edit
`camera`	`ids` / `target` / `position` — moves open editors’ view; not stored

curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' $API/scenes/SCENE_ID/ops -d '{
  "origin": "my-script",
  "ops": [
    { "op": "env", "environment": { "preset": "sunset", "ground": "sand" } },
    { "op": "add", "object": { "type": "asset", "asset_id": "3ma8qmgw", "position": [0, 0, -5] } }
  ]}'

Terrain

Terrain is endless and procedural: height at any point = a seeded base + sculpt edits. Clients compute the base themselves from terrain, so only edits travel — as 64 m chunks in terrain_chunks (and in terrain_chunks ops: { "op": "terrain_chunks", "chunks": [...], "reset": true? }):

h: 65 × 65 int16 little-endian centimetre deltas, row-major by z (row j = z, column i = x; sample (i, j) is at x = cx·64 + i, z = cz·64 + j), raw-deflated, base64.
p: 65 × 65 uint8 paint layers (auto grass dirt sand rock snow path moss), same encoding, or null.

The base is value-noise fBm with a 32-bit integer hash — specified in app/services/playground/terrain.rb, ported to app/javascript/playground/terrain_math.js and the Unity package’s TerrainMath.cs, all checked against one fixture. Port that if you render terrain yourself.

Building with the agent

Hand the server agent a goal; it plans and calls tools until it’s done.

curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     -d '{"goal":"build a desert oasis at sunset"}' $API/scenes/SCENE_ID/command
# → 202 { "turn": { "id": "k2j3h4g5", "status": "queued", ... } }

curl -s -H "Authorization: Bearer $KEY" $API/scenes/SCENE_ID/turns/k2j3h4g5   # poll
curl -s -X POST -H "Authorization: Bearer $KEY" $API/scenes/SCENE_ID/cancel     # halt

A turn’s status goes queued → running → done | failed | cancelled; steps grows as it works ({ tool, args, result, ok, at }) and reply is its closing line. One goal at a time per scene (409 otherwise).

What it cost — per turn and for the scene (steps, model calls, tokens, USD):

curl -s -H "Authorization: Bearer $KEY" $API/scenes/SCENE_ID/usage
# → { "turns": [ { "id", "goal", "status", "steps", "tools": { "place_asset": 12, … }, "llm_calls",
#                  "input_tokens", "output_tokens", "cost_usd", "seconds" } ], "totals": { … "total_cost_usd" } }

Building with your own loop: tools

Run your own LLM (or plain code) and call the same tools the agent uses:

curl -s -H "Authorization: Bearer $KEY" $API/tools          # { tools: [OpenAI function schemas], system_prompt }

curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     $API/scenes/SCENE_ID/tools/search_assets -d '{"query":"pine tree","limit":5}'

curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     $API/scenes/SCENE_ID/tools/scatter_assets -d '{
       "asset_ids": ["3ma8qmgw"], "count": 40, "group": "forest",
       "area": { "center": { "x": 0, "z": 0 }, "radius": 30, "inner_radius": 6 } }'

The tool schemas plug straight into OpenAI-style tools: (Grok, OpenRouter, OpenAI) — or Anthropic’s input_schema (use function.parameters). Results are JSON with a human summary; failures are 422 { "error": … }. Tool calls made this way show up in the browser console as “Tool calls via the API”. See the tool reference.

Assets for placement

curl -s -H "Authorization: Bearer $KEY" "$API/assets?q=pine+tree&lock=@synty&limit=8"
curl -s -H "Authorization: Bearer $KEY" $API/assets/3ma8qmgw
curl -s -H "Authorization: Bearer $KEY" $API/vendors

Search returns prefabs first, then models with no prefab, each with asset_id, kind, name, path, unity_guid, pack_id, size_m, thumbnail_url. Render info (assets/:id, and a scene’s assets map) tells a renderer how to draw it:

kind	fields
`model`	`glb_url` (untextured unless self-hosted), `model_url` (FBX/OBJ), `format`, `unit_scale` (metres per file unit), `textures` (atlas URL per material slot — apply to every mesh material with `extras.slot` = index)
`prefab`	`prefab_tree` (nodes with three.js-space transforms) + `parts` `{ files: guid → {url, format, unit_scale}, materials: guid → {texture_url} }` — assemble like `app/javascript/playground/prefab.js`; in metres

Both carry unity_guid and path, so a Unity project that already contains the pack can place the real prefab directly (AssetDatabase.GUIDToAssetPath), and anything missing can be fetched as a Unity package from GET /api/v1/assets/:id/package.

Live updates (WebSocket)

Scenes stream changes over ActionCable. Connect to wss://<host>/cable?token=<key> (send an Origin header equal to the server’s base URL), then:

→ { "command": "subscribe", "identifier": "{\"channel\":\"PlaygroundChannel\",\"id\":\"SCENE_ID\"}" }
← { "type": "welcome" }
← { "type": "confirm_subscription", "identifier": "…" }
← { "type": "ping", "message": 1791500000 }
← { "identifier": "…", "message": { "type": "ops", "ops": [ … ], "assets": { "asset_id": { … } }, "origin": "my-script" } }
← { "identifier": "…", "message": { "type": "turn", "turn": { "id": "…", "status": "running", "steps": [ … ] } } }

Apply ops messages in order (skip ones whose origin is your own client id — you already applied them). After a reconnect, re-fetch the scene. turn messages carry agent progress.

Voice

POST /transcribe — multipart audio (webm/opus, wav, mp3, mp4, ogg) → { "text": … }
POST /speak — { "text": "…", "audio_format": "mp3" | "wav" } → audio bytes (the computer’s voice)

Textures

POST /scenes/:id/texture { "prompt": "mossy cobblestone", "tile_size": 2 } paints a seamless texture and adds it to the scene as a material ({ ops: [ { op: "material", material } ] }). Then update an object with { "material": "<id>" } or set environment.ground_material.

Limits

600 requests/hour per key across the API (and MCP); 60/hour for speech; agent goals and generated textures have daily caps per user (defaults 200 and 50). 429 comes with Retry-After.