Unity client

The Holodeck computer inside the Unity editor. Pick a scene, say or type what you want, and the Holodeck agent builds it in your open Unity scene with real prefabs. It’s the same scene the web Playground shows: build in the browser and it appears in Unity; move a tree in Unity and the browser sees it move.

Install

Unity 2021.3 or newer (built-in render pipeline or URP).

  1. Download holodeck-unity.zip and unzip it anywhere.
  2. In Unity: Window ▸ Package Manager ▸ + ▸ Add package from disk… and pick com.holodeck.playground/package.json.

(With access to the Holodeck repo you can instead use Add package from git URL… with <repo>.git?path=/clients/unity/com.holodeck.playground.)

Connect

  1. Window ▸ Holodeck ▸ Playground.
  2. Settings: server https://arx.dev and an API key — Get an API key opens Account → API keys. The key is stored in your editor preferences, never in the project.
  3. Pick a scene (or New) and press Connect. The scene is built under a Holodeck · <name> object in the open Unity scene. Browser opens the same scene on the web.

Build

  • Type a goal and press Engage (or Enter). Halt stops it. The log shows each step.
  • ● Talk records one spoken command. Voice mode is hands-free: it listens when the computer is idle and speaks the replies (Grok voice).
  • Library: search the asset library and Place at the Scene view’s pivot.
  • Assets dropdown: lock the scene to one vendor or pack for a consistent style.
  • Cameras: view the scene’s cameras in the Scene view.

Editing by hand

Synced objects are normal GameObjects. Click one (the whole object is selected, not just a mesh), then move, rotate, scale, rename or delete it as usual. About half a second after you stop, the change is saved to the server and shows up in every browser and Unity editor on that scene. Anything you add outside the Holodeck · … object stays local.

Folders, static scenery and terrain

  • Scene folders (Nature/Trees) become nested 📁 GameObjects under the scene root. Drag a synced object to another folder in Unity’s Hierarchy and the move syncs back.
  • Static objects get Unity’s static flags (batching, occlusion). Hidden folders are hidden in the Scene view; locked folders can’t be picked there.
  • Terrain becomes a real Unity Terrain around the scene origin (Settings → Terrain to build: 256, 512 or 1024 m), sampled from the same maths the server and browser use, with the ground layers as TerrainLayers (custom layer colours and materials included — a layer bound to a generated texture uses that texture) and a water plane. It rebuilds when the terrain changes on the server. Edits you make with Unity’s own terrain tools stay local — sculpt in the browser, by voice, or over MCP to share them.

Where assets come from

The asset library knows each asset’s Unity GUID. So:

  1. If your project already has the pack (e.g. the Synty packs), the agent’s choices appear as the real prefabs, linked to your project’s assets.
  2. If not, and Import assets that aren’t in this project is on, the window downloads that asset’s Unity package — the prefab plus its meshes, materials and textures, with their .meta files — and imports it under Assets/. The next copy is instant.
  3. Otherwise it downloads the model file itself (FBX or OBJ; GLB if glTFast is installed) into Assets/Holodeck/Imported/, and applies its pack’s texture.
  4. If nothing works, you get a sized yellow placeholder named after the asset.

Scene materials and AI-generated textures become real .mat / image assets under Assets/Holodeck/Generated/<scene id>/. Changing a material anywhere updates the asset.

The scene’s environment — sky, fog, ambient light, sun and ground — is written to RenderSettings and a Holodeck Ground plane. Turn off Apply the scene’s sky, fog and ground to keep your own lighting.

Coordinates

Holodeck scenes use three.js coordinates (right-handed, y up, metres; positions are an object’s base). A prefab is offset inside its wrapper so the wrapper sits at the bottom centre of the prefab’s renderer bounds — the same anchor the browser uses — whatever the prefab’s own pivot.

Doors (interaction) get a HolodeckInteractable component, posed open or closed to match the scene. In Play mode it eases between the two: call Toggle(), click it (legacy Input Manager, needs a collider), or let auto doors open when the main camera comes near. Parts move without re-parenting, so prefab instances stay intact. The client mirrors them for Unity (left-handed): x is negated, rotations are mirrored, and cameras turn 180° because Unity cameras look down +z. Holodeck.Core.SceneMath has the conversions if you build your own tools.

Building your own

Holodeck.Core (in the package’s Runtime/Core) is plain .NET with no UnityEngine dependency:

using Holodeck.Core;

var api = new HolodeckApi("https://arx.dev", apiKey);
var scene = await api.CreateScene("From code");
var trees = await api.SearchAssets("pine tree", scene.AssetLock);
await api.CallTool(scene.Id, "scatter_assets", new JObject {
    ["asset_ids"] = new JArray(trees.Take(3).Select(t => t.AssetId)),
    ["count"] = 40,
    ["area"] = new JObject { ["radius"] = 25 }
});

var cable = new CableClient("https://arx.dev", apiKey, scene.Id);
cable.Message += msg => Console.WriteLine(msg["type"]);   // "ops" / "turn" (background thread)
cable.Start();

The same calls work in a runtime player, a server or a console app. See the Playground API and the tool reference.