Skip to content

agentic-sim-world-gen

An agent (Claude Code) builds an Isaac Sim world (USD) from a description in Japanese or English, a floor plan or photos: it searches for assets, picks them, lays them out, checks the result with physics in Isaac Sim and records the license of every asset. Assets come from SimReady Central, the Isaac Sim bundled assets, Poly Haven CC0 models and your own USD folders.

flowchart TB
  IN["<b>Input</b><br/>text (Japanese or English)<br/>floor plan: JSON / DXF / IFC / image<br/>photos"]
  CC["<b>Claude Code</b><br/>reads the request, decides which tools to call"]
  subgraph WG["agentic-sim-world-gen (this repository): MCP server + skill, the same operations from the CLI"]
    direction LR
    S["Search and pick assets"] --> L["Lay out<br/>relations, arrays, shelves"] --> V["Review in the browser<br/>thumbnails, top view"] --> U["Write the USD world<br/>and LICENSES.md"]
  end
  SRC["<b>Asset sources</b><br/>SimReady Central<br/>Isaac Sim bundled assets<br/>Poly Haven CC0<br/>your USD folders"]
  ISAAC["<b>Isaac Sim 6.1</b><br/>runs the check script:<br/>physics, screenshots"]
  IN --> CC
  CC -- "calls the tools" --> WG
  SRC -- "searched" --> WG
  WG -- "USD" --> ISAAC
  ISAAC -- "report: fix and check again" --> CC
  style WG fill:#536dfe14,stroke:#536dfe,stroke-width:3px
  classDef repo fill:#536dfe2e,stroke:#536dfe
  classDef input fill:#9e9e9e26,stroke:#9e9e9e,stroke-dasharray:5 5
  classDef claude fill:#d977572e,stroke:#d97757,stroke-dasharray:5 5
  classDef assets fill:#e3a0082e,stroke:#e3a008,stroke-dasharray:5 5
  classDef isaac fill:#76b9002e,stroke:#76b900,stroke-dasharray:5 5
  class S,L,V,U repo
  class IN input
  class CC claude
  class SRC assets
  class ISAAC isaac

Blue with a thick frame: this repository. Dashed: outside it (orange: Claude Code, yellow: asset sources, green: Isaac Sim, gray: your input).

Important

This is a personal project. It is not an official NVIDIA product and is not sponsored or endorsed by NVIDIA. Isaac Sim, Omniverse, SimReady and Replicator are trademarks of NVIDIA Corporation.

Warning

SimReady Central is searched through the unpublished API its web page uses. It is unofficial and may change or stop working without notice. The official endpoint (search.simready.omniverse.nvidia.com, used by the simready-search package) indexes a different catalog, mostly the Isaac Sim bundled assets, and does not return the SimReady Central datasets on Hugging Face, so it cannot replace this API.

Caution

Asset licenses differ per asset, and non-commercial ones (e.g. CC-BY-NC) can be picked. Check the LICENSES.md written for each scene before you use or share it. The Isaac Sim bundled assets are licensed by NVIDIA and are not covered by this repository's license; handing out a USD with assets packed in is redistribution.

Caution

Do not use photos that show people as input.

Demo: a warehouse

Building a warehouse: Claude Code, the browser preview and the Isaac Sim check

One turn, asked in Japanese, waits fast-forwarded: "Make a 24 m × 16 m warehouse. Put pallet racks in 4 rows with aisles between them and cardboard boxes on the shelves; pallets and a forklift in the loading area at the front. Check it with physics in Isaac Sim and sum up the licenses." Claude fills in the defaults, searches and lays out everything at once, swaps the stocked racks for empty ones from the candidates (select_asset) and puts boxes on the shelves (set_fill). The physics check in Isaac Sim 6.1 finds 2 boxes that slid and a forklift that moved: Claude spaces the boxes out, tries three forklifts (their seat or handle parts come loose) and settles on a walk-behind electric stacker, and the check passes (221 objects). It ends with the request checklist, the licenses (all CC BY 4.0) and a view from 24 directions. Assets from SimReady Central only (AGENTIC_SIM_WORLD_GEN_SOURCES=simready-central, AGENTIC_SIM_WORLD_GEN_MDL=0). A screen recording on Ubuntu (Isaac Sim not opened, the check runs headless); full quality: docs/videos/warehouse_demo.mp4.

Assets (all CC BY 4.0, details in LICENSES.md): "sm_rack_h3m_s02_01", "sm_box_cardboard_b02_01" and "sm_vehicle_forklift_orange_a01_01" by NVIDIA (nvidia/simready-assets), "pallet" by ForgeCAD (ForgeCAD/simready-assets), CC BY 4.0.

One run end to end (the example in Getting Started; ②–④ are real screens):

From request to world: ask in words, look at the candidates and pick, lay out by relations and check, physics check in Isaac Sim, scene.usda and LICENSES.md

Features

  • MCP server + skill for Claude Code (62 tools); the same operations from the CLI agentic-sim-world-gen
  • Input: text (rooms, warehouses, outdoor sites), floor plans (JSON / DXF / IFC, or traced from an image) and photos
  • Layout: relations (against a wall, in front of, next to, on, facing), arrays (rack rows with aisles), stocking shelves, overlap / out-of-room / door clearance checks
  • Look: floor and wall materials (Isaac Sim MDL or CC0 PBR), lighting presets, moods, outdoor ground / roads / buildings / sky
  • Review: candidate thumbnails and a top-down layout in a browser preview; physics check and screenshots in Isaac Sim 6.1 (headless or live)
  • Synthetic data with Replicator: RGB, 2D boxes, segmentation, depth (COCO / KITTI)
  • Licenses: source and license of every asset recorded in the USD and in LICENSES.md

Objects are placed by relation, not by coordinates; the solver works out the position and the direction, and keeps them when an asset is swapped for one of another size:

Layout relations: against_wall, in_front_of, next_to, on, facing, array

When no asset fits, a close one is used instead and reported (no 3D generation). Your own asset folders can be added with local:.

Requirements

Tested Not tested
Isaac Sim 6.1 pip install, Ubuntu 24.04, RTX 2080 Ti the standalone (binary) build, other versions, Windows
The rest (search, preview, USD) Ubuntu 24.04 (Python 3.14), macOS on Apple Silicon (Python 3.12) Windows

Python 3.12+ and uv; Claude Code to use it as an agent; Isaac Sim 6.1 only for the physics check, screenshots and synthetic data. Without a GPU you can still search, preview and write USD.

Install

git clone https://github.com/atinfinity/agentic-sim-world-gen.git
cd agentic-sim-world-gen
uv sync --extra usd --extra hf --extra mcp --extra dev
source .venv/bin/activate

agentic-sim-world-gen resolve examples/p0_study_room.json -o out/study   # search -> candidates.html, layout.svg
agentic-sim-world-gen compose out/study/scene.resolved.json              # -> out/study/scene.usda
agentic-sim-world-gen check out/study                                    # physics check in Isaac Sim (Linux)

Install usd-core only in this venv, never in Isaac Sim's Python. Point to Isaac Sim with ISAACSIM_PYTHON (pip install) or ISAACSIM_PATH (default ~/isaacsim), and set OMNI_KIT_ACCEPT_EULA=YES to run it non-interactively. The other environment variables are in Getting Started "3. Environment variables".

With Claude Code: .mcp.json registers the agentic-sim-world-gen MCP server and .claude/skills/world-gen is the skill. To only make worlds, create a workspace outside the repository with agentic-sim-world-gen workspace <dir>, start claude there and ask, e.g. "Make a 10 m² study with a desk, a chair and a bookshelf".

Guides

To use it:

Guide
Getting Started from the install to your first world
Everything on one machine Ubuntu with an RTX GPU and a screen (live mode too)
Over ssh the GPU machine is an Ubuntu box on the LAN; you use ssh and a browser from a Mac or similar

To change the code:

Guide
How it works the flow, data, search, layout, checks, USD, Isaac Sim, the conversation and rooms from photos, with diagrams
Contributing setup, tests, where things are, pull requests, the project page, GitHub Actions

The Japanese versions are in docs/.

License

The code is under the Apache License 2.0 (see NOTICE). Not covered by it:

  • Assets are not included. A generated world refers to assets that keep their own licenses (the scene's LICENSES.md).
  • Images (docs/images, src/agentic_sim_world_gen/data/moods) show assets under their own licenses (the demo's attribution is above).
  • Bundled indexes (src/agentic_sim_world_gen/data/*.gz) are lists of names, sizes and URLs; their origin is in NOTICE.
  • Example photos (examples/photos) are CC0 from Wikimedia Commons (SOURCES.md).
  • Optional dependency: IFC import (extra ifc) uses IfcOpenShell, LGPL-3.0-or-later (imported, not bundled).
  • Optional models (not bundled; downloaded only when you allow it): SigLIP 2 for the taste check (Apache-2.0), MoGe-2 for photo depth (MIT).

Contributions: Contributing. Security: SECURITY.md.