Scene Engine

Scene Engine#

Scene Engine reconstructs a table-top scene from one image. It identifies the table and visible objects, segments their masks, generates simulation-ready meshes, refines the object layout on the table, and exports a scene that can be loaded by EmbodiChain.

Quick Start#

Install EmbodiChain with the gensim extra first; see Installation.

Configure the required services in embodichain/gen_sim/.env, then generate a scene:

embodichain scene-engine \
    --image /path/to/scene.png \
    --output_root /path/to/scene_output

The same command is available through the package entry point:

python -m embodichain scene-engine \
    --image /path/to/scene.png \
    --output_root /path/to/scene_output

Scene Editing#

Edit an existing valid Scene Engine output with an instruction:

embodichain scene-engine \
    --output_root /path/to/scene_output \
    --edit_prompt "add a red cup to the front-center of the tabletop"

--image and --edit_prompt may also be provided together. Scene Engine then generates the image-based scene first and applies the edit to that export. An edit-only invocation requires an existing scene_export directory. The edit overwrites its scene_config.json, scene_graph.json, scene.json, and final mesh_assets; intermediate generation and edit artifacts remain available for debugging.

Edit Flow#

  • Import and understanding: reads and validates the existing export, then resolves the instruction into add, move, and delete operations and an updated scene graph.

  • Asset preparation: only add operations generate an image, segmentation, geometry, and SimReady asset. Move-only and delete-only edits skip this stage.

  • Layout and export: refines the edited layout using the updated scene graph and writes the resulting scene back to scene_export.

Configuration#

Scene Engine reads the LLM, segmentation, image-generation, and geometry-generation settings from embodichain/gen_sim/.env. The same file also contains the optional articulation-server connection used by the articulated-asset generation client:

OPENAI_API_KEY="your-api-key"
OPENAI_MODEL="your-model"
OPENAI_BASE_URL="https://api.openai.com/v1"
SCENE_ENGINE_OPENAI_DEFAULT_QUERY="{}"
OPENAI_MAX_ATTEMPTS=3

SCENE_ENGINE_IMAGE_SEGMENTATION_BASE_URL="http://host:port"
SCENE_ENGINE_IMAGE_SEGMENTATION_TIMEOUT_S=30
SCENE_ENGINE_IMAGE_SEGMENTATION_MAX_ATTEMPTS=3
SCENE_ENGINE_IMAGE_SEGMENTATION_HEALTH_PATH="/health"
SCENE_ENGINE_IMAGE_SEGMENTATION_BY_PROMPT_PATH="/segment_by_prompt"

SCENE_ENGINE_IMAGE_GENERATION_BASE_URL="http://host:port"
SCENE_ENGINE_IMAGE_GENERATION_TIMEOUT_S=120
SCENE_ENGINE_IMAGE_GENERATION_MAX_ATTEMPTS=3
SCENE_ENGINE_IMAGE_GENERATION_HEALTH_PATH="/health"
SCENE_ENGINE_IMAGE_GENERATION_BY_PROMPT_PATH="/generate_image_by_prompt"

SCENE_ENGINE_GEOMETRY_GENERATION_BASE_URL="http://host:port"
SCENE_ENGINE_GEOMETRY_GENERATION_TIMEOUT_S=600
SCENE_ENGINE_GEOMETRY_GENERATION_MAX_ATTEMPTS=3
SCENE_ENGINE_GEOMETRY_GENERATION_HEALTH_PATH="/health"
SCENE_ENGINE_GEOMETRY_GENERATION_OBJECTS_PATH="/generate_multiple_objects"

SCENE_ENGINE_ARTICULATED_GENERATION_BASE_URL="http://host:port"
SCENE_ENGINE_ARTICULATED_GENERATION_TIMEOUT_S=7200
SCENE_ENGINE_ARTICULATED_GENERATION_MAX_ATTEMPTS=3
SCENE_ENGINE_ARTICULATED_GENERATION_HEALTH_PATH="/health"
SCENE_ENGINE_ARTICULATED_GENERATION_GENERATE_PATH="/generate_articulation"

Processing Flow#

  • Scene understanding: analyzes the image and segments the table and visible objects.

  • Scene generation: generates meshes, prepares SimReady geometry, detects the table support surface, and refines the table-top layout.

  • Scene export: copies the final GLBs and writes a portable z-up scene export.

Output and Preview#

The important final outputs are:

scene_output/
|-- scene_understanding/     # Object analysis, masks, and stage JSON
|-- scene_generation/        # Generated, SimReady, and layout-debug artifacts
|-- scene_editing/           # Present after edits; asset-preparation and layout-optimization artifacts
`-- scene_export/
    |-- mesh_assets/         # Final GLBs
    |-- scene_config.json    # Exported z-up scene description
    |-- scene_graph.json     # Table support and planar relation graph
    `-- scene.json           # Scene Engine object metadata and y-up poses

Validate the export without opening a window:

embodichain preview-scene \
    --output_root /path/to/scene_output \
    --headless

For an interactive preview, omit --headless. Add --viser to publish the scene through Viser.

Contents