Configuration-defined manipulation RL

Configuration-defined manipulation RL#

A supported manipulation task can select observations, actions, rewards, reset events and measured outcome conditions without defining a Python environment. The loader registers the common EmbodiedEnv under the deployment’s id and sets EmbodiedEnvCfg.rl_task. This optional declaration enables TerminationManager on the existing environment; ordinary environments keep their existing success behavior. The original Python-owned task route remains available.

Deployment#

Use an ordinary inline Gym configuration, or select existing physical environment and embodiment components. Keep exactly one file-owned physics backend. Add a root rl_task mapping; it cannot coexist with task_program in one deployment. Task Program adapters and acceptance remain separate from RL outcomes. Existing env.actions, env.observations, env.rewards and env.events retain their current manager contracts and ownership rules.

rl_task:
  success:
    goal:
      func: object_near_target
      params:
        entity_cfg: {uid: cube}
        target_pose_key: goal_pose
        axes: [0, 1]
        tolerance: 0.1
  failure:
    fallen:
      func: object_below_height
      params:
        entity_cfg: {uid: cube}
        height: -0.1

Success requires every declared success condition; an empty set never defaults to success. Any failure or custom truncation condition ends its own row. max_episode_steps adds the ordinary time limit. Physical success remains visible in info["task_outcome"]["physical_success"]; accepted success excludes invalid evidence, failure and truncation. The imminent time limit is applied before success rewards and termination flags are computed.

The initial built-ins use environment-local metres. object_near_target checks a Euclidean distance over selected axes; its default bound is strict. object_below_height checks a strict local Z bound. Task Program’s existing near-target validator shares the measured implementation and retains inclusive XYZ distance semantics.

Goal state and lifecycle#

Declare virtual target poses with the existing randomize_target_pose event in reset mode and a matching store_key. Virtual goals that change during an episode are outside this initial replay contract. Missing or malformed targets fail after reset events, before returning the initial observation.

TerminationManager.compute() runs once after physics and interval events, with the host’s time-limit mask for that transition. Observations, rewards, Gym termination flags and episode metadata consume the same cache; recording does not alter acceptance. The environment owns step timing, reset targets and replay context. The manager owns condition execution, aggregation and selected row reset. Queries of info, evaluation and final task success return owned values. ignore_terminations suppresses Gym termination flags while preserving measured success/failure and time-limit truncation. Stateful conditions subclass Functor and implement reset(env_ids); selected row resets do not sample or advance another row’s condition history.

New condition functors implement func(env, **params) -> ConditionResult. Register their module through the existing manager-module mechanism, use SceneEntityCfg for entities, and return boolean (num_envs,) masks and batch-aligned metric tensors on the environment device. Metric names, shapes and dtypes must stay stable during an episode. Use a class functor when entity or geometric parameters need validation at construction, as the built-in conditions do. Validate those parameters in __init__(cfg, env) without sampling reset targets; the generic manager does not know individual condition implementations.

Registration is process-local: load the deployment through config_to_cfg before calling gym.make(id). Reloading the same declaration is idempotent; changing conditions, manager ordering, physical declarations or episode limits under the same ID fails. Device, batch size and seed are runtime choices.

PushCube#

The new PushCubeConfiguredRL-v1 deployment preserves the existing concrete robot, TCP, scene, action, observation and reward setup. It classifies falling as task failure; the legacy PushCubeRL keeps its original falling-as-truncation behavior. Existing task IDs and agent files are unchanged.

embodichain train-rl --config extensions/embodichain_tasks/configs/tasks/manipulation/push_cube/agents/ppo.configured_rl.yaml

Recording and dynamic replay#

Trajectory metadata includes the original virtual-target context as task_initial_state. Dynamic replay restores that context after restoring the physical scene and before returning observations or reseeding recordings. Measured task replay requires one environment and full resets. Missing context is rejected; a newly randomized goal cannot stand in for the recorded goal. These requirements apply only when the environment enables RL outcomes; ordinary and Task Program replay keep their existing contracts. Kinematic/control playback does not establish measured physical acceptance. Replay exhaustion stops playback and is distinct from the task’s own time limit.

Contents