Validate, split, and recover expert datasets

Use the offline dataset tools after collecting expert demonstrations to check their consistency, create quality-filtered training partitions, and diagnose interrupted commits. They read recorded files without starting a simulator or loading PyTorch. Numeric scans use NumPy/PyArrow; video decoding uses PyAV. These lightweight imports do not change the dependencies of a normal SDK install.

Task language is authored before collection. See Task and segment language for expert data for overall and segment instructions, and Expert Data Expansion for collection. Export to the independently installed LeRobot 0.6.1 runtime is covered in Export expert data for LeRobot 0.6.1.

Start with a finalized dataset

Finish collection and finalize the recorder before validation or splitting. For an interrupted run, stop every writer before inspection or repair. A recorder’s finalize() drains committed work; it does not save a live episode.

Use the exact dataset directory printed by the runner, containing meta/info.json, rather than its parent of auto-numbered datasets. The dataset subcommands do not have the preview command’s --latest option.

EXPERT_DATASET=/absolute/path/to/finalized-dataset

embodichain dataset validate "$EXPERT_DATASET" \
    --output /tmp/expert-validation.json

embodichain dataset split "$EXPERT_DATASET" /tmp/expert-split.json \
    --fractions 0.8 0.1 0.1 --seed 42 --success-only

If validation reports an incomplete commit, follow the recovery procedure below and validate again before splitting. A split manifest references the original episodes; it does not create new dataset directories. Reports and manifests must not overwrite dataset-owned files under meta/, data/, videos/, images/, or depth_videos/.

python -m embodichain dataset ... is equivalent to the console command. Use embodichain dataset validate --help, split --help, or recover --help for the argument list.

Validate structure and media

The basic scan checks actual frame and episode Parquet against the declared features and EmbodiChain sidecars:

Check	Examples of detected problems
Episode/frame indexing	Missing episodes, wrong lengths, interleaved global ranges, discontinuous frame indices
Timing	Non-finite timestamps or timestamps inconsistent with the frame/FPS grid
Numeric features	Missing fields, invalid shapes, non-finite actions, normalized gripper values outside [-1, 1]
Language references	Unknown task_index or subtask_index, inconsistent segment labels
Segment annotations	Overlapping/out-of-range spans or inconsistent start/end markers
Recording evidence	Pending journals, contradictory sidecars, invalid replay links or depth decoder metadata

Validation establishes file and annotation consistency. Physical success still depends on measured evidence recorded by the task; a successful scan does not measure whether a grasp, placement, or liquid transfer succeeded.

embodichain dataset validate "$EXPERT_DATASET" \
    --check-media --media-frame-counts \
    --batch-size 4096 --output /tmp/expert-media-validation.json

Argument	Default	Meaning
--check-media	Off	Check referenced RGB/image/depth files
--media-frame-counts	Off	Decode referenced videos to check frame counts and cadence; implies media checks and requires PyAV
--batch-size	4096	Positive maximum number of numeric frame rows read per Arrow batch
--output	None	Write a separate JSON report; the report is also printed

Decoding can be substantially more expensive than inspecting metadata. Shared video shards are probed once per inspection, without retaining all decoded frames. The recovery tool always requires usable media evidence for repair; its media checks are not disabled by these validation flags.

Read the diagnostic report

Reports contain ok, root, episodes, frames, errors, warnings, and issues. Each detailed issue has severity, code, location, and message. For example, language_reference identifies a vocabulary-reference problem, while pending_commit indicates that a recording needs diagnosis before use.

import json
from pathlib import Path

report = json.loads(Path("/tmp/expert-validation.json").read_text())
print(report["ok"], report["errors"], report["warnings"])
for issue in report["issues"]:
    print(issue["severity"], issue["code"], issue["location"], issue["message"])

Issue details are capped at 1,000 entries; aggregate counts can be larger. Missing optional lineage in an old recording can produce warnings, while contradictory references or incomplete commits produce errors. ok means no errors, so review warnings before combining legacy data with new recordings.

Command exit status	Meaning
0	Validation found no errors, splitting succeeded, or recovery inspection is complete
1	Validation errors or unresolved/incomplete recording evidence
2	Invalid arguments or a request that cannot be performed

Filter quality and keep related episodes together

embodichain dataset split "$EXPERT_DATASET" /tmp/expert-quality-split.json \
    --fractions 0.8 0.1 0.1 --seed 42 \
    --success-only --max-recoveries 1 --require-physical-valid

Only use the extra quality filters when the recording contains their evidence. The command fails if no episodes satisfy the requested filters.

Argument	Meaning
--fractions TRAIN VALIDATION TEST	Three finite nonnegative episode fractions summing to one; default 0.8 0.1 0.1
--seed	Deterministic group assignment; default 0
--success-only / --failure-only	Keep explicitly successful or failed episodes; mutually exclusive
--max-recoveries N	Nonnegative limit on execution recovery/replanning counts; unknown counts are excluded
--require-physical-valid	Require explicit accepted physical evidence; unknown evidence is excluded
--group-by lineage	Default: preserve source/parent/expansion relationships
--group-by scene --scene-field scene_id	Also join episodes with the same explicit scene identity; every episode needs that field
--allow-legacy-independent	Explicitly assume unidentified legacy episodes are independent; record the assumption in the manifest

Recovery counts refer to execution retries/replans and workflow recovery cycles, not the number of times an operator repaired a recording. Canonical Task Program traces support row-local counts without double-counting retries. Physical filtering recognizes explicit physical_valid or physical_validation results, and successful ordered_stable_regions objectives with valid measured evidence. A declared objective or projected effect alone is insufficient.

Grouping happens before filtering. If B derives from A and C derives from B, all three remain in one partition even when B is excluded. Shared source_episode_uuid, multiple sources, parent identities, and recorded expansion references establish these transitive groups. Scene grouping joins additional groups; it never separates derivatives of the same source.

Unknown legacy origins are grouped conservatively using their source fingerprint, run identity, or the whole dataset. This can leave validation or test empty. Fractions are targets, because source groups are indivisible and episode lengths differ. Supply --allow-legacy-independent only when you have established that their derivations cannot leak between partitions.

Use a split manifest in a reader

The JSON records splits, episode/frame counts, group assignments, excluded episodes with reasons, quality settings, and provenance assumptions. The training application must explicitly apply the episode selection. In the recording environment with LeRobot 0.4.x:

import json
from pathlib import Path

from lerobot.datasets.lerobot_dataset import LeRobotDataset

manifest = json.loads(Path("/tmp/expert-split.json").read_text())
dataset_root = Path(manifest["dataset_root"])
train_episodes = manifest["splits"]["train"]
if not train_episodes:
    raise ValueError("The grouped split has no training episodes")

train_dataset = LeRobotDataset(
    repo_id=dataset_root.name,
    root=dataset_root,
    episodes=train_episodes,
)
print(train_dataset[0]["task"], train_dataset[0]["subtask"])

The split command checks episode metadata and pending commits; it does not replace a full numeric/media validation scan. If you export or relocate the dataset, validate that destination and generate a manifest for its root.

Trace sources, fragments, and replay artifacts

New recordings add stable identities and configuration/runtime fingerprints to meta/embodichain_episodes.jsonl:

Field	Meaning
run_uuid	Collection run identity
episode_uuid	Identity of this full episode or fragment
source_episode_uuid / source_episode_uuids	Original source or sources
parent_episode_uuid	Parent of a derived record, when present
config_hash, program_hash	Stable configuration/program fingerprints; program information is conditional on a Task Program
provenance	Available SDK/Python/LeRobot/PyTorch versions, physics backend and control timing
expansion	Candidate records and selected source/case identities, when expansion is used
replay_artifact, trajectory_path	Explicit link to optional replay state and its source alignment

A fragment retains the original source identity and gets a deterministic derived UUID for its span, attempt, and continuity region. Language vocabulary indices identify descriptions, not episodes or source trajectories. Fingerprints support traceability but do not replace the full initial scene state.

Enable the existing replay recorder when that state is needed:

embodichain run-task \
    --gym_config embodichain_tasks/configs/tasks/manipulation/tableware/pour_water/task.cobotmagic.yaml \
    --headless --max_episodes 1 \
    --record_trajectory --trajectory_save_dir outputs/replay

Replay files are atomically saved before the dataset recorder snapshots their paths, including async snapshots. A fragment points to the full source trajectory and its source frame range. Preserve external .pt files when moving a dataset; offline inspection/recovery checks their links without loading pickle state. A seed or fingerprint alone does not reconstruct a scene, and replay does not guarantee identical physics across versions/backends.

Diagnose and repair interrupted commits

After stopping all writers:

embodichain dataset recover "$EXPERT_DATASET"

Each episode has atomic, fsynced evidence under meta/embodichain_commits/. Its prepared snapshot includes the frozen task/segment language and provenance before frames are submitted.

Journal phase	Last confirmed progress
prepared	Metadata captured; SDK commit has not begun
lerobot_committing	SDK save was entered; an interruption requires disk inspection
lerobot_committed	SDK save returned
depth_committed	Applicable depth resources finalized
complete	Episode sidecar appended and the journal advanced

These phases are not a transaction across Parquet, videos, and JSONL. SDK buffering can leave even a complete journal without durable frame/episode files. Inspection verifies the actual global/local frame identities, SDK ranges, sidecar ownership, replay references, and decodable RGB/depth coverage.

Reported commit status	Operator action
complete	Verified evidence and matching sidecar are present
repairable	Committed artifacts verify; the missing sidecar can be restored
retryable	Preparation has no committed frames; the tool does not execute the retry
unresolved	Missing, contradictory, unreadable, or insufficient evidence needs investigation

Request repair explicitly:

embodichain dataset recover "$EXPERT_DATASET" --repair
embodichain dataset validate "$EXPERT_DATASET" --media-frame-counts

Repair can append a missing episode sidecar from verified journal evidence and advance its journal. It does not reconstruct Parquet/videos, rewrite SDK metadata, truncate malformed JSONL, load replay state, or resume collection. Duplicate/conflicting sidecar indices and unknown SDK outcomes remain unresolved when disk evidence is insufficient. Preserve the report and original files for investigation. A live recorder with an uncertain SDK write blocks later submissions rather than retrying an ambiguous append.

Configure recording buffers

This fragment is merged into an existing handwritten task’s Gym config:

env:
  task_instruction: Pick and place the object.
  dataset:
    lerobot:
      func: AsyncLeRobotRecorder
      mode: save
      save_failed_episodes: false
      params:
        save_path: outputs/lerobot/expert_demos
        use_videos: true
        metadata_buffer_size: 1
        async_queue_maxsize: 4

For a Task Program, keep overall language at its program root instead. metadata_buffer_size is a positive integer, defaults to 1, and controls LeRobot episode-metadata batching for both recorders. Larger values can delay visibility of recent metadata. The async-only async_queue_maxsize limits pending payload count, not bytes; its default 0 is unbounded. Payload sizes depend on camera count, resolution, and episode length. See Dataset Functors for image-writer and depth settings.

Segment lookup builds a validated frame map once per episode instead of scanning every segment for each frame. The CPU microbenchmark checks exact annotation equivalence and reports lookup time and sampled process RSS:

python -m scripts.benchmark.data_pipeline.benchmark_segment_annotations \
    --frames 10000 --segments 100 --repeats 5 \
    --report /tmp/segment-annotations.md

It does not measure full simulation or disk throughput.

Python API and common failures

The file-only API mirrors the CLI:

from embodichain.data_pipeline.datasets.inspection import (
    create_split_manifest,
    validate_dataset,
)
from embodichain.data_pipeline.recording import inspect_recording

report = validate_dataset(dataset_root, media_frame_counts=True)
report.raise_for_errors()
split = create_split_manifest(dataset_root, "/tmp/python-split.json", seed=42)
commit_report = inspect_recording(dataset_root)
print(split["episode_counts"], commit_report["ok"])

Here dataset_root is the finalized root used above. Full signatures and report types are in embodichain.data_pipeline.datasets and embodichain.data_pipeline.recording.

Symptom	Check
meta/info.json not found	Use the exact auto-numbered dataset root
language_reference or segment errors	Inspect task/subtask tables, sidecar spans and frame annotations together
pending_commit	Stop writers, inspect/recover, then revalidate
Quality selection returns no episodes	Check whether success/recovery/physical evidence is actually present
A partition is empty	Inspect indivisible source groups and legacy-origin assumptions
Media blocks repair	Install PyAV and verify the complete referenced streams; nonempty files are insufficient
External replay path is missing	Retain or correctly relocate the linked source trajectory

For official persistent-language and native-depth conversion, continue with Export expert data for LeRobot 0.6.1. Automatic physical-state restore, suffix continuation, and Mimic/robomimic export are outside these tools.