[python] Speed up LeRobot video window reads

[XiaoHongbo-Hope]

Purpose

Speed up overlapping LeRobot video windows by decoding sorted unique frame requests once per payload, then gathering independent, contiguous outputs. Preserve fallback handling for cross-file windows and decoders without batch support.

Validation

89 tests and Flake8 passed; real-video uint8/float32 pixels and mutation isolation verified. Tests assert one decoder call per payload.

CPU overlap benchmark: 37.5 ms for the original unique-frame/stack path, 154.7 ms for the previous PR head, 23.4 ms after this fix (7-run medians). Random 4-camera throughput decreased from 177.9 to 165.1 samples/s versus the previous PR head (3-run medians, controls run earlier). Kept draft while evaluating this tradeoff.

[JingsongLi]

[P2] Avoid materializing the entire batch of windows before image transforms

When image_transforms returns new storage, such as resizing frames to a smaller resolution, these eager paths allocate every full-resolution window before the first transform runs. visual_windows then keeps all those input tensors alive until get_items() returns, even after the transformed outputs have replaced them in each item. Previously, window assembly and transforms ran one sample at a time.

I reproduced this with 32 samples, two image cameras, 16-frame windows, and a resize transform: the baseline kept at most 2 original window tensors alive simultaneously, while this version kept all 64 alive through the last transform. This significantly increases peak memory in DataLoader workers when preprocessing reduces the frame size.

Could we retain the per-sample path when image_transforms is configured, or assemble windows in bounded groups? This should cover both _decode_video_windows and _stack_visual_windows. Clearing list entries after consumption alone would not eliminate the initial batch-wide allocation peak.

[XiaoHongbo-Hope]

[P2] Avoid materializing the entire batch of windows before image transforms

When image_transforms returns new storage, such as resizing frames to a smaller resolution, these eager paths allocate every full-resolution window before the first transform runs. visual_windows then keeps all those input tensors alive until get_items() returns, even after the transformed outputs have replaced them in each item. Previously, window assembly and transforms ran one sample at a time.

I reproduced this with 32 samples, two image cameras, 16-frame windows, and a resize transform: the baseline kept at most 2 original window tensors alive simultaneously, while this version kept all 64 alive through the last transform. This significantly increases peak memory in DataLoader workers when preprocessing reduces the frame size.

Could we retain the per-sample path when image_transforms is configured, or assemble windows in bounded groups? This should cover both _decode_video_windows and _stack_visual_windows. Clearing list entries after consumption alone would not eliminate the initial batch-wide allocation peak.

Fixed by falling back to per-sample window assembly and transforms when image_transforms is configured, for both image and video windows. I consider we can set up a basic benchmark first. Optimizing the image_transforms path can be handled in a follow-up PR.

[JingsongLi]

+1