A mobile manipulator episode
示例
A warehouse mobile manipulator records one pick-and-place episode. The dataset contains: camera frames, LiDAR scans, IMU readings, tactile pad readings, and audio. The task is to pick a box from a shelf and place it on a conveyor. Assume the streams are time-synchronized. Which modality supports which part of the task, and what happens if the IMU stream is missing?
演示讲解
Step 1 — Identify the task's demands. The robot must find the box, move to it, grasp it, and place it. Each demand needs a different kind of information. Step 2 — Match modalities to demands. Camera frames supply appearance, so they support recognizing which object is the box and confirming it is the right one. LiDAR supplies 3D geometry, so it supports locating the box in space and avoiding the shelf frame. Tactile pads supply contact force, so they support detecting a secure grip and adjusting pressure. Audio supplies event sounds, so it can flag a slip or a hard contact. Step 3 — Consider the IMU. The IMU measures acceleration and angular rate, which is what lets the system track how the base and arm are moving and tilting. Step 4 — Remove the IMU. Without it, the record no longer contains direct motion measurements. The remaining streams still show what was seen and touched, but reconstructing how the body moved and stayed balanced during the episode becomes much harder, because that information was never captured.