Worked example: a grasp dataset with a quality gate
Example
A team records 500 grasp episodes with a robot arm. Each episode contains camera frames, joint positions, gripper force, and the operator's commands. The plan is to label every frame with the object's pose and a grip outcome (success or slip), then sell the dataset to a manipulation-policy team.
Assumptions: the object is sometimes occluded by the gripper or by the arm itself; two annotators are available; the buyer needs frame-level pose and a per-episode outcome label.
Walkthrough
Step 1 — Attach the targets. For each frame, an annotator marks the object's pose (position and orientation) and, at the end of the episode, records whether the grip held or slipped. This produces the two label types the buyer asked for: a per-frame pose label and a per-episode outcome label.
Step 2 — Run a consistency check. A second annotator independently labels a sample of frames. Where the two annotators agree on the object's pose within a small tolerance, the frame is treated as reliable. Where they disagree, the frame is flagged.
Step 3 — Diagnose the flagged frames. Most disagreements cluster on frames where the object is occluded by the gripper. The annotators were guessing at a pose they could not see. This is not random noise; it is a systematic gap tied to a specific condition.
Step 4 — Decide per flagged frame. Frames where the object is fully hidden are excluded from the pose-labeled set, because a guessed pose would teach the model a target that does not match what the sensors actually show. Frames where the object is partially visible are re-labeled with a tighter annotation protocol, and the re-labeled version replaces the original.
Step 5 — Record the outcome. After the gate, the dataset has a pose-labeled subset with documented exclusions and a per-episode outcome label for all 500 episodes. The exclusions are part of the dataset's description, not a hidden cleanup.