Annotation Independence
Definition:
This criterion only applies to tasks with 2 or more annotations on the same image. Watch for two things:
- Each annotation must work in a vacuum. An annotation should never pick up where another left off, reuse wording, instructions, or clarifications from another prompt, or describe things relative to instructions given elsewhere. To test it, imagine each annotation is the only one on the image. If that causes problems, it fails.
- Annotations must not be overly similar. The most common ways to keep them distinct are focusing on different parts of the image or underlying information, using a different question structure, and choosing a different mix of ontologies. None of these is a hard requirement, but the goal is relatively unique annotations — duplicates and near-duplicates will be rejected.
✅ Pass
- Each annotation explores a unique data point or visual element with different cognitive steps, fully understandable in isolation.
❌ Fail
- References to "the other one," "it," or a prior question; repetitive structures or the same formula with only minor swaps.
Examples live in the appendix under Bad Examples → Not standalone.