Minimum Annotation Requirements

    Definition:

    Every valid annotation must meet basic submission requirements.

    • Model Generated Answer (MGA): The model response rating must be 👎. If 👍 is set, the annotation is invalid. The Model Generated Answer box must contain a response — not blank due to a technical error, API timeout, or system glitch.
    • Ontology: Every VQA prompt must challenge the model on at least 2 skills. Prompts involving Enumeration require 3+ skills. Every VQA prompt must include at least one of Logical Reasoning, Table/Chart/Graph Understanding, or World Knowledge.
    • Answer type: Every annotation must be tagged as either MCQ or short answer. An untagged annotation is invalid.
    • Validation freshness: Check My Work (also called Verify Submission) must be run after the last time the model was run. If you regenerate the model response, run it again before submitting.

    ✅ Pass

    1. Model response is inaccurate; rating is set to 👎.
    2. At least 2 skills tagged (3+ if Enumeration is one of them), with at least one anchor skill.
    3. Response field contains text.
    4. Annotation tagged as MCQ or short answer.
    5. Check My Work (Verify Submission) run after the final model run.
    6. Rewrite Explanation is left blank.

    ❌ Fail

    1. Model response is correct, so 👎 cannot be validly applied.
    2. Fewer than 2 skills tagged.
    3. Enumeration prompts with fewer than 3 skills.
    4. No Logical Reasoning / Table-Chart-Graph Understanding / World Knowledge tagged.
    5. Response field is entirely empty or null.
    6. Answer type left untagged — neither MCQ nor short answer.
    7. Model regenerated after the last Check My Work (Verify Submission) run.

    Reminders

    Verify Submission helps, but it isn't the final word

    The Verify Submission check at the top of the task page in SuperAnnotate is a useful safety net, but it can still make mistakes. Check every requirement yourself before you submit.

    Changed the prompt? Rerun everything

    After any change to the prompt:
    1. Rerun the model.
    2. Confirm the model still fails.
    3. Update the question type or ontology if the change requires it.
    4. Rerun Check My Work (CMW).

    Knowledge Check

    On a VQA annotation, you tag a prompt with Spatial Reasoning and Enumeration. The model fails and you rate it 👎. Does this meet the minimum annotation requirements?

    Common ErrorsImage Feasibility