The Reviewer Playbook

    Your edits and feedback drive quality alignment across annotators. The reviewer role is fundamentally a teaching position — with every review you write, you have the opportunity to help annotators improve their craft. Every revision is an opportunity to calibrate annotators to the standards we expect. This playbook covers what you can edit, how to deliver high-impact feedback, and how to handle skipped or returned tasks.

    Ground every review against the SQS (Task Quality) Guide — it is the rubric your SQS scores and edits must align to. For live calibration walkthroughs from recent reviewer syncs, see Recordings → Reviewer Syncs in the Appendix. Newly claimed tasks have 4 hours from claim for Annotators, Reviewers/QC, and Auditors. Returned tasks have 12 hours from return before being recirculated. Finish and submit within the applicable window; once it expires, the task can be pulled from your queue.

    The Feedback Protocol

    To maintain a consistent quality control pipeline, adhere to the following workflow:

    Document in both places:

    Record all changes and their rationale in both the SuperAnnotate QC feedback box and the Feedback field of the Shadow Task on Handshake.

    Keep the two entries consistent.

    Core Operational Mandates

    ↩️ Follow the SQS guide's send-back rules

    Score every annotation against the SQS task quality guide and let the score decide where it goes. These send-back rules apply to standard QC. Client audit / Audit_II failures routed to Audit_I use the no-send-back workflow below instead.

    SQS 2 or below — send it back: Return the annotation to the annotator for rework. Don't rescue a task the score says is broken.
    SQS 3 or above — fix and approve: Correct the small issues yourself and move it forward. Approving means vouching that the task is high quality.
    • Be explicit in your feedback: Name what is wrong, which criterion it fails, and exactly what the annotator must change. "Needs work" is not feedback.
    • One issue, one instruction: If there are several problems, list each one so the annotator can work through them in order.
    • Write it so it teaches: The annotator should finish reading your feedback knowing how to avoid the same error next time.

    🔄 Skip vs. unusable workflows

    When an annotator skips an image, it lands in your queue and you have 2 options:

    Image seems usable → send it to Skipped: It will eventually be recirculated to another annotator.
    Image isn't usable → send it to Unusable: It stays there.

    Nothing returns to the original annotator from the Annotator_Skip workflow.

    • Critical action: Delete any existing annotations before routing to Skipped or Unusable.
    • Only annotators can skip an image. Reviewers cannot skip a task with the QualityCheck status.
    • No more hold: Never move an annotator-skipped task to Hold under any circumstance.

    ⏱️ Shadow Task logging (Handshake only)

    Your Handshake AI task is your time-tracking record — never log these in SuperAnnotate. The new Model Stump Check (Stump the Model) in the Handshake AI Shadow Task is currently for annotators only. It is not part of review/QC or audit workflows. Reviewers and auditors do not run it, verify a passing result, or apply its three-wrong-answer threshold. If this changes, there will be an announcement and the playbook will be updated.

    • Annotator-skipped images: Do not submit a Handshake AI task. Reviewers do not log tasks for processing skipped tasks or unusable images, and no time can be claimed for them — whether you route the task to Skipped or Unusable.
    • Standard QC reviews: Log 1 Handshake AI task per individual annotation (i.e., per question on the image), not just one for the overall image.

    ✅ Double-check the Rewrite Answer before submitting

    Rewrite answer correctness is one of the top audit failure modes. Before approving any annotation, verify the Rewrite Answer yourself against the image.

    1. Work the prompt first: Solve the prompt from the image before looking at the annotator's Rewrite Answer, so you don't anchor to their value.
    2. Then compare: Confirm the Rewrite Answer matches what you derived — value, format, units, rounding, and (for MCQs) the bare letter with no extra text.
    3. Standard QC: If the Rewrite Answer is wrong, return the annotation to the annotator and state the correct format and what they got wrong. For an Audit_II failure routed to Audit_I, correct it yourself using the audit-return workflow below.

    🔄 Prompt edits & model regeneration

    If you modify an annotator's prompt, you must trigger a model regeneration immediately.

    • Verify consistency: Always ensure the new model answer is generated from the absolute latest version of your edited prompt.
    • No unchanged retries: If the model answers correctly, do not regenerate with the same prompt to try to get a wrong answer.
    • Follow the SQS limit in standard QC: If the model answers correctly after your prompt edit, assign SQS 2 and return the annotation to the annotator. Do not spend extra time rewriting it to force a failure. This send-back instruction does not apply to Audit_II failures routed to Audit_I; follow the correction or scrapped-task workflow below instead. Prompt changes still require a rerun, and unchanged retries are not allowed.

    ⚖️ Handling client audit / Audit_II failures

    These tasks are currently routed back to Audit_I, where they appear like otherwise normal Audit_I tasks. Always check the Audit_II feedback tab on every Audit_I task to see whether it is an Audit_II failure.

    1. If the Audit_II feedback is incorrect: At the bottom of the Audit_I feedback for the specific prompt, select Yes for Do you want to rebuttal Audit_II feedback? This opens a box for your rebuttal. Clearly and thoroughly explain the issues with the Audit_II feedback, using specific evidence from the prompt, image, answer, and relevant guidance. Then approve and submit as you would any other task passing Audit_I.
    2. If the Audit_II feedback is correct: Try to fix the issue yourself and resubmit using the standard no-send-back flow. Do not return the task to the annotator. After changing a prompt, rerun the model and check the latest answer; do not retry an unchanged prompt after a correct answer.
    3. If you are stuck and the image is truly unusable or unsalvageable: Report it in the task thread in the reviewer/auditor Slack channel. Do not spend too much time on scrapped tasks: scrapped tasks are unpaid because they are not tied to a submitted annotation. Do not log a Handshake AI task for scrapped work; abandon it if you already started one.

    🛠️ Quick actions & escalations

    • Resource / Action

      LLM violations

      Protocol

      If you suspect an annotation is AI-generated, review it as normal — then fill out the LLM Usage Report Form to flag it. Do not skip, reject, or send back the task on suspicion alone.

      Channel / Link

      LLM Usage Report Form ↗

    • Resource / Action

      Audit disagreements

      Protocol

      For incorrect Audit_II feedback, select Yes for Do you want to rebuttal Audit_II feedback? at the bottom of the specific prompt's Audit_I feedback. Explain clearly and thoroughly, then approve and submit.

      Channel / Link

      Per-prompt Audit_I feedback rebuttal section

    • Resource / Action

      Process ideas & feedback

      Protocol

      We want to hear your ideas for improving the review process. Reach out directly or post publicly.

      Channel / Link

      Slack DM Kaz or post in #lizard-v2-reviewers

    • Resource / Action

      Reviewer syncs

      Protocol

      Catch up on recent live calibration sessions and Q&A with DOLs.

      Channel / Link

      Recordings → Reviewer Syncs ↗
    Announcements⭐ Use This: VQA QC & Audit Checklist