Announcements

    1. — Answer ranges and tolerances are now allowed for chartography tasks only

      New: Chartography tasks may use answer ranges or tolerances. Check the task's category to confirm its type. If the category does not say chartography, this exception does not apply—even if the image looks like a chart.

      For a chartography task, if a number is not printed on the chart and you estimate it from the gridlines, give your estimated answer plus the range you would accept. State the accepted range or tolerance clearly in the prompt. The model's wrong answer must fall outside that accepted range; an answer within the range is not a stump.

      Example prompt:

      In the stress-strain chart, what stress does the Alloy B curve reach at 0.6% strain? Round to the nearest 5 MPa; estimates within ±10 MPa are acceptable. Answer as <number> MPa (e.g., 245 MPa).

      If your estimated answer is 245 MPa, the accepted range is 235–255 MPa. A model answer of 250 MPa is within the range and is not wrong; 270 MPa is outside it.

      All non-chartography tasks retain the previous guidance: use one single, clear, verifiable final answer. Do not use ranges or tolerances that allow multiple accepted answer values. This announcement replaces the previous blanket restriction on answer ranges only for chartography tasks.

    2. — 📣 Update to the Shadow Task flow: new stump check before you submit

      Hi everyone,

      We’ve updated the Shadow Task flow. A new automated check now runs before you can submit, so please read the steps below before your next annotation.

      Currently for annotators only: The new Model Stump Check (Stump the Model) in the Shadow Task is not part of review/QC or audit workflows. Reviewers and auditors do not run it, verify a passing result, or apply its three-wrong-answer threshold. If this changes, there will be an announcement and the playbook will be updated.

      What’s changing

      Before you submit your annotation/task in SuperAnnotate, paste your prompt, answer, and image into the Shadow Task. They must match what’s in SuperAnnotate exactly.

      Click Continue once you’ve confirmed they match.

      The “Model Stump Check” will run. It tests whether your prompt is difficult enough to stump the model.

      What the result means

      The model gets it wrong 3 or more times: Your prompt is difficult enough. Submit the Shadow Task, then submit the task in SuperAnnotate.

      The model gets it wrong fewer than 3 times: Your prompt didn’t stump it enough. Revise your prompt and try again.

      Watch the updated Logging a Shadow Task video on Recorded Trainings for a walkthrough of the new flow.

    3. — Client audit failures stay at Audit_I

      New: All Audit_II / client audit failures must be handled strictly at the Audit_I layer. Never return them to QC or to the annotator. These tasks appear like normal Audit_I tasks, so always check the Audit_II feedback tab.

      There are three outcomes at Audit_I:

      • Rebut incorrect feedback: In the specific prompt's Audit_I feedback, select Yes for Do you want to rebuttal Audit_II feedback? Explain clearly and thoroughly why the feedback is incorrect, then approve and submit.
      • Fix and approve: If the feedback is correct, fix the issue yourself, then approve and resubmit without sending the task back. If you change the prompt, rerun the model and check the latest answer.
      • Scrap unsalvageable work: If the image is truly unusable or the task cannot be salvaged, report it in the reviewer/auditor Slack thread. Scrapped tasks are unpaid: do not log a Handshake AI task, and abandon any you already started. Do not spend too much time on them.

      See the updated Reviewer Playbook: Handling client audit / Audit_II failures for the full procedure.

    4. — Action needed: write prompts the model almost never gets right

      Post checks are being run on the prompts we deliver, and too often the model is getting them right too many times. This is not acceptable. Many prompts are scoring 16/16, meaning the model got the correct answer every time.

      Our target: the model is right only 1 or 2 times out of 16.

      Aim for most of your prompts to be short answer

      The model gets multiple choice right much more often, because four options make it easy to land on the right one. Short answer questions, where the answer has to be worked out and written exactly, are much harder for it to get right.

      What the model tends to miss

      • Looking closely. Counting points, lines or connections that meet a condition, and tracking where things line up across a chart.
      • Longer is not harder. Adding more steps or making the question longer does not make it harder for the model.

      Important: generate once

      • Your prompt should stump the model the first time you generate.
      • If the model answers correctly, do not rerun the same prompt. Regenerating an unchanged prompt until the model misses does not make it a hard prompt.
      • Revise the prompt first, then generate again.

      What to do

      1. Aim for a prompt the model almost always misses, not one it misses once. A single miss can be a fluke.
      2. Make most of your prompts short answer instead of multiple choice.
      3. Build the question around something the model has to look at closely. Count items that meet a condition, compare positions across parts of the chart, or combine two details into one exact answer.
      4. If the model gets it right, rewrite the question to make it harder. Do not rerun it unchanged.
      5. If you cannot make a hard question from an image, mark it unusable and skip it. That is expected.

      Please make sure to read this. Failure to manage how many times you regenerate the same prompt may result in offboarding.

    5. — Workflow Update: Unusable Tasks

      New: Skip too-easy images early. Too-easy images belong in Unusable, not back in the rework loop.

      Annotators

      If the image is too easy or unstumpable — the model gets it right with no real effort — do not annotate it. Instead:

      • Type unusable in the prompt section.
      • Select Annotator_Skip in the top right corner of the task in SuperAnnotate.

      Fully annotating a task that can't be used wastes your time, and unusable tasks aren't paid. Skipping early saves that effort and keeps these tasks out of the rework loop.

      This is expected behavior, not a mark against you. If you're unsure whether a task qualifies, ask in #lizard-v3-tasking.

      Reviewers

      If a task has issues and the image is too easy — it will be challenging to stump the model — move it to Unusable. Do not send it back for rework: that sends the same task back to fellows, who then spend time on something that can't be used.

      Annotators are now being asked to skip too-easy tasks with unusable in the prompt section, so you may see that on tasks you review. If you agree the image is too easy, move it to Unusable. If the issues are fixable and the image is a fair challenge, review it as normal.

      • Every task moved to Unusable must have a taxonomy comment.
      • Too-easy images go to Unusable, not back for rework.

    6. — Use Short Answer for most annotations

      New: Make Short Answer your default. Across submitted annotations, aim for about 90% Short Answer and 10% MCQ.

      When the image supports a numerical answer, ask for the number directly instead of listing choices. Text short answers are still 1–5 words. Color questions still require MCQ, and every MCQ must have at least four valid options.

      See Step 2: Answer types and Prompt Writing Strategies for the updated guidance.

    7. — New: recirculated task claim limits

      Updated November 3, 2026: Newly claimed tasks have 4 hours from claim for Annotators, Reviewers/QC, and Auditors. Returned tasks have 12 hours from return before being recirculated. This replaces the previous returned-task window; the 12-hour limit now applies regardless of how close the batch is to closing. Finish and submit within the applicable window; once it expires, the task can be pulled from your queue. See Task Access FAQ for the current limits.

    8. — Reviewers: send tasks back, scored against the SQS guide

      Send-backs are allowed again, and the SQS (Task Quality) Guide decides which way a task goes.

      • Let the score decide. SQS 2 or below goes back to the annotator. SQS 3 or above moves forward.
      • Moving a task forward means vouching for it. If you score 3+ and approve, the task must be genuinely high quality — fix the small things yourself before you pass it on.
      • Send it back at 2 or below. Don't rescue a task the score says is broken.
      • Be clear in your feedback. Name the problem, the criterion it fails, and exactly what the annotator must change — for every issue on the task.
      • Audit II returns (Returned_to_QC). When an audit sends a task back it returns to the review layer marked Returned_to_QC. Pick it up, apply the audit's reasoning, and resubmit.
      • Annotators: expect returns. A task can come back to you. Read the reviewer's feedback in the SuperAnnotate QC box, make the named change, and resubmit.

      Every page that previously said corrections happen in one pass with no send-backs has been updated: the Reviewer Playbook, Reviewer Simulation, Writing Strong Justifications, Glossary, After Your Review, and the common-errors criteria pages.

    Live Meetings + SlackReviewer Playbook