Use This: VQA QC & Audit Checklist

Every "What to do" follows the SQS guide send-back rules: SQS 2 or below goes back to the annotator with feedback; SQS 3 or 4 you fix and approve. Use this checklist every time you review or audit a task. It covers every item you should run through before you submit your review. When you write feedback for the annotator, name the failure mode (for example, "A7 Missing example answer") so they know exactly what to fix.

Audit_II return exception: Client audit / Audit_II failures routed to Audit_I use a no-send-back flow, rather than the standard QC send-back rules below. Always check the Audit_II feedback tab. Rebut incorrect feedback in the specific prompt's Audit_I feedback, or fix correct feedback yourself and resubmit. See the Reviewer Playbook for the full procedure and unpaid scrapped-task guidance.

Initial Checks

✓	Failure mode	Review / Audit question (no = issue)	What to do
G1	Image not clear	Is the image clearly visible? No blur, pixelation, or low resolution, and watermarks don't obscure a subject of the prompt (faint watermarks are fine). It should not take more than one zoom-in to find the answer.	Route to Unusable (SQS 2). Delete any annotations first.
G2	Image not workable	Does the image contain enough content for a meaningful Q&A? Reject generic photos, videos, movie posters, overly simple scenes, inappropriate or offensive material, sideways images (slanted or tilted is fine if clear), and photos of a screen or graph inside a wider generic photo.	Route to Unusable (SQS 2). Delete any annotations first.
G3	Wrong model rating	Is the Model Rating set to Thumbs Down?	If the model answer is wrong but not rated Thumbs Down, change the rating to Thumbs Down and approve (SQS 4) if there are no other issues. If the model answer is correct but rated as wrong, change the rating to Thumbs Up and treat it as a G5 Model correct error.
G4	Empty model response	Is the model-generated answer box non-empty?	Run the model once. If the model is stumped, no send-back is needed: fix and approve (SQS 3). If it answers correctly, send back (SQS 2) and tell the annotator the model response box was empty and the model answered correctly when run.
G5	Model correct	Is the model response incorrect? A "FINISH_REASON_LENGTH" error counts as incorrect. The model must fail beyond formatting — if the answer is "2, 3" and the model says "ANSWER: 2, 3", the model is correct and the task fails.	Send back (SQS 2): if the model response is correct, the task will need to be redone. Do not rerun the unchanged prompt to try to get a wrong answer. Return it to the annotator with feedback.

A. Prompt Quality

✓	Failure mode	Review / Audit question (no = issue)	What to do
A1	Question unclear	Is the prompt clearly written with one interpretation and one verifiable answer? If two people could disagree on what's asked, it's too ambiguous. Multi-panel images must name the panel. Answer variance or tolerance ranges such as “any value within 5% is acceptable” are unacceptable because they create multiple accepted final answers, even with whole-percent rounding. Chartography exception: if the task category says Chartography, an estimated gridline value may use a stated accepted range or tolerance; the model's wrong answer must fall outside it. See Announcements.	Fix and approve (SQS 3): reword the prompt to one interpretation; name the panel if multi-panel. If the intent can't be recovered, send back (SQS 2).
A1	Chart trend ambiguity	For line or time-series charts, does the question ask about a specific value or comparison rather than an overall trend? Prompts with more than one possible answer fail.	Fix and approve (SQS 3): ask about a specific value or comparison instead of the trend. If that changes what the question tests, send back (SQS 2).
A2	No image dependency	Is the prompt tied to the visual content, so it can't be answered without the image (not from the title or caption alone)?	Send back (SQS 2): tell the annotator to rewrite so the answer requires reading the image. If the image is too simple or otherwise unusable, route to Unusable (SQS 2) instead of sending it back. A poor image is a common reason a prompt barely depends on the image.
A3	Not standalone	Is the prompt fully standalone, with no reliance on a "previous question/answer" or other annotations on the task?	Send back (SQS 2): tell the annotator to remove references to other questions or annotations. The prompt being referenced is likely fine to leave as is. Only the prompt that references it needs to change.
A4	Giveaway	Does the prompt avoid pointing so directly at the answer that visual reasoning disappears? An answer key in the image must not reveal the exact Rewrite answer. The example answer must differ from the Rewrite answer.	Send back (SQS 2): tell the annotator to remove the hint and make sure the example answer differs from the Rewrite answer.
A5	Trivial question	Does the question require multi-step reasoning or interpretation, not surface-level extraction (or extraction plus basic arithmetic)? Triviality can take several forms (one-step questions, answers from the title or a printed label, extraction plus one subtraction, and more). See Trivial Questions for the full list.	Send back (SQS 2): tell the annotator which Trivial Questions pattern applies and to rework it so it needs at least 2 reasoning steps.
A5	Too few options	Does the question have at least 4 real or implied answer options? Binary and true/false questions are not allowed.	Send back (SQS 2): tell the annotator to expand to at least 4 options and remove binary framing.
A5	Catch-all options	Does the question avoid "none of the above," "all of the above," and "cannot be determined" as options? Zero is an acceptable answer.	Fix and approve (SQS 3): replace catch-all options with real answers.
A6	Unnecessarily long	Does the question avoid unnecessary procedural steps and performative complexity? No more than 5 steps.	Fix and approve (SQS 3): cut steps down to 5 or fewer.
A6	Excessive enumeration	Does the question avoid excessive enumeration and math? No counting over 50 items — count only objects the model would consider based on the prompt. Counting by multiplication (e.g., a 10x10 grid) is fine.	Fix and approve (SQS 3): narrow the count to 50 items or fewer. If that changes the answer and un-stumps the model, send back (SQS 2).
A7	Ambiguous answer format	Does the prompt specify the exact answer format — string, percent, integer, MCQ, lowercase, comma separators, whether to include the percent or dollar sign? Percentages must say "answer as a percentage" (99.3 vs. 0.9930).	Fix and approve (SQS 3): state the exact answer format in the prompt. Because the prompt was edited, rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.
A7	Missing example answer	Is there an example answer? For multi-item answers, does it specify the exact ordering (e.g., alphabetical), and are multiple values asked for explicitly (commas only when multiple items)?	Fix and approve (SQS 3): add an example answer with ordering and separators. Because the prompt was edited, rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.
A7	Invalid example answer	Is the example answer one of the MCQ options or listed short-answer options? If options are A–D, the example can't be E.	Fix and approve (SQS 3): change the example to a valid option. Because the prompt was edited, rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.
A7	Special characters	Does the Rewrite answer avoid special characters (e.g., Greek letters, accent marks)?	Fix and approve (SQS 4): replace special characters with plain text. If this changes the prompt, it becomes SQS 3: rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.
A8	Example format	Is the example answer formatted exactly as "(e.g., X)" — no missing periods, commas, or parentheses?	Fix and approve (SQS 3): fix to "(e.g., X)" and note it. Because the prompt was edited, rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.
A9	Fine-grained precision	Does the prompt avoid subjective or unmeasurable spatial precision? Prefer Short Answer with clear approximation wording. If MCQ is needed, use clearly distinct options; the overall target is about 90% Short Answer / 10% MCQ.	Fix and approve (SQS 3): clarify the approximation while keeping Short Answer where possible, or use distinct MCQ options if needed. Any prompt change requires a model rerun; if it answers correctly, assign SQS 2 and send back.
A9	Color naming	Is color handled as an MCQ, with specific shade names where needed (e.g., "light blue" vs. "dark blue")? Color is never a short answer.	Fix and approve (SQS 3): convert to MCQ with specific shade names.
A10	First-person phrasing	Is the question written without first-person phrasing (e.g., "I want you to…", "Let's look…")?	Fix and approve (SQS 3): rephrase neutrally and note it. Because the prompt was edited, rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.
A11	Second-person phrasing	Is the question written without second-person phrasing (e.g., "You are looking at…", "Provide your answer…")?	Fix and approve (SQS 3): rephrase neutrally and note it. Because the prompt was edited, rerun the model and make sure it is still stumped. If it answers correctly, downgrade to SQS 2 and send back.

B. Answer Quality

✓	Failure mode	Review / Audit question (no = issue)	What to do
B1	Answer incorrect	Is the rewritten answer factually correct, and does it match the example answer's format 1:1? Different spacing or capitalization makes it incorrect (e.g., "4, 2" vs. "4,2").	Fix and approve (SQS 3): correct the answer and match the example format 1:1, if the question is still valid. SQS 4: if only the Rewrite Answer needs changing (no prompt tweaks) and the model answer is still incorrect, it may be an SQS 4. The model doesn't need a rerun, so it's a quick, low-risk fix. If the question or image is flawed, send back (SQS 2).
B2	Final answer only	Does the Answer field contain ONLY the final answer, with no reasoning, explanation, or commentary?	Fix and approve (SQS 3): delete everything except the final answer.
B3	Answer length	If it's a short answer, is it 1–5 words? Comma- or hyphen-separated items count as words (e.g., "A-1, B-4, C-2, D-5, E-3" is 5 words).	Fix and approve (SQS 4): shorten to 1–5 words, if no prompt changes are needed. SQS 3: if the prompt must change to allow a shorter answer, edit it, rerun the model, and make sure it still stumps. If the model now answers correctly, downgrade to SQS 2 and send back.
B4	Derivable answer	Is the answer derivable from the image — do the needed values, shapes, and labels exist in it?	Send back (SQS 2): tell the annotator to rewrite so the answer comes from the image.

C. Category Specific

✓	Failure mode	Review / Audit question (no = issue)	What to do
C1	Multi-step format	If the prompt has multiple steps, is it a clear sequence (Command 1 → Command 2 → Final Question), not unrelated separate questions? A question mark isn't required.	Fix and approve (SQS 3): restructure into one clear sequence ending in one question.
C2	MCQ format	Do options use strict A./B./C./D. format (letter + period, each on its own line — not "A)" or "(A)")? Is the Rewrite answer the bare letter only? Are there at least 4 options, each 1–5 words?	Fix and approve (SQS 3): reformat to A./B./C./D. lines; set the answer to the bare letter. This will likely require prompt changes, so rerun the model and make sure it still stumps. If the model now answers correctly, downgrade to SQS 2 and send back.
C3	Numeric rounding	If the answer needs rounding, is the rule stated in the prompt and does the answer match it? Is intermediate rounding specified for multi-step arithmetic? Rounding must yield one verifiable final answer. Do not allow variance or tolerance ranges that accept multiple final values. Chartography exception: if the task category says Chartography, an estimated gridline value may use a stated accepted range or tolerance; the model's wrong answer must fall outside it. See Announcements.	Fix and approve (SQS 3): state the rounding rule and fix the answer to match. This will likely require prompt changes, so rerun the model and make sure it still stumps. If the model now answers correctly, downgrade to SQS 2 and send back.
C4	Verbatim label text	Do label, category, or legend prompts and answers use the exact chart text — no paraphrase, abbreviation, or different capitalization?	Fix and approve (SQS 3): match the chart text exactly and note it. This will likely require prompt changes, so rerun the model and make sure it still stumps. If the model now answers correctly, downgrade to SQS 2 and send back.

D. Structure, Skills, and Hygiene

✓	Failure mode	Review / Audit question (no = issue)	What to do
D1	Similar annotations	If a task has multiple prompts, is each unique — different skill combinations or entirely different visual components? Pass the first annotation; fail only later ones that are too similar.	The first of the similar annotations can be left as is. Send back (SQS 2): tell the annotator to rewrite only the later annotations to test different skills or components.
D2	Single Q&A pair	Does each annotation contain exactly one Question and one Answer? Multi-step is fine, with one final question and answer.	Fix and approve (SQS 3): trim to one question and one answer. If it can't be trimmed to one, send back (SQS 2).
D4	Explanation field	Is the Explanation field empty?	Fix and approve (SQS 4): clear the Explanation field and note it.
D5	Question type	Is the correct Question Type selected (MCQ or short answer)?	Fix and approve (SQS 4): select the correct type and note it.
D6	Mechanical writing	Are the prompt and answer free of grammar mistakes, typos, and misspellings? If an error changes the meaning, flag it as a Major ambiguity error.	Fix and approve (SQS 4): fix the errors and note it, only if the fix doesn't require rerunning the model (e.g., errors in the Rewrite Answer only). SQS 3: if the errors are in the prompt, any change means rerunning the model and making sure it still stumps. If the model now answers correctly, downgrade to SQS 2 and send back.
D7	Required skill	Is at least one core skill tagged — Logical Reasoning, Table/Chart/Graph Understanding, or World Knowledge?	Send back (SQS 2): skill requirements not met; tell the annotator to tag at least one core skill.
D8	Wrong skill count	Are at least 2 skills tagged, and at least 3 if Enumeration is used?	Send back (SQS 2): skill requirements not met; tell the annotator to add skills to meet the minimum.
D9	Inaccurate skills	Do the tagged skills accurately reflect the reasoning the question requires?	Fix and approve (SQS 4): correct the tags and note it.
D10	Unit ambiguity	Are the question and answer free of ambiguous units (e.g., M vs. m, or M/K for both magnitude and physical units)?	Fix and approve (SQS 3): clarify the units in the prompt and note it. This is a prompt change, so rerun the model and make sure it still stumps. If the model now answers correctly, downgrade to SQS 2 and send back.

Not sure what to do?

If you're ever unsure how to handle a task, ask in Slack or bring it to Office Hours. You can always get a second opinion before you decide.

Reviewer Playbook SQS (Task Quality) Guide