Annotator Pre-submission Checklist
Run through this checklist before you submit every task. It mirrors the checklist reviewers and auditors use, so every "no" here is an issue they will catch. Fix it now and your task moves forward instead of coming back to you.
Initial Checks
| ✓ | Failure mode | Self-check question (no = fix before submitting) | What to do |
|---|---|---|---|
| Image not clear | Is the image clearly visible? No blur, pixelation, or low resolution, and watermarks don't obscure a subject of the prompt (faint watermarks are fine). It should not take more than one zoom-in to find the answer. | Skip the image or mark it unusable. Don't write a prompt for it. | |
| Image not workable | Does the image contain enough content for a meaningful Q&A? Reject generic photos, videos, movie posters, overly simple scenes, inappropriate or offensive material, sideways images (slanted or tilted is fine if clear), and photos of a screen or graph inside a wider generic photo. | Skip the image or mark it unusable. Don't write a prompt for it. | |
| Wrong model rating | Is the Model Rating set to Thumbs Down? | If the model's answer is wrong, set the rating to Thumbs Down. If the model's answer is correct, the prompt needs redoing (see G5). | |
| Empty model response | Is the model-generated answer box non-empty? | Rerun the model once. If it still gets the answer wrong, you're fine. If it answers correctly, rewrite the prompt so it's harder. | |
| Model correct | Is the model response incorrect? A "FINISH_REASON_LENGTH" error counts as incorrect. The model must fail beyond formatting. If the answer is "2, 3" and the model says "ANSWER: 2, 3", the model is correct and the task fails. | The task has to be redone. If the model answers correctly, do not rerun it with the same prompt. Change the prompt before generating another answer. |
A. Prompt Quality
| ✓ | Failure mode | Self-check question (no = fix before submitting) | What to do |
|---|---|---|---|
| Question unclear | Is the prompt clearly written with one interpretation and one answer anyone can check? If two people could disagree on what's asked, it's too ambiguous. Multi-panel images must name the panel. Answer variance or tolerance ranges such as “any value within 5% is acceptable” are unacceptable because they create multiple accepted final answers, even with whole-percent rounding. Chartography exception: if the task category says Chartography, an estimated gridline value may use a stated accepted range or tolerance; the model's wrong answer must fall outside it. See Announcements. | Reword the prompt so it has one interpretation; name the panel if it's a multi-panel image. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Chart trend ambiguity | For line or time-series charts, does the question ask about a specific value or comparison rather than an overall trend? Prompts with more than one possible answer fail. | Ask about a specific value or comparison instead of the trend. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| No image dependency | Is the prompt tied to the visual content, so it can't be answered without the image (not from the title or caption alone)? | Rewrite so the answer requires reading the image. If the image is too simple to support a question that depends on it, skip it or mark it unusable. | |
| Not standalone | Is the prompt fully standalone, with no reliance on a "previous question/answer" or other annotations on the task? | Remove references to other questions or annotations. The prompt being referenced is likely fine as is. Only rewrite the prompt that refers to it. | |
| Giveaway | Does the prompt avoid pointing so directly at the answer that visual reasoning disappears? An answer key in the image must not reveal the exact Rewrite answer. The example answer must differ from the Rewrite answer. | Remove the hint and make sure the example answer differs from the Rewrite Answer. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Trivial question | Does the question require multi-step reasoning or interpretation, not surface-level extraction (or extraction plus basic arithmetic)? Triviality can take several forms (one-step questions, answers from the title or a printed label, extraction plus one subtraction, and more). See Trivial Questions for the full list. | Check which Trivial Questions pattern applies and rework the question so it needs at least 2 reasoning steps. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Too few options | Does the question have at least 4 real or implied answer options? Binary and true/false questions are not allowed. | Expand to at least 4 options and remove binary framing. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Catch-all options | Does the question avoid "none of the above," "all of the above," and "cannot be determined" as options? Zero is an acceptable answer. | Replace catch-all options with real answers. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Unnecessarily long | Does the question avoid unnecessary procedural steps and performative complexity? No more than 5 steps. | Cut the steps down to 5 or fewer. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Excessive enumeration | Does the question avoid excessive enumeration and math? No counting over 50 items. Count only objects the model would consider based on the prompt. Counting by multiplication (e.g., a 10x10 grid) is fine. | Narrow the count to 50 items or fewer. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Ambiguous answer format | Does the prompt specify the exact answer format. String, percent, integer, MCQ, lowercase, comma separators, whether to include the percent or dollar sign? Percentages must say "answer as a percentage" (99.3 vs. 0.9930). | State the exact answer format in the prompt. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Missing example answer | Is there an example answer? For multi-item answers, does it specify the exact ordering (e.g., alphabetical), and are multiple values asked for explicitly (commas only when multiple items)? | Add an example answer showing ordering and separators. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Invalid example answer | Is the example answer one of the MCQ options or listed short-answer options? If options are A–D, the example can't be E. | Change the example to a valid option. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Special characters | Does the Rewrite answer avoid special characters (e.g., Greek letters, accent marks)? | Replace special characters with plain text. If it's only in the Rewrite Answer, no rerun is needed; if you change the prompt, rerun the model and confirm it still gets the answer wrong. | |
| Example format | Is the example answer formatted exactly as "(e.g., X)". No missing periods, commas, or parentheses? | Fix the example to "(e.g., X)". Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Fine-grained precision | Does the prompt avoid subjective or unmeasurable spatial precision? Prefer Short Answer with clear approximation wording. If MCQ is needed, use clearly distinct options; the overall target is about 90% Short Answer / 10% MCQ. | Clarify the approximation while keeping Short Answer where possible. Use clearly distinct MCQ options only if needed. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Color naming | Is color handled as an MCQ, with specific shade names where needed (e.g., "light blue" vs. "dark blue")? Color is never a short answer. | Convert to MCQ with specific shade names. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| First-person phrasing | Is the question written without first-person phrasing (e.g., "I want you to…", "Let's look…")? | Rephrase neutrally. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Second-person phrasing | Is the question written without second-person phrasing (e.g., "You are looking at…", "Provide your answer…")? | Rephrase neutrally. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. |
B. Answer Quality
| ✓ | Failure mode | Self-check question (no = fix before submitting) | What to do |
|---|---|---|---|
| Answer incorrect | Is the rewritten answer factually correct, and does it match the example answer's format 1:1? Different spacing or capitalization makes it incorrect (e.g., "4, 2" vs. "4,2"). | Correct the Rewrite Answer and match the example format 1:1. You don't need to rerun the model. | |
| Final answer only | Does the Answer field contain ONLY the final answer, with no reasoning, explanation, or commentary? | Delete everything except the final answer. | |
| Answer length | If it's a short answer, is it 1–5 words? Comma- or hyphen-separated items count as words (e.g., "A-1, B-4, C-2, D-5, E-3" is 5 words). | Shorten to 1–5 words. If only the Rewrite Answer changes, no rerun is needed. If the prompt must change, rerun the model and confirm it still gets the answer wrong. | |
| Derivable answer | Is the answer derivable from the image. Do the needed values, shapes, and labels exist in it? | Rewrite so the answer comes from the image. |
C. Category Specific
| ✓ | Failure mode | Self-check question (no = fix before submitting) | What to do |
|---|---|---|---|
| Multi-step format | If the prompt has multiple steps, is it a clear sequence (Command 1 → Command 2 → Final Question), not unrelated separate questions? A question mark isn't required. | Restructure into one clear sequence ending in one question. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| MCQ format | Do options use strict A./B./C./D. format (letter + period, each on its own line. Not "A)" or "(A)")? Is the Rewrite answer the bare letter only? Are there at least 4 options, each 1–5 words? | Reformat to A./B./C./D. lines and set the answer to the bare letter. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Numeric rounding | If the answer needs rounding, is the rule stated in the prompt and does the answer match it? Is intermediate rounding specified for multi-step arithmetic? Rounding must yield one verifiable final answer. Do not allow variance or tolerance ranges that accept multiple final values. Chartography exception: if the task category says Chartography, an estimated gridline value may use a stated accepted range or tolerance; the model's wrong answer must fall outside it. See Announcements. | State the rounding rule in the prompt and make the answer match it. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Verbatim label text | Do label, category, or legend prompts and answers use the exact chart text. No paraphrase, abbreviation, or different capitalization? | Match the chart text exactly in the prompt and answer. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. |
D. Structure, Skills, and Hygiene
| ✓ | Failure mode | Self-check question (no = fix before submitting) | What to do |
|---|---|---|---|
| Similar annotations | If a task has multiple prompts, is each unique. Different skill combinations or entirely different visual components? Pass the first annotation; fail only later ones that are too similar. | Keep the first annotation as is. Rewrite only the later ones so they test different skills or parts of the image. | |
| Single Q&A pair | Does each annotation contain exactly one Question and one Answer? Multi-step is fine, with one final question and answer. | Trim to one question and one answer. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. | |
| Explanation field | Is the Explanation field empty? | Clear the Explanation field. | |
| Question type | Is the correct Question Type selected (MCQ or short answer)? | Select the correct question type. | |
| Mechanical writing | Are the prompt and answer free of grammar mistakes, typos, and misspellings? If an error changes the meaning, flag it as a Major ambiguity error. | Fix the errors. If they're only in the Rewrite Answer, no rerun is needed. If you change the prompt, rerun the model and confirm it still gets the answer wrong. | |
| Required skill | Is at least one core skill tagged. Logical Reasoning, Table/Chart/Graph Understanding, or World Knowledge? | Tag at least one core skill the question actually uses. | |
| Wrong skill count | Are at least 2 skills tagged, and at least 3 if Enumeration is used? | Add skills until you meet the minimum. | |
| Inaccurate skills | Do the tagged skills accurately reflect the reasoning the question requires? | Correct the skill tags so they match what the question requires. | |
| Unit ambiguity | Are the question and answer free of ambiguous units (e.g., M vs. m, or M/K for both magnitude and physical units)? | Clarify the units in the prompt. Because you changed the prompt, rerun the model and confirm it still gets the answer wrong. If it now answers correctly, write a harder question. |
Not sure?
If you're ever unsure how to handle a task, ask in Slack or bring it to Office Hours. You can always get a second opinion before you decide.