Key Terms

    These are the words you'll see in SuperAnnotate, Handshake AI and this guide. Come back here whenever a word is new to you.

    Core terms

    • VQA
      (visual question answering)
      You write a question about an image that the AI has to look at to answer, and the AI gets it wrong.
    • Task
      One pre-loaded image in SuperAnnotate with 1–3 annotations. For annotators: Submit each Handshake AI task (Shadow Task) as soon as its annotation is complete and has passed the Model Stump Check. Only submit the full SuperAnnotate task after all individual annotations are complete and all their Handshake AI tasks are submitted. The new Model Stump Check (Stump the Model) is currently for annotators only and is not part of review/QC or audit workflows. If this changes, there will be an announcement and the playbook will be updated.
    • Annotation
      One question plus its correct answer for the image. A task holds 1 to 3, and each must be clearly different from the others.
    • Prompt
      The text of your question, including what format the answer should be in.
    • MGA
      (model generated answer = the AI's answer)
      The AI's response to your prompt. It must be wrong, and you give it a thumbs-down.
    • Rewrite Answer
      The correct final answer, in exactly the format your prompt asks for, with no explanation.
    • Skills (ontology)
      (thinking skills)
      Tags that name the kind of thinking your question needs, such as comparing or calculating.
    • Question type
      Every annotation is tagged either MCQ (multiple choice) or short answer.
    • Shadow Task
      (your Handshake AI submission form)
      The form on Handshake AI where you log each task so it counts toward your pay.
    • SuperAnnotate
      (the annotation tool)
      The website where you open tasks, write annotations and submit them.
    • Thumbs-down
      The rating you give the AI's wrong answer. Every annotation you submit needs one.
    • Stumping the model
      Getting the AI to answer wrongly. If the AI answers correctly, rewrite your question.

    In SuperAnnotate

    • Annotator_Skip
      The button you use to skip an image that is truly unusable. The task leaves your queue and can't be recovered.
    • Check My Work
      A SuperAnnotate button that flags likely mistakes. You mark each flag Agree or Disagree. It doesn't grade you.
    • Verify Submission
      The final check you run after all annotations are done. It confirms every required item is filled in.
    • Submit_to_QC
      The button that sends your finished task to a reviewer.
    • Returned_to_Annotator_Fail
      A tag on a task a reviewer sent back for fixes. Fix it using the reviewer's feedback.
    • FINISH_REASON_LENGTH
      Shows up when the AI couldn't finish its answer. It counts as the AI getting it wrong.
    • Anchor skill
      Every prompt must use at least one of these three skills: Logical Reasoning, Table/Chart/Graph Understanding or World Knowledge.
    • Enumeration
      The skill tag for counting or listing things. Counts can't go above 50.
    • MCQ
      (multiple choice question)
      A question with the answer choices written in the prompt. Give at least 4 options. Color questions must always be MCQ.
    • Short answer
      A free answer of 1 to 5 words, in the format your prompt asks for.
    • Example answer
      A sample in your prompt that shows the answer format, like "(e.g., 2)". It must never be the real answer.
    • Giveaway
      A prompt that hints at its own answer. It gets rejected.

    Review and quality

    • QC
      (quality control)
      A reviewer checks your submitted task and scores it.
    • SQS
      (Submission Quality Score)
      The 1 to 5 score a reviewer gives your task. 5 means no edits were needed.
    • Thumbs Down rule
      Only submit annotations the AI got wrong. A right AI answer means the task fails.

    People and places

    • Handshake AI
      The platform where you claim work, fill in your Shadow Task and get paid.
    • Fellow
      You. A fellow writes questions and answers in SuperAnnotate.
    • Reviewer
      A senior fellow who checks submitted tasks.
    • SCA
      (Slack Community Ambassador)
      Your first contact on Slack for questions about the project.
    • DOL
      A project lead. Contact a DOL only when an SCA tells you to, or when it's urgent.
    • Bootcamp
      A live Zoom onboarding call for new fellows. Times are on Live Meetings.
    • Live Training Workshop
      A live session where you work through real tasks with project leads. See Live Meetings.
    • Golden Example
      A strong, approved annotation you can copy the approach of. See Golden Examples.
    • Yardstik
      The background check service. You fill it in on Handshake AI.
    • PT
      (Pacific Time)
      The time zone used for all meetings and deadlines.
    Recorded TrainingsAssessment Help