Writing Strong Justifications

    Every submitted Handshake AI task needs a justification that explains why you scored it the way you did. A clear, specific justification is your evidence when a decision is challenged. Anchor your reasoning in the SQS (Task Quality) Guide so the rubric — not personal judgment — backs each score.

    Client audit / Audit_II failures currently return to Audit_I as otherwise normal tasks. Always check the Audit_II feedback tab. If that feedback is incorrect, select Yes for Do you want to rebuttal Audit_II feedback? at the bottom of the Audit_I feedback for the specific prompt. Explain each issue clearly and thoroughly in the rebuttal box, with specific evidence, then approve and submit as a passing Audit_I task. If the feedback is correct, fix the issue yourself and resubmit without sending it back to the annotator. See the Reviewer Playbook for the full workflow.

    Recommended Reviewer Mindset

    Focus on the actual goal of the project when reviewing. The purpose is to:

    • Improve model understanding of charts, tables, and graphs.
    • Avoid pure data extraction (e.g., "What is the title?" or "Count the number of categories").
    • Avoid confusing "puzzle-style" prompts (artificial complexity).
    Applies to Every Submitted Handshake AI Task

    This standard applies to all submitted Handshake AI tasks for completed annotation work — passes, corrections, and standard QC rejects. Truly unusable or unsalvageable audit-return images should be reported in the reviewer/auditor Slack thread. Scrapped tasks are unpaid because they are not tied to a submitted annotation: do not log a Handshake AI task, and abandon any you already started. Do not spend too much time on scrapped work.

    A strong justification does three things:

    • Names the error mode — use the exact category (e.g., Question Unclear, Giveaways, Image un-annotatable), not a vague word like "wrong."
    • Points to the specific problem — quote or locate the part of the prompt, answer, or image that fails.
    • Explains why it fails the rubric
    Good

    Question Unclear: The prompt chained two operations without telling the model what single value to output. I reworded it to ask for one final answer (the industry name) so the target is unambiguous.

    Bad

    "unclear, fixed it"

    Both describe the same edit — but only the strong version names the error mode, points to the specific failure, and ties it to the rubric. The weak one gives an auditor nothing to defend in a rebuttal.

    Templates you can adapt:

    [Original Prompt]

Feedback:
Error 1
Error 2
etc.

[Revised Prompt]
    [Error type]: [what is wrong] in [where]. I [edit made] so it now [meets which criterion].

Un-annotatable: Image un-annotatable — [why, e.g. diffuse overlapping color with no clear quadrant], so no reliable answer exists from the image. Routed to Unusable; no Handshake AI task submitted.
    Handle LLM Judge FeedbackReviewer Simulation