SQS (Task Quality) Guide

    This guide explains how tasks are evaluated and graded. Each quality level is defined by the types of issues present. Select a score to see the full rubric and a real annotated example.

    **Audit_II return exception:** The send-back instructions in this rubric apply to standard QC. Client audit / `Audit_II` failures currently return to `Audit_I` and use a **no-send-back flow**: check the `Audit_II` feedback tab, rebut incorrect feedback or fix correct feedback yourself. Report truly unsalvageable images in the reviewer/auditor Slack thread; scrapped tasks are unpaid and must not have a submitted Handshake AI task. See [Handling client audit / Audit_II failures](/reviewer-playbook#mandate-6) for the full procedure.

    Where to Find Your Scores

    Quality scores can be found on the Project Lizard Dashboard on the Handshake AI website. To find them, access the Handshake AI Work Fellow Dashboard, navigate to the Project Lizard v2 card, and click the View Project button. The Quality Scores will be summarized in a bar above the My tasks section.

    Handshake AI Work Fellow Dashboard with the View Project button highlighted on the Project Lizard v2 cardQuality score summary bar above the My tasks section on the Project Lizard dashboard

    Error types are a guide, not a firm rule

    The Error Type Examples under each score are typical cases, not fixed assignments. What matters is how much you need to change to make the task usable. An Ambiguous Answer Format fix can leave the model answering correctly, so the task ends up at SQS 2. Skill requirements can be met but untagged, so fixing the tags with no prompt change gives SQS 4. Score the degree of change, not the error type.

    Quality score rubric

    SQS 5 — Excellent

    PASS, deliver as is

    Approve and deliver as is. Zero edits of any kind, including tags and metadata — no wording changes, no formatting cleanup, no category, ontology, or question-type corrections. If you touch anything, score it 4 or lower.

    Error type examples

    None — zero edits required

    Issues

    0issues

    Example 1

    Report_Dashboard_Server_Dashboard_136
    Grafana server monitoring dashboard

    Prompt

    In the Network Traffic graph, focus on the "eth0 transmitted" line. Treat a plateau as a stretch where the line stays at roughly the same level, avoiding large spikes or drops. Among the labeled x-axis intervals, which best represents the longest relatively stable plateau? Answer with a single letter (E.g., C)
    • AA. 08:15–08:25
    • BB. 08:15–08:20
    • CC. 08:30–08:35
    • DD. 08:05–08:10

    Answer

    B

    Why it scores 5

    Image-essential, multi-skill (locate the correct series, judge stability across intervals), explicit MCQ format, no giveaway, cleanly written.

    How to fix it

    None. Approve and deliver as-is.

    ⭐ Use This: VQA QC & Audit ChecklistReviewer Flow