These research-backed approaches add complexity and make questions more likely to stump the model. Don't force a strategy onto an image that can't support it — a clean, high-quality annotation always comes first.
1
1. Hypotheticals
A hypothetical serves three purposes:
Image lacks information for a multilayer prompt? A hypothetical can introduce more.
Struggling with the mandatory ontologies? A hypothetical is a clean way to introduce logic.
Struggling with layers of reasoning? A hypothetical is an easy second layer beyond the base question.
2
2. Good image types
Spotting a strong image quickly helps you skip low-quality images fast. Image types that pro-annotators love:
Stacked line charts — the model struggles with these in particular.
Multiple charts that share a series or axis — easy to draw comparisons between and across them.
Dashboards with statistics tied to visible charts — easy to pose questions that combine different parts of the image and information in different formats.
3
3. Elements to focus on
Once you have a good image, decide which parts are worth narrowing in on. Good ones:
Row
Column
Panel title
Legend color
Nearest labeled tick
Section heading
Grid lines
These are almost always relevant to the image, so combining two or more of them with a comparison often does the trick. They're usually unambiguous — watch for edge cases like similar colors.
4
4. Question types
Make most prompts ask for a numerical answer when the image supports it. Ask for that number directly using Short Answer. Across submissions, aim for about 90% Short Answer and 10% MCQ; a chart label can also be a short answer when it follows the answer-format rules.
No one size fits all, but pro-annotators often turn to:
1Comparing trends in charts (instead of values directly)
2Greatest increase or decrease in a graph, or among several
3Locating intersections between lines
4Determining traits that apply to specific intervals in a graph, or comparing among intervals
Bonus tip: Finding the greatest or least is usually easy for the model. Ask it to rank several things and take the second most or second smallest for a better chance at stumping.
5
5. Rephrasing
Keep prompts clear and concise, but there's almost always more than one way to phrase a question. If a question is strong but isn't stumping the model, change the wording — as long as the new version is still unambiguous. Keep Short Answer as the default. The MCQ example below shows how to adjust options for the roughly 10% of submissions that use MCQ, not a reason to switch formats.
Prompt Writing Strategies by Image Type
Here are some useful frameworks that help in generating questions. These types of strategies add complexity and increase the likelihood of stumping the model.
These strategies target clear, repeatable chart benchmark failures. The benchmark images are examples only. Use a strategy only when the new image supports one clear, independently verifiable answer. For multiple questions on one image, use a different failure family.
Use these strategies for exact labels, cross-panel questions, strict cutoffs, boundaries, and line crossings.
Use exact legend names
Use when:
The chart has similar labels and the model may mix them up.
How:
Ask for one label at a clear point. Require the exact text from the legend.
Benchmark example
Question:At epoch 0, which algorithm has the highest average caching reward? Answer with the algorithm name exactly as shown in the legend (e.g., Algorithm A).
Find one answer across two panels
Use when:
The answer must meet one condition in each of two panels.
How:
Name both panels and say that the same item must meet both conditions. Ask for one answer.
Benchmark example
Question:Across the Frappe and MovieLens panels, find the same n_e value that has the lowest loss in Frappe and the second-lowest AUC in MovieLens. Answer with one whole number exactly as shown in the legend (e.g., 20).
Count items above a strict cutoff
Use when:
The task depends on words such as over, under, or at least.
How:
Name the panel and cutoff. Say whether an item counts if it crosses the cutoff at any point.
Benchmark example
Question:In the left K400 panel, count the models that rise strictly above 80% Top-1 Accuracy at least once. A value equal to 80% does not count. Answer with one whole number (e.g., 4).
Count items inside or outside a boundary
Use when:
Points or shapes sit close to a contour or marked area.
How:
Say whether to use the point center or the whole marker. Also say whether touching the line counts as inside.
Benchmark example
Question:Count the blue point centers that lie outside all colored contour areas. A point center on a contour boundary counts as inside. Answer with one whole number (e.g., 2).
Find the start and end of a crossing
Use when:
One line moves above another for part of the chart.
How:
Name both lines and the panel. Ask for the first and last qualifying x-values in a fixed order.
Benchmark example
Question:In the Hubei panel, find the first and last rounds when the green intervention: fine line is strictly above the orange intervention: nudge line. A tie does not count. Round each endpoint to the nearest 10 rounds; estimates within +/-5 rounds are acceptable. Answer as First: <round>; Last: <round> (e.g., First: 20; Last: 30).
