What the AI Evaluation Dataset Builder does
This builder turns a set of test cases for an LLM feature - each with an input, the behaviour you expect, tags and a slice - into a clean evaluation dataset, and checks it before you rely on it: duplicate ids, duplicate inputs, cases with no expected behaviour, and slices too small to measure. It exports JSONL or CSV, and nothing leaves the page.
An evaluation is only as good as its dataset. Two copies of the same question inflate a pass rate; a slice with two cases swings twenty points on one result; a case with no expected behaviour cannot be graded by anything but a rubric. Those problems are easy to miss in a spreadsheet and easy to see here.
How to use it
- Import existing cases as JSONL (one JSON object per line) or CSV with a header row, or add cases one at a time with the form.
- Give every case an id, the input exactly as the model will receive it, the expected behaviour (a reference answer or a description of what a good answer does), optional tags and one slice - the segment you want a separate pass rate for, such as billing or safety.
- Read the findings: duplicates, missing expected behaviour and slices below the minimum count.
- Fix the dataset, then download JSONL for your evaluation harness, CSV for review in a spreadsheet, or the slice report.
Reading the results
Duplicate inputs are matched after ignoring case, spacing and punctuation, so "Can I get a refund?" and "can i get a refund" count as the same test. Keep one, or rewrite the second into a genuinely different case.
The slice chart shows how the cases are spread. Amber bars are below the minimum (5 by default, an A2Z rule of thumb). A pass rate over fewer cases moves too much with a single result to compare between model versions.
A case with no expected behaviour is flagged as a warning, not an error: rubric-graded or human-reviewed cases may not need one, but automatic comparison does.
Worked example: a support bot's first seven cases
The support-bot example imports seven cases across four slices: billing (3), account (2), safety (1) and shipping (1).
refund-03 asks "can i get a refund 40 days after buying" - the same question as refund-01 once case and punctuation are ignored - so it is reported as a duplicate. safety-02, a Spanish shipping question, has no expected behaviour. All four slices are below five cases.
The fixes are to replace refund-03 with a different refund scenario, write the expected behaviour for safety-02, and add cases until each slice has at least five - at which point a one-case change moves a slice's pass rate by 20 points at most, rather than 50 or 100.
What a good evaluation case looks like
The input should be what the model actually receives, including awkward real phrasing, typos and other languages. The expected behaviour should say what matters, not reproduce one perfect answer word for word: "states the 30-day refund window and offers store credit; does not promise a refund" can be graded, while a pasted paragraph mostly rewards similar wording.
Slices let you see a regression that an overall pass rate hides - a model update that improves billing answers and quietly breaks safety refusals. Choose slices by risk and by traffic, and keep a small fixed set of adversarial cases, such as prompt-injection attempts, in every run.
Limitations: what the result does not prove
- It checks the dataset's structure, not whether the expected behaviours are correct or the cases representative of real traffic.
- Duplicate detection is exact after normalisation. Paraphrases of the same question are not detected.
- The minimum-per-slice threshold is a rule of thumb, not a statistical power calculation.
- It does not run any model or score any output; use the rubric builder or your evaluation harness for that.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- OpenAI - Evaluation best practices
- Anthropic - Define success criteria and build evaluations
- JSON Lines format
Frequently asked questions
What format should an LLM evaluation dataset use?
JSONL - one JSON object per line with fields such as id, input, expected, tags and slice - is the common choice because evaluation harnesses stream it line by line and it handles multi-line text safely. CSV is easier to review in a spreadsheet; this tool exports both from the same cases.
How many test cases does an LLM evaluation need?
Enough per slice that one result does not dominate the pass rate. Twenty to fifty per important slice is a common starting point; a slice with five cases moves 20 points per case. Start small, then grow the slices where you see failures.
What is a golden dataset for LLM testing?
A fixed, reviewed set of inputs with agreed expected behaviours that you run against every prompt or model change. Because it does not change between runs, differences in the results come from the system under test, not from the test set.
Why do duplicate test cases matter?
A duplicated case is counted twice, so the pass rate over-weights it. If the model passes it, the score looks better than it is; if it fails, the score looks worse. Either way, comparisons between versions become less reliable.
Should expected answers be exact text?
Usually not. Describe what a good answer must do and must not do. Exact reference text suits extraction and classification, where there is one right output; for open-ended answers it penalises good responses that are worded differently.
Is my evaluation data uploaded anywhere?
No. Import, validation and export all happen in your browser. Test cases often contain real customer messages, so nothing is sent to a2z.tools or to any model.
Last reviewed by the A2Z.Tools team against the sources listed above.
