What the LLM Evaluation Rubric Builder does
This builder helps you write a scoring rubric for AI outputs: weighted criteria, an anchored description for every score level, critical gates that fail an answer on their own, and a pass threshold. It previews the weighted score for any set of scores and exports the rubric as JSON or CSV plus a ready-to-use LLM-as-judge prompt.
The scoring is deterministic and shown in full, so a human reviewer and an automated judge apply exactly the same arithmetic. Nothing is sent to any model; the judge prompt is text for you to use in your own evaluation harness.
How to use it
- Start from an example or add your own criteria. Give each one a name, a weight and a score scale.
- Write an anchor for every level - what a 1, a 3 and a 5 actually look like for this criterion. Anchors are what make two graders agree.
- Mark any criterion that must never fail, such as safety or privacy, as critical and set its minimum score.
- Set the pass threshold, then enter scores in the preview to see the weighted result, each criterion's points and whether any gate failed.
- Download the rubric JSON, the CSV for reviewers, or copy the judge prompt.
Reading the results
Each score is first scaled to 0-1 within its own range, so a 1-5 criterion and a 0-1 criterion are comparable. The weighted score is 100 times the weighted average of those scaled scores.
An answer passes only if the weighted score reaches the threshold and every critical criterion meets its minimum. A gate stops a polished but unsafe answer from passing on average.
The points table shows how much each criterion can contribute. If a criterion you care about can only move the total by a few points, raise its weight or make it a gate.
Worked example: grading a support answer
The support example has four criteria: policy accuracy (weight 3, scored 1-5), resolution (weight 2, 1-5), tone (weight 1, 1-5) and safety and privacy (weight 1, 0-1, critical, minimum 1). The weights add up to 7 and the threshold is 70.
An answer scored 4 for accuracy, 3 for resolution, 4 for tone and 1 for safety scales to 0.75, 0.5, 0.75 and 1. The weighted score is 100 x (3 x 0.75 + 2 x 0.5 + 1 x 0.75 + 1 x 1) / 7 = 100 x 5 / 7 = 71.4, which passes.
Change only the safety score to 0 - the answer asked for a full card number - and the weighted score drops to 57.1, but the result would fail even if the other three were perfect, because the safety gate is not met.
Formulas and scoring rules
- Scaled score
norm_i = (s_i - lowest_i) / (highest_i - lowest_i)Between 0 and 1 for every criterion, whatever its scale.- Weighted score
score = 100 x sum(w_i x norm_i) / sum(w_i)Shown to one decimal place; nothing is rounded before comparing with the threshold.- Pass rule
pass = score >= threshold AND every critical criterion has s_i >= minScore_i
Using the rubric with an LLM judge
The exported judge prompt lists every criterion with its anchors and asks for one integer score per criterion with a short justification, in JSON. Keep the arithmetic out of the model: let the judge score each criterion and compute the weighted total and the gates in code, exactly as this page does. Models are inconsistent at arithmetic and at applying thresholds, but reasonably consistent at matching an answer to a clearly written anchor.
Check the judge against people before trusting it. Have two people score thirty or so answers with the same rubric, compare their scores with the judge's, and rewrite the anchors where they disagree most.
Limitations: what the result does not prove
- It builds and applies a rubric; it does not score any output itself. The scores in the preview are the ones you enter.
- A rubric measures what it describes. Poorly written anchors produce consistent but meaningless scores.
- LLM judges have known biases, such as favouring longer answers or answers similar to their own style. Validate against human scores.
- Weighted averages hide trade-offs; look at per-criterion results as well as the total.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- Anthropic - Building evaluations (LLM-based grading)
- OpenAI - Evaluation best practices
- Zheng et al. - Judging LLM-as-a-judge
Frequently asked questions
What is an LLM evaluation rubric?
A set of criteria, each with a score scale and a written description of what every level looks like, plus weights and a pass rule. It turns "is this answer good?" into specific questions that different graders - people or a model - can answer the same way.
How do I write good rubric anchors?
Describe observable features, not adjectives. "Correct but omits a condition the customer needs" is gradable; "fairly good" is not. Write the lowest and highest anchors first, then the middle, and test them on real answers that sit on the boundaries.
Why use a critical gate instead of a high weight?
Because a weight can always be outvoted. With a gate, an answer that leaks personal data or gives dangerous advice fails no matter how accurate and friendly the rest is, which matches how such failures are judged in practice.
Should an LLM judge use a 1-5 or a 1-10 scale?
Short scales with written anchors give more consistent results; judges and people struggle to separate a 6 from a 7 without a description of each. Use 1-5, or a binary pass/fail for criteria that are really yes or no.
How is the weighted score calculated?
Each criterion's score is scaled to between 0 and 1 across its own range, multiplied by its weight, summed, divided by the total weight and multiplied by 100. The formula and every intermediate value are shown on the page.
Can I use this rubric with any evaluation framework?
Yes. The JSON export is a plain description of criteria, levels, weights, gates and threshold, and the judge prompt is plain text. Map them into whatever harness you use; the scoring rule is simple to reproduce in a few lines of code.
Last reviewed by the A2Z.Tools team against the sources listed above.
