What the Multi-Model Response Comparator does
This comparator lines up two to six answers to the same prompt - from different models, settings or prompt versions - and measures what can be counted: which of your key points each answer covers, how it is structured, how readable it is, how many checkable claims and citations it contains, how much it hedges, and the latency and cost you record for it.
Nothing is sent to any model and nothing here decides which answer is true. The metrics are deterministic counts in your browser, designed to make a side-by-side review faster and more consistent; the judgement about quality stays with you, ideally backed by a written rubric.
How to use it
- List the key points a good answer must include, one per line - the facts, steps or caveats you would mark it on.
- Paste each model's answer with a label. Add the latency and cost from the API response if you want them in the comparison.
- Press Compare. The coverage bars and matrix show which points each answer hit; the findings flag points nobody covered, points only one answer covered, and claims without sources.
- Read the metrics table, then read the answers themselves where the numbers disagree with your impression. Download CSV or JSON to keep the comparison with your evaluation notes.
Reading the results
Coverage is the share of key points found in an answer, by exact phrase or by at least 60% of the point's content words. It rewards answers that use your words; a correct paraphrase can be marked missing, and a wrong statement using the right words can be marked covered.
Reading ease is the Flesch score: higher is easier, 60-70 is plain English, below 30 is dense technical prose. Claims are sentences containing numbers, dates or superlatives - statements someone should verify. Hedges count words like may, might and probably; some hedging is honest, a lot is evasive.
Latency and cost are only what you enter. Compare them per useful answer, not per call: a cheaper model that misses half the key points may cost more once you add a retry or a human fix.
Worked example: three answers about API key security
Key points: never commit keys to source control; store keys in a secrets manager; rotate keys regularly; use least-privilege scopes; revoke a leaked key immediately.
Model A (55 words, reading ease 74) covers 4 of 5 points - 80% - and is the only one to mention source control and revocation. Its line "Give each key the smallest scope it needs" is marked missing for least privilege, because it shares too few words with the key point: a real miss by the heuristic that reading catches. Its 3 claim sentences, including "every 90 days", carry no citation.
Model C covers 3 of 5 (60%), is the only one to say "least-privilege scopes", and cites its sources. Model B covers 1 of 5 (20%), uses two hedges (might, probably) and a superlative ("the best single step") with nothing to back it. The numbers point to A and C; reading A shows it actually covers all five.
Formulas and scoring rules
- Coverage
coverage = points covered / key points; covered = exact phrase OR shared content words / point content words >= 0.6- Flesch Reading Ease
206.835 - 1.015 x (words / sentences) - 84.6 x (syllables / words)Syllables are estimated from vowel groups; English only. Rounded to a whole number for display.
Limitations: what the result does not prove
- It does not check facts. A confident, well-structured, fully covered answer can still be wrong; verify claims against sources.
- Coverage is word overlap. Paraphrases, synonyms and other languages are missed; phrase key points the way a correct answer would.
- Readability and hedge counts are English-only measures and say nothing about accuracy.
- One prompt is an anecdote. Compare models on a set of prompts, with a rubric, before choosing one.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology
- OpenAI - Evaluation best practices
- Anthropic - Define success criteria and build evaluations
Frequently asked questions
Does this tool run the models for me?
No. You paste answers you already have, so any model, setting or provider can be compared, including ones behind your own gateway. Nothing is sent anywhere; the comparison runs entirely in your browser.
How is coverage decided without an AI judge?
Each key point is looked for as an exact phrase, then as a set of content words with common words removed and simple endings stripped. If at least 60% of those words appear, the point counts as covered. It is fast and repeatable, but it misses paraphrases, so read the matrix critically.
What makes a good key point for comparing LLM answers?
A short statement in the words a correct answer would use, covering one idea each: "revoke a leaked key immediately" rather than "security best practice". Five to ten points usually capture what matters; more makes coverage noisy.
Why count claims and citations?
Because numbers, dates and superlatives are where models most often invent things, and a citation is what lets a reader check them. An answer with many claims and no sources needs more verification before it is used, whichever model wrote it.
Is a higher reading ease score better?
Only relative to the audience. A reading ease of 70 suits a customer-facing answer; an explanation for engineers can reasonably score 40. Use it to spot an answer that is much denser or much simpler than the others for the same question.
How should I include latency and cost in a model choice?
Record them from the API response - most providers return token usage, and you can time the call - and compare them per acceptable answer. A model that is twice as fast but needs a retry one time in three is not twice as fast in practice.
Last reviewed by the A2Z.Tools team against the sources listed above.
