LLM-as-a-judge
Also known as: LLM as a judge, AI judge, model-graded evaluation, LLM judge
LLM-as-a-judge is a technique where an AI language model grades the outputs of another AI system, using instructions or a rubric that describe what a good answer looks like.
It lets teams evaluate thousands of responses for qualities like helpfulness, accuracy, or tone without paying people to review each one. While AI judges often agree with human reviewers, they have known biases, such as favouring longer answers, and need to be checked against human judgment.
LLM-as-a-judge, explained at three levels
At home
Imagine a teacher with a thousand essays to mark and a clear marking guide. Instead of reading every one, the teacher hands the guide to a capable assistant who marks them all and explains each grade. LLM-as-a-judge works that way: one AI model, given a rubric, reads another AI's answers and scores them, so testing can happen at a scale no person could manage.
At work
LLM-as-a-judge means using a language model to grade the outputs of an AI system against written criteria such as accuracy, helpfulness, or tone. It lets teams evaluate thousands of responses in minutes for a fraction of the cost of human review. Businesses use it to compare models, test prompt changes, and monitor live quality, while still checking the judge against human reviewers.
At home
The judge gets a carefully written prompt containing the original question, the answer to grade, a rubric explaining what earns each score, and sometimes a reference answer. It writes a short explanation, then a score, such as 4 out of 5. Change the rubric and the judge changes with it: stricter wording produces stricter grades. That's why judge prompts need the same care as the prompts they test.
At work
Three formats are common: single-answer grading against a rubric, pairwise comparison where the judge picks the better of two answers, and reference-based grading against a known-good answer. Bias reduction is built into the setup: ask for reasoning before the score, use specific criteria instead of 'is this good?', run pairwise comparisons in both orders, and use a different model family from the one being tested. Tools such as OpenAI Evals, promptfoo, and Ragas include judge templates.
At home
A judge model has habits learned from its own training. Studies have found judges favour the first answer shown, prefer longer responses, and rate answers from their own model family more highly. They also can't verify facts they don't know, so a fluent wrong answer can score well. While AI judges agree with people surprisingly often, those habits add up across thousands of grades. In my opinion, they're best used as a helper for human reviewers, not a replacement.
At work
Treat a judge as a measuring instrument that needs calibration. Label a few hundred examples with human reviewers, then measure agreement, for example with Cohen's kappa, before trusting the judge. Re-check whenever the judge model or rubric changes, since scores can drift. Avoid having a model grade its own outputs. Judges are weakest in specialized domains like law or medicine, on factual accuracy without reference answers, and on subtle safety issues, so those still need expert review.
Real-world examples
At home
Carlos asked an AI to compare two versions of his cover letter and explain which was stronger. It picked the longer one, so he asked again with the order swapped to check it wasn't just favouring length.
At work
Sophie runs quality for an AI support assistant at a Montreal telecom. She uses an LLM judge to score every test conversation, then has staff review a random 10 percent to confirm the judge agrees with people.
How it compares
| feature | LLM-as-a-judge | Human review |
|---|---|---|
| Speed | Seconds per answer | Minutes per answer |
| Cost | Low | High |
| Consistency | High but biased | Varies by reviewer |
| Domain expertise | Limited | Strong with experts |
AI judges scale. Humans set the standard the judges must match.
For analysts
I'd never accept LLM-judge scores without first checking them against human reviewers on a sample. Once agreement is measured, the judge can safely handle the volume.
History & origin
Scoring open-ended text has long been difficult; older automatic metrics such as BLEU (2002) and ROUGE (2004) compared word overlap and correlated poorly with human preferences. As models like GPT-4 became capable in 2023, researchers tested using them as graders. A 2023 paper by Lianmin Zheng and colleagues in the LMSYS research group, centred at UC Berkeley, 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,' found strong agreement with human ratings while documenting position, verbosity, and self-preference biases. The approach became standard in evaluation tools and in training methods that use AI feedback. In my view, LLM judges are a powerful shortcut, but a shortcut you haven't validated is just a fast way to be wrong.