Skip to main content
LLM as a Judge evaluators are a powerful approach to evaluating language model outputs by using another language model to assess the quality, correctness, and appropriateness of responses. This method has become increasingly important in the field of AI evaluation due to its ability to capture complex patterns and relationships between inputs and outputs.

​
Why LLM as a Judge is Important

LLM as a Judge evaluators are crucial because they:
  1. Capture Nuance: They can understand and evaluate complex, context-dependent aspects of responses that traditional metrics might miss.
  2. Flexible Assessment: They can adapt to different evaluation criteria and domains without requiring extensive retraining.
  3. Human-like Judgment: They can provide evaluations that more closely resemble human judgment compared to rule-based approaches.
  4. Comprehensive Analysis: They can assess multiple aspects of a response simultaneously, including correctness, completeness, tone, and relevance.
But there are some drawbacks:
  • Cost: Requires additional LLM API calls, which can increase operational costs
  • Latency: Evaluation time is dependent on the LLM’s response time
  • Potential Bias: May inherit biases from the judging LLM
  • Consistency: May show some variation in evaluations across different runs
  • Dependency: Relies on the availability and reliability of the judging LLM

​
Current TrustTest LLM as a Judge Evaluators

TrustTest provides these LLM as a Judge evaluators:
  1. Correctness: Factual accuracy against an expected response
  2. Completeness: Coverage of the expected response
  3. Tone: Tone and style
  4. URL Correctness: URLs in the response
  5. True/False: Pass/fail from ObjectiveContext descriptions (default for most catalog threat probes)
  6. Answer Relevance: How well the answer addresses the question (QuestionContext)
  7. RAG Poisoning: Whether the answer follows poisoned retrieval context
  8. Custom: CustomEvaluatorExpected and CustomEvaluatorObjective — there is no CustomEvaluator class
We recommend using LLM as a Judge evaluators instead of Heuristic evaluators because they can understand semantic relationships and reason about the content. Unlike rule-based approaches.