Interface Judge

All Known Implementing Classes:
ChatModelJudge, DecisionModelJudge
@Experimental public interface Judge
Checks whether a text, for example the response of an AI Service, satisfies criteria written in natural language, using a model as the judge: 
Judge judge = Judge.from(judgeModel);

judge.assertThat(answer)
        .given(question)
        .satisfies("says that the booking cannot be found", "does not mention any dates");
The judge can be a ChatModel, which explains each verdict, or a DecisionModel, which answers each criterion with the probability that it is satisfied. The assertion fails when at least one criterion is not satisfied, and the message names each failed criterion with the judge's reasoning or probability. A judge is immutable: create it once and reuse it across tests.

To check that a RAG answer is grounded, give the judge the retrieved sources: 

judge.assertThat(result.content())
        .basedOnContents(result.sources())
        .satisfies("contains only claims that are stated in the sources");
Prefer this wording to "is supported by the sources": judges often pass that one even when the text adds a claim that is not in the sources.

Vague criteria give inconsistent verdicts. Write one specific criterion per thing that can go wrong, say exactly what you mean (for example "says that the return period is 14 days" rather than "answers the question") and, when its meaning could be disputed, define when it passes and fails with JudgeCriterion.

Every assertion calls the judge model, so it takes time and costs money like any other model call (to track the cost, add a ChatModelListener to the judge model). Judges make mistakes and their answers can vary between runs. Use a capable model, preferably from a different model family than the one being tested, pin a specific model version (not an alias the provider can point to a newer model) so that results stay comparable, set its temperature to 0 where the model supports it, and prefer criteria that can be judged from what the judge is given. For a ChatModel judge, enable strict JSON schema where the provider supports it (for example OpenAiChatModel.builder().strictJsonSchema(true)), so that the model follows the response format exactly.

Use assertThat(String) in tests. Use evaluate(JudgeRequest) to get the verdicts as values, for example to compute pass rates over a dataset. To use another way of judging, for example several judges that vote, implement evaluate(JudgeRequest) and return JudgeResponse.from(List): assertThat(String) works with any implementation.

  • Method Summary

    Modifier and Type
    Method
    Description
    default JudgedTextAssert
    assertThat(String text)
    Starts assertions on the given text, for example the response of an AI Service.
    JudgeResponse
    evaluate(JudgeRequest request)
    Judges the text of the request against each criterion, without failing.
    static ChatModelJudge
    from(ChatModel chatModel)
    Returns a judge backed by the given chat model.
    static DecisionModelJudge
    from(DecisionModel decisionModel)
    Returns a judge backed by the given decision model: each criterion is asked as a yes/no question, all in a single request.