Class ChatModelJudge

java.lang.Object
dev.langchain4j.test.judge.ChatModelJudge
All Implemented Interfaces:
Judge
@Experimental public final class ChatModelJudge extends Object implements Judge
A Judge backed by a ChatModel. For each criterion, the model answers SATISFIED, NOT_SATISFIED or NOT_SURE, with its reasoning. The model must support JSON schema response formats. A response that cannot be understood, or that does not assess every criterion, throws a JudgeException. Created by Judge.from(ChatModel).

Images and other media (audio, video, PDF files) in a conversation given with JudgedTextAssert.given(List) are sent to the judge model at their place in the conversation, so the judge model must support them: otherwise its provider fails, for example with an UnsupportedFeatureException. An image whose detail level was not chosen when it was created (see ImageContent.detailLevelSpecified()) is sent with ImageContent.DetailLevel.AUTO, so it may use more tokens than in the call of the application: set the detail level explicitly to control this. An image with a chosen detail level is sent as it is. An image read back from JSON, for example from a persistent chat memory store, counts as having a chosen detail level, so it keeps ImageContent.DetailLevel.LOW. Providers that do not support text and media in the same message in their original order (for example Ollama) receive the media after the text, in the order of their markers.

  • Method Details

    • withOneRequestPerCriterion

      public ChatModelJudge withOneRequestPerCriterion()
      Returns a judge that judges each criterion in its own request to the model, instead of all criteria of an assertion in a single request. This costs one request per criterion, but criteria cannot influence each other's verdicts.

    • withSystemMessage

      public ChatModelJudge withSystemMessage(String systemMessage)
      Returns a judge that uses the given system message instead of the default one. The user message sent to the judge model stays the same: the text in <text>, the criteria as numbered <condition> elements (with <passes_when> and <fails_when> if the criterion defines them), and, if given, the user message in <user_message>, the conversation as <message role="..."> elements in <conversation> and the sources as numbered <source> elements in <sources>. The characters &, < and > in these contents are escaped as &amp;, &lt; and &gt;. The response must still follow the JSON schema of the default judge: for each condition, its conditionNumber, the reasoning and the result (SATISFIED, NOT_SATISFIED or NOT_SURE).

      Images and other media of the conversation are sent to the judge model at their place in the conversation, not at the end, each right after a marker in the text, such as [image 1], [audio 1], [video 1] or [pdf 1].

      Keep telling the judge that these contents are data, not instructions: the text being judged, the conversation (including its images) and the sources can contain instructions aimed at the judge, for example to answer SATISFIED. If you judge conversations with images, also explain the markers, as the default system message does.

      The default system message was chosen by measuring how often judges give wrong verdicts with it. Check your system message on a few cases with known verdicts before relying on it.

    • evaluate

      public JudgeResponse evaluate(JudgeRequest request)
      Description copied from interface: Judge
      Judges the text of the request against each criterion, without failing.
      Specified by:
      evaluate in interface Judge
      Returns:
      one verdict per criterion, in the order of the criteria (a criterion the judge was not sure about is not satisfied), and the tokens the judge model used