Class DecisionModelJudge

java.lang.Object
dev.langchain4j.test.judge.DecisionModelJudge
All Implemented Interfaces:
Judge
@Experimental public final class DecisionModelJudge extends Object implements Judge
A Judge backed by a DecisionModel. Each criterion is asked as a yes/no question, and is satisfied when the probability of "yes" is at least the threshold, 0.5 by default. Unless the criterion defines when it passes and fails, the question tells the model to answer "no" when the criterion is satisfied only partly or when it is unclear, so a passing criterion means the model considers it clearly satisfied. All criteria of an assertion are asked in a single request; whether the answer to one criterion can be influenced by the others depends on the decision model. So does how closely the model follows the pass and fail definitions of a JudgeCriterion: check a few known cases before relying on them. Created by Judge.from(DecisionModel).

A conversation given with JudgedTextAssert.given(List) is sent to the decision model as messages, with its images and other media: an image whose detail level was not chosen when it was created (see ImageContent.detailLevelSpecified()) is sent with ImageContent.DetailLevel.AUTO, an image with a chosen detail level is sent as it is, the same way as by a ChatModelJudge. An image read back from JSON, for example from a persistent chat memory store, counts as having a chosen detail level, so it keeps ImageContent.DetailLevel.LOW.

  • Method Details

    • withThreshold

      public DecisionModelJudge withThreshold(double threshold)
      Returns a judge with the given threshold: a criterion is satisfied when the probability of "yes" is greater than or equal to it.
      Parameters:
      threshold - greater than 0 and at most 1.

    • withQuestion

      public DecisionModelJudge withQuestion(Function<JudgeCriterion, YesNoQuestion> questions)
      Returns a judge that asks each criterion with the question returned by the given function, instead of the default question. The decision model receives the text to judge as response, and, if given, the user message as userMessage and the sources as sources: the question can refer to them by these names. When a conversation is given (see JudgedTextAssert.given(List)), the decision model instead receives the conversation followed by the text to judge as the last assistant message, so the question should refer to "the last assistant message". For example: 
      DecisionModelJudge judge = Judge.from(decisionModel).withQuestion(criterion -> YesNoQuestion.builder()
        .text("Would a support agent accept this response? It must: " + criterion.text())
        .yesWhen(criterion.satisfiedWhen() != null ? criterion.satisfiedWhen() : "it clearly does")
        .noWhen(criterion.notSatisfiedWhen() != null ? criterion.notSatisfiedWhen() : "it does not, or only partly")
        .build());
      A judge can be created once and shared by all tests, for example in a constant or a test configuration. The default question was chosen by measuring how often judges give wrong verdicts with it. Check your question on a few cases with known verdicts before relying on it.
      Parameters:
      questions - returns the question to ask for a criterion

    • evaluate

      public JudgeResponse evaluate(JudgeRequest judgeRequest)
      Description copied from interface: Judge
      Judges the text of the request against each criterion, without failing.
      Specified by:
      evaluate in interface Judge
      Returns:
      one verdict per criterion, in the order of the criteria (a criterion the judge was not sure about is not satisfied), and the tokens the judge model used