Class AiServiceInvocationAssert

java.lang.Object
org.assertj.core.api.AbstractAssert<AiServiceInvocationAssert, AiServiceInvocation>
org.assertj.core.api.AbstractObjectAssert<AiServiceInvocationAssert, AiServiceInvocation>
dev.langchain4j.test.service.AiServiceInvocationAssert
All Implemented Interfaces:
org.assertj.core.api.Assert<AiServiceInvocationAssert, AiServiceInvocation>, org.assertj.core.api.Descriptable<AiServiceInvocationAssert>, org.assertj.core.api.ExtensionPoints<AiServiceInvocationAssert, AiServiceInvocation>
@Experimental public final class AiServiceInvocationAssert extends org.assertj.core.api.AbstractObjectAssert<AiServiceInvocationAssert, AiServiceInvocation>
Assertions on what an AI Service did: which tools were executed, with which arguments and in which order, which sources were retrieved, how many times the chat model and the other models were called, why the response finished and whether the invocation succeeded. These assertions are deterministic: they do not call a model. To check the response text with natural-language criteria, using a model as a judge, see Judge.

Capture the invocation with AiServiceCapture, which works with any AI Service method: 

import static dev.langchain4j.test.LangChain4jAssertions.assertThat;

AiServiceCapture capture = AiServiceCapture.capture(() -> assistant.chat("Cancel my booking MS-777"));

assertThat(capture)
        .hasToolsExecutedExactlyInAnyOrder("getBookingDetails", "cancelBooking")
        .hasToolExecutedWith("cancelBooking", Map.of("bookingNumber", "MS-777"))
        .hasNoFailedToolExecutions();
This works for every return type of the AI Service method, also Result<...>: capture the call.

hasToolExecuted(String), hasToolExecutedTimes(String, int), hasToolsExecutedInOrder(String...), hasToolsExecutedExactlyInAnyOrder(String...) and both hasToolExecutedWith(...) count only the tool executions that succeeded. A tool that threw an exception counts as a failed execution, also when the AI Service sent the error to the model and then succeeded: check it with hasToolExecutionFailed(String) or hasNoFailedToolExecutions(). They fail if the AI Service did not report whether its tool executions failed (see ToolExecutedEvent.isError()).

When the invocation started other AI Service invocations (for example an AI Service called by one of its tools, see AiServiceInvocation.nestedInvocations()), say which ones to check first: includingNested() checks the whole tree, topLevelOnly() only what the invocation did itself. Without it, the assertions on tools, sources and model calls fail, because both readings are plausible and a wrong one could hide a problem: a sub-agent that ran a forbidden tool, or called the model many times. An invocation without nested invocations needs neither.

Like Mockito's verifyNoMoreInteractions, hasNoMoreToolExecutions() checks that every tool execution was checked by a tool assertion before, also by an earlier assertThat(...) on the same capture: 

assertThat(capture).hasToolExecutedWith("cancelBooking", Map.of("bookingNumber", "MS-777"));
assertThat(capture).hasToolExecuted("getBookingDetails").hasNoMoreToolExecutions();

When an assertion fails, the message names the AI Service method and lists all executed tools with their arguments and results. All assertions fail if the invocation was cancelled. If the invocation failed, they fail until hasFailedWithMessageContaining(String) or hasFailedWith(Class) has checked the failure; after it, they check what happened before the failure.

These assertions need the langchain4j module, where AI Services are defined. langchain4j-test does not bring it in. If the compiler reports that a class of dev.langchain4j.service is not found, add it, with the same version as your other LangChain4j modules (managed by langchain4j-bom if you import it): 

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j</artifactId>
</dependency>

  • Constructor Details

  • Method Details

    • includingNested

      public AiServiceInvocationAssert includingNested()
      Makes the assertions that follow check the whole tree of invocations: this invocation and all the invocations it started, directly or not (see AiServiceInvocation.nestedInvocations()). Their tool executions, sources and model calls add up.

    • topLevelOnly

      public AiServiceInvocationAssert topLevelOnly()
      Makes the assertions that follow check only what this invocation did itself, not the invocations it started (see AiServiceInvocation.nestedInvocations()).

    • hasNoToolExecutions

      public AiServiceInvocationAssert hasNoToolExecutions()
      Asserts that no tool was executed, not even one that failed.

    • hasToolExecuted

      public AiServiceInvocationAssert hasToolExecuted(String toolName)
      Asserts that the tool with the given name was executed successfully at least once. Marks all its successful executions as checked (see hasNoMoreToolExecutions()).

    • hasToolExecutionFailed

      public AiServiceInvocationAssert hasToolExecutionFailed(String toolName)
      Asserts that the tool with the given name was executed at least once and that this execution failed: the tool threw an exception, or its arguments were invalid. Use it to check how the AI Service handles a failing tool. Marks all failed executions of the tool as checked (see hasNoMoreToolExecutions()).

    • hasToolExecutedTimes

      public AiServiceInvocationAssert hasToolExecutedTimes(String toolName, int times)
      Asserts that the tool with the given name was executed successfully exactly the given number of times. 0 checks that it was not executed successfully. Marks its successful executions as checked (see hasNoMoreToolExecutions()).

    • hasToolsExecutedInOrder

      public AiServiceInvocationAssert hasToolsExecutedInOrder(String... toolNames)
      Asserts that the given tools were executed successfully in the given order. Other tools may have been executed before, between or after them, for example hasToolsExecutedInOrder("getBookingDetails", "cancelBooking") checks that the booking was looked up before it was cancelled. When the model asked for several tools in one response (parallel tool calls), their order is the order in which the model listed them: it does not show that one tool used the result of the other. To check that exactly these tools were executed, in this order, use executedToolNames().containsExactly(...) (this includes failed executions). Marks the executions that matched as checked (see hasNoMoreToolExecutions()).
      Throws:
      IllegalStateException - after includingNested(): the order of the tool executions of different invocations is not known

    • hasToolsExecutedExactlyInAnyOrder

      public AiServiceInvocationAssert hasToolsExecutedExactlyInAnyOrder(String... toolNames)
      Asserts that exactly the given tools were executed successfully, in any order, and no other tools: a tool that is not given must not have been executed at all, not even in an execution that failed. Failed executions of the given tools are not counted: check them with hasNoFailedToolExecutions(). A tool executed several times must be given as many times: hasToolsExecutedExactlyInAnyOrder("getWeather", "getWeather") passes only when getWeather was executed twice and no other tool was executed. To also check the order, use executedToolNames().containsExactly(...) (this includes failed executions). Marks all successful executions as checked (see hasNoMoreToolExecutions()).

    • hasToolExecutedWith

      public AiServiceInvocationAssert hasToolExecutedWith(String toolName, Map<String,?> expectedArguments)
      Asserts that the tool with the given name was executed successfully at least once with arguments that contain all the given arguments. Other arguments are ignored. Values are compared as JSON: Map.of("count", 3) matches an argument "count": 3 as well as "count": 3.0. Nested objects and arrays must match exactly. A number or a boolean sent as a string matches too, as the tool receives it converted to the type of its parameter: Map.of("count", 3) matches "count": "3", and Map.of("urgent", true) matches "urgent": "true". Text must match exactly: when a real model can write a value in several ways, use hasToolExecutedWith(String, Consumer).

      The argument names are the names of the tool method's parameters. Compile with -parameters or name the parameters with @P(name = ...), otherwise they are arg0, arg1 and so on.

      Marks the matching executions as checked (see hasNoMoreToolExecutions()).

    • hasToolExecutedWith

      public AiServiceInvocationAssert hasToolExecutedWith(String toolName, Consumer<Map<String,Object>> argumentsRequirements)
      Asserts that the tool with the given name was executed successfully at least once with arguments that pass the given requirements. Use it when a real model can write an argument in several ways, for example "Munich" or "München", so that hasToolExecutedWith(String, Map) would fail for a correct call. Accept only what your tool can really handle: a looser check can hide a call that the tool cannot process. 
      assertThat(capture).hasToolExecutedWith("getWeather", arguments -> {
    assertThat(arguments.get("city")).isIn("Munich", "München");
    Judge.from(judgeModel).assertThat(arguments.get("country").toString()).satisfies("refers to Germany");
});
      The requirements receive the arguments as the model sent them, parsed from JSON: a number is a Number, a nested object a Map and an array a List. They run once for every successful execution of the tool, and an execution passes unless they throw an AssertionError. Any other exception is rethrown, for example when a judge cannot give a verdict.

      Marks the executions that passed as checked (see hasNoMoreToolExecutions()).

    • hasNoMoreToolExecutions

      public AiServiceInvocationAssert hasNoMoreToolExecutions()
      Asserts that every tool execution was checked by one of the tool assertions above, called before on the same invocation, like Mockito's verifyNoMoreInteractions: the assertions add up across separate assertThat(...) statements on the same AiServiceCapture. Use it to check that the AI Service executed no other tools than the expected ones, also with other arguments or failing.

    • hasNoFailedToolExecutions

      public AiServiceInvocationAssert hasNoFailedToolExecutions()
      Asserts that no tool execution failed (see ToolExecution.hasFailed()). Fails if the AI Service did not report whether its tool executions failed (see ToolExecutedEvent.isError()). A tool that throws an exception, or an AssertionError in a test, counts as a failed tool execution when the AI Service sends the error to the model instead of failing. Whether it does depends on the kind of AI Service method and on its tool execution error handler (see AiServices.toolExecutionErrorHandler(ToolExecutionErrorHandler)).

    • hasRetrievedSourceContaining

      public AiServiceInvocationAssert hasRetrievedSourceContaining(String text)
      Asserts that at least one retrieved source (see AiServiceInvocation.sources()) contains the given text (case-sensitive). Fails if no sources were captured.

    • hasNoRetrievedSources

      public AiServiceInvocationAssert hasNoRetrievedSources()
      Asserts that no sources were retrieved for retrieval-augmented generation (RAG), for example for a question that the retrieved knowledge should not answer. Fails if the AI Service has no retrieval augmentor or did not report what it retrieved (see AiServiceInvocation.sources()).

    • hasOwnChatModelCallCount

      public AiServiceInvocationAssert hasOwnChatModelCallCount(int expectedCount)
      Asserts that the AI Service called its own chat model exactly the given number of times (see AiServiceInvocation.ownChatModelCalls()), also calls that failed. For example, an AI Service that executes tools once calls its chat model twice: once for the tool calls and once more for the final answer. To also count the calls of the components the AI Service runs (for example an embedding or a moderation model), use hasModelCallCount(int).

    • hasOwnChatModelCallCountAtMost

      public AiServiceInvocationAssert hasOwnChatModelCallCountAtMost(int maxCount)
      Asserts that the AI Service called its own chat model at most the given number of times (see AiServiceInvocation.ownChatModelCalls()), also calls that failed, for example to check that the AI Service does not call tools in a loop.

    • hasFinishReason

      public AiServiceInvocationAssert hasFinishReason(FinishReason finishReason)
      Asserts that the final response of this invocation finished for the given reason. For example, FinishReason.LENGTH means that the response was cut off because it reached the maximum number of output tokens, so hasFinishReason(FinishReason.STOP) checks that the response is complete. It is about the final answer of this invocation, so nested invocations are never checked.

    • hasModelCallCount

      public AiServiceInvocationAssert hasModelCallCount(int expectedCount)
      Asserts that exactly the given number of model calls were made (see AiServiceInvocation.modelCalls()): those of the AI Service to its own chat model and those of the components it runs, of every model type, also calls that failed. To count only the calls of the AI Service to its own chat model, use hasOwnChatModelCallCount(int).

    • hasModelCallCount

      public AiServiceInvocationAssert hasModelCallCount(Class<? extends ModelCall> type, int expectedCount)
      Asserts that exactly the given number of calls were made to models of the given type (see AiServiceInvocation.modelCalls()), also calls that failed. For example, hasModelCallCount(EmbeddingModelCall.class, 1) checks that the query was embedded once.
      Parameters:
      type - the type of model call, for example ChatModelCall.class or EmbeddingModelCall.class

    • hasSucceeded

      public AiServiceInvocationAssert hasSucceeded()
      Asserts that the invocation succeeded: it neither failed nor was cancelled. Other assertions check this too; this one is for when nothing else is checked, for example for a streaming method whose failure the test would otherwise not notice. Nested invocations are not checked: a failure of a nested invocation that the AI Service handled (for example a tool that sent the error to the model) does not make this invocation fail.

    • hasFailedWithMessageContaining

      public AiServiceInvocationAssert hasFailedWithMessageContaining(String text)
      Asserts that the invocation failed with an error whose message, or the message of one of its causes, contains the given text (case-sensitive). The assertions that follow it check what happened before the failure, for example which tools were executed.

    • hasFailedWith

      public AiServiceInvocationAssert hasFailedWith(Class<? extends Throwable> type)
      Asserts that the invocation failed with an error of the given type, or with an error caused by one. The assertions that follow it check what happened before the failure, for example which tools were executed.

    • executedToolNames

      public org.assertj.core.api.ListAssert<String> executedToolNames()
      Returns assertions on the names of all executed tools, in the order of execution, for example executedToolNames().containsExactly("getBookingDetails", "cancelBooking"). Unlike the hasTool... assertions, this includes the executions that failed: add hasNoFailedToolExecutions() if they must all have succeeded. After includingNested(), the names of the nested invocations come after those of this invocation.

    • toolExecutions

      public org.assertj.core.api.ListAssert<ToolExecution> toolExecutions()
      Returns assertions on the tool executions, in the order of execution. After includingNested(), the executions of the nested invocations come after those of this invocation.
      Throws:
      AssertionError - if the AI Service did not report whether its tool executions failed (see AiServiceInvocation.toolExecutions())