Class AiServiceCapture

java.lang.Object
dev.langchain4j.test.service.AiServiceCapture
@Experimental public final class AiServiceCapture extends Object
Captures what AI Services do while a lambda runs: what they produce, which tools they execute, which sources they retrieve, which model calls they make, how many tokens they use and how they end. Works with any AI Service method, whatever it returns. 
import static dev.langchain4j.test.LangChain4jAssertions.assertThat;
import static dev.langchain4j.test.service.AiServiceCapture.capture;

AiServiceCapture capture = capture(() -> assistant.chat("Cancel booking MS-777"));

assertThat(capture)
        .hasToolExecutedWith("cancelBooking", Map.of("bookingNumber", "MS-777"))
        .hasNoFailedToolExecutions();
assertThat(capture.outputText()).contains("cancelled");
output() and outputText() are what the AI Service produced, whatever the lambda returned: for example the object of a method with structured output (also when the method returns it in a CompletableFuture, a Mono or a Uni), or the text of the final response of a streaming method: 
AiServiceCapture capture = capture(() -> assistant.extractBookingAsync("..."));
Booking booking = capture.output(Booking.class);
lambdaOutput() is what the lambda returned, which can be something else, for example the result of a controller called with MockMvc.

The top-level invocations (see invocations()) are those not started by another AI Service invocation, in the order they started: the AI Services that the lambda called, directly or through your code, and the AI Services that a framework uses internally, for example the planner of an agentic supervisor. The invocations started by their work (for example by their tools, guardrails or content retrievers) are nested in them (see AiServiceInvocation.nestedInvocations()): an AI Service called by a tool is found there, not in invocations(). The shortcuts of this class, such as output(), outputText() and lastRequest(), are those of the only top-level invocation.

Streaming and asynchronous methods

capture(...) returns when every captured invocation has finished, so streaming and asynchronous AI Service methods need little extra code:
  • a TokenStream must be started in the lambda: set the handlers it needs and call start(), for example capture(() -> assistant.chatStream("Hello").ignoreErrors().start()). If the lambda returns a TokenStream that was not started, capture(...) fails;
  • a Flow.Publisher returned by the lambda (for example a Mutiny Multi), or a Reactive Streams publisher (for example a Reactor Flux), is subscribed to and read to the end, and its items are captured (see lambdaOutputItems()). Do not return a publisher that the lambda already subscribed to: it would be subscribed to a second time, which may call the model again or fail;
  • a CompletableFuture (or other CompletionStage) returned by the lambda does not need to be waited for;
  • a Mutiny Uni returned by the lambda is subscribed to and waited for.
AiServiceCapture capture = capture(() -> assistant.chatStream("Hello").ignoreErrors().start());
String answer = capture.outputText();
While an invocation is still running, a warning naming it is logged every 30 seconds. After 10 minutes, capture(...) cancels the future or publisher returned by the lambda and fails, naming the invocations that have not finished. Change this timeout with the JVM system property langchain4j.test.capture.timeout-millis (in milliseconds, 0 for no timeout). capture(...) also stops waiting when its thread is interrupted, for example by JUnit's @Timeout, and then cancels the future or publisher returned by the lambda too. Call capture(...) on a thread that may block, not on an event loop. If the future or publisher returned by the lambda has ended, but an invocation still has not reported its end 5 seconds later, capture(...) fails, because some frameworks do not report it in every case. If a TokenStream was created but is not started within 2 seconds, capture(...) fails, because nothing would ever start it. Both waits can be changed, for example on a slow build machine, with the JVM system properties langchain4j.test.capture.end-report-grace-millis and langchain4j.test.capture.stream-start-grace-millis (in milliseconds), for example with -D... or the systemPropertyVariables of the Maven Surefire plugin. These properties, and the timeout above, are not Spring or Quarkus configuration properties: setting them in application.properties has no effect.

Failures

If an AI Service invocation fails, capture(...) throws its error once all invocations have finished, also when the error was passed to an onError handler or to a returned CompletableFuture, or was caught by the lambda. Any other exception thrown by the lambda is rethrown too, for example an AssertionError thrown in a mock model, or by an assertion in the lambda (such as mockMvc.perform(...).andExpect(status().isOk())). An Error is rethrown even when a failure is expected. An exception or AssertionError thrown by a tool can be different: whether it fails the invocation or is sent to the model as the result of the tool depends on the kind of AI Service method and on its tool execution error handler (see AiServices.toolExecutionErrorHandler(ToolExecutionErrorHandler)). When it is sent to the model, it is not thrown: check it with AiServiceInvocationAssert.hasToolExecutionFailed(String) or AiServiceInvocationAssert.hasNoFailedToolExecutions(). A failed invocation makes capture(...) throw even when the code recovered from it, for example an agentic workflow whose error handler retries the agent or returns a fallback result. To check a failure that the test expects, use captureExpectingFailure(Callable): 
AiServiceCapture capture = captureExpectingFailure(() -> assistant.chat("Cancel MS-777"));

assertThat(capture)
        .hasFailedWithMessageContaining("Booking MS-777 not found")
        .hasToolExecutionFailed("getBookingDetails");
A cancelled invocation does not make capture(...) throw, but every assertion on it fails, and AiServiceInvocation.output() and AiServiceInvocation.outputText() throw. Invocations started by another invocation (see AiServiceInvocation.nestedInvocations()) do not make capture(...) throw when they fail, because the invocation that started them can handle their failure, for example a tool that reports the error to the model. This relies on the AI Service reporting which invocation started another one (see InvocationContext.parentInvocationId()).

Pay attention to what the invocation captured rather than to the side effects of your own handlers: for a TokenStream, the invocation finishes before the onCompleteResponse handler you gave it has run, so when capture(...) returns, that handler may still be running. AiServiceInvocation.outputText() has the final text in any case.

Threads

An invocation is captured when it starts on the thread that runs the lambda, or when it is started while a captured invocation runs, for example by its tool, guardrail or content retriever (see AiServiceInvocation.nestedInvocations()). Invocations started on other threads are not captured: for example by an HTTP request to a running server, by an @Async method, by subscribeOn(...), by a parallel stream, by an executor, or the agents of an agentic workflow that run on other threads, such as the agents of a parallel workflow. Captures on different threads do not see each other's invocations, so tests can run in parallel. langchain4j-test listens to AI Services with listeners that ignore all events while nothing is being captured.

capture(...) blocks until the captured invocations have finished, also when the lambda returns a CompletableFuture or a stream. So call it from the test thread, never from an event-loop thread (for example on a Vert.x context): an invocation that needs that thread to finish could then not finish.

  • Method Details

    • capture

      public static AiServiceCapture capture(Callable<?> code)
      Runs the given lambda and captures the AI Service invocations it starts. If the lambda returns a publisher, a future or a Uni, it is subscribed to. Then waits until all captured invocations have finished.
      Parameters:
      code - the code to run, for example () -> assistant.chat("Hello")
      Returns:
      the captured invocations, and what the lambda returned

    • capture

      public static AiServiceCapture capture(AiServiceCapture.ThrowingRunnable code)
      Runs the given lambda and captures the AI Service invocations it starts, then waits until all of them have finished.
      Parameters:
      code - the code to run, for example () -> assistant.chatStream("Hi").onPartialResponse(tokens::add).onError(errors::add).start()
      Returns:
      the captured invocations

    • captureExpectingFailure

      public static AiServiceCapture captureExpectingFailure(Callable<?> code)
      Like capture(Callable), for a lambda that is expected to make an AI Service invocation fail: the failed invocation is captured instead of its error being thrown, so that what happened before the failure can be checked.
      Parameters:
      code - the code to run, for example () -> assistant.chat("Cancel booking MS-777")
      Returns:
      the captured invocations, and what the lambda returned
      Throws:
      AssertionError - if no AI Service invocation started by the lambda failed, also when nothing was captured while the lambda threw an exception (the exception is then the cause)

    • captureExpectingFailure

      public static AiServiceCapture captureExpectingFailure(AiServiceCapture.ThrowingRunnable code)
      Like capture(ThrowingRunnable), for a lambda that is expected to make an AI Service invocation fail: the failed invocation is captured instead of its error being thrown, so that what happened before the failure can be checked.
      Parameters:
      code - the code to run, for example () -> assistant.chatStream("Hi").onPartialResponse(tokens::add).onError(errors::add).start()
      Returns:
      the captured invocations
      Throws:
      AssertionError - if no AI Service invocation started by the lambda failed, also when nothing was captured while the lambda threw an exception (the exception is then the cause)

    • output

      public Object output()
      Returns what the only top-level AI Service invocation produced (see AiServiceInvocation.output()): for example the String answer, the object of a method with structured output, the value inside a Result, a CompletableFuture, a Mono or a Uni, or the text of the final response of a streaming method. It does not depend on what the lambda returned (see lambdaOutput() for that). 
      assertThat(capture.output()).isEqualTo(new Booking("MS-777"));
      Use output(Class) to get it as a given type, also when the lambda started several invocations.
      Throws:
      AssertionError - if no top-level invocation or more than one was captured, or if the invocation failed or was cancelled

    • output

      public <T> T output(Class<T> type)
      Returns what a top-level AI Service invocation produced, as the given type: the output of the only top-level invocation whose output is of this type (see AiServiceInvocation.output()). This works also when the lambda started several invocations, as long as only one of them produced a value of this type: 
      // the lambda extracts a Booking with one AI Service, then creates a Client with another one
Booking booking = capture.output(Booking.class);
Client client = capture.output(Client.class);
      Invocations that failed or were cancelled produced nothing, so they never match.
      Type Parameters:
      T - the type of the output
      Parameters:
      type - the type of the output, for example Booking.class or String.class
      Throws:
      AssertionError - if no top-level invocation or more than one produced a value of the given type

    • outputText

      public String outputText()
      Returns the text produced by the only top-level AI Service invocation: the answer of a method that returns a String (also inside a Result or a CompletableFuture), or the text of the final response of a streaming method. A shortcut for invocation().outputText(), see AiServiceInvocation.outputText().
      Throws:
      AssertionError - if no top-level invocation or more than one was captured, if the invocation failed or was cancelled, or if the AI Service method does not produce text (see AiServiceInvocation.outputText())

    • lambdaOutput

      public Object lambdaOutput()
      Returns what the lambda returned. When the lambda returns the result of an AI Service method, this is the same as output(), except for a method that returns a Result, a stream or an asynchronous value (such as a TokenStream, a Flux, a CompletableFuture or a Uni): then this is that Result (whose content is output()), or that stream or value, which has already been started and has ended. Do not start or subscribe to it again: this would call the model again, or fail. When the lambda calls something else, for example a controller with MockMvc, this is what that call returned: 
      AiServiceCapture capture = AiServiceCapture.capture(() -> mockMvc.perform(get("/chat?message=Hi")).andReturn());
MvcResult mvcResult = capture.lambdaOutput(MvcResult.class);
      Throws:
      AssertionError - if the lambda returns nothing (for example () -> assistant.handle("...") with a void method), or if it threw an exception (possible with captureExpectingFailure(Callable))

    • lambdaOutput

      public <T> T lambdaOutput(Class<T> type)
      Returns what the lambda returned, as the given type. See lambdaOutput().
      Type Parameters:
      T - the type of the value returned by the lambda
      Parameters:
      type - the type of the value returned by the lambda, for example MvcResult.class
      Throws:
      AssertionError - if the lambda threw an exception, or returned null or a value of another type

    • lambdaOutputItems

      public List<Object> lambdaOutputItems()
      Returns the items of the stream returned by the lambda, in the order they arrived. capture(...) subscribes to a returned publisher (such as a Flux, a Multi or a Flow.Publisher) and keeps every item it receives, whatever its type: for example the Strings streamed by an AI Service method, or the server-sent events that a controller built from them. 
      AiServiceCapture capture = AiServiceCapture.capture(() -> streamingController.chat("Hi"));
assertThat(capture.lambdaOutputItems()).hasSizeGreaterThan(1);
      Throws:
      AssertionError - if the lambda did not return a stream, for example when it returned a TokenStream, a Mono or a value, or when it threw an exception

    • invocation

      public AiServiceInvocation invocation()
      Returns the only top-level AI Service invocation (see invocations()).
      Throws:
      AssertionError - if no top-level invocation or more than one was captured: use invocations() then

    • invocations

      public List<AiServiceInvocation> invocations()
      Returns the top-level AI Service invocations: those not started by another captured invocation, in the order they started. Empty if none was captured. The invocations started by their work (for example an AI Service called by a tool) are nested in them, at any depth: see AiServiceInvocation.nestedInvocations().

    • invocation

      public AiServiceInvocation invocation(Class<?> aiService)
      Returns the only top-level AI Service invocation of the given AI Service interface (see invocations()). Useful when the lambda starts several invocations, for example an agentic workflow or supervisor, which calls each of its agents as a separate top-level invocation: 
      AiServiceCapture capture = capture(() -> supervisor.invoke("Book a flight"));
assertThat(capture.invocation(BookingAgent.class)).hasToolExecuted("bookFlight");
      Parameters:
      aiService - the AI Service interface, for example BookingAgent.class
      Throws:
      AssertionError - if no top-level invocation of this interface or more than one was captured: use invocations(Class) then

    • invocations

      public List<AiServiceInvocation> invocations(Class<?> aiService)
      Returns the top-level AI Service invocations of the given AI Service interface (see invocations()), in the order they started. Empty if none was captured.
      Parameters:
      aiService - the AI Service interface, for example BookingAgent.class

    • firstRequest

      public ChatRequest firstRequest()
      Returns the first request sent to the chat model by the only top-level AI Service invocation. A shortcut for invocation().firstRequest(), see AiServiceInvocation.firstRequest(). 
      assertThat(capture.firstRequest()).hasSystemMessageContaining("support agent");
      Throws:
      AssertionError - if no top-level invocation or more than one was captured, or if the model did not respond to any request of the invocation

    • lastRequest

      public ChatRequest lastRequest()
      Returns the last request sent to the chat model by the only top-level AI Service invocation. A shortcut for invocation().lastRequest(), see AiServiceInvocation.lastRequest(). Its messages are the conversation that led to the last response, which can be given to a judge: 
      Judge.from(judgeModel)
        .assertThat(capture.outputText())
        .given(capture.lastRequest().messages())
        .satisfies("only uses facts from the tool results");
      Throws:
      AssertionError - if no top-level invocation or more than one was captured, or if the model did not respond to any request of the invocation

    • lastResponse

      public ChatResponse lastResponse()
      Returns the last response of the chat model to the only top-level AI Service invocation. A shortcut for invocation().lastResponse(), see AiServiceInvocation.lastResponse().
      Throws:
      AssertionError - if no top-level invocation or more than one was captured, or if the model did not respond

    • toString

      public String toString()
      Overrides:
      toString in class Object