Interface TokenStream

All Known Implementing Classes:
AiServiceTokenStream
public interface TokenStream
Represents a token stream from the model to which you can subscribe and receive updates when a new partial response (usually a single token) is available, when the model finishes streaming, or when an error occurs during streaming. It is intended to be used as a return type in AI Service.

  • Method Details

    • onPartialResponse

      TokenStream onPartialResponse(Consumer<String> partialResponseHandler)
      The provided consumer will be invoked every time a new partial textual response (usually a single token) from a language model is available.

      Either this or the onPartialResponseWithContext(BiConsumer) callback can be used if you want to consume tokens as soon as they become available.

      Parameters:
      partialResponseHandler - lambda that will be invoked when a model generates a new partial textual response
      Returns:
      token stream instance used to configure or start stream processing
      See Also:

    • onPartialResponseWithContext

      @Experimental default TokenStream onPartialResponseWithContext(BiConsumer<PartialResponse, PartialResponseContext> handler)
      The provided consumer will be invoked every time a new partial textual response (usually a single token) from a language model is available.

      Either this or the onPartialResponse(Consumer) callback can be used if you want to consume tokens as soon as they become available.

      Parameters:
      handler - lambda that will be invoked when a model generates a new partial textual response
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.8.0
      See Also:

    • onPartialThinking

      @Experimental default TokenStream onPartialThinking(Consumer<PartialThinking> partialThinkingHandler)
      The provided consumer will be invoked every time a new partial thinking/reasoning text (usually a single token) from a language model is available.

      Either this or the onPartialThinkingWithContext(BiConsumer) callback can be used if you want to consume thinking tokens as soon as they become available.

      Parameters:
      partialThinkingHandler - lambda that will be invoked when a model generates a new partial thinking/reasoning text
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.2.0
      See Also:

    • onPartialThinkingWithContext

      @Experimental default TokenStream onPartialThinkingWithContext(BiConsumer<PartialThinking, PartialThinkingContext> handler)
      The provided consumer will be invoked every time a new partial thinking/reasoning text (usually a single token) from a language model is available.

      Either this or the onPartialThinking(Consumer) callback can be used if you want to consume thinking tokens as soon as they become available.

      Parameters:
      handler - lambda that will be invoked when a model generates a new partial thinking/reasoning text
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.8.0
      See Also:

    • onPartialToolCall

      @Experimental default TokenStream onPartialToolCall(Consumer<PartialToolCall> partialToolCallHandler)
      The provided consumer will be invoked every time a new partial tool call (usually containing a single token of the tool's arguments) from a language model is available.

      Either this or the onPartialToolCallWithContext(BiConsumer) callback can be used if you want to consume partial tool calls as soon as they become available.

      Parameters:
      partialToolCallHandler - lambda that will be invoked when a model generates a new partial tool call
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.11.0
      See Also:

    • onPartialToolCallWithContext

      @Experimental default TokenStream onPartialToolCallWithContext(BiConsumer<PartialToolCall, PartialToolCallContext> handler)
      The provided consumer will be invoked every time a new partial tool call (usually containing a single token of the tool's arguments) from a language model is available.

      Either this or the onPartialToolCall(Consumer) callback can be used if you want to consume partial tool calls as soon as they become available.

      Parameters:
      handler - lambda that will be invoked when a model generates a new partial tool call
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.11.0
      See Also:

    • onRetrieved

      TokenStream onRetrieved(Consumer<List<Content>> contentHandler)
      The provided consumer will be invoked if any Contents are retrieved using RetrievalAugmentor.

      The invocation happens before any call is made to the language model.

      Parameters:
      contentHandler - lambda that consumes all retrieved contents
      Returns:
      token stream instance used to configure or start stream processing

    • onIntermediateResponse

      default TokenStream onIntermediateResponse(Consumer<ChatResponse> intermediateResponseHandler)
      The provided consumer will be invoked when a language model finishes streaming the intermediate chat response, as opposed to the final response (see onCompleteResponse(Consumer)). Intermediate chat responses contain ToolExecutionRequests, AI service will execute them after returning from this consumer.
      Parameters:
      intermediateResponseHandler - lambda that consumes intermediate chat responses
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.2.0
      See Also:

    • beforeToolExecution

      default TokenStream beforeToolExecution(Consumer<BeforeToolExecution> beforeToolExecutionHandler)
      The provided consumer will be invoked right before a tool is executed.
      Parameters:
      beforeToolExecutionHandler - lambda that consumes BeforeToolExecution
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.2.0

    • onUnmappedRawEvent

      @Experimental default TokenStream onUnmappedRawEvent(Consumer<Object> rawEventHandler)
      The provided consumer will be invoked when a provider emits a raw streaming event that is not already exposed through one of the typed callbacks (such as onPartialResponse(Consumer), onPartialThinking(Consumer) or onToolExecuted(Consumer)).

      This acts as an escape hatch for provider-specific events that langchain4j does not model, such as server-tool lifecycle events (e.g., OpenAI's web_search_call.in_progress). Events that are already delivered as partial responses, thinking or tool calls are not repeated here.

      The event type depends on the provider implementation. Implementations using the dev.langchain4j.http.client.HttpClient abstraction (e.g., OpenAI, Anthropic, Google AI Gemini) typically expose ServerSentEvent; other implementations can expose provider-specific event objects (e.g., the OpenAI official Responses model exposes the SDK's ResponseStreamEvent).

      Parameters:
      rawEventHandler - lambda that consumes raw provider streaming events
      Returns:
      token stream instance used to configure or start stream processing
      Since:
      1.17.0

    • onToolExecuted

      TokenStream onToolExecuted(Consumer<ToolExecution> toolExecuteHandler)
      The provided consumer will be invoked right after a tool is executed.

      The invocation happens after the tool method has finished and before any other tool is executed.

      Parameters:
      toolExecuteHandler - lambda that consumes ToolExecution
      Returns:
      token stream instance used to configure or start stream processing

    • onCompleteResponse

      TokenStream onCompleteResponse(Consumer<ChatResponse> completeResponseHandler)
      The provided consumer will be invoked when a language model finishes streaming the final chat response, as opposed to the intermediate response (see onIntermediateResponse(Consumer)).

      Please note that ChatResponse.tokenUsage() contains aggregate token usage across all calls to the LLM. It is a sum of ChatResponse.tokenUsage()s of all intermediate responses (onIntermediateResponse(Consumer)).

      Parameters:
      completeResponseHandler - lambda that will be invoked when language model finishes streaming
      Returns:
      token stream instance used to configure or start stream processing
      See Also:

    • onError

      TokenStream onError(Consumer<Throwable> errorHandler)
      The provided consumer will be invoked when an error occurs during streaming.
      Parameters:
      errorHandler - lambda that will be invoked when an error occurs
      Returns:
      token stream instance used to configure or start stream processing

    • ignoreErrors

      TokenStream ignoreErrors()
      All errors during streaming will be ignored (but will be logged with a WARN log level).
      Returns:
      token stream instance used to configure or start stream processing

    • start

      void start()
      Completes the current token stream building and starts processing.

      Will send a request to LLM and start response streaming.