Interface StreamingChatModel

All Known Implementing Classes:
AnthropicStreamingChatModel, AzureOpenAiStreamingChatModel, BedrockStreamingChatModel, DisabledStreamingChatModel, GoogleAiGeminiStreamingChatModel, GoogleGenAiStreamingChatModel, GPULlama3StreamingChatModel, JlamaStreamingChatModel, LocalAiStreamingChatModel, MistralAiStreamingChatModel, OllamaStreamingChatModel, OpenAiOfficialResponsesStreamingChatModel, OpenAiOfficialStreamingChatModel, OpenAiResponsesStreamingChatModel, OpenAiStreamingChatModel, VertexAiAnthropicStreamingChatModel, VertexAiGeminiStreamingChatModel, WatsonxDeploymentStreamingChatModel, WatsonxGatewayStreamingChatModel, WatsonxStreamingChatModel
public interface StreamingChatModel
Represents a language model that has a chat API and can stream a response one token at a time.
See Also:

  • Method Details

    • chat

      default void chat(ChatRequest request, StreamingChatResponseHandler handler)
      This is the main API to interact with the chat model.
      Parameters:
      request - a ChatRequest, containing all the inputs to the LLM
      handler - a StreamingChatResponseHandler that will handle streaming response from the LLM

    • chat

      default void chat(ChatRequest request, ChatRequestOptions options, StreamingChatResponseHandler handler)
      Sends a streaming chat request with additional invocation options.
      Parameters:
      request - a ChatRequest, containing all the inputs to the LLM
      options - a ChatRequestOptions carrying listener attributes and other per-call metadata
      handler - a StreamingChatResponseHandler that will handle streaming response from the LLM
      Since:
      1.13.0

    • doChat

      default void doChat(ChatRequest chatRequest, StreamingChatResponseHandler handler)

    • chat

      @Experimental default Flow.Publisher<ChatModelStreamingEvent> chat(ChatRequest request)
      Reactive entry point: sends a chat request and returns a Flow.Publisher of ChatModelStreamingEvents.

      The publisher is cold: nothing happens until you subscribe, and each subscribe() call initiates a new LLM request. It emits, in this order (each CompleteToolCall arriving as soon as that tool call finishes assembling, so it interleaves with the next call's chunks):

      followed by onComplete. On failure, onError is signaled after onSubscribe.

      Registered ChatModelListeners are invoked: onRequest on each new subscription (just before the underlying request goes out), onResponse after the terminal ChatResponse is emitted, onError on failure.

      If the Flow.Subscriber throws from onNext (or any other signal method), it violates the Reactive Streams contract (Rule 2.13): the stream is cancelled and no further events are delivered, and no ChatModelListener callback fires for it — neither onResponse nor onError. This differs from the handler-based chat(ChatRequest, StreamingChatResponseHandler) path, which catches exceptions thrown from handler callbacks, reports them to onError, and keeps streaming.

      Subscribers must be prepared to receive ChatModelStreamingEvent subtypes they do not recognize and ignore them. New event types may be introduced over time (and providers may surface unmapped events as RawStreamingEvent), so consuming this stream with an exhaustive type switch that lacks a default branch is unsafe.

      Demand and back-pressure. This streams a finite, bounded-rate source — an LLM response over HTTP. Implementations are not required to propagate subscriber demand to the model: meaningfully throttling an LLM is impractical (its work and cost are incurred regardless of how fast the response is read, and stalling the transport to slow it down only risks provider/proxy idle timeouts). An implementation therefore typically consumes the response eagerly and relays events through a bounded internal buffer. A subscriber that requests fewer items than are produced may thus cause buffering and, once the buffer is exhausted, a terminal error. Subscribers should request liberally (e.g. Long.MAX_VALUE) and must not block or perform heavy work in onNext — offload it to another thread.

      Threading. Events are delivered on the model's own thread — for HTTP models, the transport's I/O worker that reads the response (the JDK HTTP client's HttpClient-* workers), the same scarce, shared threads a ChatModelListener callback runs on. Blocking there stalls this stream and, under concurrency, degrades throughput for every in-flight call.

      Since:
      1.20.0

    • doChat

      @Experimental default Flow.Publisher<ChatModelStreamingEvent> doChat(ChatRequest chatRequest)
      Provider-specific implementation of the reactive stream returned by chat(ChatRequest) (which wraps it with ChatModelListener invocation). Implementations must honor the event ordering and the demand / back-pressure expectations documented on chat(ChatRequest) — in particular, they typically consume the response eagerly and relay ChatModelStreamingEvents through a bounded buffer rather than propagating subscriber demand to the model.

      The default implementation returns an immediately-failing Publisher carrying AsyncNotSupportedException to signal that this model has no native reactive-streaming implementation; a provider that does not support reactive streaming leaves it unimplemented (consistent with ChatModel#doChatAsync and the other async defaults).

      Since:
      1.20.0

    • defaultRequestParameters

      default ChatRequestParameters defaultRequestParameters()

    • listeners

      default List<ChatModelListener> listeners()

    • provider

      default ModelProvider provider()

    • chat

      default void chat(String userMessage, StreamingChatResponseHandler handler)

    • chat

      default void chat(List<ChatMessage> messages, StreamingChatResponseHandler handler)

    • chat

      @Experimental default Flow.Publisher<String> chat(String userMessage)
      Reactive convenience counterpart of chat(String, StreamingChatResponseHandler): returns a cold Publisher that streams the model's textual response to userMessage, token by token.

      This is the streaming analog of the simplified ChatModel.chat(String) (which returns the response String): it emits only the text chunks (PartialResponse.text()), filtering out the other ChatModelStreamingEvents of the underlying chat(ChatRequest) stream. For the full event stream, use chat(ChatMessage...) / chat(List) / chat(ChatRequest).

      Nothing happens until you subscribe: the request is sent on each subscribe() call.

      Since:
      1.20.0

    • chat

      @Experimental default Flow.Publisher<ChatModelStreamingEvent> chat(ChatMessage... messages)
      Reactive convenience overload accepting the messages directly: returns a cold Publisher that streams the response to messages.

      Nothing happens until you subscribe: the request is sent on each subscribe() call.

      Since:
      1.20.0

    • chat

      @Experimental default Flow.Publisher<ChatModelStreamingEvent> chat(List<ChatMessage> messages)
      Reactive convenience counterpart of chat(List, StreamingChatResponseHandler): returns a cold Publisher that streams the response to messages.

      Nothing happens until you subscribe: the request is sent on each subscribe() call.

      Since:
      1.20.0

    • supportedCapabilities

      default Set<Capability> supportedCapabilities()