Interface StreamingHandle

@Experimental
public interface StreamingHandle

Handle that can be used to cancel the streaming done via StreamingChatResponseHandler.

The handle is passed to the StreamingChatResponseHandler callbacks that accept a context: StreamingChatResponseHandler.onPartialResponse(PartialResponse, PartialResponseContext), StreamingChatResponseHandler.onPartialThinking(PartialThinking, PartialThinkingContext) and StreamingChatResponseHandler.onPartialToolCall(PartialToolCall, PartialToolCallContext), so it becomes available as soon as the model streams its first partial response (there is nothing to cancel through it before that). You can call cancel() right inside the callback, or keep a reference to the handle and call cancel() later, for example when the user presses a "Stop" button:

AtomicReference<StreamingHandle> streamingHandle = new AtomicReference<>();

model.chat("Tell me a long story", new StreamingChatResponseHandler() {

    @Override
    public void onPartialResponse(PartialResponse partialResponse, PartialResponseContext context) {
        streamingHandle.set(context.streamingHandle());
        System.out.print(partialResponse.text());
    }

    @Override
    public void onCompleteResponse(ChatResponse completeResponse) {
    }

    @Override
    public void onError(Throwable error) {
        error.printStackTrace();
    }
});

// later, for example when the user presses "Stop":
StreamingHandle handle = streamingHandle.get();
if (handle != null) { // null until the first partial response arrives
    handle.cancel();
}

Depending on the StreamingChatModel implementation, cancelling also closes the connection to the LLM provider, so that it can stop generating further tokens.

When using AI Services, the same handle is available in the TokenStream callbacks whose names end with WithContext, such as TokenStream.onPartialResponseWithContext(BiConsumer).

Since:

1.8.0

Method Summary

Modifier and Type	Method	Description
void	cancel()	Cancels the streaming.
boolean	isCancelled()	Returns true if streaming was cancelled by calling cancel().

Method Details

cancel

void cancel()

Cancels the streaming. Once this method has been called, the StreamingChatResponseHandler does not receive any further callbacks, including StreamingChatResponseHandler.onCompleteResponse(ChatResponse) and StreamingChatResponseHandler.onError(Throwable).

Throws:

UnsupportedFeatureException - if the StreamingChatModel implementation does not support cancellation

isCancelled

boolean isCancelled()

Returns true if streaming was cancelled by calling cancel().