Class OpenAiOfficialBatchChatModel

java.lang.Object
dev.langchain4j.model.openaiofficial.OpenAiOfficialBatchChatModel
All Implemented Interfaces:
BatchChatModel
@Experimental public final class OpenAiOfficialBatchChatModel extends Object implements BatchChatModel
Submits chat requests to the OpenAI Batch API, which processes them asynchronously at a reduced cost compared to real-time requests.

Requests are serialized to a JSONL file, uploaded through the Files API with the batch purpose, and submitted against the chat completions endpoint. Results are correlated back to the submitted requests by custom_id, so BatchResponse.results() is in submission order even though OpenAI does not guarantee the order of the output file. See retrieve(String) for the one case in which that correlation is not possible.

Every request in a batch must use the same model, so submitting requests that resolve to different models fails fast with an IllegalArgumentException.

Azure OpenAI is supported. Its batch and file operations are scoped to the resource rather than to a deployment, it expects /chat/completions as the batch endpoint, and it identifies the model by deployment name, all of which are handled here. GitHub Models has no Batch API and is rejected with an UnsupportedFeatureException.

See Also:

  • Field Details

    • client

      protected com.openai.client.OpenAIClient client

    • asyncClient

      protected com.openai.client.OpenAIClientAsync asyncClient

    • modelName

      protected String modelName

    • defaultRequestParameters

      protected OpenAiOfficialChatRequestParameters defaultRequestParameters

    • responseFormat

      protected String responseFormat

    • strictJsonSchema

      protected Boolean strictJsonSchema

    • strictTools

      protected Boolean strictTools

    • tokenCountEstimator

      protected TokenCountEstimator tokenCountEstimator

    • listeners

      protected List<ChatModelListener> listeners

    • supportedCapabilities

      protected Set<Capability> supportedCapabilities

    • modelProvider

      protected ModelProvider modelProvider

  • Constructor Details

  • Method Details

    • submit

      public BatchResponse<ChatResponse> submit(BatchRequest<ChatRequest> request)
      Creates a batch of chat requests and submits them for asynchronous processing.

      The returned BatchResponse represents the status of the batch operation.

      Submitting a batch uploads a JSONL input file through the provider's Files API before creating the batch, so that API's storage, retention and quota limits apply in addition to the batch limits, and may change independently of them. Uploaded input files are retained until they expire or are deleted, which OpenAiOfficialBatchChatModel.Builder.inputFileExpiresAfter(Duration) controls.

      OpenAI currently limits a batch to 50,000 requests and its input file to 200 MB; exceeding either is rejected by the API.

      Specified by:
      submit in interface BatchChatModel
      Parameters:
      request - the list of chat requests to process in the batch
      Returns:
      a BatchResponse representing the initial state of the batch operation
      Throws:
      IllegalArgumentException - if the batch is empty, or if the requests do not all resolve to the same model.

    • retrieve

      public BatchResponse<ChatResponse> retrieve(String batchId)
      Retrieves the current state and results of a chat batch operation.

      The response indicates whether the batch is still processing, completed successfully, or failed. Clients should poll this method at intervals until the batch completes.

      Both the output file and the error file are streamed, but every parsed result is held in memory, so a batch close to OpenAI's 200 MB limit requires a correspondingly large heap. Results are returned as soon as OpenAI has produced them, which includes the partial results of an expired or cancelled batch.

      Results are correlated by custom_id, using the identifiers that submit(BatchRequest) generates. If any of them is missing, malformed, duplicated or outside the submitted range - which happens for a batch that was not submitted through this model, for example one created from the OpenAI dashboard - correlation is abandoned for the whole batch and the results are returned in the order the result files list them, with a warning logged. Results are never discarded, so a batch that was paid for is always readable; only their correspondence to the submitted requests is lost.

      BatchError.code() holds the HTTP status code for a request that OpenAI attempted and rejected, and 0 for a request that never ran (for example an expired one) or for a batch-level failure, because OpenAI reports those with a string code instead. That string code is available under the "code" key of BatchError.details().

      Specified by:
      retrieve in interface BatchChatModel
      Parameters:
      batchId - the batch identifier obtained from BatchChatModel.submit(BatchRequest)
      Returns:
      a BatchResponse representing the current state of the chat batch operation

    • cancel

      public void cancel(String batchId)
      Description copied from interface: BatchChatModel
      Cancels a chat batch operation that is currently pending or running.
      Specified by:
      cancel in interface BatchChatModel
      Parameters:
      batchId - the batch identifier to cancel

    • list

      public BatchPage<ChatResponse> list(@Nullable BatchPagination pagination)
      Description copied from interface: BatchChatModel
      Lists chat batch jobs with optional pagination.
      Specified by:
      list in interface BatchChatModel
      Parameters:
      pagination - the maximum number of batch jobs to return and token for retrieving a specific page; if null, uses server default
      Returns:
      a BatchPage containing chat batch responses and pagination information

    • builder

      public static OpenAiOfficialBatchChatModel.Builder builder()
      Returns:
      a new OpenAiOfficialBatchChatModel.Builder for OpenAiOfficialBatchChatModel.

    • init

      public void init(String baseUrl, String apiKey, com.openai.credential.Credential credential, String microsoftFoundryDeploymentName, com.openai.azure.AzureOpenAIServiceVersion azureOpenAIServiceVersion, String organizationId, boolean isAzure, boolean isGitHubModels, ChatRequestParameters defaultRequestParameters, String modelName, Double temperature, Double topP, List<String> stop, Integer maxCompletionTokens, Double presencePenalty, Double frequencyPenalty, Map<String,Integer> logitBias, String responseFormat, Boolean strictJsonSchema, Integer seed, String user, Boolean strictTools, Boolean parallelToolCalls, Boolean store, Map<String,String> metadata, String serviceTier, Duration timeout, Integer maxRetries, Proxy proxy, TokenCountEstimator tokenCountEstimator, Map<String,String> customHeaders, List<ChatModelListener> listeners, Set<Capability> capabilities, boolean isAsync)

    • defaultRequestParameters

      public OpenAiOfficialChatRequestParameters defaultRequestParameters()

    • supportedCapabilities

      public Set<Capability> supportedCapabilities()

    • listeners

      public List<ChatModelListener> listeners()

    • provider

      public ModelProvider provider()