Class BedrockBatchChatModel

java.lang.Object
dev.langchain4j.model.bedrock.BedrockBatchChatModel
All Implemented Interfaces:
BatchChatModel, Closeable, AutoCloseable
@Experimental public final class BedrockBatchChatModel extends Object implements BatchChatModel, Closeable
A BatchChatModel for the Amazon Bedrock batch inference API, which processes multiple chat requests asynchronously at a lower price than on-demand inference for supported models.

Jobs are created with the Converse invocation type. Each ChatRequest is written as a JSONL record to Amazon S3, a model invocation job is submitted, and the results are read back from the S3 output location. All requests in a batch run against the single model configured on this batch model, because Bedrock runs one model per job.

Bedrock batch inference does not support tool calling, structured output or prompt caching, so a ChatRequest that specifies tools, a JSON response format or cache points is rejected with an UnsupportedFeatureException. So is a request that specifies a service tier or a model other than BedrockBatchChatModel.Builder.modelId(String).

See Also:

  • Method Details

    • submit

      public BatchResponse<ChatResponse> submit(BatchRequest<ChatRequest> request)
      Writes the requests as one JSONL file to Amazon S3 and creates a Bedrock model invocation job for it.

      The input file is uploaded under BedrockBatchChatModel.Builder.inputS3Uri(String) and is not deleted afterwards, so it is stored and billed by S3 until you remove it; the same applies to the output Bedrock writes. Bedrock enforces a minimum and a maximum number of records per job as service quotas, and rejects a job that falls outside them.

      Specified by:
      submit in interface BatchChatModel
      Parameters:
      request - the list of chat requests to process in the batch
      Returns:
      a BatchResponse representing the initial state of the batch operation

    • retrieve

      public BatchResponse<ChatResponse> retrieve(String batchId)
      Retrieves the job and, once it has ended, its results.

      Results are read from the output location recorded on the job, for every ended job that produced output. This includes the partial results of a stopped or expired job, since Bedrock charges for the records it processed. Every parsed result is held in memory.

      Results are correlated by recordId, using the identifiers that submit(BatchRequest) generates, and a request that produced no result is represented by a failure. If any identifier is missing, malformed, duplicated or outside the submitted range - which happens for a job that was not submitted through this model - correlation is abandoned for the whole job and the results are returned in the order the output files list them, with a warning logged. Results are never discarded.

      BatchError.code() holds the errorCode Bedrock reports for a record that failed, and 0 for a request that produced no result or for a job-level failure.

      Specified by:
      retrieve in interface BatchChatModel
      Parameters:
      batchId - the batch identifier obtained from BatchChatModel.submit(BatchRequest)
      Returns:
      a BatchResponse representing the current state of the chat batch operation

    • cancel

      public void cancel(String batchId)
      Description copied from interface: BatchChatModel
      Cancels a chat batch operation that is currently pending or running.
      Specified by:
      cancel in interface BatchChatModel
      Parameters:
      batchId - the batch identifier to cancel

    • list

      public BatchPage<ChatResponse> list(@Nullable BatchPagination pagination)
      Description copied from interface: BatchChatModel
      Lists chat batch jobs with optional pagination.
      Specified by:
      list in interface BatchChatModel
      Parameters:
      pagination - the maximum number of batch jobs to return and token for retrieving a specific page; if null, uses server default
      Returns:
      a BatchPage containing chat batch responses and pagination information

    • close

      public void close()
      Closes the Bedrock and S3 clients this model created. A client or credentials provider passed to the BedrockBatchChatModel.Builder belongs to the caller and is left open.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • builder

      public static BedrockBatchChatModel.Builder builder()
      Returns:
      a new BedrockBatchChatModel.Builder for BedrockBatchChatModel.