Package dev.langchain4j.model

Interface Tokenizer

All Known Implementing Classes:
AzureOpenAiTokenizer, OpenAiTokenizer, QwenTokenizer
public interface Tokenizer
Represents an interface for estimating the count of tokens in various text types such as a text, prompt, text segment, etc. This can be useful when it's necessary to know in advance the cost of processing a specified text by the LLM.

  • Method Details

    • estimateTokenCountInText

      int estimateTokenCountInText(String text)
      Estimates the count of tokens in the given text.
      Parameters:
      text - the text.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInMessage

      int estimateTokenCountInMessage(ChatMessage message)
      Estimates the count of tokens in the given message.
      Parameters:
      message - the message.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInMessages

      int estimateTokenCountInMessages(Iterable<ChatMessage> messages)
      Estimates the count of tokens in the given messages.
      Parameters:
      messages - the messages.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInTools

      default int estimateTokenCountInTools(Object objectWithTools)
      Estimates the count of tokens in Tool annotations of the given object.
      Parameters:
      objectWithTools - the object.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInTools

      default int estimateTokenCountInTools(Iterable<Object> objectsWithTools)
      Estimates the count of tokens in Tool annotations of the given objects.
      Parameters:
      objectsWithTools - the objects.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInToolSpecifications

      int estimateTokenCountInToolSpecifications(Iterable<ToolSpecification> toolSpecifications)
      Estimates the count of tokens in the given tool specifications.
      Parameters:
      toolSpecifications - the tool specifications.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInForcefulToolSpecification

      default int estimateTokenCountInForcefulToolSpecification(ToolSpecification toolSpecification)
      Estimates the count of tokens in the given tool specification.
      Parameters:
      toolSpecification - the tool specification.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInToolExecutionRequests

      int estimateTokenCountInToolExecutionRequests(Iterable<ToolExecutionRequest> toolExecutionRequests)
      Estimates the count of tokens in the given tool execution requests.
      Parameters:
      toolExecutionRequests - the tool execution request.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInForcefulToolExecutionRequest

      default int estimateTokenCountInForcefulToolExecutionRequest(ToolExecutionRequest toolExecutionRequest)
      Estimates the count of tokens in the given tool execution request.
      Parameters:
      toolExecutionRequest - the tool execution request.
      Returns:
      the estimated count of tokens.