Package dev.langchain4j.model

Interface Tokenizer

All Known Implementing Classes:
AzureOpenAiTokenizer, GoogleAiGeminiTokenizer, OpenAiTokenizer
public interface Tokenizer
Represents an interface for estimating the count of tokens in various text types such as a text, prompt, text segment, etc. This can be useful when it's necessary to know in advance the cost of processing a specified text by the LLM.

  • Method Details

    • estimateTokenCountInText

      int estimateTokenCountInText(String text)
      Estimates the count of tokens in the given text.
      Parameters:
      text - the text.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInMessage

      int estimateTokenCountInMessage(ChatMessage message)
      Estimates the count of tokens in the given message.
      Parameters:
      message - the message.
      Returns:
      the estimated count of tokens.

    • estimateTokenCountInMessages

      int estimateTokenCountInMessages(Iterable<ChatMessage> messages)
      Estimates the count of tokens in the given messages.
      Parameters:
      messages - the messages.
      Returns:
      the estimated count of tokens.