Class QwenTokenizer

java.lang.Object
dev.langchain4j.model.dashscope.QwenTokenizer
All Implemented Interfaces:
Tokenizer

public class QwenTokenizer extends Object implements Tokenizer
  • Constructor Details

    • QwenTokenizer

      public QwenTokenizer(String apiKey, String modelName)
  • Method Details

    • estimateTokenCountInText

      public int estimateTokenCountInText(String text)
      Description copied from interface: Tokenizer
      Estimates the count of tokens in the given text.
      Specified by:
      estimateTokenCountInText in interface Tokenizer
      Parameters:
      text - the text.
      Returns:
      the estimated count of tokens.
    • estimateTokenCountInMessage

      public int estimateTokenCountInMessage(ChatMessage message)
      Description copied from interface: Tokenizer
      Estimates the count of tokens in the given message.
      Specified by:
      estimateTokenCountInMessage in interface Tokenizer
      Parameters:
      message - the message.
      Returns:
      the estimated count of tokens.
    • estimateTokenCountInMessages

      public int estimateTokenCountInMessages(Iterable<ChatMessage> messages)
      Description copied from interface: Tokenizer
      Estimates the count of tokens in the given messages.
      Specified by:
      estimateTokenCountInMessages in interface Tokenizer
      Parameters:
      messages - the messages.
      Returns:
      the estimated count of tokens.
    • estimateTokenCountInToolSpecifications

      public int estimateTokenCountInToolSpecifications(Iterable<ToolSpecification> toolSpecifications)
      Description copied from interface: Tokenizer
      Estimates the count of tokens in the given tool specifications.
      Specified by:
      estimateTokenCountInToolSpecifications in interface Tokenizer
      Parameters:
      toolSpecifications - the tool specifications.
      Returns:
      the estimated count of tokens.
    • estimateTokenCountInToolExecutionRequests

      public int estimateTokenCountInToolExecutionRequests(Iterable<ToolExecutionRequest> toolExecutionRequests)
      Description copied from interface: Tokenizer
      Estimates the count of tokens in the given tool execution requests.
      Specified by:
      estimateTokenCountInToolExecutionRequests in interface Tokenizer
      Parameters:
      toolExecutionRequests - the tool execution request.
      Returns:
      the estimated count of tokens.
    • isBlank

      public static boolean isBlank(CharSequence cs)
    • builder

      public static QwenTokenizer.QwenTokenizerBuilder builder()