Package dev.langchain4j.model.dashscope

Class QwenTokenizer

java.lang.Object

dev.langchain4j.model.dashscope.QwenTokenizer

All Implemented Interfaces:

Tokenizer

public class QwenTokenizer
extends Object
implements Tokenizer

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	QwenTokenizer.QwenTokenizerBuilder

Constructor Summary

Constructors

Constructor	Description
QwenTokenizer(String apiKey, String modelName)

Method Summary

Modifier and Type	Method	Description
static QwenTokenizer.QwenTokenizerBuilder	builder()
int	estimateTokenCountInMessage(ChatMessage message)	Estimates the count of tokens in the given message.
int	estimateTokenCountInMessages(Iterable<ChatMessage> messages)	Estimates the count of tokens in the given messages.
int	estimateTokenCountInText(String text)	Estimates the count of tokens in the given text.
int	estimateTokenCountInToolExecutionRequests(Iterable<ToolExecutionRequest> toolExecutionRequests)	Estimates the count of tokens in the given tool execution requests.
int	estimateTokenCountInToolSpecifications(Iterable<ToolSpecification> toolSpecifications)	Estimates the count of tokens in the given tool specifications.
static boolean	isBlank(CharSequence cs)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolExecutionRequest, estimateTokenCountInForcefulToolSpecification, estimateTokenCountInTools, estimateTokenCountInTools

Constructor Details

QwenTokenizer

public QwenTokenizer(String apiKey,
 String modelName)

Method Details

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Description copied from interface: Tokenizer

Estimates the count of tokens in the given text.

Specified by:

estimateTokenCountInText in interface Tokenizer

Parameters:

text - the text.

Returns:

the estimated count of tokens.

estimateTokenCountInMessage

public int estimateTokenCountInMessage(ChatMessage message)

Description copied from interface: Tokenizer

Estimates the count of tokens in the given message.

Specified by:

estimateTokenCountInMessage in interface Tokenizer

Parameters:

message - the message.

Returns:

the estimated count of tokens.

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<ChatMessage> messages)

Description copied from interface: Tokenizer

Estimates the count of tokens in the given messages.

Specified by:

estimateTokenCountInMessages in interface Tokenizer

Parameters:

messages - the messages.

Returns:

the estimated count of tokens.

estimateTokenCountInToolSpecifications

public int estimateTokenCountInToolSpecifications(Iterable<ToolSpecification> toolSpecifications)

Description copied from interface: Tokenizer

Estimates the count of tokens in the given tool specifications.

Specified by:

estimateTokenCountInToolSpecifications in interface Tokenizer

Parameters:

toolSpecifications - the tool specifications.

Returns:

the estimated count of tokens.

estimateTokenCountInToolExecutionRequests

public int estimateTokenCountInToolExecutionRequests(Iterable<ToolExecutionRequest> toolExecutionRequests)

Description copied from interface: Tokenizer

Estimates the count of tokens in the given tool execution requests.

Specified by:

estimateTokenCountInToolExecutionRequests in interface Tokenizer

Parameters:

toolExecutionRequests - the tool execution request.

Returns:

the estimated count of tokens.

isBlank

public static boolean isBlank(CharSequence cs)

builder

public static QwenTokenizer.QwenTokenizerBuilder builder()