Package dev.langchain4j.model
Interface Tokenizer
- All Known Implementing Classes:
AzureOpenAiTokenizer
,OpenAiTokenizer
,QwenTokenizer
public interface Tokenizer
Represents an interface for estimating the count of tokens in various text types such as a text, prompt, text segment, etc.
This can be useful when it's necessary to know in advance the cost of processing a specified text by the LLM.
-
Method Summary
Modifier and TypeMethodDescriptiondefault int
estimateTokenCountInForcefulToolExecutionRequest
(ToolExecutionRequest toolExecutionRequest) Estimates the count of tokens in the given tool execution request.default int
estimateTokenCountInForcefulToolSpecification
(ToolSpecification toolSpecification) Estimates the count of tokens in the given tool specification.int
estimateTokenCountInMessage
(ChatMessage message) Estimates the count of tokens in the given message.int
estimateTokenCountInMessages
(Iterable<ChatMessage> messages) Estimates the count of tokens in the given messages.int
Estimates the count of tokens in the given text.int
estimateTokenCountInToolExecutionRequests
(Iterable<ToolExecutionRequest> toolExecutionRequests) Estimates the count of tokens in the given tool execution requests.default int
estimateTokenCountInTools
(Iterable<Object> objectsWithTools) Estimates the count of tokens inTool
annotations of the given objects.default int
estimateTokenCountInTools
(Object objectWithTools) Estimates the count of tokens inTool
annotations of the given object.int
estimateTokenCountInToolSpecifications
(Iterable<ToolSpecification> toolSpecifications) Estimates the count of tokens in the given tool specifications.
-
Method Details
-
estimateTokenCountInText
Estimates the count of tokens in the given text.- Parameters:
text
- the text.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInMessage
Estimates the count of tokens in the given message.- Parameters:
message
- the message.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInMessages
Estimates the count of tokens in the given messages.- Parameters:
messages
- the messages.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInTools
Estimates the count of tokens inTool
annotations of the given object.- Parameters:
objectWithTools
- the object.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInTools
Estimates the count of tokens inTool
annotations of the given objects.- Parameters:
objectsWithTools
- the objects.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInToolSpecifications
Estimates the count of tokens in the given tool specifications.- Parameters:
toolSpecifications
- the tool specifications.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInForcefulToolSpecification
Estimates the count of tokens in the given tool specification.- Parameters:
toolSpecification
- the tool specification.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInToolExecutionRequests
Estimates the count of tokens in the given tool execution requests.- Parameters:
toolExecutionRequests
- the tool execution request.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInForcefulToolExecutionRequest
default int estimateTokenCountInForcefulToolExecutionRequest(ToolExecutionRequest toolExecutionRequest) Estimates the count of tokens in the given tool execution request.- Parameters:
toolExecutionRequest
- the tool execution request.- Returns:
- the estimated count of tokens.
-