Package dev.langchain4j.model.azure
Class AzureOpenAiTokenizer
java.lang.Object
dev.langchain4j.model.azure.AzureOpenAiTokenizer
- All Implemented Interfaces:
Tokenizer
This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming.
Magic numbers present in this class were found empirically while testing.
There are integration tests in place that are making sure that the calculations here are very close to that of OpenAI.
-
Constructor Summary
ConstructorsConstructorDescriptionCreates an instance of theAzureOpenAiTokenizer
for the "gpt-3.5-turbo" model.AzureOpenAiTokenizer
(AzureOpenAiChatModelName modelName) Creates an instance of theAzureOpenAiTokenizer
for a givenAzureOpenAiChatModelName
.Creates an instance of theAzureOpenAiTokenizer
for a givenAzureOpenAiEmbeddingModelName
.AzureOpenAiTokenizer
(AzureOpenAiLanguageModelName modelName) Creates an instance of theAzureOpenAiTokenizer
for a givenAzureOpenAiLanguageModelName
.AzureOpenAiTokenizer
(String modelName) Creates an instance of theAzureOpenAiTokenizer
for a given model name. -
Method Summary
Modifier and TypeMethodDescriptionint
estimateTokenCountInMessage
(ChatMessage message) Estimates the count of tokens in the given message.int
estimateTokenCountInMessages
(Iterable<ChatMessage> messages) Estimates the count of tokens in the given messages.int
Estimates the count of tokens in the given text.
-
Constructor Details
-
AzureOpenAiTokenizer
public AzureOpenAiTokenizer()Creates an instance of theAzureOpenAiTokenizer
for the "gpt-3.5-turbo" model. It should be suitable for most OpenAI models, as most of them use the same cl100k_base encoding (except for GPT-4o). -
AzureOpenAiTokenizer
Creates an instance of theAzureOpenAiTokenizer
for a givenAzureOpenAiChatModelName
. -
AzureOpenAiTokenizer
Creates an instance of theAzureOpenAiTokenizer
for a givenAzureOpenAiEmbeddingModelName
. -
AzureOpenAiTokenizer
Creates an instance of theAzureOpenAiTokenizer
for a givenAzureOpenAiLanguageModelName
. -
AzureOpenAiTokenizer
Creates an instance of theAzureOpenAiTokenizer
for a given model name.
-
-
Method Details
-
estimateTokenCountInText
Description copied from interface:Tokenizer
Estimates the count of tokens in the given text.- Specified by:
estimateTokenCountInText
in interfaceTokenizer
- Parameters:
text
- the text.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInMessage
Description copied from interface:Tokenizer
Estimates the count of tokens in the given message.- Specified by:
estimateTokenCountInMessage
in interfaceTokenizer
- Parameters:
message
- the message.- Returns:
- the estimated count of tokens.
-
estimateTokenCountInMessages
Description copied from interface:Tokenizer
Estimates the count of tokens in the given messages.- Specified by:
estimateTokenCountInMessages
in interfaceTokenizer
- Parameters:
messages
- the messages.- Returns:
- the estimated count of tokens.
-