Uses of Interface
dev.langchain4j.model.Tokenizer

Packages that use Tokenizer

Package	Description
dev.langchain4j.data.document.splitter
dev.langchain4j.memory.chat
dev.langchain4j.model.azure
dev.langchain4j.model.dashscope
dev.langchain4j.model.openai
dev.langchain4j.model.qianfan.client

Uses of Tokenizer in dev.langchain4j.data.document.splitter

Fields in dev.langchain4j.data.document.splitter declared as Tokenizer

Modifier and Type	Field	Description
protected final Tokenizer	HierarchicalDocumentSplitter.tokenizer

Methods in dev.langchain4j.data.document.splitter with parameters of type Tokenizer

Modifier and Type	Method	Description
static DocumentSplitter	DocumentSplitters.recursive(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)	This is a recommended DocumentSplitter for generic text.

Constructors in dev.langchain4j.data.document.splitter with parameters of type Tokenizer

Modifier	Constructor	Description
	DocumentByCharacterSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)
	DocumentByCharacterSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)
	DocumentByLineSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)
	DocumentByLineSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)
	DocumentByParagraphSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)
	DocumentByParagraphSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)
	DocumentByRegexSplitter(String regex, String joinDelimiter, int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)
	DocumentByRegexSplitter(String regex, String joinDelimiter, int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)
	DocumentBySentenceSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)
	DocumentBySentenceSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)
	DocumentByWordSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)
	DocumentByWordSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)
protected	HierarchicalDocumentSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer)	Creates a new instance of HierarchicalDocumentSplitter.
protected	HierarchicalDocumentSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, Tokenizer tokenizer, DocumentSplitter subSplitter)	Creates a new instance of HierarchicalDocumentSplitter.

Uses of Tokenizer in dev.langchain4j.memory.chat

Methods in dev.langchain4j.memory.chat with parameters of type Tokenizer

Modifier and Type	Method	Description
TokenWindowChatMemory.Builder	TokenWindowChatMemory.Builder.maxTokens(Integer maxTokens, Tokenizer tokenizer)
static TokenWindowChatMemory	TokenWindowChatMemory.withMaxTokens(int maxTokens, Tokenizer tokenizer)

Uses of Tokenizer in dev.langchain4j.model.azure

Classes in dev.langchain4j.model.azure that implement Tokenizer

Modifier and Type	Class	Description
class	AzureOpenAiTokenizer	This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming.

Methods in dev.langchain4j.model.azure with parameters of type Tokenizer

Modifier and Type	Method	Description
AzureOpenAiChatModel.Builder	AzureOpenAiChatModel.Builder.tokenizer(Tokenizer tokenizer)
AzureOpenAiEmbeddingModel.Builder	AzureOpenAiEmbeddingModel.Builder.tokenizer(Tokenizer tokenizer)
AzureOpenAiLanguageModel.Builder	AzureOpenAiLanguageModel.Builder.tokenizer(Tokenizer tokenizer)
AzureOpenAiStreamingChatModel.Builder	AzureOpenAiStreamingChatModel.Builder.tokenizer(Tokenizer tokenizer)
AzureOpenAiStreamingLanguageModel.Builder	AzureOpenAiStreamingLanguageModel.Builder.tokenizer(Tokenizer tokenizer)

Constructors in dev.langchain4j.model.azure with parameters of type Tokenizer

Modifier	Constructor	Description
	AzureOpenAiChatModel(com.azure.ai.openai.OpenAIClient client, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, List<ChatModelListener> listeners)
	AzureOpenAiChatModel(String endpoint, String serviceVersion, com.azure.core.credential.KeyCredential keyCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, List<ChatModelListener> listeners, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiChatModel(String endpoint, String serviceVersion, com.azure.core.credential.TokenCredential tokenCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, List<ChatModelListener> listeners, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiChatModel(String endpoint, String serviceVersion, String apiKey, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, List<ChatModelListener> listeners, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiEmbeddingModel(String endpoint, String serviceVersion, com.azure.core.credential.KeyCredential keyCredential, String deploymentName, Tokenizer tokenizer, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Integer dimensions, Map<String,String> customHeaders)
	AzureOpenAiEmbeddingModel(String endpoint, String serviceVersion, com.azure.core.credential.TokenCredential tokenCredential, String deploymentName, Tokenizer tokenizer, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Integer dimensions, Map<String,String> customHeaders)
	AzureOpenAiEmbeddingModel(String endpoint, String serviceVersion, String apiKey, String deploymentName, Tokenizer tokenizer, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Integer dimensions, Map<String,String> customHeaders)
	AzureOpenAiLanguageModel(com.azure.ai.openai.OpenAIClient client, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Integer bestOf)
	AzureOpenAiLanguageModel(String endpoint, String serviceVersion, com.azure.core.credential.KeyCredential keyCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Integer bestOf, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiLanguageModel(String endpoint, String serviceVersion, com.azure.core.credential.TokenCredential tokenCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Integer bestOf, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiLanguageModel(String endpoint, String serviceVersion, String apiKey, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Integer bestOf, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiStreamingChatModel(com.azure.ai.openai.OpenAIClient client, com.azure.ai.openai.OpenAIAsyncClient asyncClient, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, List<ChatModelListener> listeners)
	AzureOpenAiStreamingChatModel(String endpoint, String serviceVersion, com.azure.core.credential.KeyCredential keyCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, boolean useAsyncClient, List<ChatModelListener> listeners, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiStreamingChatModel(String endpoint, String serviceVersion, com.azure.core.credential.TokenCredential tokenCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, boolean useAsyncClient, List<ChatModelListener> listeners, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiStreamingChatModel(String endpoint, String serviceVersion, String apiKey, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, List<String> stop, Double presencePenalty, Double frequencyPenalty, List<com.azure.ai.openai.models.AzureChatExtensionConfiguration> dataSources, com.azure.ai.openai.models.AzureChatEnhancementConfiguration enhancements, Long seed, com.azure.ai.openai.models.ChatCompletionsResponseFormat responseFormat, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, boolean useAsyncClient, List<ChatModelListener> listeners, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiStreamingLanguageModel(com.azure.ai.openai.OpenAIClient client, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty)
	AzureOpenAiStreamingLanguageModel(String endpoint, String serviceVersion, com.azure.core.credential.KeyCredential keyCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiStreamingLanguageModel(String endpoint, String serviceVersion, com.azure.core.credential.TokenCredential tokenCredential, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Map<String,String> customHeaders)
	AzureOpenAiStreamingLanguageModel(String endpoint, String serviceVersion, String apiKey, String deploymentName, Tokenizer tokenizer, Integer maxTokens, Double temperature, Double topP, Map<String,Integer> logitBias, String user, Integer n, Integer logprobs, Boolean echo, List<String> stop, Double presencePenalty, Double frequencyPenalty, Duration timeout, Integer maxRetries, com.azure.core.http.ProxyOptions proxyOptions, boolean logRequestsAndResponses, String userAgentSuffix, Map<String,String> customHeaders)

Uses of Tokenizer in dev.langchain4j.model.dashscope

Classes in dev.langchain4j.model.dashscope that implement Tokenizer

Modifier and Type	Class	Description
class	QwenTokenizer

Uses of Tokenizer in dev.langchain4j.model.openai

Classes in dev.langchain4j.model.openai that implement Tokenizer

Modifier and Type	Class	Description
class	OpenAiTokenizer	This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming.

Methods in dev.langchain4j.model.openai with parameters of type Tokenizer

Modifier and Type	Method	Description
Response<AiMessage>	OpenAiStreamingResponseBuilder.build(Tokenizer tokenizer, boolean forcefulToolExecution)

Constructors in dev.langchain4j.model.openai with parameters of type Tokenizer

Modifier	Constructor	Description
	OpenAiChatModel(String baseUrl, String apiKey, String organizationId, String modelName, Double temperature, Double topP, List<String> stop, Integer maxTokens, Double presencePenalty, Double frequencyPenalty, Map<String,Integer> logitBias, String responseFormat, Boolean strictJsonSchema, Integer seed, String user, Boolean strictTools, Boolean parallelToolCalls, Duration timeout, Integer maxRetries, Proxy proxy, Boolean logRequests, Boolean logResponses, Tokenizer tokenizer, Map<String,String> customHeaders, List<ChatModelListener> listeners)
	OpenAiEmbeddingModel(String baseUrl, String apiKey, String organizationId, String modelName, Integer dimensions, String user, Duration timeout, Integer maxRetries, Proxy proxy, Boolean logRequests, Boolean logResponses, Tokenizer tokenizer, Map<String,String> customHeaders)
	OpenAiLanguageModel(String baseUrl, String apiKey, String organizationId, String modelName, Double temperature, Duration timeout, Integer maxRetries, Proxy proxy, Boolean logRequests, Boolean logResponses, Tokenizer tokenizer, Map<String,String> customHeaders)
	OpenAiStreamingChatModel(String baseUrl, String apiKey, String organizationId, String modelName, Double temperature, Double topP, List<String> stop, Integer maxTokens, Double presencePenalty, Double frequencyPenalty, Map<String,Integer> logitBias, String responseFormat, Integer seed, String user, Boolean strictTools, Boolean parallelToolCalls, Duration timeout, Proxy proxy, Boolean logRequests, Boolean logResponses, Tokenizer tokenizer, Map<String,String> customHeaders, List<ChatModelListener> listeners)
	OpenAiStreamingLanguageModel(String baseUrl, String apiKey, String organizationId, String modelName, Double temperature, Duration timeout, Proxy proxy, Boolean logRequests, Boolean logResponses, Tokenizer tokenizer, Map<String,String> customHeaders)

Uses of Tokenizer in dev.langchain4j.model.qianfan.client

Methods in dev.langchain4j.model.qianfan.client with parameters of type Tokenizer

Modifier and Type	Method	Description
Response<String>	QianfanStreamingResponseBuilder.build(Tokenizer tokenizer)
Response<AiMessage>	QianfanStreamingResponseBuilder.build(Tokenizer tokenizer, boolean forcefulToolExecution)