Class HuggingFaceTokenCountEstimator

java.lang.Object
dev.langchain4j.model.embedding.onnx.HuggingFaceTokenCountEstimator
All Implemented Interfaces:
TokenCountEstimator
public class HuggingFaceTokenCountEstimator extends Object implements TokenCountEstimator
A token count estimator for models that can be found on HuggingFace.
Uses DJL's HuggingFaceTokenizer under the hood.
Requires tokenizer.json to instantiate. An example.

  • Constructor Details

    • HuggingFaceTokenCountEstimator

      public HuggingFaceTokenCountEstimator()
      Creates an instance of a HuggingFaceTokenCountEstimator using a built-in tokenizer.json file.

    • HuggingFaceTokenCountEstimator

      public HuggingFaceTokenCountEstimator(Path pathToTokenizer)
      Creates an instance of a HuggingFaceTokenCountEstimator using a provided tokenizer.json file.
      Parameters:
      pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")

    • HuggingFaceTokenCountEstimator

      public HuggingFaceTokenCountEstimator(Path pathToTokenizer, Map<String,String> options)
      Creates an instance of a HuggingFaceTokenCountEstimator using a provided tokenizer.json file and a map of DJL's tokenizer options.
      Parameters:
      pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")
      options - The DJL's tokenizer options

    • HuggingFaceTokenCountEstimator

      public HuggingFaceTokenCountEstimator(String pathToTokenizer)
      Creates an instance of a HuggingFaceTokenCountEstimator using a provided tokenizer.json file.
      Parameters:
      pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")

    • HuggingFaceTokenCountEstimator

      public HuggingFaceTokenCountEstimator(String pathToTokenizer, Map<String,String> options)
      Creates an instance of a HuggingFaceTokenCountEstimator using a provided tokenizer.json file and a map of DJL's tokenizer options.
      Parameters:
      pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")
      options - The DJL's tokenizer options

  • Method Details