Skip to main content

In-process (ONNX)

LangChain4j provides a few popular local embedding models packaged as maven dependencies. They are powered by ONNX runtime and are running in the same java process.

Each model is provided in 2 flavours: original and quantized (has a -q suffix in maven artifact name and Quantized in the class name).

For example:

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId>
    <version>1.23.0-beta33</version>
</dependency>
EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();
Response<Embedding> response = embeddingModel.embed("test");
Embedding embedding = response.content();

Or quantized:

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-embeddings-all-minilm-l6-v2-q</artifactId>
    <version>1.23.0-beta33</version>
</dependency>
EmbeddingModel embeddingModel = new AllMiniLmL6V2QuantizedEmbeddingModel();
Response<Embedding> response = embeddingModel.embed("test");
Embedding embedding = response.content();

The complete list of all embedding models can be found here.

Windows

ONNX runtime needs version 14.40 or newer of the Microsoft Visual C++ runtime library msvcp140.dll. Many JDKs ship their own copy of this file in their bin folder, and Windows uses that copy first. If your JDK ships an older copy, the model fails to load, and the root cause of the error is java.lang.UnsatisfiedLinkError: ...\onnxruntime.dll: A dynamic link library (DLL) initialization routine failed. To fix it, update your JDK to a recent build that ships msvcp140.dll version 14.40 or newer.

Parallelization​

By default, the embedding process is parallelized using all available CPU cores, so each TextSegment is embedded in a separate thread.

The parallelization is done by using an Executor. By default, in-process embedding models use a cached thread pool with the number of threads equal to the number of available processors. Threads are cached for 1 second.

You can provide a custom instance of the Executor when creating a model:

Executor = ...;
EmbeddingModel embeddingModel = new AllMiniLmL6V2QuantizedEmbeddingModel(executor);

Embedding using GPU is not supported yet.

Custom models​

Many models (e.g., from Hugging Face) can be used, as long as they are in the ONNX format.

Information on how to convert models into ONNX format can be found here.

Many models already converted to ONNX format are available here.

Example of using custom embedding model:

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-embeddings</artifactId>
    <version>1.23.0-beta33</version>
</dependency>
String pathToModel = "/home/langchain4j/model.onnx";
String pathToTokenizer = "/home/langchain4j/tokenizer.json";
PoolingMode poolingMode = PoolingMode.MEAN;
EmbeddingModel embeddingModel = new OnnxEmbeddingModel(pathToModel, pathToTokenizer, poolingMode);

Response<Embedding> response = embeddingModel.embed("test");
Embedding embedding = response.content();

Examples​