In-process (ONNX)

LangChain4j provides local scoring (reranking) models, powered by ONNX runtime, running in the same Java process.

Many models (e.g., from Hugging Face) can be used, as long as they are in the ONNX format.

Information on how to convert models into ONNX format can be found here.

Many models already converted to ONNX format are available here.

Usage

By default, scoring (reranking) model uses the CPU.

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-onnx-scoring</artifactId>
    <version>1.23.0-beta33</version>
</dependency>

String pathToModel = "/home/langchain4j/model.onnx";
String pathToTokenizer = "/home/langchain4j/tokenizer.json";
OnnxScoringModel scoringModel = new OnnxScoringModel(pathToModel, pathToTokenizer);

Response<Double> response = scoringModel.score("query", "passage");
Double score = response.content();

OnnxScoringModel implements AutoCloseable; close it when it is no longer needed to release its native resources.

Windows

ONNX runtime needs version 14.40 or newer of the Microsoft Visual C++ runtime library msvcp140.dll. Many JDKs ship their own copy of this file in their bin folder, and Windows uses that copy first. If your JDK ships an older copy, the model fails to load, and the root cause of the error is java.lang.UnsatisfiedLinkError: ...\onnxruntime.dll: A dynamic link library (DLL) initialization routine failed. To fix it, update your JDK to a recent build that ships msvcp140.dll version 14.40 or newer.

If you want to use the GPU, onnxruntime_gpu version can be found here.

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-onnx-scoring</artifactId>
    <version>1.23.0-beta33</version>
    <exclusions>
        <exclusion>
            <groupId>com.microsoft.onnxruntime</groupId>
            <artifactId>onnxruntime</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<!-- use the same version as the onnxruntime dependency of langchain4j-onnx-scoring; 1.23.2 needs CUDA 12.x (12.8 or newer) and cuDNN 9 -->
<dependency>
    <groupId>com.microsoft.onnxruntime</groupId>
    <artifactId>onnxruntime_gpu</artifactId>
    <version>1.23.2</version>
</dependency>

String pathToModel = "/home/langchain4j/model.onnx";
String pathToTokenizer = "/home/langchain4j/tokenizer.json";

OrtSession.SessionOptions options = new OrtSession.SessionOptions();
options.addCUDA(0);
OnnxScoringModel scoringModel = new OnnxScoringModel(pathToModel, options, pathToTokenizer);

Response<Double> response = scoringModel.score("query", "passage");
Double score = response.content();

Usage​

Usage