Skip to main content

Ollama

Ollama runs decision models, such as Nimble, Tev1 and Clef, locally on your machine, with no API key and no cost per request. OllamaDecisionModel implements the DecisionModel API on top of Ollama's System One API (POST /v1/systemone), available in Ollama v0.35.0 or later.

note

This integration is experimental and may change in future releases.

Maven Dependency​

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-ollama</artifactId>
    <version>1.23.0</version>
</dependency>

Usage​

Download a decision model first:

ollama pull nimble

Then:

DecisionModel decisionModel = OllamaDecisionModel.builder()
        .baseUrl("http://localhost:11434")
        .modelName("nimble")
        .build();

DecisionRequest request = DecisionRequest.builder()
        .input("Help! My payouts have been failing for 3 days and nobody answers my emails.")
        .question("team", ChoiceQuestion.builder()
                .text("Which team should handle this ticket?")
                .option("billing", "Payments, payouts, invoices, refunds")
                .option("support", "Problems using the product")
                .option("sales", "Pricing, upgrades, new accounts")
                .build())
        .question("urgent", YesNoQuestion.of("Does this need attention today?"))
        .build();

DecisionResponse response = decisionModel.decide(request);

response.choice("team").value();          // "billing"
response.yesNo("urgent").probability();   // 0.97

The model name must be set, either on the builder or on each request. A model that has not been downloaded is rejected by Ollama, and the request fails with a ModelNotFoundException.

Without a GPU, or while Ollama loads the model, a request can take several seconds, so consider a longer timeout(...). Ollama unloads a model after 5 minutes without requests (unless configured otherwise on the server): to keep it loaded longer, set keepAlive(...) in seconds, or a negative value to keep it loaded. The other settings are httpClientBuilder, maxRetries, logRequests, logResponses, logger, customHeaders and listeners.

Both decide() and decideAsync() are supported.

Input​

The input can be:

  • text;
  • a Map of named values, which is sent as a JSON object;
  • a list of TextContents and ImageContents, whose texts are joined with blank lines.

The whole input must fit in the context window the model is loaded with (it depends on the model and on the Ollama server's configuration): Ollama never truncates it. The request body must also be at most 64 KiB, or 32 MiB with images.

Images​

Models with vision, such as Clef and Clef Flash (Ollama v0.35.1 or later), can judge PNG, JPEG and WebP images:

DecisionModel decisionModel = OllamaDecisionModel.builder()
        .baseUrl("http://localhost:11434")
        .modelName("clef-flash")
        .build();

DecisionRequest request = DecisionRequest.builder()
        .input(List.of(
                TextContent.from("The customer says the package arrived like this."),
                ImageContent.from(base64Image, "image/jpeg")))
        .question("damaged", YesNoQuestion.of("Is the item visibly damaged?"))
        .build();

Ollama receives the images separately from the rest of the input, so each image is replaced in the input by its position among the images: the input above is sent as The customer says the package arrived like this. followed by [image 1]. In a Map, an image is replaced the same way, so that its name says what it is: Map.of("comment", comment, "photo", ImageContent.from(...)) is sent as {"comment": "...", "photo": "[image 1]"}. Images are numbered from 1 in the order they appear in the input, across all values of the Map, so questions can refer to them, for example "Does [image 2] show the same item as [image 1]?". This [image N] convention comes from LangChain4j, not from Ollama.

Images can be given as base64 data, or by an http, https or file URL, in which case they are downloaded or read by your application, on the calling thread (also for decideAsync()), before the request is sent. data: URLs are not supported. Other contents, such as audio, are rejected with an UnsupportedFeatureException before calling Ollama.

caution

Do not pass image URLs that come from untrusted sources, such as users, tools or retrieved content: your application would download any http URL and read any local file URL, and send the content to Ollama. Download and check such images yourself, and pass them as base64 data.

Questions and answers​

DecisionModelOllamaNotes
YesNoQuestionnoulyesWhen and noWhen are sent as the criteria when both are set; a single one is appended to the question instead (for example, Answer yes when: ...)
ChoiceQuestionchoiceoptions are sent with their descriptions
ScaleQuestionscorelevels are sent as text, as label: description when they have a description

Yes/no answers include the probability of "yes". Choice and scale answers include the probability of each option or level, and a confidence.

A request can have at most 64 questions. A scale question can have 2 to 10 levels, and a choice question needs at least 2 options; the maximum number of options depends on the model (for example, 26 for Tev1).

Because of these limits, set maxToolsPerDecisionRequest(64) on the tool selection components when there are more than 64 tools, and for re-ranking, lower maxSegmentsPerRequest(...) so that the segments of one request fit in the context window.

response.tokenUsage() reports the input and output tokens counted by Ollama.

Whatever the model, OllamaDecisionModel reports ModelProvider.OLLAMA as its provider, for example in listeners and metrics.

Switching from TypeSafeDecisionModel​

Ollama's decision models can also be called with TypeSafeDecisionModel and a custom base URL, but Ollama rejects some of its requests, for example scale levels with descriptions, and it does not support images. To switch to OllamaDecisionModel, replace the class name: the base URL and the model name stay the same. Listeners and metrics then report OLLAMA instead of TYPESAFE as the provider.