Skip to main content

Structured Outputs

note

The term "Structured Outputs" is overloaded and can refer to two things:

  • The general ability of the LLM to generate outputs in a structured format (this is what we cover on this page)
  • The Structured Outputs feature of OpenAI, which applies to both response format and tools (function calling).

Many LLMs and LLM providers support generating outputs in a structured format, typically JSON. These outputs can be easily mapped to Java objects and used in other parts of your application.

For instance, let’s assume we have a Person class:

record Person(String name, int age, double height, boolean married) {
}

We aim to extract a Person object from unstructured text like this:

John is 42 years old and lives an independent life.
He stands 1.75 meters tall and carries himself with confidence.
Currently unmarried, he enjoys the freedom to focus on his personal goals and interests.

Currently, depending on the LLM and the LLM provider, there are four ways how this can be achieved (from most to least reliable):

JSON Schema​

Some LLM providers (e.g., OpenAI and Google Gemini) allow specifying JSON schema for the desired output. You can view all supported LLM providers here in the "JSON Schema" column.

When a JSON schema is specified in the request, the LLM is expected to generate an output that adheres to this schema.

note

Please note that the JSON schema is specified in a dedicated attribute in the request to the LLM provider's API and does not require any free-form instructions to be included in the prompt (e.g., in system or user messages).

LangChain4j supports the JSON Schema feature in both the low-level ChatLanguageModel API and the high-level AI Service API.

Using JSON Schema with ChatLanguageModel​

In the low-level ChatLanguageModel API, JSON schema can be specified using LLM-provider-agnostic ResponseFormat and JsonSchema when creating a ChatRequest:

ResponseFormat responseFormat = ResponseFormat.builder()
        .type(JSON) // type can be either TEXT (default) or JSON
        .jsonSchema(JsonSchema.builder()
                .name("Person") // OpenAI requires specifying the name for the schema
                .rootElement(JsonObjectSchema.builder() // see [1] below
                        .addStringProperty("name")
                        .addIntegerProperty("age")
                        .addNumberProperty("height")
                        .addBooleanProperty("married")
                        .required("name", "age", "height", "married") // see [2] below
                        .build())
                .build())
        .build();

UserMessage userMessage = UserMessage.from("""
        John is 42 years old and lives an independent life.
        He stands 1.75 meters tall and carries himself with confidence.
        Currently unmarried, he enjoys the freedom to focus on his personal goals and interests.
        """);

ChatRequest chatRequest = ChatRequest.builder()
        .responseFormat(responseFormat)
        .messages(userMessage)
        .build();

ChatLanguageModel chatModel = OpenAiChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        .logRequests(true)
        .logResponses(true)
        .build();
// OR
ChatLanguageModel chatModel = GoogleAiGeminiChatModel.builder()
        .apiKey(System.getenv("GOOGLE_AI_GEMINI_API_KEY"))
        .modelName("gemini-1.5-flash")
        .temperature(0.0)
        .logRequestsAndResponses(true)
        .build();

ChatResponse chatResponse = chatModel.chat(chatRequest);

String output = chatResponse.aiMessage().text();
System.out.println(output); // {"name":"John","age":42,"height":1.75,"married":false}

Person person = new ObjectMapper().readValue(output, Person.class);
System.out.println(person); // Person[name=John, age=42, height=1.75, married=false]

Notes:

  • [1] - In most cases, the root element must be of JsonObjectSchema type, however Gemini allows JsonEnumSchema and JsonArraySchema as well.
  • [2] - Required properties must be explicitly specified; otherwise, they are considered optional.

The structure of the JSON schema is defined using JsonSchemaElement interface, with the following subtypes:

  • JsonObjectSchema - for object types.
  • JsonStringSchema - for String, char/Character types.
  • JsonIntegerSchema - for int/Integer, long/Long, BigInteger types.
  • JsonNumberSchema - for float/Float, double/Double, BigDecimal types.
  • JsonBooleanSchema - for boolean/Boolean types.
  • JsonEnumSchema - for enum types.
  • JsonArraySchema - for arrays and collections (e.g., List, Set).
  • JsonReferenceSchema - to support recursion (e.g., Person has a Set<Person> children field).
  • JsonAnyOfSchema - to support polymorphism (e.g., Shape can be either Circle or Rectangle).

JsonObjectSchema​

The JsonObjectSchema represents an object with nested properties. It is usually the root element of the JsonSchema.

There are several ways to add properties to a JsonObjectSchema:

  1. You can add all the properties at once using the properties(Map<String, JsonSchemaElement> properties) method:
JsonSchemaElement citySchema = JsonStringSchema.builder()
        .description("The city for which the weather forecast should be returned")
        .build();

JsonSchemaElement temperatureUnitSchema = JsonEnumSchema.builder()
        .enumValues("CELSIUS", "FAHRENHEIT")
        .build();

Map<String, JsonSchemaElement> properties = Map.of(
        "city", citySchema,
        "temperatureUnit", temperatureUnitSchema
);

JsonSchemaElement rootElement = JsonObjectSchema.builder()
        .properties(properties)
        .required("city") // required properties should be specified explicitly
        .build();
  1. You can add properties individually using the addProperty(String name, JsonSchemaElement jsonSchemaElement) method:
JsonSchemaElement rootElement = JsonObjectSchema.builder()
        .addProperty("city", citySchema)
        .addProperty("temperatureUnit", temperatureUnitSchema)
        .required("city")
        .build();
  1. You can add properties individually using one of the add{Type}Property(String name) or add{Type}Property(String name, String description) methods:
JsonSchemaElement rootElement = JsonObjectSchema.builder()
        .addStringProperty("city", "The city for which the weather forecast should be returned")
        .addEnumProperty("temperatureUnit", List.of("CELSIUS", "FAHRENHEIT"))
        .required("city")
        .build();

Please refer to the Javadoc of the JsonObjectSchema for more details.

JsonStringSchema​

An example of creating JsonStringSchema:

JsonSchemaElement stringSchema = JsonStringSchema.builder()
        .description("The name of the person")
        .build();

JsonIntegerSchema​

An example of creating JsonIntegerSchema:

JsonSchemaElement integerSchema = JsonIntegerSchema.builder()
        .description("The age of the person")
        .build();

JsonNumberSchema​

An example of creating JsonNumberSchema:

JsonSchemaElement numberSchema = JsonNumberSchema.builder()
        .description("The height of the person")
        .build();

JsonBooleanSchema​

An example of creating JsonBooleanSchema:

JsonSchemaElement booleanSchema = JsonBooleanSchema.builder()
        .description("Is the person married?")
        .build();

JsonEnumSchema​

An example of creating JsonEnumSchema:

JsonSchemaElement enumSchema = JsonEnumSchema.builder()
        .description("Marital status of the person")
        .enumValues(List.of("SINGLE", "MARRIED", "DIVORCED"))
        .build();

JsonArraySchema​

An example of creating JsonArraySchema to define an array of strings:

JsonSchemaElement itemSchema = JsonStringSchema.builder()
        .description("The name of the person")
        .build();

JsonSchemaElement arraySchema = JsonArraySchema.builder()
        .description("All names of the people found in the text")
        .items(itemSchema)
        .build();

JsonReferenceSchema​

The JsonReferenceSchema can be used to support recursion:

String reference = "person"; // reference should be unique withing the schema

JsonObjectSchema jsonObjectSchema = JsonObjectSchema.builder()
        .addStringProperty("name")
        .addProperty("children", JsonArraySchema.builder()
                .items(JsonReferenceSchema.builder()
                        .reference(reference)
                        .build())
                .build())
        .required("name", "children")
        .definitions(Map.of(reference, JsonObjectSchema.builder()
                .addStringProperty("name")
                .addProperty("children", JsonArraySchema.builder()
                        .items(JsonReferenceSchema.builder()
                                .reference(reference)
                                .build())
                        .build())
                .required("name", "children")
                .build()))
        .build();
note

The JsonReferenceSchema is currently supported only by OpenAI.

JsonAnyOfSchema​

The JsonAnyOfSchema can be used to support polymorphism:

JsonSchemaElement circleSchema = JsonObjectSchema.builder()
        .addNumberProperty("radius")
        .build();

JsonSchemaElement rectangleSchema = JsonObjectSchema.builder()
        .addNumberProperty("width")
        .addNumberProperty("height")
        .build();

JsonSchemaElement shapeSchema = JsonAnyOfSchema.builder()
        .anyOf(circleSchema, rectangleSchema)
        .build();

JsonSchema jsonSchema = JsonSchema.builder()
        .name("Shapes")
        .rootElement(JsonObjectSchema.builder()
                .addProperty("shapes", JsonArraySchema.builder()
                        .items(shapeSchema)
                        .build())
                .required(List.of("shapes"))
                .build())
        .build();

ResponseFormat responseFormat = ResponseFormat.builder()
        .type(ResponseFormatType.JSON)
        .jsonSchema(jsonSchema)
        .build();

UserMessage userMessage = UserMessage.from("""
        Extract information from the following text:
        1. A circle with a radius of 5
        2. A rectangle with a width of 10 and a height of 20
        """);

ChatRequest chatRequest = ChatRequest.builder()
        .messages(userMessage)
        .responseFormat(responseFormat)
        .build();

ChatResponse chatResponse = model.chat(chatRequest);

System.out.println(chatResponse.aiMessage().text()); // {"shapes":[{"radius":5},{"width":10,"height":20}]}
note

The JsonAnyOfSchema is currently supported only by OpenAI.

Adding Description​

All of the JsonSchemaElement subtypes, except for JsonReferenceSchema, have a description property. If an LLM does not provide the desired output, descriptions can be provided to give more instructions and examples of correct outputs to the LLM, for example:

JsonSchemaElement stringSchema = JsonStringSchema.builder()
        .description("The name of the person, for example: John Doe")
        .build();

Limitations​

When using JSON Schema with ChatLanguageModel, there are some limitations:

  • It works only with supported OpenAI and Gemini models.
  • It does not work in the streaming mode yet.
  • JsonReferenceSchema and JsonAnyOfSchema are currently supported only by OpenAI.

Using JSON Schema with AI Services​

When using AI Services, one can achieve the same much easier and with less code:

interface PersonExtractor {
    
    Person extractPersonFrom(String text);
}

ChatLanguageModel chatModel = OpenAiChatModel.builder() // see [1] below
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        .responseFormat("json_schema") // see [2] below
        .strictJsonSchema(true) // see [2] below
        .logRequests(true)
        .logResponses(true)
        .build();
// OR
ChatLanguageModel chatModel = GoogleAiGeminiChatModel.builder() // see [1] below
        .apiKey(System.getenv("GOOGLE_AI_GEMINI_API_KEY"))
        .modelName("gemini-1.5-flash")
        .responseFormat(ResponseFormat.JSON) // see [3] below
        .temperature(0.0)
        .logRequestsAndResponses(true)
        .build();

PersonExtractor personExtractor = AiServices.create(PersonExtractor.class, chatModel); // see [1] below

String text = """
        John is 42 years old and lives an independent life.
        He stands 1.75 meters tall and carries himself with confidence.
        Currently unmarried, he enjoys the freedom to focus on his personal goals and interests.
        """;

Person person = personExtractor.extractPersonFrom(text);

System.out.println(person); // Person[name=John, age=42, height=1.75, married=false]

Notes:

  • [1] - In a Quarkus or a Spring Boot application, there is no need to explicitly create the ChatLanguageModel and the AI Service, as these beans are created automatically. More info on this: for Quarkus, for Spring Boot.
  • [2] - This is required to enable the JSON Schema feature for OpenAI, see more details here.
  • [3] - This is required to enable the JSON Schema feature for Google AI Gemini.

When all the following conditions are met:

  • AI Service method returns a POJO
  • The used ChatLanguageModel supports the JSON Schema feature
  • The JSON Schema feature is enabled on the used ChatLanguageModel

then the ResponseFormat with JsonSchema will be generated automatically based on the specified return type.

note

Make sure to explicitly enable JSON Schema feature when configuring ChatLanguageModel, as it is disabled by default.

The name of the generated JsonSchema is a simple name of the return type (getClass().getSimpleName()), in this case: "Person".

Once LLM responds, the output is parsed into an object and returned from the AI Service method.

note

While we are gradually migrating to Jackson, Gson is still used for parsing the outputs in AI Service, so Jackson annotations on your POJOs will have no effect.

You can find many examples of supported use cases here and here.

Adding Description​

If an LLM does not provide the desired output, classes and fields can be annotated with @Description to give more instructions and examples of correct outputs to the LLM, for example:

@Description("a person")
record Person(@Description("person's first and last name, for example: John Doe") String name,
              @Description("person's age, for example: 42") int age,
              @Description("person's height in meters, hor example: 1.78") double height,
              @Description("is person married or not, for example: false") boolean married) {
}

Limitations​

When using JSON Schema with AI Services, there are some limitations:

  • It works only with supported OpenAI and Gemini models.
  • Support for JSON Schema needs to be enabled explicitly when configuring ChatLanguageModel.
  • It does not work in the streaming mode.
  • Currently, it works only when return type is a (single) POJO or a Result<POJO>. If you need other types (e.g., List<POJO>, enum, etc.), please wrap these into a POJO. We are working on supporting more return types soon.
  • POJOs can contain:
    • Scalar/simple types (e.g., String, int/Integer, double/Double, boolean/Boolean, etc.)
    • enums
    • Nested POJOs
    • List<T>, Set<T> and T[], where T is a scalar, an enum or a POJO
  • All fields and sub-fields in the generated JsonSchema are automatically marked as required, there is currently no way to make them optional.
  • When LLM does not support JSON Schema feature, or it is not enabled, or return type is not a POJO, AI Service will fall back to prompting.
  • Recursion is currently supported only by OpenAI.
  • Polymorphism is not supported yet. The returned POJO and its nested POJOs must be concrete classes; interfaces or abstract classes are not supported.

Tools (Function Calling)​

This method assumes (mis)using tools to produce structured outputs. A single tool is specified in the request to the LLM, and the tool parameters describe the desired structure of the output. Once the LLM returns an AiMessage containing a ToolExecutionRequest, the JSON string in ToolExecutionRequest.arguments() is parsed into a POJO.

More info is coming soon.

In the meantime, please read this section and this article.

Prompting + JSON Mode​

More info is coming soon. In the meantime, please read this section and this article.

Prompting​

When using prompting, one has to specify the format of the desired output in free-form text within a system or user message and hope that the LLM will abide. This approach is quite unreliable. If LLM and LLM provider supports the methods described above, it is better to use those.

More info is coming soon.

In the meantime, please read this section and this article.