Package dev.langchain4j.store.embedding

Class EmbeddingStoreIngestor

java.lang.Object

dev.langchain4j.store.embedding.EmbeddingStoreIngestor

public class EmbeddingStoreIngestor
extends Object

The EmbeddingStoreIngestor represents an ingestion pipeline and is responsible for ingesting Documents into an EmbeddingStore.

In the simplest configuration, EmbeddingStoreIngestor embeds provided documents using a provided EmbeddingModel and stores them, along with their Embeddings in an EmbeddingStore.

Optionally, the EmbeddingStoreIngestor can transform documents using a provided DocumentTransformer. This can be useful if you want to clean, enrich, or format documents before embedding them.

Optionally, the EmbeddingStoreIngestor can split documents into TextSegments using a provided DocumentSplitter. This can be useful if documents are big, and you want to split them into smaller segments to improve the quality of similarity searches and reduce the size and cost of a prompt sent to the LLM.

Optionally, the EmbeddingStoreIngestor can transform TextSegments using a TextSegmentTransformer. This can be useful if you want to clean, enrich, or format TextSegments before embedding them.
Including a document title or a short summary in each TextSegment is a common technique to improve the quality of similarity searches.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	EmbeddingStoreIngestor.Builder	EmbeddingStoreIngestor builder.

Constructor Summary

Constructors

Constructor	Description
EmbeddingStoreIngestor(DocumentTransformer documentTransformer, DocumentSplitter documentSplitter, TextSegmentTransformer textSegmentTransformer, EmbeddingModel embeddingModel, EmbeddingStore<TextSegment> embeddingStore)	Creates an instance of an EmbeddingStoreIngestor.

Method Summary

Modifier and Type	Method	Description
static EmbeddingStoreIngestor.Builder	builder()	Creates a new EmbeddingStoreIngestor builder.
void	ingest(Document document)	Ingests a specified document into an EmbeddingStore that was specified during the creation of this EmbeddingStoreIngestor.
void	ingest(Document... documents)	Ingests specified documents into an EmbeddingStore that was specified during the creation of this EmbeddingStoreIngestor.
static void	ingest(Document document, EmbeddingStore<TextSegment> embeddingStore)	Ingests a specified Document into a specified EmbeddingStore.
void	ingest(List<Document> documents)	Ingests specified documents into an EmbeddingStore that was specified during the creation of this EmbeddingStoreIngestor.
static void	ingest(List<Document> documents, EmbeddingStore<TextSegment> embeddingStore)	Ingests specified Documents into a specified EmbeddingStore.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

EmbeddingStoreIngestor

public EmbeddingStoreIngestor(DocumentTransformer documentTransformer,
 DocumentSplitter documentSplitter,
 TextSegmentTransformer textSegmentTransformer,
 EmbeddingModel embeddingModel,
 EmbeddingStore<TextSegment> embeddingStore)

Creates an instance of an EmbeddingStoreIngestor.

Parameters:

documentTransformer - The DocumentTransformer to use. Optional.

documentSplitter - The DocumentSplitter to use. Optional. If none is specified, it tries to load one through SPI (see DocumentSplitterFactory).

textSegmentTransformer - The TextSegmentTransformer to use. Optional.

embeddingModel - The EmbeddingModel to use. Mandatory. If none is specified, it tries to load one through SPI (see EmbeddingModelFactory).

embeddingStore - The EmbeddingStore to use. Mandatory.

Method Details

ingest

public static void ingest(Document document,
 EmbeddingStore<TextSegment> embeddingStore)

Ingests a specified Document into a specified EmbeddingStore.
Uses DocumentSplitter and EmbeddingModel found through SPIs (see DocumentSplitterFactory and EmbeddingModelFactory).
For the "Easy RAG", import langchain4j-easy-rag module, which contains a DocumentSplitterFactory and EmbeddingModelFactory implementations.

ingest

public static void ingest(List<Document> documents,
 EmbeddingStore<TextSegment> embeddingStore)

Ingests specified Documents into a specified EmbeddingStore.
Uses DocumentSplitter and EmbeddingModel found through SPIs (see DocumentSplitterFactory and EmbeddingModelFactory).
For the "Easy RAG", import langchain4j-easy-rag module, which contains a DocumentSplitterFactory and EmbeddingModelFactory implementations.

ingest

public void ingest(Document document)

Ingests a specified document into an EmbeddingStore that was specified during the creation of this EmbeddingStoreIngestor.

Parameters:

document - the document to ingest.

ingest

public void ingest(Document... documents)

Ingests specified documents into an EmbeddingStore that was specified during the creation of this EmbeddingStoreIngestor.

Parameters:

documents - the documents to ingest.

ingest

public void ingest(List<Document> documents)

Ingests specified documents into an EmbeddingStore that was specified during the creation of this EmbeddingStoreIngestor.

Parameters:

documents - the documents to ingest.

builder

public static EmbeddingStoreIngestor.Builder builder()

Creates a new EmbeddingStoreIngestor builder.

Returns:

the builder.