Class DocumentByParagraphSplitter

java.lang.Object

dev.langchain4j.data.document.splitter.HierarchicalDocumentSplitter

dev.langchain4j.data.document.splitter.DocumentByParagraphSplitter

All Implemented Interfaces:

DocumentSplitter

public class DocumentByParagraphSplitter
extends HierarchicalDocumentSplitter

Splits the provided Document into paragraphs and attempts to fit as many paragraphs as possible into a single TextSegment, adhering to the limit set by maxSegmentSize.

The maxSegmentSize can be defined in terms of characters (default) or tokens. For token-based limit, a TokenCountEstimator must be provided.

Paragraph boundaries are detected by a minimum of two newline characters ("\n\n"). Any additional whitespaces before, between, or after are ignored. So, the following examples are all valid paragraph separators: "\n\n", "\n\n\n", "\n \n", " \n \n ", and so on.

If multiple paragraphs fit within maxSegmentSize, they are joined together using a double newline ("\n\n").

If a single paragraph is too long and exceeds maxSegmentSize, the subSplitter (DocumentBySentenceSplitter by default) is used to split it into smaller parts and place them into multiple segments. Such segments contain only the parts of the split long paragraph.

Each TextSegment inherits all metadata from the Document and includes an "index" metadata key representing its position within the document (starting from 0).

Field Summary

Fields inherited from class HierarchicalDocumentSplitter

maxOverlapSize, maxSegmentSize, subSplitter, tokenCountEstimator

Constructor Summary

Constructors

Constructor	Description
DocumentByParagraphSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars)
DocumentByParagraphSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars, DocumentSplitter subSplitter)
DocumentByParagraphSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, TokenCountEstimator tokenCountEstimator)
DocumentByParagraphSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, TokenCountEstimator tokenCountEstimator, DocumentSplitter subSplitter)

Method Summary

Modifier and Type	Method	Description
protected DocumentSplitter	defaultSubSplitter()	The default sub-splitter to use when a single segment is too long.
String	joinDelimiter()	Delimiter string to use to re-join the parts.
String[]	split(String text)	Splits the provided text into parts.

Methods inherited from class HierarchicalDocumentSplitter

split

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface DocumentSplitter

splitAll, splitAll

Constructor Details

DocumentByParagraphSplitter

public DocumentByParagraphSplitter(int maxSegmentSizeInChars,
 int maxOverlapSizeInChars)

DocumentByParagraphSplitter

public DocumentByParagraphSplitter(int maxSegmentSizeInChars,
 int maxOverlapSizeInChars,
 DocumentSplitter subSplitter)

DocumentByParagraphSplitter

public DocumentByParagraphSplitter(int maxSegmentSizeInTokens,
 int maxOverlapSizeInTokens,
 TokenCountEstimator tokenCountEstimator)

DocumentByParagraphSplitter

public DocumentByParagraphSplitter(int maxSegmentSizeInTokens,
 int maxOverlapSizeInTokens,
 TokenCountEstimator tokenCountEstimator,
 DocumentSplitter subSplitter)

Method Details

split

public String[] split(String text)

Description copied from class: HierarchicalDocumentSplitter

Splits the provided text into parts. Implementation API.

Specified by:

split in class HierarchicalDocumentSplitter

Parameters:

text - The text to be split.

Returns:

An array of parts.

joinDelimiter

public String joinDelimiter()

Description copied from class: HierarchicalDocumentSplitter

Delimiter string to use to re-join the parts.

Specified by:

joinDelimiter in class HierarchicalDocumentSplitter

Returns:

The delimiter.

defaultSubSplitter

protected DocumentSplitter defaultSubSplitter()

Description copied from class: HierarchicalDocumentSplitter

The default sub-splitter to use when a single segment is too long.

Specified by:

defaultSubSplitter in class HierarchicalDocumentSplitter

Returns:

The default sub-splitter.