Class DocumentByCharacterSplitter
java.lang.Object
dev.langchain4j.data.document.splitter.HierarchicalDocumentSplitter
dev.langchain4j.data.document.splitter.DocumentByCharacterSplitter
- All Implemented Interfaces:
DocumentSplitter
Splits the provided
Document into characters and attempts to fit as many characters as possible
into a single TextSegment, adhering to the limit set by maxSegmentSize.
The maxSegmentSize can be defined in terms of characters (default) or tokens.
For token-based limit, a TokenCountEstimator must be provided.
If multiple characters fit within maxSegmentSize, they are joined together without delimiters.
Each TextSegment inherits all metadata from the Document and includes an "index" metadata key
representing its position within the document (starting from 0).
-
Field Summary
Fields inherited from class HierarchicalDocumentSplitter
maxOverlapSize, maxSegmentSize, subSplitter, tokenCountEstimator -
Constructor Summary
ConstructorsConstructorDescriptionDocumentByCharacterSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars) DocumentByCharacterSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars, DocumentSplitter subSplitter) DocumentByCharacterSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, TokenCountEstimator tokenCountEstimator) DocumentByCharacterSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, TokenCountEstimator tokenCountEstimator, DocumentSplitter subSplitter) -
Method Summary
Modifier and TypeMethodDescriptionprotected DocumentSplitterThe default sub-splitter to use when a single segment is too long.Delimiter string to use to re-join the parts.String[]Splits the provided text into parts.Methods inherited from class HierarchicalDocumentSplitter
splitMethods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface DocumentSplitter
splitAll, splitAll
-
Constructor Details
-
DocumentByCharacterSplitter
public DocumentByCharacterSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars) -
DocumentByCharacterSplitter
public DocumentByCharacterSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars, DocumentSplitter subSplitter) -
DocumentByCharacterSplitter
public DocumentByCharacterSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, TokenCountEstimator tokenCountEstimator) -
DocumentByCharacterSplitter
public DocumentByCharacterSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, TokenCountEstimator tokenCountEstimator, DocumentSplitter subSplitter)
-
-
Method Details
-
split
Description copied from class:HierarchicalDocumentSplitterSplits the provided text into parts. Implementation API.- Specified by:
splitin classHierarchicalDocumentSplitter- Parameters:
text- The text to be split.- Returns:
- An array of parts.
-
joinDelimiter
Description copied from class:HierarchicalDocumentSplitterDelimiter string to use to re-join the parts.- Specified by:
joinDelimiterin classHierarchicalDocumentSplitter- Returns:
- The delimiter.
-
defaultSubSplitter
Description copied from class:HierarchicalDocumentSplitterThe default sub-splitter to use when a single segment is too long.- Specified by:
defaultSubSplitterin classHierarchicalDocumentSplitter- Returns:
- The default sub-splitter.
-