Interface WhiteSpaceConfig

Configuration for a white space chunking algorithm.

interface WhiteSpaceConfig {
    maxOverlapTokens?: number;
    maxTokensPerChunk?: number;
}

Properties

maxOverlapTokens?: number

Maximum number of overlapping tokens between two adjacent chunks.

maxTokensPerChunk?: number

Maximum number of tokens per chunk.