Document Chunking/Splitting in Langroid

Langroid's [ParsingConfig][langroid.parsing.parser.ParsingConfig] provides several document chunking strategies through the Splitter enum:

1. MARKDOWN (Splitter.MARKDOWN) (The default)

Purpose: Structure-aware splitting that preserves markdown formatting.

How it works: - Preserves document hierarchy (headers and sections) - Enriches chunks with header information - Uses word count instead of token count (with adjustment factor) - Supports "rollup" to maintain document structure - Ideal for markdown documents where preserving formatting is important

2. TOKENS (Splitter.TOKENS)

Purpose: Creates chunks of approximately equal token size.

How it works: - Tokenizes the text using tiktoken - Aims for chunks of size chunk_size tokens (default: 200) - Looks for natural breakpoints like punctuation or newlines - Prefers splitting at sentence/paragraph boundaries - Ensures chunks are at least min_chunk_chars long (default: 350)

3. PARA_SENTENCE (Splitter.PARA_SENTENCE)

Purpose: Splits documents respecting paragraph and sentence boundaries.

How it works: - Recursively splits documents until chunks are below 1.3× the target size - Maintains document structure by preserving natural paragraph breaks - Adjusts chunk boundaries to avoid cutting in the middle of sentences - Stops when it can't split chunks further without breaking coherence

4. SIMPLE (Splitter.SIMPLE)

Purpose: Basic splitting using predefined separators.

How it works: - Uses a list of separators to split text (default: ["\n\n", "\n", " ", ""]) - Splits on the first separator in the list - Doesn't attempt to balance chunk sizes - Simplest and fastest splitting method

Basic Configuration

from langroid.parsing.parser import ParsingConfig, Splitter

config = ParsingConfig(
    splitter=Splitter.MARKDOWN,  # Most feature-rich option
    chunk_size=200,              # Target tokens per chunk
    chunk_size_variation=0.30,   # Allowed variation from target
    overlap=50,                  # Token overlap between chunks
    token_encoding_model="text-embedding-3-small"
)

Format-Specific Configuration

# Customize PDF parsing
config = ParsingConfig(
    splitter=Splitter.PARA_SENTENCE,
    pdf=PdfParsingConfig(
        library="pymupdf4llm"  # Default PDF parser
    )
)

# Use Gemini for PDF parsing
config = ParsingConfig(
    pdf=PdfParsingConfig(
        library="gemini",
        gemini_config=GeminiConfig(
            model_name="gemini-2.0-flash",
            requests_per_minute=5
        )
    )
)

Setting Up Parsing Config in DocChatAgentConfig

You can configure document parsing when creating a DocChatAgent by customizing the parsing field within the DocChatAgentConfig. Here's how to do it:

from langroid.agent.special.doc_chat_agent import DocChatAgentConfig  
from langroid.parsing.parser import ParsingConfig, Splitter, PdfParsingConfig

# Create a DocChatAgent with custom parsing configuration
agent_config = DocChatAgentConfig(
    parsing=ParsingConfig(
        # Choose the splitting strategy
        splitter=Splitter.MARKDOWN,  # Structure-aware splitting with header context

        # Configure chunk sizes
        chunk_size=800,              # Target tokens per chunk
        overlap=150,                 # Overlap between chunks

        # Configure chunk behavior
        max_chunks=5000,             # Maximum number of chunks to create
        min_chunk_chars=250,         # Minimum characters when truncating at punctuation
        discard_chunk_chars=10,      # Discard chunks smaller than this

        # Configure context window
        n_neighbor_ids=3,            # Store 3 chunk IDs on either side

        # Configure PDF parsing specifically
        pdf=PdfParsingConfig(
            library="pymupdf4llm",   # Choose PDF parsing library
        )
    )
)