Chonkie chunkers

CocoIndex Plus wrappers around the chonkie chunking library — recursive, semantic, neural, and code chunking with byte/char/line/column positions.

Version: v 1.0.14

The cocoindex.ops.chonkie module wraps the chonkie chunking library and returns position-tracked Chunk objects that drop straight into CocoIndex target states.

python

from cocoindex.ops.chonkie import (
    ChonkieRecursiveChunker,
    ChonkieSemanticChunker,
    ChonkieNeuralChunker,
    ChonkieCodeChunker,
)

CocoIndex Plus only

These chunkers ship with cocoindex-plus and depend on the optional chonkie library. Install with:

pip install "cocoindex-plus[chonkie]"

Or install the chonkie extras directly:

pip install chonkie                 # recursive only
pip install "chonkie[code]"         # adds CodeChunker
pip install "chonkie[semantic]"     # adds SemanticChunker
pip install "chonkie[neural]"       # adds NeuralChunker

Usage shape

All chunkers follow the same shape as RecursiveSplitter in cocoindex.ops.text: construct once, then call .split(...) repeatedly. They each implement __coco_memo_key__, so a memoized @coco.fn cache-busts when chunker parameters change between runs.

python

import cocoindex as coco
from cocoindex.ops.chonkie import ChonkieRecursiveChunker
from cocoindex.resources.file import FileLike

chunker = ChonkieRecursiveChunker(chunk_size=512)

@coco.fn(memo=True)
async def process_doc(file: FileLike, target: coco.TableTarget) -> None:
    text = await file.read_text()
    for c in chunker.split(text):
        target.declare_row(
            DocChunk(text=c.text, line=c.start.line, char_offset=c.start.char_offset)
        )

.split(...) returns list[cocoindex.resources.chunk.Chunk]. Each Chunk has text, start, and end, where the positions carry both byte and character offsets along with line/column.

`ChonkieRecursiveChunker`

Token-aware recursive chunking via chonkie’s RecursiveChunker.

python

ChonkieRecursiveChunker(
    *,
    tokenizer: str = "character",
    chunk_size: int = 2048,
    min_characters_per_chunk: int = 24,
)

Parameter	Default	Description
`tokenizer`	`"character"`	Tokenizer name (`"character"`, `"word"`, or HF model name).
`chunk_size`	`2048`	Maximum number of tokens per chunk.
`min_characters_per_chunk`	`24`	Minimum chunk length in characters.

`ChonkieSemanticChunker`

Semantic-similarity chunking via chonkie’s SemanticChunker.

python

ChonkieSemanticChunker(
    *,
    embedding_model: str = "minishlab/potion-base-32M",
    threshold: float = 0.8,
    chunk_size: int = 2048,
    similarity_window: int = 3,
    min_sentences_per_chunk: int = 1,
    skip_window: int = 0,
    filter_window: int = 5,
    filter_polyorder: int = 3,
)

Parameter	Default	Description
`embedding_model`	`"minishlab/potion-base-32M"`	Model for semantic embeddings.
`threshold`	`0.8`	Similarity threshold in `[0, 1]`.
`chunk_size`	`2048`	Maximum tokens per chunk.
`similarity_window`	`3`	Sentences to consider for similarity.
`min_sentences_per_chunk`	`1`	Minimum sentences per chunk.
`skip_window`	`0`	Groups to skip when merging.
`filter_window`	`5`	Savitzky-Golay filter window length.
`filter_polyorder`	`3`	Polynomial order for the filter.

`ChonkieNeuralChunker`

BERT-based semantic-shift chunking via chonkie’s NeuralChunker.

python

ChonkieNeuralChunker(
    *,
    model: str = "mirth/chonky_modernbert_base_1",
    device_map: str = "cpu",
    min_characters_per_chunk: int = 10,
)

Parameter	Default	Description
`model`	`"mirth/chonky_modernbert_base_1"`	Fine-tuned BERT model name.
`device_map`	`"cpu"`	Inference device (`"cpu"`, `"cuda"`, `"mps"`).
`min_characters_per_chunk`	`10`	Minimum chunk length in characters.

`ChonkieCodeChunker`

AST-aware code chunking via chonkie’s CodeChunker. A separate underlying chonkie chunker is constructed lazily per language and cached on the instance, so repeated .split(..., language=...) calls for the same language reuse the same parser.

python

ChonkieCodeChunker(
    *,
    tokenizer: str = "character",
    chunk_size: int = 2048,
)

chunker.split(text, *, language="python")

Parameter	Default	Description
`tokenizer`	`"character"`	Tokenizer name.
`chunk_size`	`2048`	Maximum tokens per chunk.

language accepts any chonkie-supported language identifier ("python", "javascript", "rust", etc.).