# Chonkie chunkers

> **CocoIndex v1.** This page documents CocoIndex **v1** — a ground-up redesign from v0. When writing code, ignore any v0 flow-builder DSL or deprecated decorators.
>
> Source: https://cocoindex.io/docs/ops/chonkie/ · Docs index: https://cocoindex.io/docs/llms.txt · Agent skill: https://cocoindex.io/docs/skill.md
>
> v0→v1 quick map — if you reach for these v0 symbols, stop and use the v1 form: `@cocoindex.flow_def`/`FlowBuilder` → `coco.App` + a `@coco.fn` main function; `add_collector()`/`collect()`/`export()` → declare target states (`declare_row`, `declare_file`); `cocoindex.sources/functions/targets.*` → connector APIs (`localfs.walk_dir`, `coco.ops.*`, `postgres.declare_table_target`). Full mapping + API reference: https://cocoindex.io/docs/skill.md.

The `cocoindex.ops.chonkie` module wraps the
[chonkie](https://github.com/chonkie-inc/chonkie) chunking library and returns
position-tracked `Chunk` objects that drop straight into CocoIndex target
states.

```python
from cocoindex.ops.chonkie import (
    ChonkieRecursiveChunker,
    ChonkieSemanticChunker,
    ChonkieNeuralChunker,
    ChonkieCodeChunker,
)
```

**Note — CocoIndex Plus only**
These chunkers ship with `cocoindex-plus` and depend on the optional
`chonkie` library. Install with:

```sh
pip install "cocoindex-plus[chonkie]"
```

Or install the chonkie extras directly:

```sh
pip install chonkie                 # recursive only
pip install "chonkie[code]"         # adds CodeChunker
pip install "chonkie[semantic]"     # adds SemanticChunker
pip install "chonkie[neural]"       # adds NeuralChunker
```

## Usage shape

All chunkers follow the same shape as `RecursiveSplitter` in
`cocoindex.ops.text`: construct once, then call `.split(...)` repeatedly.
They each implement `__coco_memo_key__`, so a memoized `@coco.fn` cache-busts
when chunker parameters change between runs.

```python
import cocoindex as coco
from cocoindex.ops.chonkie import ChonkieRecursiveChunker
from cocoindex.resources.file import FileLike

chunker = ChonkieRecursiveChunker(chunk_size=512)

@coco.fn(memo=True)
async def process_doc(file: FileLike, target: coco.TableTarget) -> None:
    text = await file.read_text()
    for c in chunker.split(text):
        target.declare_row(
            DocChunk(text=c.text, line=c.start.line, char_offset=c.start.char_offset)
        )
```

`.split(...)` returns `list[cocoindex.resources.chunk.Chunk]`. Each `Chunk`
has `text`, `start`, and `end`, where the positions carry both byte and
character offsets along with line/column.

## `ChonkieRecursiveChunker`

Token-aware recursive chunking via chonkie's
[`RecursiveChunker`](https://docs.chonkie.ai/oss/chunkers/recursive-chunker).

```python
ChonkieRecursiveChunker(
    *,
    tokenizer: str = "character",
    chunk_size: int = 2048,
    min_characters_per_chunk: int = 24,
)
```

| Parameter                  | Default       | Description                                                   |
| -------------------------- | ------------- | ------------------------------------------------------------- |
| `tokenizer`                | `"character"` | Tokenizer name (`"character"`, `"word"`, or HF model name).   |
| `chunk_size`               | `2048`        | Maximum number of tokens per chunk.                           |
| `min_characters_per_chunk` | `24`          | Minimum chunk length in characters.                           |

## `ChonkieSemanticChunker`

Semantic-similarity chunking via chonkie's
[`SemanticChunker`](https://docs.chonkie.ai/oss/chunkers/semantic-chunker).

```python
ChonkieSemanticChunker(
    *,
    embedding_model: str = "minishlab/potion-base-32M",
    threshold: float = 0.8,
    chunk_size: int = 2048,
    similarity_window: int = 3,
    min_sentences_per_chunk: int = 1,
    skip_window: int = 0,
    filter_window: int = 5,
    filter_polyorder: int = 3,
)
```

| Parameter                 | Default                          | Description                                       |
| ------------------------- | -------------------------------- | ------------------------------------------------- |
| `embedding_model`         | `"minishlab/potion-base-32M"`    | Model for semantic embeddings.                    |
| `threshold`               | `0.8`                            | Similarity threshold in `[0, 1]`.                 |
| `chunk_size`              | `2048`                           | Maximum tokens per chunk.                         |
| `similarity_window`       | `3`                              | Sentences to consider for similarity.             |
| `min_sentences_per_chunk` | `1`                              | Minimum sentences per chunk.                      |
| `skip_window`             | `0`                              | Groups to skip when merging.                      |
| `filter_window`           | `5`                              | Savitzky-Golay filter window length.              |
| `filter_polyorder`        | `3`                              | Polynomial order for the filter.                  |

## `ChonkieNeuralChunker`

BERT-based semantic-shift chunking via chonkie's
[`NeuralChunker`](https://docs.chonkie.ai/oss/chunkers/neural-chunker).

```python
ChonkieNeuralChunker(
    *,
    model: str = "mirth/chonky_modernbert_base_1",
    device_map: str = "cpu",
    min_characters_per_chunk: int = 10,
)
```

| Parameter                  | Default                              | Description                                         |
| -------------------------- | ------------------------------------ | --------------------------------------------------- |
| `model`                    | `"mirth/chonky_modernbert_base_1"`   | Fine-tuned BERT model name.                         |
| `device_map`               | `"cpu"`                              | Inference device (`"cpu"`, `"cuda"`, `"mps"`).      |
| `min_characters_per_chunk` | `10`                                 | Minimum chunk length in characters.                 |

## `ChonkieCodeChunker`

AST-aware code chunking via chonkie's
[`CodeChunker`](https://docs.chonkie.ai/oss/chunkers/code-chunker). A separate
underlying chonkie chunker is constructed lazily per `language` and cached on
the instance, so repeated `.split(..., language=...)` calls for the same
language reuse the same parser.

```python
ChonkieCodeChunker(
    *,
    tokenizer: str = "character",
    chunk_size: int = 2048,
)

chunker.split(text, *, language="python")
```

| Parameter    | Default       | Description                          |
| ------------ | ------------- | ------------------------------------ |
| `tokenizer`  | `"character"` | Tokenizer name.                      |
| `chunk_size` | `2048`        | Maximum tokens per chunk.            |

`language` accepts any chonkie-supported language identifier
(`"python"`, `"javascript"`, `"rust"`, etc.).