Text operations

Built-in text utilities — code language detection, regex-based SeparatorSplitter, and syntax-aware RecursiveSplitter that returns position-tracked Chunks with optional custom domain-specific splitting rules.

Version
v 1.0.14
Last reviewed
Jul 4, 2026

The cocoindex.ops.text module provides operations for text processing.

python
from cocoindex.ops.text import RecursiveSplitter, SeparatorSplitter

Features include:

  • Code language detection
  • Text chunking and splitting
  • Syntax-aware code splitting

Available functions and classes

detect_code_language()

Detect the programming language from a filename.

Usage:

python
from cocoindex.ops.text import detect_code_language

language = detect_code_language(filename="main.py")
print(language)  # "python"

language = detect_code_language(filename="app.rs")
print(language)  # "rust"

language = detect_code_language(filename="unknown.xyz")
print(language)  # None

SeparatorSplitter

Split text by regex separators.

Usage:

python
from cocoindex.ops.text import SeparatorSplitter

splitter = SeparatorSplitter()

text = "First sentence. Second sentence. Third sentence."
chunks = splitter.split(
    text,
    chunk_size=100,
    chunk_overlap=20,
    separators=[r"\.\s+"]  # Split on periods followed by whitespace
)

for chunk in chunks:
    print(chunk.text)

RecursiveSplitter

Advanced text chunking with language awareness and syntax-aware splitting for code. Returns Chunk objects with position information.

Features:

  • Supports many programming languages
  • Preserves code structure
  • Customizable chunk sizes and overlap
  • Returns Chunk objects with start/end positions (line, column, byte/char offsets)

Usage:

python
from cocoindex.ops.text import RecursiveSplitter

splitter = RecursiveSplitter()

# Split markdown text
text = "# Title\n\nParagraph 1.\n\nParagraph 2."
chunks = splitter.split(
    text,
    chunk_size=2000,
    chunk_overlap=500,
    language="markdown"
)

for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Start: line {chunk.start.line}, char {chunk.start.char_offset}")
    print(f"End: line {chunk.end.line}, char {chunk.end.char_offset}")

Language-aware code splitting:

python
# Split Python code
python_code = '''
def hello():
    print("Hello, world!")

def goodbye():
    print("Goodbye!")
'''

chunks = splitter.split(
    python_code,
    chunk_size=1000,
    min_chunk_size=300,
    chunk_overlap=300,
    language="python"
)

Reusing one parse across APIs:

When the same code is passed to several parse-consuming APIs (e.g. splitting and structural pattern matching), wrap it in a CodeSource so it is parsed at most once. The CodeSource carries its own language, so the language= argument must be omitted:

python
from cocoindex.ops.code import CodePattern, CodeSource

src = CodeSource(python_code, language="python")
chunks = splitter.split(src, chunk_size=1000)          # parses once
defs = CodePattern(r"def \NAME(\(A*\)):", language="python").match_source(src)  # reuses the parse

Supported languages:

language= accepts a language name, an alternate name, or a file extension — "python", "py" and ".py" all select Python. Matching is case-insensitive. Use detect_code_language() to infer the value from a filename.

Languages with syntax-aware (tree-sitter) splitting — splits at logical boundaries like functions, classes, and blocks:

Languagelanguage= valueExtensions
Astro"astro".astro
Bash"bash".sh, .bash
C"c".c, .cats, .idc
C#"csharp".cs, .csx, .cake, .linq
C++"cpp".cpp, .cc, .cxx, .c++, .h, .hpp, .hh, .hxx
CMake"cmake".cmake, .cmake.in
CSS"css".css, .scss
Dart"dart".dart
DTD"dtd".dtd
Elm"elm".elm
Fortran"fortran".f, .f90, .f95, .f03
Go"go".go
HCL / Terraform"hcl".hcl, .tf
HTML"html".html, .htm, .xhtml
Java"java".java, .jav, .jsh
JavaScript"javascript".js, .mjs, .cjs, .jsx
JSON"json".json, .jsonl, .geojson
Julia"julia".jl
Kotlin"kotlin".kt, .kts, .ktm
Markdown"markdown".md, .markdown, .mdx
Objective-C"objc".m
Objective-C++"objcpp".mm
Pascal / Delphi"pascal".pas, .dpr, .lpr, .dfm
PHP"php".php
Python"python".py, .pyi, .pyw, .pyx, .pxd
R"r".r, .R
Ruby"ruby".rb
Rust"rust".rs
Scala"scala".scala
Solidity"solidity".sol
SQL"sql".sql
Svelte"svelte".svelte
Swift"swift".swift
TOML"toml".toml
TSX"tsx".tsx
TypeScript"typescript".ts
Vue"vue".vue
XML"xml".xml
YAML"yaml".yaml, .yml

Note that .h maps to C++, not C — the C++ grammar also parses C headers. Pass language="c" explicitly to force the C grammar.

Over 110 additional languages use separator-based splitting (e.g. "elixir", "erlang", "haskell", "lua", "nix", "perl", "powershell", "proto", "zig", and more). A language= value that matches no known language also falls back to separator-based splitting rather than raising.

CustomLanguageConfig

Define custom language splitting rules.

Usage:

python
from cocoindex.ops.text import CustomLanguageConfig, RecursiveSplitter

# Create custom language config for abstracts
abstract_config = CustomLanguageConfig(
    language_name="abstract",
    separators_regex=[
        r"[.?!]+\s+",  # Sentence boundaries
        r"[:;]\s+",     # Clause boundaries
        r",\s+",        # Comma boundaries
        r"\s+",         # Whitespace
    ]
)

splitter = RecursiveSplitter(custom_languages=[abstract_config])

chunks = splitter.split(
    "This is a sample abstract. It has multiple sentences...",
    chunk_size=500,
    chunk_overlap=150,
    language="abstract"
)

API reference

For detailed API documentation, refer to the module docstrings:

python
from cocoindex.ops import text

help(text.RecursiveSplitter)
help(text.SeparatorSplitter)
help(text.detect_code_language)
help(text.CustomLanguageConfig)
PreviousSentence transformers
CocoIndex DocsEdit this pageReport issue