Text operations

Built-in text utilities — code language detection, regex-based SeparatorSplitter, and syntax-aware RecursiveSplitter that returns position-tracked Chunks with optional custom domain-specific splitting rules.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The cocoindex::ops::text module provides operations for text processing.

use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter, SeparatorSplitter};

cocoindex = { version = "...", features = ["text"] }

Features include:

• Code language detection
• Text chunking and splitting
• Syntax-aware code splitting

Available functions and classes

detect_code_language()

Detect the programming language from a filename.

Usage:

use cocoindex::ops::text::detect_code_language;

assert_eq!(detect_code_language("main.py").as_deref(), Some("python"));
assert_eq!(detect_code_language("app.rs").as_deref(), Some("rust"));
assert_eq!(detect_code_language("file.xyz"), None);

It returns Option<String>: None when the extension is not recognized.

SeparatorSplitter

Split text by regex separators.

Usage:

use cocoindex::ops::text::{KeepSeparator, SeparatorSplitConfig, SeparatorSplitter};

// Separators only.
let splitter = SeparatorSplitter::new([r"\n\n+"])?;
let text = "Para1\n\nPara2\n\nPara3";
let chunks = splitter.split(text);
for chunk in &chunks {
    println!("{}", chunk.text(text));
}

// Keep the separator on the left of the split.
let splitter = SeparatorSplitter::with_config(SeparatorSplitConfig {
    separators_regex: vec![r"\.".to_string()],
    keep_separator: Some(KeepSeparator::Left),
    ..Default::default()
})?;

SeparatorSplitter::new(separators) takes the regex list; the other options (keep_separator, trim) go through SeparatorSplitConfig and with_config. A Chunk does not hold its text: call chunk.text(text) with the string you split.

Use RecursiveSplitter below to configure chunk size and overlap.

RecursiveSplitter

Advanced text chunking with language awareness and syntax-aware splitting for code. Returns Chunk objects with position information.

Features:

• Supports many programming languages
• Preserves code structure
• Customizable chunk sizes and overlap
• Returns Chunk objects with start/end positions (line, column, byte/char offsets)

Usage:

use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter};

let splitter = RecursiveSplitter::new()?;

// Split markdown text
let text = "# Title\n\nParagraph 1.\n\nParagraph 2.";
let chunks = splitter.split_with(
    text,
    RecursiveChunkConfig {
        chunk_size: 2000,
        min_chunk_size: None,
        chunk_overlap: Some(500),
        language: Some("markdown".to_string()),
    },
);

for chunk in &chunks {
    println!("Chunk: {}", chunk.text(text));
    println!("Start: line {}, char {}", chunk.start.line, chunk.start.char_offset);
    println!("End: line {}, char {}", chunk.end.line, chunk.end.char_offset);
}

// With only a chunk size: default minimum, no overlap, no language hint.
let chunks = splitter.split("Short text.", 100);

RecursiveSplitter::new() returns a Result. split(text, chunk_size) uses the defaults; split_with(text, config) takes a RecursiveChunkConfig with chunk_size, min_chunk_size (default chunk_size / 2), chunk_overlap and language. Sizes are in bytes.

Language-aware code splitting:

// Split Python code
let python_code = "def foo():\n    pass\n\ndef bar():\n    pass";

let chunks = splitter.split_with(
    python_code,
    RecursiveChunkConfig {
        chunk_size: 1000,
        min_chunk_size: Some(300),
        chunk_overlap: Some(300),
        language: Some("python".to_string()),
    },
);

Reusing one parse across APIs:

When the same code is passed to several parse-consuming APIs (e.g. splitting and structural pattern matching), wrap it in a CodeSource so it is parsed at most once. The CodeSource carries its own language, so config.language is ignored:

use cocoindex::ops::text::{CodeSource, RecursiveChunkConfig, RecursiveSplitter};

let src = CodeSource::with_language(python_code, "python");
let chunks = splitter.split_source(
    &src,
    RecursiveChunkConfig {
        chunk_size: 1000,
        min_chunk_size: None,
        chunk_overlap: None,
        language: None, // the source carries its own language
    },
);

RecursiveSplitter::split_source(&source, config) reuses the parse cached in a CodeSource; the source’s language wins, so config.language is ignored. The structural matcher lives in cocoindex::ops::code (feature code_match): CodeAst::new(code, "python")? parses once and offers matches(pattern), matches_with(&CodePattern::compile(pattern, "python")?) and split(chunk_size, min_chunk_size, chunk_overlap) over that one parse. render_match(&ast, &matches[0]) renders a match as a SourceView: the head lines of its enclosing scopes as SegmentKind::Frame segments, an elision cue, then the matched code verbatim, with view.text ready to embed or display and view.segments mapping each piece back to its source range.

Supported languages:

language accepts a language name, an alternate name, or a file extension — "python", "py" and ".py" all select Python. Matching is case-insensitive. Use detect_code_language() to infer the value from a filename.

Languages with syntax-aware (tree-sitter) splitting — splits at logical boundaries like functions, classes, and blocks:

Note that .h maps to C++, not C — the C++ grammar also parses C headers. Pass language: Some("c".to_string()) explicitly to force the C grammar.

110 additional languages use separator-based splitting (e.g. "elixir", "erlang", "haskell", "nix", "perl", "powershell", "proto", "zig", and more). A language value that matches no known language also falls back to separator-based splitting rather than failing.

CustomLanguageConfig

Define custom language splitting rules.

Usage:

use cocoindex::ops::text::{CustomLanguageConfig, RecursiveChunkConfig, RecursiveSplitter};

let config = CustomLanguageConfig {
    language_name: "myformat".to_string(),
    aliases: vec!["mf".to_string()],
    separators_regex: vec![r"---".to_string()],
};
let splitter = RecursiveSplitter::with_custom_languages(vec![config])?;

let text = "Part1---Part2---Part3";
let chunks = splitter.split_with(
    text,
    RecursiveChunkConfig {
        chunk_size: 10,
        min_chunk_size: Some(3),
        chunk_overlap: None,
        language: Some("myformat".to_string()),
    },
);

CustomLanguageConfig has the three fields language_name, aliases and separators_regex; RecursiveSplitter::with_custom_languages(configs)? registers them.

API reference

For detailed API documentation, build the crate documentation with cargo doc --features text --open and open the cocoindex::ops::text module.