# Text operations

> **CocoIndex v1, Rust SDK (experimental).** This page documents the CocoIndex **v1** Rust SDK (the `cocoindex` crate). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are at https://cocoindex.io/docs/.
>
> Source: https://cocoindex.io/docs/rust/ops/text/ · Rust docs index: https://cocoindex.io/docs/rust/llms.txt · Rust SDK tour: https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/SHOWCASE.md · Rust examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust

The `cocoindex::ops::text` module provides operations for text processing.

```rust
use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter, SeparatorSplitter};
```

**Note — Cargo feature**
The module is behind the `text` feature of the `cocoindex` crate:

```toml
cocoindex = { version = "...", features = ["text"] }
```

Features include:

- Code language detection
- Text chunking and splitting
- Syntax-aware code splitting

## Available functions and classes

### `detect_code_language()`

Detect the programming language from a filename.

**Usage:**

```rust
use cocoindex::ops::text::detect_code_language;

assert_eq!(detect_code_language("main.py").as_deref(), Some("python"));
assert_eq!(detect_code_language("app.rs").as_deref(), Some("rust"));
assert_eq!(detect_code_language("file.xyz"), None);
```

It returns `Option<String>`: `None` when the extension is not recognized.

### `SeparatorSplitter`

Split text by regex separators.

**Usage:**

```rust
use cocoindex::ops::text::{KeepSeparator, SeparatorSplitConfig, SeparatorSplitter};

// Separators only.
let splitter = SeparatorSplitter::new([r"\n\n+"])?;
let text = "Para1\n\nPara2\n\nPara3";
let chunks = splitter.split(text);
for chunk in &chunks {
    println!("{}", chunk.text(text));
}

// Keep the separator on the left of the split.
let splitter = SeparatorSplitter::with_config(SeparatorSplitConfig {
    separators_regex: vec![r"\.".to_string()],
    keep_separator: Some(KeepSeparator::Left),
    ..Default::default()
})?;
```

`SeparatorSplitter::new(separators)` takes the regex list; the other options (`keep_separator`, `trim`) go through `SeparatorSplitConfig` and `with_config`. A `Chunk` does not hold its text: call `chunk.text(text)` with the string you split.

Use `RecursiveSplitter` below to configure chunk size and overlap.

### `RecursiveSplitter`

Advanced text chunking with language awareness and syntax-aware splitting for code. Returns [`Chunk`](/docs/rust/common_resources/data_types/#chunk) objects with position information.

**Features:**
- Supports many programming languages
- Preserves code structure
- Customizable chunk sizes and overlap
- Returns [`Chunk`](/docs/rust/common_resources/data_types/#chunk) objects with start/end positions (line, column, byte/char offsets)

**Usage:**

```rust
use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter};

let splitter = RecursiveSplitter::new()?;

// Split markdown text
let text = "# Title\n\nParagraph 1.\n\nParagraph 2.";
let chunks = splitter.split_with(
    text,
    RecursiveChunkConfig {
        chunk_size: 2000,
        min_chunk_size: None,
        chunk_overlap: Some(500),
        language: Some("markdown".to_string()),
    },
);

for chunk in &chunks {
    println!("Chunk: {}", chunk.text(text));
    println!("Start: line {}, char {}", chunk.start.line, chunk.start.char_offset);
    println!("End: line {}, char {}", chunk.end.line, chunk.end.char_offset);
}

// With only a chunk size: default minimum, no overlap, no language hint.
let chunks = splitter.split("Short text.", 100);
```

`RecursiveSplitter::new()` returns a `Result`. `split(text, chunk_size)` uses the defaults; `split_with(text, config)` takes a `RecursiveChunkConfig` with `chunk_size`, `min_chunk_size` (default `chunk_size / 2`), `chunk_overlap` and `language`. Sizes are in bytes.

**Language-aware code splitting:**

```rust
// Split Python code
let python_code = "def foo():\n    pass\n\ndef bar():\n    pass";

let chunks = splitter.split_with(
    python_code,
    RecursiveChunkConfig {
        chunk_size: 1000,
        min_chunk_size: Some(300),
        chunk_overlap: Some(300),
        language: Some("python".to_string()),
    },
);
```

**Reusing one parse across APIs:**

When the same code is passed to several parse-consuming APIs (e.g. splitting *and*
structural pattern matching), wrap it in a `CodeSource` so it is parsed at most once.
The `CodeSource` carries its own language, so `config.language` is ignored:

```rust
use cocoindex::ops::text::{CodeSource, RecursiveChunkConfig, RecursiveSplitter};

let src = CodeSource::with_language(python_code, "python");
let chunks = splitter.split_source(
    &src,
    RecursiveChunkConfig {
        chunk_size: 1000,
        min_chunk_size: None,
        chunk_overlap: None,
        language: None, // the source carries its own language
    },
);
```

`RecursiveSplitter::split_source(&source, config)` reuses the parse cached in a `CodeSource`; the source's language wins, so `config.language` is ignored. The structural matcher lives in `cocoindex::ops::code` (feature `code_match`): `CodeAst::new(code, "python")?` parses once and offers `matches(pattern)`, `matches_with(&CodePattern::compile(pattern, "python")?)` and `split(chunk_size, min_chunk_size, chunk_overlap)` over that one parse. `render_match(&ast, &matches[0])` renders a match as a `SourceView`: the head lines of its enclosing scopes as `SegmentKind::Frame` segments, an elision cue, then the matched code verbatim, with `view.text` ready to embed or display and `view.segments` mapping each piece back to its source range.

**Supported languages:**

`language` accepts a language name, an alternate name, or a file extension — `"python"`, `"py"` and `".py"` all select Python. Matching is case-insensitive. Use [`detect_code_language()`](#detect_code_language) to infer the value from a filename.

Languages with syntax-aware (tree-sitter) splitting — splits at logical boundaries like functions, classes, and blocks:

| Language | `language` value | Extensions |
|---|---|---|
| Astro | `"astro"` | `.astro` |
| Bash | `"bash"` | `.sh`, `.bash` |
| C | `"c"` | `.c`, `.cats`, `.idc` |
| C# | `"csharp"` | `.cs`, `.csx`, `.cake`, `.linq` |
| C++ | `"cpp"` | `.cpp`, `.cc`, `.cxx`, `.c++`, `.h`, `.hpp`, `.hh`, `.hxx` |
| CMake | `"cmake"` | `.cmake`, `.cmake.in` |
| CSS | `"css"` | `.css`, `.scss` |
| Dart | `"dart"` | `.dart` |
| DTD | `"dtd"` | `.dtd` |
| Elm | `"elm"` | `.elm` |
| Fortran | `"fortran"` | `.f`, `.f90`, `.f95`, `.f03` |
| Go | `"go"` | `.go` |
| HCL / Terraform | `"hcl"` | `.hcl`, `.tf` |
| HTML | `"html"` | `.html`, `.htm`, `.xhtml` |
| Java | `"java"` | `.java`, `.jav`, `.jsh` |
| JavaScript | `"javascript"` | `.js`, `.mjs`, `.cjs`, `.jsx` |
| JSON | `"json"` | `.json`, `.jsonl`, `.geojson` |
| Julia | `"julia"` | `.jl` |
| Kotlin | `"kotlin"` | `.kt`, `.kts`, `.ktm` |
| Lua | `"lua"` | `.lua`, `.nse`, `.p8`, `.pd_lua`, `.rbxs`, `.rockspec`, `.wlua` |
| Markdown | `"markdown"` | `.md`, `.markdown`, `.mdx` |
| Nix | `"nix"` | `.nix` |
| Pascal / Delphi | `"pascal"` | `.pas`, `.dpr`, `.lpr`, `.dfm` |
| PHP | `"php"` | `.php` |
| Python | `"python"` | `.py`, `.pyi`, `.pyw`, `.pyx`, `.pxd` |
| R | `"r"` | `.r`, `.R` |
| Ruby | `"ruby"` | `.rb` |
| Rust | `"rust"` | `.rs` |
| Scala | `"scala"` | `.scala` |
| Solidity | `"solidity"` | `.sol` |
| SQL | `"sql"` | `.sql` |
| Svelte | `"svelte"` | `.svelte` |
| Swift | `"swift"` | `.swift` |
| TOML | `"toml"` | `.toml` |
| TSX | `"tsx"` | `.tsx` |
| TypeScript | `"typescript"` | `.ts` |
| Vue | `"vue"` | `.vue` |
| XML | `"xml"` | `.xml` |
| YAML | `"yaml"` | `.yaml`, `.yml` |

Note that `.h` maps to C++, not C — the C++ grammar also parses C headers. Pass `language: Some("c".to_string())` explicitly to force the C grammar.

110 additional languages use separator-based splitting (e.g. `"elixir"`, `"erlang"`, `"haskell"`, `"nix"`, `"perl"`, `"powershell"`, `"proto"`, `"zig"`, and more). A `language` value that matches no known language also falls back to separator-based splitting rather than failing.

### `CustomLanguageConfig`

Define custom language splitting rules.

**Usage:**

```rust
use cocoindex::ops::text::{CustomLanguageConfig, RecursiveChunkConfig, RecursiveSplitter};

let config = CustomLanguageConfig {
    language_name: "myformat".to_string(),
    aliases: vec!["mf".to_string()],
    separators_regex: vec![r"---".to_string()],
};
let splitter = RecursiveSplitter::with_custom_languages(vec![config])?;

let text = "Part1---Part2---Part3";
let chunks = splitter.split_with(
    text,
    RecursiveChunkConfig {
        chunk_size: 10,
        min_chunk_size: Some(3),
        chunk_overlap: None,
        language: Some("myformat".to_string()),
    },
);
```

`CustomLanguageConfig` has the three fields `language_name`, `aliases` and `separators_regex`; `RecursiveSplitter::with_custom_languages(configs)?` registers them.

## API reference

For detailed API documentation, build the crate documentation with `cargo doc --features text --open` and open the `cocoindex::ops::text` module.