# Common data types > **CocoIndex v1, Rust SDK (experimental).** This page documents the CocoIndex **v1** Rust SDK (the `cocoindex` crate). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are at https://cocoindex.io/docs/. > > Source: https://cocoindex.io/docs/rust/common_resources/data_types/ · Rust docs index: https://cocoindex.io/docs/rust/llms.txt · Rust SDK tour: https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/SHOWCASE.md · Rust examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust The `cocoindex::resources` modules (`file`, `fs`, `chunk`, `embedder`, `id`, `live_map`, `rate_limit`, `schema`) provide common data models and abstractions shared across connectors and built-in operations. The types this page names are re-exported from `cocoindex::prelude`. Connectors provide concrete implementations — for example, the local file system (`cocoindex::resources::fs`) has `FileEntry`, which implements `FileLike`, and its `walk_dir` builds a `FilePath` for each file. See individual [connector docs](/docs/rust/connectors/localfs/) for connector-specific details. ## File The file module (`cocoindex::resources::file`) defines types and traits and utilities for working with file-like objects. ### FileLike `FileLike` is a trait for file objects with async read methods. Each connector has a type that implements it: `FileEntry` for the local file system, `AzureBlobFile` for Azure Blob Storage, and so on. A function takes the concrete type, as below, or a `&dyn FileLike`. ```rust use cocoindex::prelude::*; #[cocoindex::function(memo)] async fn read_file(ctx: &Ctx, file: &FileEntry) -> Result { file.read_text().await } ``` **Methods:** - `file_path()` — A `FilePath` representing the file's path. Access the relative path via `file_path().path()` (`&Path`). - `metadata().await?` — A `FileMetadata`: `size` in bytes, `modified`, and an optional `content_fingerprint`. - `read().await?` — The file content as `bytes::Bytes` (the cached content, shared, not copied). - `read_size(size).await?` — At most `size` bytes of the content. - `read_text().await?` — The content as text. Detects the encoding by BOM, falls back to UTF-8, and decodes with replacement characters. - `read_text_with(encoding, errors).await?` — The content as text with an explicit decoding: `encoding` is a WHATWG label such as `Some("latin1")` or `Some("utf-16le")` (or `"utf-32le"` / `"utf-32be"`), `None` detects by BOM; `errors` is `DecodeErrors::Replace` (the default of `read_text`), `Ignore` or `Strict`. - `content_fingerprint().await?` — The `Fingerprint` the memo validation uses. `decode_bytes_with` decodes bytes you already hold the same way. `FileEntry` adds the sync helpers `content()` and `content_str()`, plus `key()`, `stem()` and `relative_path()`. **Memoization:** `FileLike` objects provide a memoization key based on `file_path` (file identity). When used as arguments to a [memoized function](/docs/rust/programming_guide/function/#memoization), CocoIndex uses a two-level validation: it checks the modification time first (cheap), then computes a content fingerprint only if the modification time has changed. This means touching a file or moving it won't cause unnecessary recomputation if the content is unchanged. When the backend reports a fingerprint at no cost (for example an S3 ETag), CocoIndex compares it even when the modification time is unchanged. This catches an overwrite within the resolution of the modification time (S3 reports whole seconds). A `FileLike` is a `MemoInput` whose key is `file_path().memo_key()`, and whose state is the modification time, validated by the content fingerprint when the time changed. A `FileEntry` can therefore be an argument of a `#[cocoindex::function(memo)]` directly, or inside a `Vec`. ### FilePath `FilePath` combines a **base directory** (with a stable key) and a **relative path**. This enables stable memoization even when the entire directory tree is moved to a different location. ```rust use cocoindex::prelude::*; // A path with no base key: memoization follows the path itself. let path = FilePath::new("docs/guide.md"); // A base directory under a stable key: the tree can move between runs. let files = walk_dir(FilePath::with_base_dir("docs", root.to_path_buf(), PathBuf::new())) .recursive(true) .walk()?; // The base directory comes from a `ContextKey` provided in the lifespan. let root = FilePath::under_context_dir(&ctx, &OUT_DIR, "out")?; ``` One `FilePath` struct serves every connector. **Constructors:** - `FilePath::new(path)` — A path with no base key. - `FilePath::with_base_dir(base_key, base_dir, path)` — A path under a base directory with a stable key. - `FilePath::under_context_dir(&ctx, &KEY, path)?` — Reads the base directory from a `ContextKey` in the context and uses the key's name as the base key. **Methods:** - `path()` — The path relative to the base directory (`&Path`). - `base_key()` — The optional base key, used for stable memoization. - `resolve()` — The full path, a `PathBuf` (the base directory joined with the path). **Path Operations:** `FilePath` has the `pathlib.PurePath`-style operations as methods: `join(child)`, `name()`, `stem()`, `suffix()`, `suffixes()`, `parts()`, `parent()`, `with_name(..)`, `with_stem(..)`, `with_suffix(..)`, `parents()`, `relative_to(base)`, `is_relative_to(base)`, `matches(pattern)` and `as_posix()`. `matches` follows `PurePath.match`: a relative pattern is matched from the right, one component at a time, and an absolute pattern must match the whole path. Each method that returns a path keeps the base key. **Memoization:** `FilePath` provides a memoization key based on `(base_key, path)`, returned by `memo_key()`; it falls back to the path alone when the `FilePath` has no base key. This means: - Two `FilePath` objects with the same base directory key and relative path have the same memo key - Moving the entire project directory doesn't invalidate memoization, as long as the same base directory key is used For connector-specific usage, see the individual connector documentation like [Local File System](/docs/rust/connectors/localfs/). ### FilePathMatcher `FilePathMatcher` is a trait for filtering files and directories during traversal. ```rust pub trait FilePathMatcher: Send + Sync { fn is_dir_included(&self, _path: &Path) -> bool { true } fn is_file_included(&self, path: &Path) -> bool; } ``` `is_dir_included` returns `true` to traverse a directory (the default); `is_file_included` returns `true` to include a file. Pass an implementation to `walk_dir(..).path_matcher(..)`. `MatchAllFilePathMatcher` includes everything. #### PatternFilePathMatcher A built-in `FilePathMatcher` implementation using [globset](https://docs.rs/globset/#syntax) patterns: ```rust use cocoindex::prelude::*; // Include only Markdown files; no exclusions. let files = walk_dir(sourcedir) .recursive(true) .path_matcher(PatternFilePathMatcher::new( ["**/*.md"], std::iter::empty::<&str>(), )?); ``` `PatternFilePathMatcher::new(included, excluded)` takes two iterables of patterns and returns a `Result`: an invalid glob is an error at construction. `PatternFilePathMatcher::include(patterns)` is the form with no exclusions. **Parameters:** - `included_patterns` — Glob patterns ([globset](https://docs.rs/globset) syntax) for files to include. Use `**/*.ext` to match at any depth. An empty list includes all files. - `excluded_patterns` — Glob patterns ([globset](https://docs.rs/globset) syntax) for files/directories to exclude. Excluded directories are not traversed. A pattern prefixed with `!` negates the exclusion for matching paths (see below). **Note** Patterns use [globset](https://docs.rs/globset) semantics: `*.py` matches only in the root directory; use `**/*.py` to match at any depth. **Gitignore-style negation.** Within `excluded_patterns`, a pattern beginning with `!` *un-excludes* paths that would otherwise be excluded by a preceding pattern. This lets you exclude broadly and then carve out exceptions, instead of enumerating every directory to exclude: ```rust // Exclude all dot-entries, but keep .github through let matcher = PatternFilePathMatcher::new( std::iter::empty::<&str>(), [ "**/.*", // exclude all dot-files and dot-directories "!**/.github/**", // …except anything under .github ], )?; ``` Directory traversal honors negations correctly: a directory excluded by a normal pattern is still traversed when a `!` pattern could match its contents, so the kept paths inside it are reachable. ## Chunk The chunk module (`cocoindex::resources::chunk`) defines types for representing text chunks produced by [text splitters](/docs/rust/ops/text/). ### Chunk A `Chunk` (in the prelude) represents a piece of text with its position information in the original document. ```rust use cocoindex::resources::chunk::Chunk; ``` **Fields and methods:** - `start` (`TextPosition`) — Start position in the original text. - `end` (`TextPosition`) — End position in the original text. - `range()` — The byte range of the chunk in the original text. - `text(&source)` — The text content of the chunk: the slice of the original string. A `Chunk` does not own its text. It keeps the byte range, and `chunk.text(&source)` slices the original string, so chunking a large document does not copy its contents. ### TextPosition A struct representing a position in text. **Fields:** - `byte_offset` (`usize`) — Byte offset from the start of the text. - `char_offset` (`usize`) — Character offset from the start of the text. - `line` (`u32`) — 1-based line number. - `column` (`u32`) — 1-based column number. **Example:** ```rust use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter}; let splitter = RecursiveSplitter::new()?; let chunks = splitter.split_with( &text, RecursiveChunkConfig { chunk_size: 2000, min_chunk_size: None, chunk_overlap: Some(500), language: Some("markdown".to_string()), }, ); for chunk in &chunks { println!("[{}:{}] {}", chunk.start.line, chunk.start.column, chunk.text(&text)); } ``` ## Embedder The embedder module (`cocoindex::resources::embedder`) defines a trait for single-text async embedding. ### Embedder trait ```rust use cocoindex::prelude::*; #[async_trait] pub trait Embedder: Send + Sync { async fn embed(&self, ctx: &Ctx, text: &str) -> Result>; } ``` This is the call-site contract that consumers like [`resolve_entities`](/docs/rust/ops/entity_resolution/) rely on. Both [`SentenceTransformerEmbedder`](/docs/rust/ops/sentence_transformers/) and [`ApiEmbedder`](/docs/rust/ops/model_apis/) implement this trait — `embedder.embed(ctx, "some text").await?` returns a single `Vec`. The trait takes the `Ctx` explicitly, so an implementation can memoize each text and batch concurrent calls through the engine; the two built-in embedders do both. The trait is deliberately narrow: it does not include the vector dimension or vector schema, which are concerns of connectors and table-schema creation, not of embedding consumers. ```rust use cocoindex::entity_resolution; // Any `Embedder` works with resolve_entities. let embedder = ctx.get_key(&EMBEDDER)?; let resolved = entity_resolution::resolve_entities( ctx, names, embedder, &resolver, None, entity_resolution::ResolveOptions::default(), ) .await?; ``` `resolve_entities` takes an `EntityEmbedder`. Every `Embedder` is one through a blanket implementation, so a `SentenceTransformerEmbedder`, an `ApiEmbedder` or your own type works as is.