Common data types

Data types shared across connectors and built-in operations — FileLike, FilePath, FilePathMatcher, Chunk with text positions, and the Embedder trait for single-text async embedding.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The cocoindex::resources modules (file, fs, chunk, embedder, id, live_map, rate_limit, schema) provide common data models and abstractions shared across connectors and built-in operations. The types this page names are re-exported from cocoindex::prelude. Connectors provide concrete implementations — for example, the local file system (cocoindex::resources::fs) has FileEntry, which implements FileLike, and its walk_dir builds a FilePath for each file. See individual connector docs for connector-specific details.

File

The file module (cocoindex::resources::file) defines types and traits and utilities for working with file-like objects.

FileLike

use cocoindex::prelude::*;

#[cocoindex::function(memo)]
async fn read_file(ctx: &Ctx, file: &FileEntry) -> Result<String> {
    file.read_text().await
}

Methods:

• file_path() — A FilePath representing the file’s path. Access the relative path via file_path().path() (&Path).
• metadata().await? — A FileMetadata: size in bytes, modified, and an optional content_fingerprint.
• read().await? — The file content as bytes::Bytes (the cached content, shared, not copied).
• read_size(size).await? — At most size bytes of the content.
• read_text().await? — The content as text. Detects the encoding by BOM, falls back to UTF-8, and decodes with replacement characters.
• read_text_with(encoding, errors).await? — The content as text with an explicit decoding: encoding is a WHATWG label such as Some("latin1") or Some("utf-16le") (or "utf-32le" / "utf-32be"), None detects by BOM; errors is DecodeErrors::Replace (the default of read_text), Ignore or Strict.
• content_fingerprint().await? — The Fingerprint the memo validation uses.

decode_bytes_with decodes bytes you already hold the same way. FileEntry adds the sync helpers content() and content_str(), plus key(), stem() and relative_path().

Memoization:

FileLike objects provide a memoization key based on file_path (file identity). When used as arguments to a memoized function, CocoIndex uses a two-level validation: it checks the modification time first (cheap), then computes a content fingerprint only if the modification time has changed. This means touching a file or moving it won’t cause unnecessary recomputation if the content is unchanged. When the backend reports a fingerprint at no cost (for example an S3 ETag), CocoIndex compares it even when the modification time is unchanged. This catches an overwrite within the resolution of the modification time (S3 reports whole seconds).

A FileLike is a MemoInput whose key is file_path().memo_key(), and whose state is the modification time, validated by the content fingerprint when the time changed. A FileEntry can therefore be an argument of a #[cocoindex::function(memo)] directly, or inside a Vec.

FilePath

FilePath combines a base directory (with a stable key) and a relative path. This enables stable memoization even when the entire directory tree is moved to a different location.

use cocoindex::prelude::*;

// A path with no base key: memoization follows the path itself.
let path = FilePath::new("docs/guide.md");

// A base directory under a stable key: the tree can move between runs.
let files = walk_dir(FilePath::with_base_dir("docs", root.to_path_buf(), PathBuf::new()))
    .recursive(true)
    .walk()?;

// The base directory comes from a `ContextKey<PathBuf>` provided in the lifespan.
let root = FilePath::under_context_dir(&ctx, &OUT_DIR, "out")?;

One FilePath struct serves every connector.

Constructors:

• FilePath::new(path) — A path with no base key.
• FilePath::with_base_dir(base_key, base_dir, path) — A path under a base directory with a stable key.
• FilePath::under_context_dir(&ctx, &KEY, path)? — Reads the base directory from a ContextKey<PathBuf> in the context and uses the key’s name as the base key.

Methods:

• path() — The path relative to the base directory (&Path).
• base_key() — The optional base key, used for stable memoization.
• resolve() — The full path, a PathBuf (the base directory joined with the path).

Path Operations:

FilePath has the pathlib.PurePath-style operations as methods: join(child), name(), stem(), suffix(), suffixes(), parts(), parent(), with_name(..), with_stem(..), with_suffix(..), parents(), relative_to(base), is_relative_to(base), matches(pattern) and as_posix(). matches follows PurePath.match: a relative pattern is matched from the right, one component at a time, and an absolute pattern must match the whole path. Each method that returns a path keeps the base key.

Memoization:

FilePath provides a memoization key based on (base_key, path), returned by memo_key(); it falls back to the path alone when the FilePath has no base key. This means:

• Two FilePath objects with the same base directory key and relative path have the same memo key
• Moving the entire project directory doesn’t invalidate memoization, as long as the same base directory key is used

For connector-specific usage, see the individual connector documentation like Local File System.

FilePathMatcher

FilePathMatcher is a trait for filtering files and directories during traversal.

pub trait FilePathMatcher: Send + Sync {
    fn is_dir_included(&self, _path: &Path) -> bool {
        true
    }

    fn is_file_included(&self, path: &Path) -> bool;
}

is_dir_included returns true to traverse a directory (the default); is_file_included returns true to include a file. Pass an implementation to walk_dir(..).path_matcher(..). MatchAllFilePathMatcher includes everything.

PatternFilePathMatcher

A built-in FilePathMatcher implementation using globset patterns:

use cocoindex::prelude::*;

// Include only Markdown files; no exclusions.
let files = walk_dir(sourcedir)
    .recursive(true)
    .path_matcher(PatternFilePathMatcher::new(
        ["**/*.md"],
        std::iter::empty::<&str>(),
    )?);

PatternFilePathMatcher::new(included, excluded) takes two iterables of patterns and returns a Result: an invalid glob is an error at construction. PatternFilePathMatcher::include(patterns) is the form with no exclusions.

Parameters:

• included_patterns — Glob patterns (globset syntax) for files to include. Use **/*.ext to match at any depth. An empty list includes all files.
• excluded_patterns — Glob patterns (globset syntax) for files/directories to exclude. Excluded directories are not traversed. A pattern prefixed with ! negates the exclusion for matching paths (see below).

Gitignore-style negation. Within excluded_patterns, a pattern beginning with ! un-excludes paths that would otherwise be excluded by a preceding pattern. This lets you exclude broadly and then carve out exceptions, instead of enumerating every directory to exclude:

// Exclude all dot-entries, but keep .github through
let matcher = PatternFilePathMatcher::new(
    std::iter::empty::<&str>(),
    [
        "**/.*",          // exclude all dot-files and dot-directories
        "!**/.github/**", // …except anything under .github
    ],
)?;

Directory traversal honors negations correctly: a directory excluded by a normal pattern is still traversed when a ! pattern could match its contents, so the kept paths inside it are reachable.

Chunk

The chunk module (cocoindex::resources::chunk) defines types for representing text chunks produced by text splitters.

Chunk

A Chunk (in the prelude) represents a piece of text with its position information in the original document.

use cocoindex::resources::chunk::Chunk;

Fields and methods:

• start (TextPosition) — Start position in the original text.
• end (TextPosition) — End position in the original text.
• range() — The byte range of the chunk in the original text.
• text(&source) — The text content of the chunk: the slice of the original string.

TextPosition

Fields:

• byte_offset (usize) — Byte offset from the start of the text.
• char_offset (usize) — Character offset from the start of the text.
• line (u32) — 1-based line number.
• column (u32) — 1-based column number.

Example:

use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter};

let splitter = RecursiveSplitter::new()?;
let chunks = splitter.split_with(
    &text,
    RecursiveChunkConfig {
        chunk_size: 2000,
        min_chunk_size: None,
        chunk_overlap: Some(500),
        language: Some("markdown".to_string()),
    },
);

for chunk in &chunks {
    println!("[{}:{}] {}", chunk.start.line, chunk.start.column, chunk.text(&text));
}

Embedder

The embedder module (cocoindex::resources::embedder) defines a trait for single-text async embedding.

Embedder trait

use cocoindex::prelude::*;

#[async_trait]
pub trait Embedder: Send + Sync {
    async fn embed(&self, ctx: &Ctx, text: &str) -> Result<Vec<f32>>;
}

This is the call-site contract that consumers like resolve_entities rely on. Both SentenceTransformerEmbedder and ApiEmbedder implement this trait — embedder.embed(ctx, "some text").await? returns a single Vec<f32>.

The trait takes the Ctx explicitly, so an implementation can memoize each text and batch concurrent calls through the engine; the two built-in embedders do both. The trait is deliberately narrow: it does not include the vector dimension or vector schema, which are concerns of connectors and table-schema creation, not of embedding consumers.

use cocoindex::entity_resolution;

// Any `Embedder` works with resolve_entities.
let embedder = ctx.get_key(&EMBEDDER)?;
let resolved = entity_resolution::resolve_entities(
    ctx,
    names,
    embedder,
    &resolver,
    None,
    entity_resolution::ResolveOptions::default(),
)
.await?;

resolve_entities takes an EntityEmbedder. Every Embedder is one through a blanket implementation, so a SentenceTransformerEmbedder, an ApiEmbedder or your own type works as is.