Local filesystem connector

Walk local directories with path-matcher filtering, watch for changes in live mode via the platform's native watcher, and write outputs with declare_file() and DirTarget — with memo-stable FilePath across project moves.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The local filesystem connector provides utilities for reading files from and writing files to the local file system.

use cocoindex::resources::fs::{self, DirTarget, FileOptions, FilePath};

The connector is the module cocoindex::resources::fs. It is always compiled; cocoindex::prelude::* exports DirTarget, FileEntry, walk_dir, walk_items and PatternFilePathMatcher. Live watching (.live()) needs the fs_live Cargo feature.

Stable memoization with FilePath

A key feature of the local filesystem connector is stable memoization through FilePath. When you move your entire project directory, memoization keys remain stable as long as you use the same ContextKey key name for the base directory.

Using ContextKey for stable base directories

Declare a ContextKey<PathBuf> with context_key!; its name is the stable identifier. Provide the actual path when you build the environment, then build paths under it with FilePath::under_context_dir and pass them to walk_dir, DirTarget::mount and related functions.

use std::path::PathBuf;

use cocoindex::resources::fs::{DirTarget, FilePath};
use cocoindex::Environment;

// The key's name ("OUT_DIR") is the stable identifier.
cocoindex::context_key!(static OUT_DIR: PathBuf);

// Provide the actual directory when you build the environment:
let app = Environment::builder()
    .db_path(&db)
    .provide_key(&OUT_DIR, base)
    .build()
    .await?
    .app("fs_context_dir")
    .await?;

app.update(move |ctx| async move {
    // A path under the key's directory. Memo keys and target keys name
    // the key, not the directory.
    let root = FilePath::under_context_dir(&ctx, &OUT_DIR, "out")?;
    let t = DirTarget::mount(&ctx, root)?;
    t.declare_file(&ctx, "a.txt", b"a")?;
    Ok(())
})
.await?;

FilePath::under_context_dir(ctx, &KEY, path) builds the path under the key’s directory. Every function that takes impl Into<FilePath>, such as walk_dir and DirTarget::mount, accepts that FilePath.

When you move your project to a different location, just update the path in provide_key — memoization keys remain the same because they’re based on the stable key name (OUT_DIR), not the filesystem path.

Target files are keyed the same way, by the key name and the relative path. Because the base directory isn’t part of the key, CocoIndex can’t tell when two keys reach the same file. See One file, one declaration.

FilePath

FilePath combines an optional base directory key and a relative path. Its path operations are methods:

// Under a context directory: the key and the directory it was provided with.
let source = FilePath::under_context_dir(&ctx, &SOURCE_DIR, "")?;

// Create paths with join
let config_path = source.join("config").join("settings.json");

// Access path properties
config_path.name();      // "settings.json"
config_path.suffix();    // ".json"
config_path.parent();    // Some(FilePath pointing to "config")

// Resolve to the full path
let abs_path = config_path.resolve();  // std::path::PathBuf

FilePath::new(path) builds a path with no base directory. The other path operations are stem, suffixes, parts, with_name, with_stem, with_suffix, relative_to and as_posix.

See FilePath in Resource Types for full details.

As source

Use walk_dir() to list the files in a directory. It returns a DirWalker, a builder whose walk() and items() return the matching files.

pub fn walk_dir(path: impl Into<FilePath>) -> DirWalker

impl DirWalker {
    pub fn recursive(self, recursive: bool) -> Self
    pub fn path_matcher(self, matcher: impl FilePathMatcher + 'static) -> Self
    pub fn walk(&self) -> Result<Vec<FileEntry>>
    pub fn items(&self) -> Result<Vec<(String, FileEntry)>>
    pub fn live(self) -> LiveDirWalker   // `fs_live` feature
}

impl LiveDirWalker {
    pub fn poll_interval(self, interval: Duration) -> Self
    pub fn rescan_interval(self, interval: Option<Duration>) -> Self
}

Parameters:

• path — The root directory to walk: any impl Into<FilePath>, such as a &str, a PathBuf, or a FilePath under a context directory.
• .recursive(bool) — If true, recursively walk subdirectories. Defaults to false.
• .path_matcher(matcher) — Filter for files and directories. Defaults to match-all. See PatternFilePathMatcher.
• .live() — Turns the walker into a LiveDirWalker for ctx.mount_each_live (fs_live feature). See Live file watching.
• .poll_interval(interval), .rescan_interval(interval) — LiveDirWalker only: how often to poll where the native watcher is unavailable (default one second), and the interval between periodic full rescans (default one hour; None disables it).

Returns: A DirWalker. walk() returns the matching FileEntrys and items() the (key, file) pairs; both block on the directory walk, so call them before the pipeline starts or from a blocking context (a live feed walks on a blocking thread). A missing root directory yields no files; a root that is not a directory is an error.

Symlinked directories are followed, except into a directory that is already an ancestor (a cycle); a symlink to a file is a file. A directory the process can’t read, and an entry that vanishes or can’t be stat-ed during the walk, are skipped.

Iterating files

walk() returns FileEntry values, which implement the FileLike trait:

for file in walk_dir("/path/to/documents").recursive(true).walk()? {
    let text = file.content_str()?;
    // ...
}

walk() reads the directory and returns a Vec<FileEntry>, sorted by relative path. content() and content_str() are blocking reads. In an async function prefer the FileLike reads: file.read().await? reads on the runtime’s blocking pool and caches the content, returned as a shared bytes::Bytes, and file.read_text().await? decodes it.

Keyed iteration with items()

DirWalker::items() returns keyed (String, FileEntry) pairs, useful for associating each file with a stable string key (its relative path):

for (key, file) in walk_dir("/path/to/dir").recursive(true).items()? {
    let content = file.content()?;
}

// Shorthand: recursive walk with glob patterns, as `(key, file)` pairs
// ready for `mount_each!`.
let files = cocoindex::resources::fs::walk_items(&source_dir, &["**/*.md"])?;

The key is FileEntry::key(): the relative path with forward slashes.

Filtering files

Use PatternFilePathMatcher to filter which files and directories are included:

use cocoindex::prelude::*;

// Include only .py and .md files, exclude hidden directories and test files
let matcher = PatternFilePathMatcher::new(
    ["**/*.py", "**/*.md"],
    ["**/.*", "**/test_*", "**/__pycache__"],
)?;

for file in walk_dir("/path/to/project")
    .recursive(true)
    .path_matcher(matcher)
    .walk()?
{
    process(&ctx, file).await?;
}

PatternFilePathMatcher::new(included, excluded) takes two iterables of patterns; PatternFilePathMatcher::include(patterns) takes only included patterns. An excluded pattern that starts with ! is a negation, as in Python.

Live file watching

.live() turns the walker into a LiveDirWalker, a LiveMapView plus LiveMapFeed. Combined with ctx.mount_each_live, this enables automatic incremental file watching — new, modified, and deleted files are processed without a full rescan:

// One component per markdown file; each declares its own rows. In live
// mode (`cocoindex update -L`) the walker keeps watching the directory.
let files = walk_dir(sourcedir)
    .recursive(true)
    .path_matcher(PatternFilePathMatcher::new(
        ["**/*.md"],
        std::iter::empty::<&str>(),
    )?)
    .live()
    .poll_interval(Duration::from_secs(1));
ctx.mount_each_live(&"files", files, move |ctx, file: FileEntry| {
    let table = table.clone();
    async move {
        for row in process_file(&ctx, &file).await? {
            table.declare_row(&ctx, &row)?;
        }
        Ok(())
    }
})
.await

.live() needs the fs_live Cargo feature. The LiveDirWalker follows the directory through the platform’s native watcher: each event re-runs or removes that file’s component alone (whether its work can be skipped is the file’s memo decision), a directory that appears is walked, and a directory that moves away removes every file under it. Where the native watcher cannot be set up, it polls instead, every poll_interval (default one second). Every rescan_interval (default one hour; None disables it) it recreates the watcher and walks the whole directory, as a safety net against events the platform dropped. ctx.mount_each_live(&key, feed, |ctx, file| ...) mounts one child component per file. See examples/rust/text_embedding for the full program.

See Live Mode for how this works and how to enable it on the app.

Example

use cocoindex::prelude::*;

/// Memoized on the file: an unchanged file is not processed again.
#[cocoindex::function(memo)]
async fn process_file(_ctx: &Ctx, file: FileEntry) -> Result<()> {
    let text = file.content_str()?;
    // ... process the file content ...
    Ok(())
}

let app = cocoindex::App::open("files_transform", ".cocoindex_db").await?;
let stats = app
    .run(move |ctx| {
        let source_dir = source_dir.clone();
        async move {
            let files = cocoindex::resources::fs::walk_items(&source_dir, &["**/*.md"])?;
            // One processing component per file.
            mount_each!(files, |file| process_file(ctx, file)).await?;
            Ok(())
        }
    })
    .await?;

mount_each!(items, |item| f(ctx, item)) mounts one processing component per (key, file) pair; FileEntry::key() is the component subpath. See examples/rust/files_transform and examples/rust/text_embedding for full programs.

As target

The localfs connector provides target state APIs for writing files. CocoIndex tracks what files should exist and automatically handles creation, updates, and deletion.

declare_file

Declare a single file target. This is the simplest way to write a file.

pub fn declare_file(ctx: &Ctx, path: impl Into<FilePath>, content: &[u8]) -> Result<()>

pub fn declare_file_with_options(
    ctx: &Ctx,
    path: impl Into<FilePath>,
    content: &[u8],
    options: FileOptions,
) -> Result<()>

pub struct FileOptions {
    pub create_parent_dirs: bool,
}

Parameters:

• path — The filesystem path for the file: any impl Into<FilePath>.
• content — The file content, as bytes.
• create_parent_dirs — In FileOptions, declare_file_with_options only. If true, create parent directories if they don’t exist. Defaults to false.

Example:

app.update(move |ctx| async move {
    fs::declare_file_with_options(
        &ctx,
        path,
        b"hello",
        FileOptions {
            create_parent_dirs: true,
        },
    )
})
.await?;

path is any impl Into<FilePath>: a PathBuf, or FilePath::under_context_dir(&ctx, &OUTPUT_DIR, "readme.txt")? for a file under a context directory.

declare_dir_target

Declare a directory target for writing multiple files. Returns a DirTarget for declaring files within.

The common form is DirTarget::mount(ctx, dir):

pub fn dir_target(dir: impl Into<FilePath>) -> DirTargetState
impl DirTargetState {
    pub fn create_parent_dirs(self, create_parent_dirs: bool) -> Self
}

pub fn declare_dir_target(ctx: &Ctx, target: DirTargetState) -> Result<DirTarget>
pub fn mount_dir_target(ctx: &Ctx, dir: impl Into<FilePath>) -> Result<DirTarget>

impl DirTarget {
    pub fn mount(ctx: &Ctx, dir: impl Into<FilePath>) -> Result<Self>  // same as mount_dir_target
}

declare_dir_target(ctx, dir_target(dir).create_parent_dirs(false)) sets the option.

Parameters:

• dir — The filesystem path for the directory: any impl Into<FilePath>.
• create_parent_dirs — On DirTargetState. If true, create parent directories if they don’t exist. Defaults to true.

Returns: A DirTarget, ready at once: there is no pending state to resolve. A DirTarget is Clone and can be an argument of mount_each! or a memoized function, as in the target example.

DirTarget.declare_file

Declares a file to be written within the directory.

let target = DirTarget::mount(ctx, "./output")?;
target.declare_file(ctx, "result.txt", b"final output")?;

// With options
target.declare_file_with_options(
    &ctx,
    "made/a.txt",
    b"AAA",
    FileOptions {
        create_parent_dirs: true,
    },
)?;

Parameters:

• filename — The name of the file (can include subdirectory path). . and .. segments are normalized, so ./a.txt and a.txt are one file. A name that is absolute or leaves the directory (such as ../a.txt) returns an error.
• content — The file content (bytes).
• create_parent_dirs — In FileOptions, declare_file_with_options only. If true, create parent directories within the target directory.

DirTarget.declare_dir_target

Declares a subdirectory target within the directory.

let t = DirTarget::mount(&ctx, &out1)?;
let sub = t.declare_dir_target(&ctx, "sub", false)?;
sub.declare_file(&ctx, "b.txt", b"BBB")?;
let deep = sub.declare_dir_target(&ctx, "deep/er", true)?;
deep.declare_file(&ctx, "c.txt", b"CCC")?;

The third argument is create_parent_dirs.

Parameters:

• path — The path of the subdirectory (relative to this directory). It’s normalized and checked like filename above.
• create_parent_dirs — If true, create parent directories.

Returns: A DirTarget for the subdirectory.

Target example

use cocoindex::prelude::*;

/// Process one markdown file: render it and declare the HTML output.
#[cocoindex::function]
async fn process_file(ctx: &Ctx, file: FileEntry, target: DirTarget) -> Result<()> {
    let html = render_markdown(ctx, &file).await?;
    target.declare_file(ctx, &output_name_for(&file), html.as_bytes())?;
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    let (source_dir, output_dir) = parse_args();

    let app = cocoindex::App::open("files_transform", ".cocoindex_db").await?;
    let stats = app
        .run(move |ctx| {
            let source_dir = source_dir.clone();
            let output_dir = output_dir.clone();
            async move {
                // Declarative output: the engine reconciles these files against
                // the previous run and deletes outputs whose source disappeared.
                let target = DirTarget::mount(&ctx, &output_dir)?;
                let files = cocoindex::resources::fs::walk_items(&source_dir, &["**/*.md"])?;

                // One processing component per file.
                mount_each!(files, |file| process_file(ctx, file, target)).await?;

                Ok(())
            }
        })
        .await?;

    println!("{stats}");
    Ok(())
}

This example uses plain paths. For a root under a ContextKey, build it with FilePath::under_context_dir as shown in Using ContextKey for stable base directories. See examples/rust/files_transform for the full program.

One file, one declaration

CocoIndex identifies each target file by its key: the ContextKey name (none for a path relative to the current directory) and the path under it. The base directory isn’t part of the key, which is what lets you move it. The cost is that CocoIndex can’t tell when two keys reach the same file on disk. Declare each physical file through exactly one key.

Two keys can reach one file in these ways:

• Overlapping roots. Two root directory targets are the same directory, or one is inside the other: under two ContextKeys, a ContextKey and a plain path, or two paths that reach one directory through a symlink. CocoIndex logs a warning when it applies two such targets. Two ContextKeys that point at one base directory are fine if their roots are disjoint, such as FilePath::under_context_dir(&ctx, &A, "foo")? and FilePath::under_context_dir(&ctx, &B, "bar")?.
• One file named two ways. Names are normalized, so ./a.txt, x/../a.txt and a.txt are one key. A flat path and a nested directory target are not: target.declare_file(&ctx, "sub/a.txt", ..) and target.declare_dir_target(&ctx, "sub", false) with declare_file(&ctx, "a.txt", ..) are two keys for one file.
• Names that differ only in case on a case-insensitive file system (the default on macOS and Windows). There, A.txt and a.txt are one file.
• A file declared through a path into a directory target. If sub is a directory target and its parent also declares sub/a.txt, removing the sub directory target removes sub/a.txt too. Removing a directory target removes the whole directory.

You can switch a file from one key to another in one update if the processing component that declares the file stays the same. Examples are a case-only rename, or a flat path sub/a.txt moved to a nested sub directory target. CocoIndex writes the file under the new key and skips the delete under the old key. After a case-only rename on a case-insensitive file system, the file keeps its old spelling on disk.

The opposite switch, from a nested directory target to flat paths, takes two updates. A nested directory target is resolved in its own processing component, so CocoIndex removes it after it writes the flat files, and that removal deletes them. Remove the directory target and run an update first. Then declare the flat files and run a second update.

A relative path, such as PathBuf::from("out"), uses the current directory as its base, and the key holds only the relative path. If you run the app from another directory, the same keys reach other files: CocoIndex doesn’t write unchanged files to the new place, and the old files stay. Run the app from one directory, or use a ContextKey.