# Local filesystem connector > **CocoIndex v1, Rust SDK (experimental).** This page documents the CocoIndex **v1** Rust SDK (the `cocoindex` crate). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are at https://cocoindex.io/docs/. > > Source: https://cocoindex.io/docs/rust/connectors/localfs/ · Rust docs index: https://cocoindex.io/docs/rust/llms.txt · Rust SDK tour: https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/SHOWCASE.md · Rust examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust The local filesystem connector provides utilities for reading files from and writing files to the local file system. ```rust use cocoindex::resources::fs::{self, DirTarget, FileOptions, FilePath}; ``` The connector is the module `cocoindex::resources::fs`. It is always compiled; `cocoindex::prelude::*` exports `DirTarget`, `FileEntry`, `walk_dir`, `walk_items` and `PatternFilePathMatcher`. Live watching (`.live()`) needs the `fs_live` Cargo feature. ## Stable memoization with FilePath A key feature of the local filesystem connector is [**stable memoization**](/docs/rust/programming_guide/function/) through `FilePath`. When you move your entire project directory, memoization keys remain stable as long as you use the same `ContextKey` key name for the base directory. ### Using ContextKey for stable base directories Declare a `ContextKey` with `context_key!`; its name is the stable identifier. Provide the actual path when you build the environment, then build paths under it with `FilePath::under_context_dir` and pass them to `walk_dir`, `DirTarget::mount` and related functions. ```rust use std::path::PathBuf; use cocoindex::resources::fs::{DirTarget, FilePath}; use cocoindex::Environment; // The key's name ("OUT_DIR") is the stable identifier. cocoindex::context_key!(static OUT_DIR: PathBuf); // Provide the actual directory when you build the environment: let app = Environment::builder() .db_path(&db) .provide_key(&OUT_DIR, base) .build() .await? .app("fs_context_dir") .await?; app.update(move |ctx| async move { // A path under the key's directory. Memo keys and target keys name // the key, not the directory. let root = FilePath::under_context_dir(&ctx, &OUT_DIR, "out")?; let t = DirTarget::mount(&ctx, root)?; t.declare_file(&ctx, "a.txt", b"a")?; Ok(()) }) .await?; ``` `FilePath::under_context_dir(ctx, &KEY, path)` builds the path under the key's directory. Every function that takes `impl Into`, such as `walk_dir` and `DirTarget::mount`, accepts that `FilePath`. When you move your project to a different location, just update the path in `provide_key` — memoization keys remain the same because they're based on the stable key name (`OUT_DIR`), not the filesystem path. Target files are keyed the same way, by the key name and the relative path. Because the base directory isn't part of the key, CocoIndex can't tell when two keys reach the same file. See [One file, one declaration](#one-file-one-declaration). ### FilePath `FilePath` combines an optional base directory key and a relative path. Its path operations are methods: ```rust // Under a context directory: the key and the directory it was provided with. let source = FilePath::under_context_dir(&ctx, &SOURCE_DIR, "")?; // Create paths with join let config_path = source.join("config").join("settings.json"); // Access path properties config_path.name(); // "settings.json" config_path.suffix(); // ".json" config_path.parent(); // Some(FilePath pointing to "config") // Resolve to the full path let abs_path = config_path.resolve(); // std::path::PathBuf ``` `FilePath::new(path)` builds a path with no base directory. The other path operations are `stem`, `suffixes`, `parts`, `with_name`, `with_stem`, `with_suffix`, `relative_to` and `as_posix`. See [FilePath](/docs/rust/common_resources/data_types/#filepath) in Resource Types for full details. ## As source Use `walk_dir()` to list the files in a directory. It returns a `DirWalker`, a builder whose `walk()` and `items()` return the matching files. ```rust pub fn walk_dir(path: impl Into) -> DirWalker impl DirWalker { pub fn recursive(self, recursive: bool) -> Self pub fn path_matcher(self, matcher: impl FilePathMatcher + 'static) -> Self pub fn walk(&self) -> Result> pub fn items(&self) -> Result> pub fn live(self) -> LiveDirWalker // `fs_live` feature } impl LiveDirWalker { pub fn poll_interval(self, interval: Duration) -> Self pub fn rescan_interval(self, interval: Option) -> Self } ``` **Parameters:** - `path` — The root directory to walk: any `impl Into`, such as a `&str`, a `PathBuf`, or a `FilePath` under a context directory. - `.recursive(bool)` — If `true`, recursively walk subdirectories. Defaults to `false`. - `.path_matcher(matcher)` — Filter for files and directories. Defaults to match-all. See [PatternFilePathMatcher](/docs/rust/common_resources/data_types/#patternfilepathmatcher). - `.live()` — Turns the walker into a `LiveDirWalker` for `ctx.mount_each_live` (`fs_live` feature). See [Live file watching](#live-file-watching). - `.poll_interval(interval)`, `.rescan_interval(interval)` — `LiveDirWalker` only: how often to poll where the native watcher is unavailable (default one second), and the interval between periodic full rescans (default one hour; `None` disables it). **Returns:** A `DirWalker`. `walk()` returns the matching `FileEntry`s and `items()` the `(key, file)` pairs; both block on the directory walk, so call them before the pipeline starts or from a blocking context (a live feed walks on a blocking thread). A missing root directory yields no files; a root that is not a directory is an error. Symlinked directories are followed, except into a directory that is already an ancestor (a cycle); a symlink to a file is a file. A directory the process can't read, and an entry that vanishes or can't be stat-ed during the walk, are skipped. ### Iterating files `walk()` returns `FileEntry` values, which implement the [`FileLike`](/docs/rust/common_resources/data_types/#filelike) trait: ```rust for file in walk_dir("/path/to/documents").recursive(true).walk()? { let text = file.content_str()?; // ... } ``` `walk()` reads the directory and returns a `Vec`, sorted by relative path. `content()` and `content_str()` are blocking reads. In an async function prefer the `FileLike` reads: `file.read().await?` reads on the runtime's blocking pool and caches the content, returned as a shared `bytes::Bytes`, and `file.read_text().await?` decodes it. The walker has no async iteration: `walk()` and `items()` return the full list. ### Keyed iteration with `items()` `DirWalker::items()` returns keyed `(String, FileEntry)` pairs, useful for associating each file with a stable string key (its relative path): ```rust for (key, file) in walk_dir("/path/to/dir").recursive(true).items()? { let content = file.content()?; } // Shorthand: recursive walk with glob patterns, as `(key, file)` pairs // ready for `mount_each!`. let files = cocoindex::resources::fs::walk_items(&source_dir, &["**/*.md"])?; ``` The key is `FileEntry::key()`: the relative path with forward slashes. ### Filtering files Use `PatternFilePathMatcher` to filter which files and directories are included: ```rust use cocoindex::prelude::*; // Include only .py and .md files, exclude hidden directories and test files let matcher = PatternFilePathMatcher::new( ["**/*.py", "**/*.md"], ["**/.*", "**/test_*", "**/__pycache__"], )?; for file in walk_dir("/path/to/project") .recursive(true) .path_matcher(matcher) .walk()? { process(&ctx, file).await?; } ``` `PatternFilePathMatcher::new(included, excluded)` takes two iterables of patterns; `PatternFilePathMatcher::include(patterns)` takes only included patterns. An excluded pattern that starts with `!` is a negation, as in Python. ### Live file watching `.live()` turns the walker into a `LiveDirWalker`, a [`LiveMapView`](/docs/rust/advanced_topics/live_component/#livemapfeed-and-livemapview) plus `LiveMapFeed`. Combined with `ctx.mount_each_live`, this enables automatic incremental file watching — new, modified, and deleted files are processed without a full rescan: ```rust // One component per markdown file; each declares its own rows. In live // mode (`cocoindex update -L`) the walker keeps watching the directory. let files = walk_dir(sourcedir) .recursive(true) .path_matcher(PatternFilePathMatcher::new( ["**/*.md"], std::iter::empty::<&str>(), )?) .live() .poll_interval(Duration::from_secs(1)); ctx.mount_each_live(&"files", files, move |ctx, file: FileEntry| { let table = table.clone(); async move { for row in process_file(&ctx, &file).await? { table.declare_row(&ctx, &row)?; } Ok(()) } }) .await ``` `.live()` needs the `fs_live` Cargo feature. The `LiveDirWalker` follows the directory through the platform's native watcher: each event re-runs or removes that file's component alone (whether its work can be skipped is the file's memo decision), a directory that appears is walked, and a directory that moves away removes every file under it. Where the native watcher cannot be set up, it polls instead, every `poll_interval` (default one second). Every `rescan_interval` (default one hour; `None` disables it) it recreates the watcher and walks the whole directory, as a safety net against events the platform dropped. `ctx.mount_each_live(&key, feed, |ctx, file| ...)` mounts one child component per file. See `examples/rust/text_embedding` for the full program. See [Live Mode](/docs/rust/programming_guide/live_mode/) for how this works and how to enable it on the app. ### Example ```rust use cocoindex::prelude::*; /// Memoized on the file: an unchanged file is not processed again. #[cocoindex::function(memo)] async fn process_file(_ctx: &Ctx, file: FileEntry) -> Result<()> { let text = file.content_str()?; // ... process the file content ... Ok(()) } let app = cocoindex::App::open("files_transform", ".cocoindex_db").await?; let stats = app .run(move |ctx| { let source_dir = source_dir.clone(); async move { let files = cocoindex::resources::fs::walk_items(&source_dir, &["**/*.md"])?; // One processing component per file. mount_each!(files, |file| process_file(ctx, file)).await?; Ok(()) } }) .await?; ``` `mount_each!(items, |item| f(ctx, item))` mounts one processing component per `(key, file)` pair; `FileEntry::key()` is the component subpath. See `examples/rust/files_transform` and `examples/rust/text_embedding` for full programs. ## As target The `localfs` connector provides target state APIs for writing files. CocoIndex tracks what files should exist and automatically handles creation, updates, and deletion. ### declare_file Declare a single file target. This is the simplest way to write a file. ```rust pub fn declare_file(ctx: &Ctx, path: impl Into, content: &[u8]) -> Result<()> pub fn declare_file_with_options( ctx: &Ctx, path: impl Into, content: &[u8], options: FileOptions, ) -> Result<()> pub struct FileOptions { pub create_parent_dirs: bool, } ``` **Parameters:** - `path` — The filesystem path for the file: any `impl Into`. - `content` — The file content, as bytes. - `create_parent_dirs` — In `FileOptions`, `declare_file_with_options` only. If `true`, create parent directories if they don't exist. Defaults to `false`. **Example:** ```rust app.update(move |ctx| async move { fs::declare_file_with_options( &ctx, path, b"hello", FileOptions { create_parent_dirs: true, }, ) }) .await?; ``` `path` is any `impl Into`: a `PathBuf`, or `FilePath::under_context_dir(&ctx, &OUTPUT_DIR, "readme.txt")?` for a file under a context directory. ### declare_dir_target Declare a directory target for writing multiple files. Returns a `DirTarget` for declaring files within. The common form is `DirTarget::mount(ctx, dir)`: ```rust pub fn dir_target(dir: impl Into) -> DirTargetState impl DirTargetState { pub fn create_parent_dirs(self, create_parent_dirs: bool) -> Self } pub fn declare_dir_target(ctx: &Ctx, target: DirTargetState) -> Result pub fn mount_dir_target(ctx: &Ctx, dir: impl Into) -> Result impl DirTarget { pub fn mount(ctx: &Ctx, dir: impl Into) -> Result // same as mount_dir_target } ``` `declare_dir_target(ctx, dir_target(dir).create_parent_dirs(false))` sets the option. **Parameters:** - `dir` — The filesystem path for the directory: any `impl Into`. - `create_parent_dirs` — On `DirTargetState`. If `true`, create parent directories if they don't exist. Defaults to `true`. **Returns:** A `DirTarget`, ready at once: there is no pending state to resolve. A `DirTarget` is `Clone` and can be an argument of `mount_each!` or a memoized function, as in the [target example](#target-example). ### DirTarget.declare_file Declares a file to be written within the directory. ```rust let target = DirTarget::mount(ctx, "./output")?; target.declare_file(ctx, "result.txt", b"final output")?; // With options target.declare_file_with_options( &ctx, "made/a.txt", b"AAA", FileOptions { create_parent_dirs: true, }, )?; ``` **Parameters:** - `filename` — The name of the file (can include subdirectory path). `.` and `..` segments are normalized, so `./a.txt` and `a.txt` are one file. A name that is absolute or leaves the directory (such as `../a.txt`) returns an error. - `content` — The file content (bytes). - `create_parent_dirs` — In `FileOptions`, `declare_file_with_options` only. If `true`, create parent directories within the target directory. ### DirTarget.declare_dir_target Declares a subdirectory target within the directory. ```rust let t = DirTarget::mount(&ctx, &out1)?; let sub = t.declare_dir_target(&ctx, "sub", false)?; sub.declare_file(&ctx, "b.txt", b"BBB")?; let deep = sub.declare_dir_target(&ctx, "deep/er", true)?; deep.declare_file(&ctx, "c.txt", b"CCC")?; ``` The third argument is `create_parent_dirs`. **Parameters:** - `path` — The path of the subdirectory (relative to this directory). It's normalized and checked like `filename` above. - `create_parent_dirs` — If `true`, create parent directories. **Returns:** A `DirTarget` for the subdirectory. ### Target example ```rust use cocoindex::prelude::*; /// Process one markdown file: render it and declare the HTML output. #[cocoindex::function] async fn process_file(ctx: &Ctx, file: FileEntry, target: DirTarget) -> Result<()> { let html = render_markdown(ctx, &file).await?; target.declare_file(ctx, &output_name_for(&file), html.as_bytes())?; Ok(()) } #[tokio::main] async fn main() -> Result<()> { let (source_dir, output_dir) = parse_args(); let app = cocoindex::App::open("files_transform", ".cocoindex_db").await?; let stats = app .run(move |ctx| { let source_dir = source_dir.clone(); let output_dir = output_dir.clone(); async move { // Declarative output: the engine reconciles these files against // the previous run and deletes outputs whose source disappeared. let target = DirTarget::mount(&ctx, &output_dir)?; let files = cocoindex::resources::fs::walk_items(&source_dir, &["**/*.md"])?; // One processing component per file. mount_each!(files, |file| process_file(ctx, file, target)).await?; Ok(()) } }) .await?; println!("{stats}"); Ok(()) } ``` This example uses plain paths. For a root under a `ContextKey`, build it with `FilePath::under_context_dir` as shown in [Using ContextKey for stable base directories](#using-contextkey-for-stable-base-directories). See `examples/rust/files_transform` for the full program. ### One file, one declaration CocoIndex identifies each target file by its key: the `ContextKey` name (none for a path relative to the current directory) and the path under it. The base directory isn't part of the key, which is what lets you move it. The cost is that CocoIndex can't tell when two keys reach the same file on disk. Declare each physical file through exactly one key. Two keys can reach one file in these ways: - **Overlapping roots.** Two root directory targets are the same directory, or one is inside the other: under two `ContextKey`s, a `ContextKey` and a plain path, or two paths that reach one directory through a symlink. CocoIndex logs a warning when it applies two such targets. Two `ContextKey`s that point at one base directory are fine if their roots are disjoint, such as `FilePath::under_context_dir(&ctx, &A, "foo")?` and `FilePath::under_context_dir(&ctx, &B, "bar")?`. - **One file named two ways.** Names are normalized, so `./a.txt`, `x/../a.txt` and `a.txt` are one key. A flat path and a nested directory target are not: `target.declare_file(&ctx, "sub/a.txt", ..)` and `target.declare_dir_target(&ctx, "sub", false)` with `declare_file(&ctx, "a.txt", ..)` are two keys for one file. - **Names that differ only in case** on a case-insensitive file system (the default on macOS and Windows). There, `A.txt` and `a.txt` are one file. - **A file declared through a path into a directory target.** If `sub` is a directory target and its parent also declares `sub/a.txt`, removing the `sub` directory target removes `sub/a.txt` too. Removing a directory target removes the whole directory. You can switch a file from one key to another in one update if the processing component that declares the file stays the same. Examples are a case-only rename, or a flat path `sub/a.txt` moved to a nested `sub` directory target. CocoIndex writes the file under the new key and skips the delete under the old key. After a case-only rename on a case-insensitive file system, the file keeps its old spelling on disk. The opposite switch, from a nested directory target to flat paths, takes two updates. A nested directory target is resolved in its own processing component, so CocoIndex removes it after it writes the flat files, and that removal deletes them. Remove the directory target and run an update first. Then declare the flat files and run a second update. A relative path, such as `PathBuf::from("out")`, uses the current directory as its base, and the key holds only the relative path. If you run the app from another directory, the same keys reach other files: CocoIndex doesn't write unchanged files to the new place, and the old files stay. Run the app from one directory, or use a `ContextKey`.