Internal storage configuration

CocoIndex persists target states and memo results in LMDB. The database grows automatically when a write runs out of space; tune the initial map_size and max_dbs via environment variables or programmatic settings for large-scale deployments.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

CocoIndex uses an LMDB database to persist its internal state. This database tracks target states and memoization results from previous runs, enabling CocoIndex to detect what changed and apply only the necessary updates.

Database path

CocoIndex needs a database path (db_path) to know where to store this internal state. The simplest way to set it is via the COCOINDEX_DB environment variable:

export COCOINDEX_DB=./cocoindex.db

You can also set it programmatically in a lifespan function:

#[cocoindex::lifespan]
async fn lifespan(builder: EnvironmentBuilder) -> Result<EnvironmentBuilder> {
    Ok(builder.db_path("./cocoindex.db"))
}

let env = Environment::builder().db_path("./cocoindex.db").build().await?;
// or, for a single app:
let app = App::builder("MyApp").db_path("./cocoindex.db").build().await?;

Setting db_path in a lifespan takes precedence over the COCOINDEX_DB environment variable. Environment::from_registered() runs every #[cocoindex::lifespan] and uses the variable when no lifespan set a db_path; if neither is provided, it returns an error. That is the path the cocoindex CLI takes.

Opening and closing the database

Opening a db_path opens its LMDB environment, and LMDB allows one environment per path per process. Opening the same path again while the first one is still held fails. See Closing an Environment for when an explicit environment releases its store, and the error you get when you reopen a path too early.

LMDB tuning

The LMDB database has two tunable settings, set with the builder’s lmdb_max_dbs(..) and lmdb_map_size(..). The defaults work well for most use cases — you only need to adjust them for large-scale deployments. Environment::from_registered() reads the two environment variables when a lifespan did not set them.

When to adjust

• lmdb_map_size — most deployments never need to touch it. The map grows on its own and each step is cheap, so it is worth setting only when that growth actually costs you something: a large state re-climbing from 4 GiB on every restart, or resize pauses you can measure. If you do size it up, every enlargement logs LMDB map full, auto-resizing to N bytes and retrying — the largest N you see is the number to use. On 64-bit systems, the map size is a virtual address space reservation, so setting it larger than needed is safe and consumes neither physical memory nor disk.
• lmdb_max_dbs — raise it if you have an unusually large number of apps sharing a single database directory.

Configuration

Via environment variables:

export COCOINDEX_LMDB_MAP_SIZE=8589934592   # 8 GiB
export COCOINDEX_LMDB_MAX_DBS=2048

Or programmatically in a lifespan function:

#[cocoindex::lifespan]
async fn lifespan(builder: EnvironmentBuilder) -> Result<EnvironmentBuilder> {
    Ok(builder
        .db_path("./cocoindex.db")
        .lmdb_map_size(8 * 1024 * 1024 * 1024) // 8 GiB
        .lmdb_max_dbs(2048))
}

The same setters exist on App::builder(name) and Environment::builder().

Inspecting the database

The cocoindex ls and cocoindex show commands read the database directly when you pass --db (and --app-name for show), without loading your app:

cocoindex ls --db ./cocoindex.db
cocoindex show --db ./cocoindex.db --app-name MyApp
cocoindex show --db ./cocoindex.db --app-name MyApp --target-states
cocoindex show --db ./cocoindex.db --app-name MyApp '/"a.md"'

Without --db, ls uses COCOINDEX_DB. With an app target instead of --db and --app-name, the CLI builds the crate and asks the app for its database.

The database format does not depend on the SDK that wrote it. The Python CLI lists and shows a database written by a Rust app, and prints the same output.

Inspecting from code

cocoindex::inspect reads the same data from a program: an app’s stable paths, the details of each one, and the target states it tracks. Every function takes an App; its _by_name form takes an Environment and an app name, for an app whose pipeline the binary does not hold. The functions only read.

use cocoindex::inspect::{self, StablePath, StablePathQuery};
use futures::TryStreamExt;

for path in inspect::list_stable_paths(&app).await? {
    println!("{path}");
}

// Streams read in one transaction and buffer nothing.
let mut target_states = std::pin::pin!(inspect::iter_target_states(&app).await);
while let Some(entry) = target_states.try_next().await? {
    println!("{} owner:{}", entry.readable_path, entry.owner_component_path);
}

// One path, with the paths directly under it.
let query = StablePathQuery { include_children: true, ..Default::default() };
let details = inspect::query_stable_path_details(&app, &StablePath::root(), query).await?;

// Without the `App`: an environment and the app's name.
let one = inspect::get_stable_path_detail_by_name(&env, "MyApp", &StablePath::root()).await?;

A streaming read holds its LMDB transaction for the whole stream, so a map resize waits for it to finish (see How the map grows).