# Internal storage configuration

> **CocoIndex v1, Rust SDK (experimental).** This page documents the CocoIndex **v1** Rust SDK (the `cocoindex` crate). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are at https://cocoindex.io/docs/.
>
> Source: https://cocoindex.io/docs/rust/advanced_topics/internal_storage/ · Rust docs index: https://cocoindex.io/docs/rust/llms.txt · Rust SDK tour: https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/SHOWCASE.md · Rust examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust

CocoIndex uses an [LMDB](http://www.lmdb.tech/doc/) database to persist its internal state. This database tracks target states and [memoization](/docs/rust/programming_guide/function/) results from previous runs, enabling CocoIndex to detect what changed and apply only the necessary updates.

## Database path

CocoIndex needs a database path (`db_path`) to know where to store this internal state. The simplest way to set it is via the `COCOINDEX_DB` environment variable:

```bash
export COCOINDEX_DB=./cocoindex.db
```

You can also set it programmatically in a [lifespan function](/docs/rust/programming_guide/app/#lifespan-optional):

```rust
#[cocoindex::lifespan]
async fn lifespan(builder: EnvironmentBuilder) -> Result<EnvironmentBuilder> {
    Ok(builder.db_path("./cocoindex.db"))
}
```

Or set it on the builder when you create an [`Environment`](/docs/rust/advanced_topics/multiple_environments/) or an `App` directly:

```rust
let env = Environment::builder().db_path("./cocoindex.db").build().await?;
// or, for a single app:
let app = App::builder("MyApp").db_path("./cocoindex.db").build().await?;
```

Setting `db_path` in a lifespan takes precedence over the `COCOINDEX_DB` environment variable. `Environment::from_registered()` runs every `#[cocoindex::lifespan]` and uses the variable when no lifespan set a `db_path`; if neither is provided, it returns an error. That is the path the `cocoindex` CLI takes.

Only `Environment::from_registered()` reads `COCOINDEX_DB`. A plain `Environment::builder().build()` or `App::builder(name).build()` reads no environment variables and defaults `db_path` to `./coco_state`.

## Opening and closing the database

Opening a `db_path` opens its LMDB environment, and LMDB allows one environment per path per process. Opening the same path again while the first one is still held fails. See [Closing an Environment](/docs/rust/advanced_topics/multiple_environments/#closing-an-environment) for when an explicit environment releases its store, and the error you get when you reopen a path too early.

## LMDB tuning

The LMDB database has two tunable settings, set with the builder's `lmdb_max_dbs(..)` and `lmdb_map_size(..)`. The defaults work well for most use cases — you only need to adjust them for large-scale deployments. `Environment::from_registered()` reads the two environment variables when a lifespan did not set them.

| Setting | Default | Env Variable | Description |
|---------|---------|-------------|-------------|
| `lmdb_max_dbs` | `1024` | `COCOINDEX_LMDB_MAX_DBS` | Maximum number of apps (named LMDB sub-databases) sharing one `db_path`. Unlike the map size, this **is** a hard cap — creating one app beyond it fails. Applied at open time, so raising it takes effect on the next start. Must be &ge; 1. |
| `lmdb_map_size` | `4294967296` (4 GiB) | `COCOINDEX_LMDB_MAP_SIZE` | *Initial* size of the LMDB memory map in bytes. CocoIndex enlarges the map automatically as the state grows, so this is a starting point, not a cap. Must be &gt; 0; rounded up to the nearest multiple of the system page size. |

### When to adjust

- **`lmdb_map_size`** — most deployments never need to touch it. The map grows on its own and each step is cheap, so it is worth setting only when that growth actually costs you something: a large state re-climbing from 4 GiB on every restart, or resize pauses you can measure. If you do size it up, every enlargement logs `LMDB map full, auto-resizing to N bytes and retrying` — the largest `N` you see is the number to use. On 64-bit systems, the map size is a virtual address space reservation, so setting it larger than needed is safe and consumes neither physical memory nor disk.
- **`lmdb_max_dbs`** — raise it if you have an unusually large number of apps sharing a single database directory.

**Note — How the map grows**
When a write runs out of map space, CocoIndex doubles the map and retries, repeating until the write fits. The remap itself is cheap (no data copy, no file rewrite) but it needs exclusivity: it waits for every LMDB transaction open in the process to finish, and queues new ones behind it. That's imperceptible for ordinary reads and writes, but a long streaming read (`cocoindex show` over a large app) holds its transaction for the whole stream, and the resize waits that long.

The enlarged size doesn't carry over. The next process opens at the configured map size, raised to at least the space the data file already occupies — the file's high-water mark, not the live data, since LMDB reuses freed pages but never shrinks the file. So a database that outgrew its configured map size still opens and reads fine, just with no headroom: the next write needing new space enlarges it again. A map that grew to 16 GiB over a 10 GiB file reopens at 10 GiB, not 16, then doubles to 20 GiB on the next write that needs space.

**Note — What typically grows the store**
- **The memoization cache** — usually the bulk. Each entry stores its call's return value, so memoized functions that return large values (e.g., embedding vectors) dominate quickly.
- **Target-state bookkeeping** — grows with the number of target states your app declares.

If internal state grows faster than your targets, oversized memoized return values are the first thing to check — see the cost note and placement rule in [When to memoize](/docs/rust/programming_guide/function/#when-to-memoize).

### Configuration

Via environment variables:

```bash
export COCOINDEX_LMDB_MAP_SIZE=8589934592   # 8 GiB
export COCOINDEX_LMDB_MAX_DBS=2048
```

Or programmatically in a lifespan function:

```rust
#[cocoindex::lifespan]
async fn lifespan(builder: EnvironmentBuilder) -> Result<EnvironmentBuilder> {
    Ok(builder
        .db_path("./cocoindex.db")
        .lmdb_map_size(8 * 1024 * 1024 * 1024) // 8 GiB
        .lmdb_max_dbs(2048))
}
```

The same setters exist on `App::builder(name)` and `Environment::builder()`.

## Inspecting the database

The [`cocoindex ls` and `cocoindex show`](/docs/rust/cli/#ls) commands read the database directly when you pass `--db` (and `--app-name` for `show`), without loading your app:

```bash
cocoindex ls --db ./cocoindex.db
cocoindex show --db ./cocoindex.db --app-name MyApp
cocoindex show --db ./cocoindex.db --app-name MyApp --target-states
cocoindex show --db ./cocoindex.db --app-name MyApp '/"a.md"'
```

Without `--db`, `ls` uses `COCOINDEX_DB`. With an app target instead of `--db` and `--app-name`, the CLI builds the crate and asks the app for its database.

The database format does not depend on the SDK that wrote it. The [Python CLI](/docs/cli/) lists and shows a database written by a Rust app, and prints the same output.

### Inspecting from code

`cocoindex::inspect` reads the same data from a program: an app's stable paths, the details of each one, and the target states it tracks. Every function takes an `App`; its `_by_name` form takes an `Environment` and an app name, for an app whose pipeline the binary does not hold. The functions only read.

```rust
use cocoindex::inspect::{self, StablePath, StablePathQuery};
use futures::TryStreamExt;

for path in inspect::list_stable_paths(&app).await? {
    println!("{path}");
}

// Streams read in one transaction and buffer nothing.
let mut target_states = std::pin::pin!(inspect::iter_target_states(&app).await);
while let Some(entry) = target_states.try_next().await? {
    println!("{} owner:{}", entry.readable_path, entry.owner_component_path);
}

// One path, with the paths directly under it.
let query = StablePathQuery { include_children: true, ..Default::default() };
let details = inspect::query_stable_path_details(&app, &StablePath::root(), query).await?;

// Without the `App`: an environment and the app's name.
let one = inspect::get_stable_path_detail_by_name(&env, "MyApp", &StablePath::root()).await?;
```

A streaming read holds its LMDB transaction for the whole stream, so a map resize waits for it to finish (see [How the map grows](#lmdb-tuning)).