CocoIndex overview

CocoIndex is a real-time data transformation framework for AI with incremental processing, with a Rust SDK that builds pipelines into your own binary: only the data that actually changed is reprocessed, and lineage is observable end to end.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

CocoIndex is a framework for building data processing pipelines for AI workloads, with built-in incremental processing.

Programming model

CocoIndex uses a declarative, state-driven programming model. You specify what your target should look like as a function of your source data — not how to incrementally update it. CocoIndex handles change detection and applies only the necessary updates automatically.

If you’ve used React, spreadsheets, or materialized views, this will feel familiar:

• React: declare UI as a function of state → React re-renders what changed
• Spreadsheets: declare formulas → cells recompute when inputs change
• CocoIndex: declare target states as a function of source → CocoIndex syncs what changed

CocoIndex features

Rust engine

Rust transformations

Write transformations as ordinary async Rust functions with your existing crates. Write batch-style code; CocoIndex tracks dependencies and applies incremental updates in batch and live mode. Attribute macros add logic tracking, memoization and batching to a function; row types derive their schema.

Incremental processing

CocoIndex tracks fine-grained dependencies and recomputes work affected by changes to source data or processing code.

Lineage

Inspect processing steps, intermediate results, and execution paths to trace how the pipeline produced its output.

Open integration model

Sources and targets use a standard, open interface. Use Rust crates, models, and functions in your transformations.

Controlled concurrency

Pipelines run work concurrently with configurable limits and batch requests to external services and models.

Recovery

The engine retries transient failures and resumes from previous progress after interruptions.

Managed state

CocoIndex manages change detection, cached results, target state reconciliation, execution concurrency, and backfills.

Incremental data processing

CocoIndex stores state across runs to identify new or changed data.

What incremental processing means:

• Avoid unnecessarily recomputing work, based on multi-level change detection:
  • Component level: only reprocess source items with changes
  • Function level: within an item’s processing, memoize expensive function calls and reuse when possible
  • Target level: apply minimum necessary changes (insertions, updates, deletions) to the target
• Support multiple mechanisms to capture source changes (CDC, poll-based) out of the box

Use the updated output for serving, training, or feature computation.

The Rust SDK

The Rust SDK is the cocoindex crate. It has the same programming model as the Python SDK and runs on the same engine, with the same state database format:

• An app is an async function that receives a &Ctx and declares target states through it. App::open opens the app, and app.run(...) runs one update.
• Functions are async fns marked with #[cocoindex::function]. #[cocoindex::function(memo)] memoizes one, and #[cocoindex::function(batching)] batches calls to it.
• Processing components are mounted with mount!, use_mount! and mount_each!, at stable component paths.
• Resources, such as a database connection or a model, are shared through typed context keys (cocoindex::context_key!) and read with ctx.get_key(&KEY).
• Connectors and operations are Cargo features, so a crate builds only what it uses.
• The cocoindex CLI for Rust projects builds your crate and runs its apps.

The crate is not on crates.io yet; Installation shows how to depend on it from the repository.

Next steps

• Install CocoIndex and follow the Quickstart to build your first pipeline
• Read Core Concepts for the mental model behind CocoIndex