CocoIndex overview

CocoIndex is a real-time data transformation framework for AI with incremental processing: only the data that actually changed is reprocessed, lineage is observable end to end, and the schema is known up front.

Version
v 1.0.14
Last reviewed
Nov 4, 2026

CocoIndex is a framework for building data processing pipelines for AI workloads, with built-in incremental processing.

Programming model

CocoIndex uses a declarative, state-driven programming model. You specify what your target should look like as a function of your source data — not how to incrementally update it. CocoIndex handles change detection and applies only the necessary updates automatically.

If you’ve used React, spreadsheets, or materialized views, this will feel familiar:

  • React: declare UI as a function of state → React re-renders what changed
  • Spreadsheets: declare formulas → cells recompute when inputs change
  • CocoIndex: declare target states as a function of source → CocoIndex syncs what changed

CocoIndex features

Rust engine

CocoIndex executes pipelines with a Rust engine and a Python API.

Python transformations

Write transformations in Python with your existing libraries. Write batch-style code; CocoIndex tracks dependencies and applies incremental updates in batch and live mode.

Incremental processing

CocoIndex tracks fine-grained dependencies and recomputes work affected by changes to source data or processing code.

Lineage

Inspect processing steps, intermediate results, and execution paths to trace how the pipeline produced its output.

Open integration model

Sources and targets use a standard, open interface. Use Python models, functions, and libraries in your transformations.

Controlled concurrency

Pipelines run work concurrently with configurable limits and batch requests to external services and models.

Recovery

The engine retries transient failures and resumes from previous progress after interruptions.

Managed state

CocoIndex manages change detection, cached results, target state reconciliation, execution concurrency, and backfills.

Incremental data processing

CocoIndex stores state across runs to identify new or changed data.

What incremental processing means:

  • Avoid unnecessarily recomputing work, based on multi-level change detection:
    • Component level: only reprocess source items with changes
    • Function level: within an item’s processing, memoize expensive function calls and reuse when possible
    • Target level: apply minimum necessary changes (insertions, updates, deletions) to the target
  • Support multiple mechanisms to capture source changes (CDC, poll-based) out of the box

Use the updated output for serving, training, or feature computation.

Next steps