CocoIndex overview
CocoIndex is a real-time data transformation framework for AI with incremental processing: only the data that actually changed is reprocessed, lineage is observable end to end, and the schema is known up front.
CocoIndex is a framework for building data processing pipelines for AI workloads, with built-in incremental processing.
Programming model
CocoIndex uses a declarative, state-driven programming model. You specify what your target should look like as a function of your source data — not how to incrementally update it. CocoIndex handles change detection and applies only the necessary updates automatically.
If you’ve used React, spreadsheets, or materialized views, this will feel familiar:
- React: declare UI as a function of state → React re-renders what changed
- Spreadsheets: declare formulas → cells recompute when inputs change
- CocoIndex: declare target states as a function of source → CocoIndex syncs what changed
CocoIndex features
Rust engine
CocoIndex executes pipelines with a Rust engine and a Python API.Python transformations
Write transformations in Python with your existing libraries. Write batch-style code; CocoIndex tracks dependencies and applies incremental updates in batch and live mode.
Incremental processing
CocoIndex tracks fine-grained dependencies and recomputes work affected by changes to source data or processing code.
Lineage
Inspect processing steps, intermediate results, and execution paths to trace how the pipeline produced its output.
Open integration model
Sources and targets use a standard, open interface. Use Python models, functions, and libraries in your transformations.
Controlled concurrency
Pipelines run work concurrently with configurable limits and batch requests to external services and models.
Recovery
The engine retries transient failures and resumes from previous progress after interruptions.
Managed state
CocoIndex manages change detection, cached results, target state reconciliation, execution concurrency, and backfills.
Incremental data processing
CocoIndex stores state across runs to identify new or changed data.
What incremental processing means:
- Avoid unnecessarily recomputing work, based on multi-level change detection:
- Component level: only reprocess source items with changes
- Function level: within an item’s processing, memoize expensive function calls and reuse when possible
- Target level: apply minimum necessary changes (insertions, updates, deletions) to the target
- Support multiple mechanisms to capture source changes (CDC, poll-based) out of the box
Use the updated output for serving, training, or feature computation.
Next steps
- Install CocoIndex and follow the Quickstart to build your first pipeline
- Read Core Concepts for the mental model behind CocoIndex