CocoIndex is an open-source incremental engine for derived data. You write the transformation in Python. The Rust engine keeps its output in sync as the source data and your code change. The output is the context that AI applications and agents read: chunks and embeddings, extracted entities, knowledge graphs, summaries, and rows in a database. Examples include recommendation systems and other applications that depend on data indexing.
This blog covers how the engine works, the design decisions behind it, and examples you can run.
Agents read derived data, and it goes stale
A retrieval system, a recommendation model, or an agent that grounds itself in your documents does not read the source directly. It reads data derived from the source: a vector index, a graph, a table of extracted fields. The answer is only as current as that derived data.
Two inputs change all the time:
- The source data. Records arrive, change, and are deleted. A codebase changes on every commit. Meeting notes, tickets, and documents change during the day.
- The processing code. You add a field, change the chunk size, swap the embedding model, or fix a prompt.
Agents make this worse. An agent that works for hours writes code, records decisions, and edits files. Those outputs become source data for its next step, and for the next agent. When work resumes after a test run, a teammate’s change, or a human reply, the agent needs the current state, not the state of the last nightly build.
Why full rebuilds and hand-written deltas both fail
Most pipelines answer change with a scheduled full rebuild. The cost of a rebuild grows with the size of the corpus, not with the size of the change. Latency is bounded by the schedule, and the index is disrupted while it rebuilds.
Incremental updates by hand are correct only if you build all of these and keep them consistent:
- Change detection across files, databases, queues, and APIs.
- Deletes. When a source item disappears, every row, chunk, and vector derived from it must go too.
- Intermediate state, so that work whose inputs did not change is not recomputed.
- Code changes. When a function changes, re-run the work that used it, and only that work.
- Schema changes and backfills in the target when the output shape changes.
Each item is a place for a subtle bug. A missed delete leaves a stale vector that retrieval returns months later.
The model: target state is a function of source state
CocoIndex takes a state-driven approach. You do not write the how of an update. You declare what the target should hold as a function of the current source:
TargetState = Transform(SourceState)The engine computes the inserts, updates, and deletes that move the target to that state. If you have used spreadsheets, React, or materialized views, the model is familiar:
- Spreadsheets: you declare formulas in cells; when an upstream cell changes, downstream cells recompute.
- React: you declare your UI as a function of state; when state changes, React re-renders what changed.
- CocoIndex: you declare your target as a function of your source; CocoIndex syncs what changed.
CocoIndex core model: you declare the target as a function of the source. Source data (files, databases, APIs) flows into a declarative CocoIndex flow where target equals f of source, run on a Rust engine with Python transforms, producing a fresh vector index and knowledge graph. Underneath, an incremental engine tracks fine-grained lineage: it detects what changed, recomputes only the delta, and syncs the targets, taking end-to-end updates from hours or days down to seconds.
target = f(source)≈ spreadsheets · React · materialized viewsRust enginePython transformsfresh rowsFresh indexvector indexknowledge graphincremental engine · tracks fine-grained lineage, recomputes only the deltaDetect changediff source and codeRecompute deltaonly what changedSync targetshours or days → secondsThis excerpt summarizes and embeds each file, then declares one row per file in a Postgres table. It is plain async Python with no DSL:
import cocoindex as coco
from cocoindex.connectors import postgres
@coco.fn.as_async(batching=True)
async def embed(texts: list[str]) -> list[list[float]]: ...
@coco.fn(memo=True)
async def process_file(
file: FileLike,
target: postgres.TableTarget[DocRow],
):
summary = await summarize(file)
vector = await embed(summary)
target.declare_row(row=DocRow(
name=file.name,
summary=summary,
embedding=vector,
))declare_row does not write to the database. It declares that the row should exist. Each source file gets its own processing component, which owns the target states that file produces. After a run, CocoIndex compares the newly declared target states with those of the previous run:
| Target state | First declared | Declared differently | No longer declared |
|---|---|---|---|
| A table | Create the table | Alter the table | Drop the table |
| A row | Insert the row | Update the row | Delete the row |
Deletes follow from the same rule: if a source file disappears, its component declares nothing, and CocoIndex deletes its rows.
How the engine decides what to re-run
A function decorated with @coco.fn takes part in change detection. With memo=True, CocoIndex caches its result. On the next run, it skips the function body if three things are unchanged:
- Logic: the function’s source code (or its
version), plus the logic of every@coco.fnit calls during the run. - Inputs: the arguments.
- Context: the context values it reads with change detection on.
A memo hit returns the previous result and carries over the target states the call declared last time.
Take the excerpt above with 7 files in docs/, where only api.md changes. The 6 unchanged files are memo hits. process_file runs for api.md only, and the engine updates 1 row in Postgres. If you instead edit summarize, its logic fingerprint changes, so every process_file call that used it re-runs. If you edit an unrelated helper, nothing re-runs. In practice this can take end-to-end update times from hours or days down to seconds.
What the engine provides
- A Rust core with a Python API. Pipelines execute on a Rust runtime built for resilient, scalable data processing. You write transformations in plain Python with the types you already use.
- Catch-up and live mode from one codebase. The same code runs as a one-off
app.update()that processes what changed since the last run, or in live mode, which keeps the app running and applies source changes as they stream in. - Lineage you can inspect. Every processing step, intermediate result, and execution path is inspectable, so you can see how a given output was produced. This matters for audit and transparency requirements.
- An open integration model. Sources and targets plug in through a standard interface. The connectors include Postgres, Qdrant, LanceDB, Neo4j, Kafka, S3, and others, and you can write a custom one in Python.
Where this goes: shared context for agents
CocoIndex maintains derived data as its inputs change. We are extending that engine to support shared knowledge and agent collaboration, so that agents can build on each other’s work. One agent’s findings become fresh source data for the next, and an assumption that a later change invalidated is detected instead of reused.
We start with software engineering. CocoIndex Code keeps a codebase index current, so coding and review agents can find code and trace what a change affects.
Who it’s for
CocoIndex is for AI and data engineers who build retrieval, knowledge graphs from documents, codebase indexes, recommendation features, or the context layer for agents. You write the transformation once, and the engine keeps the target in sync as data and logic change. The same model applies to a first embedding pipeline and to a large index that updates all day.
CocoIndex is open source (Apache-2.0) and has crossed 10,000+ stars on GitHub. To learn the programming model, read the overview and core concepts.
What we will publish here
- Technical deep dives into the architecture and design decisions
- Performance benchmarks and optimization techniques
- Practices for running CocoIndex in AI systems
- Release notes for new features and improvements
- Case studies from our community
To follow development, watch the GitHub repository.
About the author
CEO and cofounder of CocoIndex. Writes about incremental data infrastructure, agents, and the engineering decisions behind the engine.
Build fresh context for your agents.
CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.