V0 is legacy and in maintenance mode. Please refer to CocoIndex V1.
Go to CocoIndex V1 →CocoIndex V0 Documentation (Legacy)
CocoIndex is an ultra-performant framework for building data pipelines for AI, with built-in incremental processing. Define a flow from your source to your target — the Rust engine keeps it in sync, reprocessing only the Δ.
$ pip install -U cocoindex Your first flow in ten minutes
Install, define a source and a target, and let the engine keep them in sync. Four ways in:
Quickstart →
Build your first flow end to end — chunk and embed markdown into Postgres with pgvector, only reprocessing changed files.
Installation →
Install the package and set up Postgres. Python 3.11+, pip / uv / poetry.
Overview →
What CocoIndex is, how flows map sources to targets, and how incremental syncing works.
Indexing basics →
Flows, data scopes, collectors, transforms, and exports — the building blocks of every pipeline.
Two steps. Then it maintains itself.
A flow is a function from source to target. Run it once; re-run it forever — CocoIndex tracks fine-grained lineage and recomputes only what changed, in batch or live mode.
Define the flow
Point at a source (LocalFile, Postgres, Google Drive, S3…), transform it — chunk, embed, extract, resolve entities — and collect the rows, vectors, or graph nodes you want, then export them to a target.
Run — only the Δ recomputes
The engine backfills on first run and keeps the target in sync on every re-run or in live mode. Change one file and only that file reprocesses. No DAGs, no orchestration to babysit.
Different views of your data
CocoIndex flows build the views an agent reasons over — vector indexes, knowledge graphs, and structured tables — and keep them fresh as the source changes.
Semantic & vector search →
Chunk, embed, and index documents into pgvector, Qdrant, LanceDB, or Pinecone.
Knowledge graphs →
Export entities and relationships into Neo4j or Kuzu — structured context for agents.
Structured extraction →
Turn PDFs and transcripts into typed rows with LLM extraction over your sources.
Live, incremental updates →
Keep targets in sync as sources change — in batch runs or continuous live mode.
Declarative Python. No DSL.
Define the flow with a decorator; CocoIndex runs it incrementally and keeps the output up to date — recomputing only the Δ.
Full quickstart →import cocoindex
@cocoindex.flow_def(name="TextEmbedding")
def text_embedding_flow(flow_builder, data_scope):
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="markdown_files"))
doc_embeddings = data_scope.add_collector()
with data_scope["documents"].row() as doc:
doc["chunks"] = doc["content"].transform(
cocoindex.functions.SplitRecursively(),
language="markdown", chunk_size=2000)
with doc["chunks"].row() as chunk:
chunk["embedding"] = chunk["text"].transform(
cocoindex.functions.SentenceTransformerEmbed(
model="sentence-transformers/all-MiniLM-L6-v2"))
doc_embeddings.collect(
filename=doc["filename"], location=chunk["location"],
text=chunk["text"], embedding=chunk["embedding"])
doc_embeddings.export(
"doc_embeddings", cocoindex.targets.Postgres(),
primary_key_fields=["filename", "location"])Sources in, transforms, targets out
Mix and match connectors and operations through one open interface — no vendor lock-in.
Build in the open
An incremental engine for AI data — Apache-2.0, built fully in the open.