We are officially open sourced! 🎉

We are thrilled to announce the open-source release of CocoIndex, the world’s first engine that supports both custom transformation logic and incremental processing specialized for data indexing.

CocoIndex combines custom transformation logic and incremental processing

CocoIndex is an ETL framework to prepare data for AI applications such as semantic search and retrieval-augmented generation (RAG). It offers a data-driven programming model that simplifies the creation and maintenance of data indexing pipelines, ensuring data freshness and consistency.

Why this is hard, and why we built an engine for it

Indexing pipelines have a property most ETL tooling doesn’t account for: the output is derived data that has to stay correct as the source keeps changing. The clean way to express that is TargetState = Transform(SourceState) — you declare what the index should look like given the current source, and something keeps it in sync. The trouble is that doing the “keeping in sync” by hand is brutal. You have to figure out what changed, compute the exact delta of inserts, updates, and deletes for the target, preserve intermediate results so you aren’t recomputing embeddings or LLM calls you’ve already paid for, and migrate the target schema when your own transformation code evolves. Get any of it wrong and you ship a stale or inconsistent index — the kind of bug that’s painful to even detect.

CocoIndex exists to own that bookkeeping. You write the transformation; the engine diffs each run against the last, skips work whose inputs and code are unchanged (memoization), and applies only the difference to the target. That’s the difference between a generic data framework and one built specifically for indexing.

CocoIndex is now open source under the Apache License 2.0. This means the core functionality of CocoIndex is freely available for anyone to use, modify, and distribute. We believe that open sourcing CocoIndex will foster innovation, enable broader adoption, and create a vibrant community of contributors who can help shape its future. By choosing the Apache License 2.0, we’re ensuring that both individual developers and enterprises can confidently build upon and integrate CocoIndex into their projects while maintaining the flexibility to create proprietary extensions.

Open source also matters specifically for infrastructure like this. An indexing pipeline sits between your data and your AI applications — it’s load-bearing. Teams should be able to read the engine that decides what gets reprocessed and what gets cached, extend it with the connectors and transforms their data actually needs, and run it without a vendor in the critical path. A permissive license is the honest way to make those commitments durable.

🔥 Key features

Data Flow Programming: Build indexing pipelines by composing transformations like building blocks, with built-in state management and observability.
Support Custom Logic: Plug in your choice of chunking, embedding, and vector stores. Extend with custom transformations like deduplication and reconciliation.
Incremental Processing: Smart state management minimizes re-computation by tracking changes at the file level, with future support for chunk-level granularity.
Python SDK: Built with a RUST core 🦀 for performance, exposed through an intuitive Python binding 🐍 for ease of use.

The Rust-core / Python-binding split is deliberate. The engine’s job — change detection, state tracking, computing deltas, and reconciling target states — is concurrent, long-running, and performance-sensitive, which is exactly where Rust earns its keep. But the people writing indexing logic want to reach for the rest of the AI and data ecosystem: chunkers, embedding models, LLM clients, vector stores. So the heavy lifting stays in Rust while everything you actually author — sources, transforms, targets — is plain Python. You get the engine’s guarantees without giving up the libraries you already use.

We are moving fast and a lot of features and improvements are coming soon.

🚀 Getting started

Installation: Install the CocoIndex Python library:
sh
```
pip install cocoindex
```

Set Up Postgres with pgvector Extension: Ensure Docker Compose is installed, then start a Postgres database:

docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/postgres.yaml) up -d

Define Your Indexing Flow: Create a flow to index your data. For example:

python

@cocoindex.flow_def(name="TextEmbedding")
def text_embedding(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
    data_scope["documents"] = flow_builder.add_source(cocoindex.sources.LocalFile(path="markdown_files"))
    doc_embeddings = data_scope.add_collector()

    with data_scope["documents"].row() as doc:
        doc["chunks"] = doc["content"].transform(
            cocoindex.functions.SplitRecursively(language="markdown", chunk_size=300, chunk_overlap=100))

        with doc["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].transform(
                cocoindex.functions.SentenceTransformerEmbed(model="sentence-transformers/all-MiniLM-L6-v2"))

            doc_embeddings.collect(filename=doc["filename"], location=chunk["location"],
                                   text=chunk["text"], embedding=chunk["embedding"])

    doc_embeddings.export(
        "doc_embeddings",
        cocoindex.storages.Postgres(),
        primary_key_fields=["filename", "location"],
        vector_indexes=[cocoindex.VectorIndex("embedding", cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])

For a detailed walkthrough, refer to our Quickstart Guide.

🤗 Community

We are super excited for community contributions of all kinds - whether it’s code improvements, documentation updates, issue reports, feature requests on GitHub, or discussions in our Discord.

Since this release the project has grown well past where it started: CocoIndex has crossed 10,000+ GitHub stars and shipped 200+ releases, a steady cadence of fixes, connectors, and capabilities driven in no small part by the community. If you find the project useful, the best way to help is to use it, tell us where it falls short, and contribute back.

GitHub: Please give us a star on the repository 🤗.
Documentation: Check out our documentation for detailed guides and API reference.
Discord: Join discussions, seek support, and share your experiences on our Discord server.
Social Media: Follow us on Twitter and LinkedIn for updates.

We would love to foster an inclusive, welcoming, and supportive environment. Contributing to CocoIndex should feel collaborative, friendly and enjoyable for everyone. Together, we can build better AI applications through robust data infrastructure.

Looking forward to seeing what you build with CocoIndex!

CocoIndex

Fresh context for long-horizon agents.

Learn the concept → View on GitHub