We have been working on CocoIndex, a real-time data framework for AI, for a while, with lots of excitement from the community. We officially crossed 1k stars earlier this week. Huge thanks to everyone who starred, forked, contributed, or shared the love ❤️!
CocoIndex is an ultra performant data transformation framework, with its core engine written in Rust. The problem it tries to solve is to make it easy to prepare fresh data for AI (either creating embeddings, building knowledge graphs, or performing other data transformations) and take real-time data pipelines beyond traditional SQL.
The philosophy is to have the framework handle the source updates, and have developers only worry about defining a series of data transformations, inspired by spreadsheets.
Here’s a bit of history about our open source journey:
What is data flow programming?
Unlike a workflow orchestration framework where data is usually opaque, in CocoIndex, data and data operations are first-class citizens. CocoIndex follows the idea of the Dataflow programming model. Each transformation creates a new field solely based on input fields, without hidden states and value mutation. All data before/after each transformation is observable, with lineage out of the box.
For example, this is a kind of data flow: Parse files -> Data Mapping -> Data Extraction -> Knowledge Graph
Particularly, users don’t define data operations like creation, update, or deletion. Rather, they define something like this: for a set of source data, this is the transformation or formula. The framework takes care of the data operations such as when to create, update, or delete.
With operations like:
# ingest
data['content'] = flow_builder.add_source(...)
# transform
data['out'] = data['content']
.transform(...)
.transform(...)
# collect data
collector.collect(...)
# export to db, vector db, graph db ...
collector.export(...) How does CocoIndex keep data fresh?
As a data framework, CocoIndex takes data freshness to the next level. Incremental processing is one of the core values provided by CocoIndex.
The framework takes care of:
- Change data capture
- Figuring out what exactly needs to be updated, and only updating that without having to recompute everything throughout, backed by memoization of computation results.
This makes it fast to reflect any source updates to the target store. If you have concerns with surfacing stale data to AI agents and are spending lots of effort working on infrastructure pieces to optimize latency, the framework actually handles it for you.
Built-in modules and custom modules
We understand preparing data is highly use-case based and there is no one-size-fits-all solution. With all the latest advancements in models, there is no single best solution for all cases, and it’s also a balance of multiple factors in terms of quality, performance, pricing, etc. There are a lot of efforts in the data infra ecosystem that optimize for different goals: parsers, embedding models, vector DBs, graph DBs, and many others.
We take the composition approach, and instead of building everything, we provide native plugins to embrace the ecosystem and make it easier to plug in and swap any module by standardizing the interface, exactly like assembling building blocks.
And we can focus on what we’re best at: building the best incremental pipeline and compute engine in the AI era.
Tooling
In addition to standard statistics and reports with CocoIndex, we are building a product called CocoInsight. It has zero pipeline data retention and connects to your on-premise CocoIndex server for pipeline insights. This makes data directly visible and makes it easy to develop ETL pipelines.
Especially, as we mentioned earlier, CocoIndex is heavily influenced by spreadsheets. This is directly reflected in CocoInsight, as you can see all the data in the spreadsheet and how it looks after each transformation.
This is an example of mapping:
In addition to the data insights for data transformation, we also aim to provide tooling to make it directly visible to understand the LLM transformations, for example:
-
Understand and troubleshoot the chunking, if there’s any, and debug why a chunk is showing in search results
-
Understand the relation extraction for Graphs
-
and more
Data should be transparent, especially for ETL frameworks.
Support us
We are constantly improving, and more features and examples are coming soon. If you love this article, please drop us a star ⭐ at the GitHub repo to help us grow. You can also find us on Discord. I try to be there 24/7 😊 Looking forward to discussing CocoIndex or any topics in Data/AI infra with you!
About the author.
CEO and cofounder of CocoIndex. Writes about incremental data infrastructure, agents, and the engineering decisions behind the engine.
Frequently asked questions.
What is CocoIndex?
CocoIndex is an ultra-performant, real-time data transformation framework for AI, with its core engine written in Rust. It makes it easy to prepare fresh data for AI — creating embeddings, building knowledge graphs, or running other transformations — and takes real-time data pipelines beyond traditional SQL.
The philosophy is that the framework handles source updates while developers only define a series of data transformations, inspired by spreadsheets.
What is dataflow programming and how does CocoIndex use it?
In CocoIndex, data and data operations are first-class citizens, following the dataflow programming model. Each transformation creates a new field solely based on input fields, with no hidden state and no value mutation, so all data before and after each transformation is observable with lineage out of the box.
How does CocoIndex keep indexed data fresh?
CocoIndex relies on incremental processing: the framework handles change data capture and then figures out exactly what needs to be updated, updating only that instead of recomputing everything. This makes it fast to reflect source updates in the target store.
Developers don't define create, update, or delete operations themselves — they define the transformation for a set of source data, and the framework takes care of when to create, update, or delete. See How does CocoIndex keep data fresh?
Does CocoIndex have built-in modules, and can I plug in my own?
Yes. CocoIndex takes a composition approach rather than building everything itself: it provides native plugins for parsers, embedding models, vector DBs, graph DBs, and more, and standardizes the interface so you can plug in and swap any module like assembling building blocks.
What is CocoInsight?
CocoInsight is a companion product for pipeline insights. It has zero pipeline data retention and connects to your on-premise CocoIndex server, making data directly visible so it is easier to develop ETL pipelines. Influenced by spreadsheets, it lets you see all the data and how it looks after each transformation, including help understanding chunking and relation extraction for graphs.
See Tooling.
Is CocoIndex open source?
Yes. CocoIndex is open source on GitHub, and this post marks the milestone of crossing 1,000 stars. You can support the project by starring the repo and joining the community on Discord.
See Support us.
