CocoIndex
CocoIndex

Data transformation for AI

Ultra performant data transformation framework for AI, with incremental processing and data lineage out-of-box. Exceptional developer velocity. Production-ready at day 0.
Data transformation flow diagram

CocoIndex

Open source ETL framework designed for AI workloads.

CocoIndex is an ultra performant data transformation framework, with core engine written in Rust. Make it super easy to transform data with AI workloads, and keep source data and target in sync effortlessly.

Either creating embedding, building knowledge graphs, or doing any data transformations - beyond traditional SQL.
CocoIndex is the worlds' first open source ETL framework / compute engine that supports 1) incremental processing 2) custom logic 3) compute heavy transformation beyond SQL, for example: LLM inference, structured extraction, vector embedding, etc.

Exceptional Velocity

Just declare transformations in dataflow, minimal code needed. Get started with ~100 lines of Python.
# import
data['content'] = flow_builder.add_source(...)

# transform
data['out'] = data['content']
    .transform(...)
    .transform(...)

# collect data
collector.collect(...)

# export to db, vector db, graph db ...
collector.export(...)
Data flow diagram showing CocoIndex's transformation pipeline

Assemble building blocks

Native building blocks for different sources, targets and transformations. Standardize interface, make it 1-line code switch between different components - as easy as assemble building blocks.
CocoIndex native components: 1) ingestion: local files, ingestion API, web pages, cloud storage. 2) indexing: parse, extract structure, chunk, embed, map, dedup, cluster, reconcile. 3) export: relational db, vector db, graph db, object db.

Incremental Processing

Out-of-box support for incremental indexing
  • minimal recomputation on source or logic change.
  • (re-)processing necessary portions; reuse cache when possible
Incremental processing diagram showing CocoIndex's efficient data flow

Flow is Single Source of Truth

Define once, run in multiple modes
  • batch
  • long running job (live update that watches your source)
  • sample based fast preview run (cocoinsight)
Automatic schema setup based on logic and data
  • data processing and schema management
CocoIndex flow is the single source of truth. once a flow is defined, it can be run in multiple modes, and schemas in target stores are automatically managed.

CocoInsight

Your Data Lineage and Observability Companion

You don't need to be a data expert. CocoInsight provides you with best in-class tools to understand your pipeline step by step, explains the process, and helps you choose the best indexing strategy.

One of the most loved feature for our users building ETL with coco, with significant boost on developer velocity, and lowering the barrier to entry for data engineering.
CocoInsight's data lineage and observability tool

️ Loved by builders

Unity logo

At Unity, we use CocoIndex to incrementally index critical unstructured assets. With it, we've managed to dramatically reduce unnecessary data processing computation and remote LLM embedding calls.


CocoIndex is our “Kubernetes moment” – it empowers us to index and operate data with exceptional efficiency, keeping everything always current and context-ready for AI.

Wenlan Yang, Staff Engineer @ Unity
Share you love

Start free and scales as you grow

Open Source

  • Self-hosted
  • Free
  • Apache 2.0 license

CocoInsight

Team/Enterprise

  • VPC / On Premise Deployments
  • Guaranteed customer support and SLA
  • Enterprise source connectors
  • Data governance - PII
  • Cost and usage optimization
  • CocoInsight
  • Support and control plane for distributed computing