Architecture
14 articles
System design and internals of CocoIndex and data indexing engines.
CocoIndex V1 is Live!
CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
A Safe pickle Alternative: Serialize Python with Type Hints
Why CocoIndex replaced pickle with serialization that reads Python type hints to choose msgspec, Pydantic or restricted pickle for each value.
Build a Python Daemon for a CLI: Auto-Start, Upgrade, Stop
Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
AI-native data pipelines: why we built CocoIndex
Why AI applications need open, AI-native data pipelines that keep context fresh as sources and code change, and the design choices behind CocoIndex.
Write a Custom Source in Python: Index Any API
Index an internal API with CocoIndex v1: plain async Python, deletes propagate, and a memo on updated_at skips unchanged items.
Thinking in Rust: Ownership, Access, and Memory Safety
A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
Control Processing Concurrency in CocoIndex
How CocoIndex v1 caps concurrent processing components with one app-level limit, lends slots to nested mounts, and adds rate and batch bounds.
Introducing CocoInsight
CocoInsight, the CocoIndex v0 tool for inspecting pipeline data and lineage step by step. It is not part of CocoIndex v1; v1 apps use cocoindex show.
Story of CocoIndex, at 1k stars
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
What Is Incremental Processing? Keep Indexes Fresh
Incremental processing re-runs only the work a change affects. See how CocoIndex v1 finds changes, reuses memoized results and deletes stale rows.
How indexing pipelines differ from other data pipelines
How indexing pipelines differ from ETL, streaming and transactional systems: an index is derived data that must stay correct as its source changes.
How CocoIndex handles data and code changes
When a file, a function, a prompt or a table schema changes, CocoIndex re-runs only the work that depends on it. See what each change costs.
Processing Large Files in Indexing Pipelines
A USPTO weekly XML file holds thousands of patents. Size CocoIndex processing components so rows land early and restarts resume, measured on Postgres.
Data Consistency in Indexing Pipelines
How a crash or two racing versions leave stale rows in a RAG index, and how CocoIndex v1 keeps its internal store and your vector index consistent.