Architecture
14 articles
System design and internals of CocoIndex and data indexing engines.
-
CocoIndex V1 is Live!
CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
-
From pickle to type-guided, safer Python serialization
How CocoIndex moved from pickle to type-guided serialization that uses Python type hints to pick the right serializer, no decorators or registration needed.
-
Invisible Daemon: architecture patterns for local dev tools
Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
-
AI-Native Data Pipeline - Why We Made It
Why the next wave of AI needs open-source, scalable, AI-native data infrastructure, and how CocoIndex is building the foundation for intelligent data pipelines.
-
Bring your own data: Index any data with Custom Sources
Read data from any system with CocoIndex: in v1 a custom source is plain async Python plus processing components. No connector API to implement.
-
Thinking in Rust: Ownership, Access, and Memory Safety
A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
-
Control Processing Concurrency in CocoIndex
How CocoIndex's layered concurrency controls optimize data-processing performance, prevent system overload, and keep pipelines stable and efficient at scale.
-
Introducing CocoInsight
Introducing CocoInsight, a data lineage and observability tool that lets you inspect, trace, and debug every step of a CocoIndex pipeline in real time.
-
Story of CocoIndex, at 1k stars 🎉
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
-
Incremental Processing with CocoIndex
Learn how CocoIndex keeps indexes current with source changes through cached processing, dependency tracking, and change detection.
-
How indexing pipelines differ from other data pipelines
What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.
-
System updates and automatic schema inference
How CocoIndex handles system updates in indexing flows: automatic schema inference and managing data + logic evolution without downtime.
-
Processing Large Files in Data Indexing Systems
Handle large files in data indexing: processing granularity, fan-in/fan-out, and memory pressure, walked through a patent XML example in CocoIndex.
-
Data Consistency in Indexing Pipelines
Data consistency in indexing pipelines: concurrent updates, exposure risks, and how CocoIndex's data-driven approach keeps indexes converging.