The CocoIndex Blog.
Tutorials, deep dives, and notes from the CocoIndex team. Incremental data infrastructure, Rust internals, knowledge graphs, and stories from production.
-
Index Your Codebase for AI Agents with CocoIndex V1
Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
-
CocoIndex Changelog 1.0.1 - 1.0.7
CocoIndex's first post-v1 releases: stable memoization keys, scheduled live refresh, scoped stats, safer SQL connectors, and more integrations.
-
Live CSV → Kafka with CocoIndex's New Kafka Target Connector
Walk through a live CocoIndex pipeline that watches a folder of CSV files and publishes each row as JSON to a Kafka topic incrementally, with no glue code.
-
Turn Podcasts into a Knowledge Graph with LLM and CocoIndex
Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
-
From pickle to type-guided, safer Python serialization
How CocoIndex moved from pickle to type-guided serialization that uses Python type hints to pick the right serializer, no decorators or registration needed.
-
Invisible Daemon: architecture patterns for local dev tools
Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
-
CocoIndex Changelog 0.3.27 - 0.3.34
Featuring five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.
-
CocoIndex joins the GitHub Secure Open Source Fund
CocoIndex joined the GitHub Secure Open Source Fund, hardening the AI data infrastructure developers depend on with threat modeling, CodeQL, and audits.
-
SEC EDGAR financial analytics with Apache Doris
A multi-source pipeline that ingests SEC filings (TXT, JSON, PDF), scrubs PII, extracts topics, and powers hybrid search with CocoIndex + Apache Doris.
-
Build a Self-Updating Wiki for Your Codebases with an LLM
Auto-generate documentation for every project in your codebase: a CocoIndex pipeline writes a wiki page per repo with an LLM, kept fresh as code changes.
-
Slides-to-speech: narrate presentations automatically
Turn slide decks into a live multimodal dataset with CocoIndex: extract notes with Gemini Vision, narrate with Piper TTS, and keep LanceDB in sync.
-
CocoIndex Changelog 0.3.11 - 0.3.26
CocoIndex updates: production-ready resilience, a structured error system, expanded integrations, and always-fresh context for agents.
-
Extract patient intake forms with DSPy and CocoIndex
Extract Pydantic-typed structured data from patient intake forms using DSPy and CocoIndex: OCR vision models with incremental processing.
-
A knowledge graph from meeting notes that auto-updates
Build a self-updating knowledge graph from meeting notes: extract decisions, tasks, owners, and relationships from your documents with CocoIndex and an LLM.
-
Real-time HackerNews trending topics detector with CocoIndex
Build a real-time HackerNews trending topics detector with CocoIndex: a deep dive into Custom Sources and AI-powered topic extraction.
-
CocoIndex Changelog 0.2.21 - 0.3.10
Featuring batching support for CocoIndex functions, execution robustness, schema & type system improvements, custom source support, and more.
-
Extract HackerNews into Postgres with a custom source
Build a custom incremental HackerNews connector with CocoIndex's Custom Source API and export to Postgres for semantic search and analytics.
-
Extracting Intake Forms with BAML and CocoIndex
How to use BAML and CocoIndex to extract structured data from patient intake forms in PDF/Word with LLMs continuously for production.
-
Adaptive Batching - 5x throughput on your data pipelines
CocoIndex now batches GPU and ML workloads automatically: 5x throughput on text embeddings and AI ops, with zero configuration required.
-
AI-Native Data Pipeline - Why We Made It
Why the next wave of AI needs open-source, scalable, AI-native data infrastructure, and how CocoIndex is building the foundation for intelligent data pipelines.
-
Index PDF elements with mixed embedding models
Extract, embed, and store multimodal PDF elements (text with SentenceTransformers, images with CLIP) for unified semantic search with traceable metadata.
-
Bring your own data: Index any data with Custom Sources
CocoIndex now supports custom sources: read data from any system and keep it incrementally fresh as knowledge for AI agents.
-
CocoIndex Changelog 2025-10-19
Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
-
Automated invoice processing with AI and Snowflake
Extract invoice fields from PDFs in Azure Blob Storage into Snowflake with an incremental CocoIndex + GPT-4o pipeline: open-source unstructured ETL.
-
Thinking in Rust: Ownership, Access, and Memory Safety
A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
-
Iterate faster: trace queries back to source data
Define query handlers in CocoIndex and trace search results back to source data in CocoInsight to close the loop on indexing strategy.
-
Incrementally transform Postgres data with AI
Build incrementally updated semantic and structured search over PostgreSQL with CocoIndex, writing pgvector embeddings back to Postgres.
-
Index PDFs, images, and slides with ColPali, no OCR
Build a unified visual document index from multiple file formats (including PDFs, images, and slides) using CocoIndex and ColPali. No OCR needed.
-
CocoIndex Changelog 2025-08-18
CocoIndex updates: production readiness, scalability, and reliability, plus more customization, native integrations, and multi-modal pipeline features.
-
Control Processing Concurrency in CocoIndex
How CocoIndex's layered concurrency controls optimize data-processing performance, prevent system overload, and keep pipelines stable and efficient at scale.
-
Index Images with ColPali: Multi-Modal Context Engineering
CocoIndex now natively integrates ColPali for multi-vector, patch-level image indexing: multi-modal context engineering for visually rich documents and PDFs.
-
Multi-Dimensional Vector Support in CocoIndex
CocoIndex natively handles typed multi-dimensional vectors, from simple arrays to multi-vector embeddings, unlocking multimodal AI pipelines at scale.
-
Custom Targets: export your data anywhere
CocoIndex now supports custom targets. Export indexed data to any destination: a local file, cloud storage, a REST API, or your own bespoke system.
-
Index faces for visual search: your own Google Photos
Build a scalable face detection and recognition pipeline with CocoIndex: embed faces, structure for search, and export to a vector DB.
-
Index academic papers and extract metadata for AI agents
How to index academic research papers by extracting metadata (e.g., title, authors, abstract) for AI agents and AI workflows using LLMs and CocoIndex
-
CocoIndex Changelog 2025-07-07
CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.
-
Introducing CocoInsight
Introducing CocoInsight, a data lineage and observability tool that lets you inspect, trace, and debug every step of a CocoIndex pipeline in real time.
-
Flow-based schema inference for Qdrant
CocoIndex sets up Qdrant collections automatically by inferring the target schema from your indexing flow: no manual config, vector sizes kept in sync.
-
CocoIndex + Kuzu: Real-time knowledge graph with Kuzu
Build a real-time knowledge graph with Kuzu as a native CocoIndex target: incremental updates, high-performance graph queries.
-
CocoIndex Changelog 2025-05-31
CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.
-
Real-time data pipeline with S3, SQS, and CocoIndex
Build a real-time data pipeline on Amazon S3 and SQS with CocoIndex: incremental indexing on object storage that reprocesses only changed files.
-
Image search in natural language with CLIP
Indexing images with CocoIndex and Vision Model in real-time: multi-modal embedding, and build vector index for efficient retrieval.
-
How to build an index with text embeddings
Build a semantic text index with CocoIndex and text embeddings, then query it with natural language: a beginner's guide to embeddings and vector search.
-
Story of CocoIndex, at 1k stars 🎉
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
-
Real-time product recommendations with LLM + graph DB
Build a real-time product recommendation engine with an LLM and a graph database, from the aspect of product category (taxonomy) understanding.
-
CocoIndex Changelog 2025-04-30
CocoIndex updates: knowledge graph support, Qdrant and Supabase targets, KTable and LTable data types, additional LLM providers, and more.
-
Build Real-Time Knowledge Graph For Documents with LLM
CocoIndex now supports knowledge graphs with incremental processing. Building live knowledge for agents is super easy with CocoIndex!
-
CocoIndex Changelog 2025-04-07
CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.
-
Continuous updates: derive data when sources change
CocoIndex continuously watches source changes and applies incremental updates to keep derived data in sync, with low latency and no full reindexing.
-
Incremental Processing with CocoIndex
CocoIndex helps to keep the index up to date with source changes, super efficient and low latency - with the support of incremental processing.
-
Structured Extraction from Patient Intake Form with LLM
Extract structured data from patient intake forms in PDF and Word documents using an LLM and CocoIndex: a practical healthcare document extraction example.
-
Build text embeddings from Google Drive for RAG
Step-by-step tutorial to build text embeddings from Google Drive docs with CocoIndex and store them in Postgres for semantic search and RAG.
-
CocoIndex Changelog 2025-03-20
First release of CocoIndex Changelog: LLM support, codebase indexing, custom functions, and assorted core/performance improvements
-
Build Real-Time Codebase Indexing for AI Code Generation
Indexing codebase for RAG with CocoIndex and Tree-sitter in real-time: chunking, embedding, semantic search, and build vector index for efficient retrieval.
-
On-premise structured extraction with LLM using Ollama
Extract structured data from PDF/Markdown with CocoIndex and Ollama's local LLMs, running on premise without sending data to external APIs.
-
We are officially open sourced! 🎉
CocoIndex is now open source: the first engine to combine custom transformation logic with incremental processing built specifically for data indexing.
-
Customizable Data Indexing Pipelines
What customizable data indexing pipelines are and why custom transformation logic matters, with practical CocoIndex examples.
-
What Makes Indexing Pipelines Different?
What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.
-
System updates and automatic schema inference
How CocoIndex handles system updates in indexing flows: automatic schema inference and managing data + logic evolution without downtime.
-
Processing Large Files in Data Indexing Systems
Handle large files in data indexing: processing granularity, fan-in/fan-out, and memory pressure, walked through a patent XML example in CocoIndex.
-
Data Consistency in Indexing Pipelines
Data consistency in indexing pipelines: concurrent updates, exposure risks, and how CocoIndex's data-driven approach keeps indexes converging.
-
Data Indexing and Common Challenges
Fundamentals of data indexing pipelines for RAG: what makes a good one, common production pitfalls, and how CocoIndex addresses them.
-
CocoIndex - A Data Indexing Platform for AI Applications
CocoIndex is a data indexing platform for AI: ingestion, chunking, embedding, and pipeline management for RAG, semantic search, and knowledge graphs.
-
Welcome to CocoIndex
Welcome to the official CocoIndex blog! We're excited to share our journey in building high-performance indexing infrastructure for AI applications.