The CocoIndex Blog.
Tutorials, deep dives, and notes from the CocoIndex team. Incremental data infrastructure, Rust internals, knowledge graphs, and stories from production.
-
CocoIndex Changelog 1.0.8 - 1.0.16 New
Changelog for CocoIndex 1.0.8-1.0.16: persistent per-component state, LiveMap, rate limiting, batched target writes, BigQuery and Snowflake connectors.
-
Index Your Codebase for AI Agents with CocoIndex V1
Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
-
CocoIndex Changelog 1.0.1 - 1.0.7
CocoIndex's first post-v1 releases: stable memoization keys, scheduled live refresh, scoped stats, safer SQL connectors, and more integrations.
-
Live CSV → Kafka with CocoIndex's New Kafka Target Connector
Walk through a live CocoIndex pipeline that watches a folder of CSV files and publishes each row as JSON to a Kafka topic incrementally, with no glue code.
-
Turn Podcasts into a Knowledge Graph with LLM and CocoIndex
Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
-
From pickle to type-guided, safer Python serialization
How CocoIndex moved from pickle to type-guided serialization that uses Python type hints to pick the right serializer, no decorators or registration needed.
-
Invisible Daemon: architecture patterns for local dev tools
Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
-
CocoIndex Changelog 0.3.27 - 0.3.34
Featuring five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.
-
CocoIndex joins the GitHub Secure Open Source Fund
CocoIndex joined the GitHub Secure Open Source Fund, hardening the AI data infrastructure developers depend on with threat modeling, CodeQL, and audits.
-
SEC EDGAR financial analytics with Apache Doris
A multi-format CocoIndex v1 pipeline that ingests SEC filings (TXT, JSON), scrubs PII, extracts topics, and powers hybrid search with Apache Doris.
-
Build a Self-Updating Wiki for Your Codebases with an LLM
Auto-generate documentation for every project in your codebase: a CocoIndex pipeline writes a wiki page per repo with an LLM, kept fresh as code changes.
-
Slides-to-speech: Bring your slide decks to life with narrated audio
Build a CocoIndex v1 pipeline that turns PDF slides into vision-generated speaker notes, local Pocket TTS narration, and searchable LanceDB records.
-
CocoIndex Changelog 0.3.11 - 0.3.26
CocoIndex updates: production-ready resilience, a structured error system, expanded integrations, and always-fresh context for agents.
-
Extract patient intake forms with DSPy and CocoIndex
Extract Pydantic-typed structured data from patient intake forms using DSPy and CocoIndex: OCR vision models with incremental processing.
-
A knowledge graph from meeting notes that auto-updates
Build a self-updating Neo4j knowledge graph from meeting notes with CocoIndex v1: LLM-extracted decisions, tasks, and owners, with embedding-based dedupe.
-
What HackerNews is talking about: trending topics with an LLM
Rank trending HackerNews topics in Postgres with an incremental CocoIndex v1 pipeline: scrape threads and comments, extract topics with an LLM.
-
CocoIndex Changelog 0.2.21 - 0.3.10
Featuring batching support for CocoIndex functions, execution robustness, schema & type system improvements, custom source support, and more.
-
Extract HackerNews into Postgres with a custom source
Build a custom CocoIndex source for the HackerNews API: fetch threads and comments in async Python, extract topics with an LLM, keep Postgres in sync.
-
Extracting Intake Forms with BAML and CocoIndex
How to use BAML and CocoIndex to extract structured data from patient intake forms in PDF/Word with LLMs continuously for production.
-
Adaptive Batching - 5x throughput on your data pipelines
CocoIndex now batches GPU and ML workloads automatically: 5x throughput on text embeddings and AI ops, with zero configuration required.
-
AI-Native Data Pipeline - Why We Made It
Why the next wave of AI needs open-source, scalable, AI-native data infrastructure, and how CocoIndex is building the foundation for intelligent data pipelines.
-
Index PDF elements with mixed embedding models
Extract, embed, and store multimodal PDF elements (text with SentenceTransformers, images with CLIP) for unified semantic search with traceable metadata.
-
Bring your own data: Index any data with Custom Sources
Read data from any system with CocoIndex: in v1 a custom source is plain async Python plus processing components. No connector API to implement.
-
CocoIndex Changelog 2025-10-19
Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
-
Automated invoice processing with AI and Snowflake
Incremental ETL from Azure Blob Storage to Snowflake with CocoIndex v1: LLM invoice extraction from PDFs, processing only new or changed files.
-
Thinking in Rust: Ownership, Access, and Memory Safety
A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
-
Trace search results back to source data
Define query handlers in CocoIndex and trace search results back to source data in CocoInsight to close the loop on indexing strategy.
-
Turn a Postgres table into a semantic index
Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.
-
Index PDFs, images, and slides with ColPali, no OCR
Build a unified visual document index from multiple file formats (including PDFs, images, and slides) using CocoIndex and ColPali. No OCR needed.
-
CocoIndex Changelog 2025-08-18
CocoIndex updates: production readiness, scalability, and reliability, plus more customization, native integrations, and multi-modal pipeline features.
-
Control Processing Concurrency in CocoIndex
How CocoIndex's layered concurrency controls optimize data-processing performance, prevent system overload, and keep pipelines stable and efficient at scale.
-
Index Images with ColPali: multi-vector visual search
Index images with ColPali multi-vector patch embeddings and a Qdrant MaxSim collection using CocoIndex v1: incremental, live, in plain async Python.
-
Multi-Dimensional Vector Support in CocoIndex
CocoIndex natively handles typed multi-dimensional vectors, from simple arrays to multi-vector embeddings, unlocking multimodal AI pipelines at scale.
-
Custom Targets: export your data anywhere
Export CocoIndex data anywhere: implement a TargetHandler with a tracking record and an action sink; the engine handles diffing, syncing, and cleanup.
-
Index faces for visual search: your own Google Photos
Build a scalable face detection and recognition pipeline with CocoIndex: embed faces, structure for search, and export to a vector DB.
-
Index academic papers and extract metadata for AI agents
Index academic papers with CocoIndex: read the first page of each PDF, extract title, authors, and abstract with an LLM, and embed them for semantic search.
-
CocoIndex Changelog 2025-07-07
CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.
-
Introducing CocoInsight
Introducing CocoInsight, a data lineage and observability tool that lets you inspect, trace, and debug every step of a CocoIndex pipeline in real time.
-
Flow-based schema inference for Qdrant
CocoIndex sets up Qdrant collections automatically by inferring the target schema from your indexing flow: no manual config, vector sizes kept in sync.
-
CocoIndex + Kuzu: Real-time knowledge graph with Kuzu
Build a real-time knowledge graph with Kuzu as a native CocoIndex target: incremental updates, high-performance graph queries.
-
CocoIndex Changelog 2025-05-31
CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.
-
Incremental ETL on Amazon S3 with CocoIndex
Index Markdown from Amazon S3 into Postgres pgvector with CocoIndex v1: incremental processing re-embeds only the files and chunks that changed.
-
Image search in natural language with CLIP
Search a folder of photos by meaning with CocoIndex v1: CLIP embeds images and text into one vector space, the index runs live inside a FastAPI app, and vectors live in Qdrant.
-
How to build an index with text embeddings
Text embeddings 101: what they are, why you chunk and embed, and how to build a semantic search index with CocoIndex v1 and Postgres pgvector.
-
Story of CocoIndex, at 1k stars 🎉
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
-
Build a product recommendation engine with LLM + Neo4j
An LLM extracts each product's taxonomy and complements; CocoIndex turns the labels into a Neo4j knowledge graph that answers 'bought this, also need…'.
-
CocoIndex Changelog 2025-04-30
CocoIndex updates: knowledge graph support, Qdrant and Supabase targets, KTable and LTable data types, additional LLM providers, and more.
-
Build Real-Time Knowledge Graph For Documents with LLM
Turn a folder of Markdown docs into a Neo4j knowledge graph: an LLM extracts subject-predicate-object triples, and CocoIndex keeps the graph in sync as the docs change.
-
CocoIndex Changelog 2025-04-07
CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.
-
Keep derived data in sync with changing sources
CocoIndex continuously watches source changes and applies incremental updates to keep derived data in sync, with low latency and no full reindexing.
-
Incremental Processing with CocoIndex
What incremental processing is, who needs it, and how CocoIndex keeps an index in sync with source changes through caching, lineage tracking, and change data capture.
-
Structured Extraction from Patient Intake Form with LLM
Extract typed Patient records from PDF and DOCX intake forms with an LLM and CocoIndex v1: the nested schema is the whole prompt; results land in Postgres.
-
Search your Google Drive by meaning
Index the documents in a shared Google Drive folder as text embeddings in Postgres with CocoIndex v1, then search them by meaning instead of by filename.
-
CocoIndex Changelog 2025-03-20
First release of CocoIndex Changelog: LLM support, codebase indexing, custom functions, and assorted core/performance improvements
-
Build Real-Time Codebase Indexing for AI Code Generation
Indexing codebase for RAG with CocoIndex and Tree-sitter in real-time: chunking, embedding, semantic search, and build vector index for efficient retrieval.
-
On-premise structured extraction from PDFs with Ollama
Extract structured data from PDF manuals locally with Ollama and CocoIndex: docling converts PDFs to Markdown, a local LLM fills typed Postgres rows.
-
We are officially open sourced! 🎉
CocoIndex is now open source: the first engine to combine custom transformation logic with incremental processing built specifically for data indexing.
-
Customizable Data Indexing Pipelines
What customizable data indexing pipelines are and why custom transformation logic matters, with practical CocoIndex examples.
-
How indexing pipelines differ from other data pipelines
What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.
-
System updates and automatic schema inference
How CocoIndex handles system updates in indexing flows: automatic schema inference and managing data + logic evolution without downtime.
-
Processing Large Files in Data Indexing Systems
Handle large files in data indexing: processing granularity, fan-in/fan-out, and memory pressure, walked through a patent XML example in CocoIndex.
-
Data Consistency in Indexing Pipelines
Data consistency in indexing pipelines: concurrent updates, exposure risks, and how CocoIndex's data-driven approach keeps indexes converging.
-
Data Indexing and Common Challenges
Fundamentals of data indexing pipelines for RAG: what makes a good one, common production pitfalls, and how CocoIndex addresses them.
-
CocoIndex - A Data Indexing Platform for AI Applications
CocoIndex is a data indexing platform for AI: ingestion, chunking, embedding, and pipeline management for RAG, semantic search, and knowledge graphs.
-
Welcome to CocoIndex
Welcome to the official CocoIndex blog! We're excited to share our journey in building high-performance indexing infrastructure for AI applications.