Incremental updates, change detection, and always-fresh derived data.
Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
Walk through a live CocoIndex pipeline that watches a folder of CSV files and publishes each row as JSON to a Kafka topic incrementally, with no glue code.
CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
Featuring five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.
Auto-generate documentation for every project in your codebase: a CocoIndex pipeline writes a wiki page per repo with an LLM, kept fresh as code changes.
Build a self-updating Neo4j knowledge graph from meeting notes with CocoIndex v1: LLM-extracted decisions, tasks, and owners, with embedding-based dedupe.
Rank trending HackerNews topics in Postgres with an incremental CocoIndex v1 pipeline: scrape threads and comments, extract topics with an LLM.
Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
Incremental ETL from Azure Blob Storage to Snowflake with CocoIndex v1: LLM invoice extraction from PDFs, processing only new or changed files.
Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.
Export CocoIndex data anywhere: implement a TargetHandler with a tracking record and an action sink; the engine handles diffing, syncing, and cleanup.
CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.
CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.
Index Markdown from Amazon S3 into Postgres pgvector with CocoIndex v1: incremental processing re-embeds only the files and chunks that changed.
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.
CocoIndex continuously watches source changes and applies incremental updates to keep derived data in sync, with low latency and no full reindexing.
What incremental processing is, who needs it, and how CocoIndex keeps an index in sync with source changes through caching, lineage tracking, and change data capture.
CocoIndex is now open source: the first engine to combine custom transformation logic with incremental processing built specifically for data indexing.
What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.