Incremental Processing
21 articles
Incremental updates, change detection, and always-fresh derived data.
-
Index Your Codebase for AI Agents with CocoIndex V1
Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
-
Live CSV → Kafka with CocoIndex's New Kafka Target Connector
Walk through a live CocoIndex pipeline that watches a folder of CSV files and publishes each row as JSON to a Kafka topic incrementally, with no glue code.
-
CocoIndex V1 is Live!
CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
-
Turn Podcasts into a Knowledge Graph with LLM and CocoIndex
Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
-
CocoIndex Changelog 0.3.27 - 0.3.34
Featuring five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.
-
Build a Self-Updating Wiki for Your Codebases with an LLM
Auto-generate documentation for every project in your codebase: a CocoIndex pipeline writes a wiki page per repo with an LLM, kept fresh as code changes.
-
A knowledge graph from meeting notes that auto-updates
Build a self-updating Neo4j knowledge graph from meeting notes with CocoIndex v1: LLM-extracted decisions, tasks, and owners, with embedding-based dedupe.
-
Trending HackerNews topics with an LLM and CocoIndex
Rank trending HackerNews topics in Postgres with an incremental CocoIndex v1 pipeline: scrape threads and comments, extract topics with an LLM.
-
CocoIndex Changelog 2025-10-19
Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
-
Automated invoice processing with AI and Snowflake
Incremental ETL from Azure Blob Storage to Snowflake with CocoIndex v1: LLM invoice extraction from PDFs, processing only new or changed files.
-
Turn a Postgres table into a semantic index
Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.
-
Custom Targets: export your data anywhere
Export CocoIndex data anywhere: implement a TargetHandler with a tracking record and an action sink; the engine handles diffing, syncing, and cleanup.
-
CocoIndex Changelog 2025-07-07
CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.
-
CocoIndex Changelog 2025-05-31
CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.
-
Incremental ETL on Amazon S3 with CocoIndex
Index Markdown from Amazon S3 into Postgres pgvector with CocoIndex v1: incremental processing re-embeds only the files and chunks that changed.
-
Story of CocoIndex, at 1k stars 🎉
The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
-
CocoIndex Changelog 2025-04-07
CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.
-
Keep derived data in sync with changing sources
CocoIndex continuously watches source changes and applies incremental updates to keep derived data in sync, with low latency and no full reindexing.
-
Incremental Processing with CocoIndex
Learn how CocoIndex keeps indexes current with source changes through cached processing, dependency tracking, and change detection.
-
We are officially open sourced! 🎉
CocoIndex is now open source: an engine that combines custom transformation logic with incremental processing for data indexing.
-
How indexing pipelines differ from other data pipelines
What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.