CocoIndex Changelog 1.0.8 - 1.0.16

This is the second changelog since the v1.0 launch, covering nine releases (1.0.8–1.0.16). The focus shifts from initial production hardening into new engine primitives for stateful and streaming pipelines, a wholly new structural code-matching subsystem, a major expansion of cloud and enterprise connectors, and continued depth on agent ergonomics.

CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.

Why this release cycle matters

For developers building retrieval, extraction, and knowledge-graph pipelines, the most significant changes are:

• Persistent per-component state. coco.use_state() lets pipeline components carry state that survives across runs, opening the door to accumulators, counters, and stateful deduplication without an external store.
• In-memory intermediate collections. LiveMap holds a mutable keyed collection in memory during a live pipeline run, enabling fan-out aggregation patterns that are impossible in pure declarative pipelines.
• Structural code matching. The new code_match crate and CodePattern Python API let pipelines search codebases by syntax structure rather than text strings, matching function calls, containment relationships, and token patterns with O(N·k) performance.
• Token-bucket rate limiting. RateLimiter is now a first-class engine primitive, replacing ad-hoc sleep loops and per-integration retry logic with a consistent, cooperative throttle shared across pipeline components.
• Enterprise cloud connectors. BigQuery, Snowflake, and Azure Blob Storage land alongside sparse-vector Qdrant support and a new Valkey vector store connector, completing coverage of the major cloud data platforms.

Engine

coco.use_state(): persistent per-component state

coco.use_state() gives any pipeline component a persistent, typed state slot that survives across update cycles. The state is stored in the LMDB state store alongside other engine metadata, keyed by a StableKey (not just a plain string, as of #2085). A type_hint parameter added in v1.0.15 enables typed deserialization so components can round-trip structured state objects cleanly.

Why it matters: the incremental engine already knows what changed between runs. use_state lets components know what they have already done. This enables accumulators (running totals, seen-entity sets), watermark trackers, and stateful deduplication, patterns that previously required an external Redis or Postgres sidecar. The net effect is that a wider class of stateful streaming logic can now live inside the pipeline declaration, subject to the same incremental reconciliation semantics as the rest of CocoIndex.

LiveMap: in-memory intermediate collections

LiveMap is a new in-memory keyed collection that lives inside a live pipeline run. Where coco.stats_group provides observability over a data slice, LiveMap provides computation over a data slice: a mutable intermediate state that components can read and write as events flow through the pipeline.

LiveComponent classes can now be used inside mount_each(), enabling dynamic live component trees where the set of active live components changes based on source data.

Token-bucket RateLimiter

RateLimiter is now a first-class engine primitive implementing a token-bucket algorithm. It provides a cooperative throttle shared across pipeline components: any component that acquires a token from the same RateLimiter instance participates in the same budget. This replaces per-integration sleep loops and ad-hoc retry logic.

Combined with the existing retry-on-429 (HTTP rate limit) behavior added in earlier cycles, RateLimiter gives pipelines deterministic, budget-aware flow control rather than reactive backpressure alone.

Preview mode for update actions

Update actions can now be previewed before they are applied. A pipeline run can describe the adds, reprocesses, and deletes it would perform, without writing to any target, enabling dry-run validation and change auditing before committing to a production store.

Batched target writes across components

Row-level writes from sibling components are now coalesced into a single batched apply. The common shape is mount_each(..., table_target): many child components each declare one row into the same target. Previously every child submitted its own leaf action directly to the sink after precommit, so the connector saw a stream of single-row calls even though its API already accepts a batch. In practice a 50-row ingest into LanceDB produced roughly 50 merge_insert commits, and 50 table versions, instead of one (#2219).

An app-scoped batcher, keyed by the target sink, now merges concurrent sibling commits that hit the same sink before the connector’s apply call (#2221). The design reuses CocoIndex’s existing Batcher primitive rather than a bespoke queue, so leaf-target batching shares the same path as the engine’s other batched work and adds no debounce latency to a lone write. Sink lifetime drives cleanup: a keeper owns the batcher one-to-one and drops its registry entry the moment the sink goes away, so a pipeline with an unbounded number of sink functions cannot leak batcher entries. Batching also preserves order, each output mapping back to its input, which let container targets route through the same batched path instead of a separate one.

Why it matters: for connectors with real per-operation latency, such as LanceDB and cloud warehouses, write cost is dominated by round-trips and commits, not row count. Collapsing N single-row applies into one batched apply cuts that overhead directly. For a store like LanceDB it also collapses N table versions into one, which keeps version history and compaction manageable under heavy fan-out.

Engine performance improvements

• Prefetch fn-memos and user-states in one read transaction: the engine now batches all LMDB reads for function memos and user states into a single read transaction per cycle, replacing N individual reads (#2076).
• Minimize serialization for UserStateCache: reduces round-trip cost for state reads during hot-path execution (#2127).
• LMDB map auto-resize: the state store now automatically resizes the LMDB map when it fills, eliminating the manual LMDB_MAP_SIZE tuning that previously required operator intervention (#2231).
• LMDB map size rounded to page size: map sizes are now aligned to the OS page boundary, preventing subtle allocation failures on some Linux kernels (#2109).

Other engine fixes

• coco.fn callable outside a component context: coco.fn-decorated functions can now be called directly in scripts and tests without being inside a mounted pipeline component (#2101).
• !-prefixed negation in excluded_patterns: source file filters now support !pattern to explicitly un-exclude a path that matched an earlier pattern, following the .gitignore-style rule ordering (#1901).
• Retry transient LiteLLM embedding errors: transient failures (network blips, upstream 5xx) during embedding calls are retried automatically (#2090).
• Reject rows with missing or null primary keys: the engine now fails fast rather than silently writing corrupt rows (#2239).

Code AST matching (new subsystem)

The most architecturally novel work in this cycle is the code_match crate: a structural code-search engine built on tree-sitter that allows CocoIndex pipelines to match source code by syntax structure, not just text content.

What it is

CodePattern compiles a structural pattern once and reuses it across an entire codebase. A pattern can match:

• Literal tokens: specific keywords, operators, identifiers
• Anonymous regex matchers: any token matching a regular expression
• Named regex matchers: \(NAME:/re/*\) binds the match to a named capture
• Quantifiers: * (zero-or-more), + (one-or-more)
• Containment: \{{ INNER \}} matches any node that contains INNER as a descendant
• Whole-node boundary: \{ P \} matches a node whose complete token sequence matches P (“is”)

Patterns are matched via CodePattern.match_file, returning all non-overlapping matches per node with source positions.

Why it matters for codebase indexing

Text-based search (grep, BM25) finds patterns anywhere in a file, including comments, strings, and variable names that happen to spell the target. Structural search respects the parse tree: a pattern matching a function call only matches actual function calls in the AST, not the word “function” in a comment. For AI pipelines extracting API usage, dependency graphs, or security-relevant patterns (SQL queries, exec calls, HTTP requests), this distinction is the difference between noisy results and precise extraction.

The prefilter system (index_terms, matches_prefiltered) extracts required-content tokens from a pattern and can drive an inverted index, so structurally-matched search avoids scanning the full tree on documents that can’t possibly match.

Performance

The implementation received substantial algorithmic attention in v1.0.13:

• Child-run scan pruned by trailing literal token
• Child-run tolerance folded into one dynamic-programming pass per candidate: O(N·k) rather than O(N²)
• GIL released across all Python-facing entry points (code-matcher is a Rust hot path)
• Byte→position index memoized, eliminating per-query full scans

Shared CodeSource (v1.0.16)

v1.0.16 introduced CodeSource, a shared parse context that performs a single tree-sitter parse and reuses it across all consumers: the splitter, the matcher, and any custom analysis functions. Previously each consumer triggered its own parse. On large codebases where both chunking and structural matching run over the same files, this is a significant throughput improvement. Deprecated compat shims keep existing CodeAst surface usage working through the transition.

Connectors

Target: Snowflake (v1.0.15)

A native Snowflake target connector lands, bringing CocoIndex’s declare-target-state model to one of the most widely deployed cloud data warehouses. Snowflake targets participate in the same incremental reconciliation as all other CocoIndex targets: rows for deleted sources are removed automatically, and only changed rows are upserted.

Target: BigQuery (v1.0.16)

A Google BigQuery target connector was added alongside the Snowflake connector in the same cycle. For organizations already running Google Cloud infrastructure, this completes cloud data warehouse coverage across the major platforms.

Source: Azure Blob Storage (v1.0.16)

An Azure Blob Storage source connector mirrors the OCI and S3 source APIs. For teams on Azure infrastructure, this enables the same live-mode, incremental bucket-watching behavior available to AWS and OCI users.

Target: Valkey vector store (v1.0.12)

Valkey, the open-source Redis fork under Linux Foundation governance, gains a target connector with vector store support. For organizations that have standardized on Valkey over Redis, this provides a fully supported vector index target without requiring a separate vector database.

Qdrant: native sparse vector support (v1.0.16)

The Qdrant connector was upgraded with native sparse vector support and eager point-ID validation. Sparse vectors are essential for hybrid search pipelines that combine dense (semantic) and sparse (BM25/TF-IDF) representations, making CocoIndex’s Qdrant integration first-class for hybrid retrieval.

LanceDB: vector and FTS index support (v1.0.11)

LanceDB gained explicit vector and full-text search (FTS) index management. The drop_index operation was added alongside a dependency bump to LanceDB 0.33.0 and PyArrow 23.0.0. Nullable schema-only row rewrites (a correctness bug that appeared on partial schema changes) were fixed in both the Python and Rust layers. In v1.0.15, LanceDB table maintenance was tied to LanceDB’s own stats API, making compaction scheduling driven by actual table metrics rather than a fixed interval.

zvec: FTS field support (v1.0.14)

The zvec target connector gained full-text search field support, allowing pipelines to write both vector and FTS fields to zvec in a single target export.

OCI: skip startup scan on unchanged logic (v1.0.11)

The OCI Object Storage source now detects when pipeline logic is unchanged since the last run and skips the expensive full-bucket startup scan, going directly to live-stream event processing. For large buckets, this can reduce cold-start time from minutes to seconds.

Local file system: periodic rescan in live mode (v1.0.15)

The local file system source in live mode now periodically rescans and recreates the watcher if it becomes stale. OS file-system watchers (inotify, kqueue) have known limitations under high churn, network mounts, and certain editors; periodic rescan provides a safety net that catches any events the watcher missed.

New connector summary

Agent & developer experience

cocoindex show: full-fledged inspection tool (v1.0.12)

cocoindex show was upgraded from a basic state dump into a full inspection tool for pipeline debugging. It provides structured, human-readable output of pipeline state, component configuration, and target schema: the production debugging workflow that previously required querying the LMDB or Postgres state store directly.

Python error messages

Runtime errors crossing the Rust-to-Python boundary now use Display (human-readable) rather than Debug (Rust internals) formatting, making error messages that surface in Python stack traces significantly more actionable.

Multiple GPU and fractional GPU allocation (v1.0.15)

GPUPool was added to support multiple GPUs and fractional GPU allocations within a pipeline. For embedding-heavy pipelines running on multi-GPU infrastructure, this allows work to be distributed across available GPU resources without hand-partitioning the pipeline.

Example walkthroughs

v1.0.13 shipped a major documentation expansion: 17 new example walkthroughs (10 full + 7 variants) covering the breadth of CocoIndex use cases:

• Meeting Notes → Knowledge Graph, with Neo4j and FalkorDB variants
• CSV → Kafka streaming pipeline
• Semantic Search over PDFs with mixed embedding models
• Search Images by Text, a multimodal CLIP walkthrough
• Face Recognition, ported from v0 to v1
• Docs to Knowledge graph construction

Each walkthrough was structured for agent followability, with consistent heading patterns, copyable commands, and explicit prerequisites, so AI coding agents can execute them as step-by-step procedures.

Bug fixes and correctness

Summary

This cycle’s theme is depth and breadth: depth in the engine (stateful components, structural code search, GPU pools, performance batching) and breadth in connectors (completing cloud warehouse coverage with Snowflake, BigQuery, and Azure Blob, plus Valkey and enhanced Qdrant). The code_match subsystem is the most significant new capability. Structural code search is a primitive that unlocks a class of precise, noise-free codebase intelligence pipelines that text-based search cannot provide.

For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.

Thanks to the community

Thanks to everyone who shipped this cycle, core maintainers and community alike, including twelve first-time contributors.

@georgeh0

Thanks @georgeh0 for the bulk of this cycle’s engine work: the code_match structural-search crate and its shared CodeSource parse, LiveMap, the token-bucket RateLimiter, and LiveComponent support inside mount_each.

@badmonster0

Thanks @badmonster0 for the documentation updates.

@hakuuww

Thanks @hakuuww for building coco.use_state(), persistent per-component state, then following it with engine-side performance work: prefetching fn-memos and user states in one read transaction and minimizing UserStateCache serialization.

@sdhilip200

Thanks @sdhilip200 for a prolific connector cycle: the Snowflake and BigQuery targets, the Azure Blob source, and Mermaid diagram rendering in the docs.

@Sujit-1509

Thanks @Sujit-1509 for LanceDB vector and FTS indexing, plus correctness fixes: preserving data and NOT NULL constraints during Postgres column-type evolution and reconciling orphaned child-existence rows.

@prrao87

Thanks @prrao87 for deep LanceDB work: tying table maintenance to LanceDB’s own stats, implementing drop_index, and fixing nullable schema-only row rewrites, plus batching leaf target actions.

@lichuang

Thanks @lichuang for GPUPool, supporting multiple and fractional GPU allocations, and for adding type_hint to use_state for typed state deserialization.

@Haleshot

Thanks @Haleshot for the initial zvec target integration and adding FTS field support to it.

@nuthalapativarun

Thanks @nuthalapativarun for switching runtime errors crossing into Python to Display formatting, adding Qdrant and local file system connector unit tests, and fixing broken documentation links.

@daric93

Thanks @daric93 for the Valkey vector store target connector, a fully supported vector index target for teams standardized on Valkey.

@prithvi-moonshot

Thanks @prithvi-moonshot for upgrading cocoindex show into a full-fledged inspection tool for debugging.

@MrAnayDongre

Thanks @MrAnayDongre for preview mode for update actions, letting a run describe the adds, reprocesses, and deletes it would perform before touching a target.

@pyjuan91

Thanks @pyjuan91 for making the OCI source skip its full startup scan when pipeline logic is unchanged, cutting cold-start time on large buckets.

@tomz-alt

Thanks @tomz-alt for native sparse vector support and eager point-ID validation in the Qdrant connector, making it first-class for hybrid retrieval.

@jordigilh

Thanks @jordigilh for adding periodic rescan and watcher recreation to the local file system source in live mode, a safety net for events an OS watcher misses.

@MaxRong

Thanks @MaxRong for making the engine reject rows with missing or null primary keys instead of silently writing corrupt rows.

@slliland

Thanks @slliland for making the state store automatically resize the LMDB map when it fills, removing the manual LMDB_MAP_SIZE tuning operators used to need.

@SuperMarioYL

Thanks @SuperMarioYL for !-prefixed negation in excluded_patterns, so a path matched by an earlier rule can be explicitly un-excluded.

@davidmyriel

Thanks @davidmyriel for documenting Tigris alongside MinIO as a supported S3-compatible service.

@dashitongzhi

Thanks @dashitongzhi for improving the docs CI with proper triggers and build verification.

CocoIndex

Fresh context for long-horizon agents.

Learn the concept → View on GitHub

About the author.

George He Cofounder & CTO of CocoIndex · @georgehe0

Maintainer of CocoIndex, Ex-Google Infra Lead. Writes about incremental data infrastructure, Rust internals, and the engineering decisions behind the engine.