CocoIndex Changelog 1.0.8 - 1.0.16

This is the second changelog since the v1.0 launch, covering nine releases (1.0.8–1.0.16). The focus shifts from initial hardening into new engine primitives for stateful, production-grade pipelines, a major expansion of cloud and enterprise connectors, and continued depth on agent ergonomics.

CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.

Why this release cycle matters

For developers building retrieval, extraction, and knowledge-graph pipelines, the most significant changes are:

• Persistent per-component state. coco.use_state() lets pipeline components carry state that survives across runs, opening the door to accumulators, counters, and stateful deduplication without an external store.
• In-memory intermediate collections. LiveMap is a live, in-memory keyed collection that bridges a producing and a consuming stage of the same run: one side declares entries, the other consumes them through mount_each, which grows and shrinks one live component per entry as the data changes — a producer/consumer hand-off that pure declarative pipelines can’t express.
• Batched target writes. Row-level writes to the same target from different components can now be coalesced into one apply, collapsing N write transactions into one — a direct win for connectors like LanceDB and cloud warehouses where per-transaction cost dominates.
• Token-bucket rate limiting. RateLimiter is now a first-class engine primitive: a consistent, cooperative throttle shared across pipeline components, replacing hand-rolled sleep pacing at each call site with one fair, budgeted rate.
• Enterprise cloud connectors. BigQuery, Snowflake, and Azure Blob Storage land alongside sparse-vector Qdrant support and a new Valkey vector store connector, completing coverage of the major cloud data platforms.

Engine

coco.use_state(): persistent per-component state

coco.use_state() gives any pipeline component a persistent, typed state slot that survives across update cycles. The state is stored in the LMDB state store alongside other engine metadata, keyed by a StableKey (not just a plain string, as of #2085). A type_hint parameter added in v1.0.15 enables typed deserialization so components can round-trip structured state objects cleanly.

Why it matters: the incremental engine already knows what changed between runs. use_state lets components know what they have already done. This enables accumulators (running totals, seen-entity sets), watermark trackers, and stateful deduplication, patterns that previously required an external Redis or Postgres sidecar. The net effect is that a wider class of stateful logic can now live inside the pipeline declaration, subject to the same incremental reconciliation semantics as the rest of CocoIndex.

LiveMap: in-memory intermediate collections

LiveMap is a new in-memory keyed collection that bridges a producing and a consuming half of the same live run. Producing components declare entries into it as target states (LiveMap.declare_entry), and the consuming side reads that same in-memory dict as a live source through mount_each. Where coco.stats_group provides observability over a data slice, LiveMap provides computation over one: a mutable intermediate collection the pipeline both writes and reads, decoupled through the shared map.

Because LiveComponent classes can now be used inside mount_each(), the consuming side is a dynamic live component tree — one live component per entry, mounted and dropped as producers add and stop declaring keys.

Token-bucket RateLimiter

RateLimiter is now a first-class engine primitive implementing a token-bucket algorithm. It provides a cooperative throttle shared across pipeline components: any component that acquires a token from the same RateLimiter instance participates in the same budget, and concurrent callers are served FIFO so none starve. This replaces hand-rolled sleep pacing at each call site with one budgeted, coordinated rate.

Combined with the existing retry-on-429 (HTTP rate limit) behavior added in earlier cycles, RateLimiter gives pipelines deterministic, budget-aware flow control rather than reactive backpressure alone.

Preview mode for update actions

Update actions can now be previewed before they are applied. A pipeline run can describe the adds, reprocesses, and deletes it would perform, without writing to any target, enabling dry-run validation and change auditing before committing to a production store.

Batched target writes across components

Row-level writes to the same target from different components can now be coalesced into a single batched apply. The common shape is mount_each(..., table_target), where many child components each declare one row into the same target — but the coalescing is keyed by the target sink, not by any sibling relationship, so any components whose writes land on the same sink concurrently are batched together. Previously each component submitted its leaf actions directly to the sink after precommit, so the connector saw a stream of single-row calls even though its API already accepts a batch. In practice a 50-row ingest into LanceDB produced roughly 50 merge_insert commits — and 50 table versions — instead of one (#2219).

An app-scoped batcher, keyed by the target sink, merges concurrent commits that hit the same sink before the connector’s apply call (#2221). It reuses CocoIndex’s existing Batcher primitive rather than a bespoke queue, so leaf-target batching shares the same path as the engine’s other batched work and adds no debounce latency to a lone write. Batching preserves order, each output mapping back to its input; container targets, which return per-action child handlers, stay on the direct apply path, while the batched path handles leaf writes. Sink lifetime drives cleanup: a keeper owns the batcher one-to-one and drops its registry entry the moment the sink goes away, so a pipeline with an unbounded number of sink functions cannot leak batcher entries.

Why it matters: for connectors with real per-operation cost, such as LanceDB and cloud warehouses, write cost is dominated by the number of write transactions — round-trips and commits — not by row count. Coalescing N single-row applies into one batched apply collapses N write transactions into one, cutting that overhead directly. For a store like LanceDB it also collapses N table versions into one, which keeps version history and compaction manageable under heavy fan-out.

Engine performance improvements

• Prefetch fn-memos and user-states in one read transaction: the engine now batches all LMDB reads for function memos and user states into a single read transaction per cycle, replacing N individual reads (#2076).
• Minimize serialization for UserStateCache: reduces round-trip cost for state reads during hot-path execution (#2127).
• LMDB map auto-resize: the state store now automatically resizes the LMDB map when it fills, eliminating the manual LMDB_MAP_SIZE tuning that previously required operator intervention (#2231).
• LMDB map size rounded to page size: map sizes are now aligned to the OS page boundary, preventing subtle allocation failures on some Linux kernels (#2109).

Other engine fixes

• coco.fn callable outside a component context: coco.fn-decorated functions can now be called directly in scripts and tests without being inside a mounted pipeline component (#2101).
• !-prefixed negation in excluded_patterns: source file filters now support !pattern to explicitly un-exclude a path that matched an earlier pattern, following the .gitignore-style rule ordering (#1901).
• Retry transient LiteLLM embedding errors: transient failures (network blips, upstream 5xx) during embedding calls are retried automatically (#2090).
• Reject rows with missing or null primary keys: the engine now fails fast rather than silently writing corrupt rows (#2239).

Connectors

Target: Snowflake (v1.0.15)

A native Snowflake target connector lands, bringing CocoIndex’s declare-target-state model to one of the most widely deployed cloud data warehouses. Snowflake targets participate in the same incremental reconciliation as all other CocoIndex targets: rows for deleted sources are removed automatically, and only changed rows are upserted.

Target: BigQuery (v1.0.16)

A Google BigQuery target connector was added alongside the Snowflake connector in the same cycle. For organizations already running Google Cloud infrastructure, this completes cloud data warehouse coverage across the major platforms.

Source: Azure Blob Storage (v1.0.16)

An Azure Blob Storage source connector mirrors the OCI and S3 source APIs. For teams on Azure infrastructure, this enables the same live-mode, incremental bucket-watching behavior available to AWS and OCI users.

Target: Valkey vector store (v1.0.12)

Valkey, the open-source Redis fork under Linux Foundation governance, gains a target connector with vector store support. For organizations that have standardized on Valkey over Redis, this provides a fully supported vector index target without requiring a separate vector database.

Qdrant: native sparse vector support (v1.0.16)

The Qdrant connector was upgraded with native sparse vector support and eager point-ID validation. Sparse vectors are essential for hybrid search pipelines that combine dense (semantic) and sparse (BM25/TF-IDF) representations, making CocoIndex’s Qdrant integration first-class for hybrid retrieval.

LanceDB: vector and FTS index support (v1.0.11)

LanceDB gained explicit vector and full-text search (FTS) index management. The drop_index operation was added alongside a dependency bump to LanceDB 0.33.0 and PyArrow 23.0.0. Nullable schema-only row rewrites (a correctness bug that appeared on partial schema changes) were fixed in both the Python and Rust layers. In v1.0.15, LanceDB table maintenance was tied to LanceDB’s own stats API, making compaction scheduling driven by actual table metrics rather than a fixed interval.

zvec: FTS field support (v1.0.14)

The zvec target connector gained full-text search field support, allowing pipelines to write both vector and FTS fields to zvec in a single target export.

OCI: skip startup scan on unchanged logic (v1.0.11)

The OCI Object Storage source now detects when pipeline logic is unchanged since the last run and skips the expensive full-bucket startup scan, going directly to live-stream event processing. For large buckets, this can reduce cold-start time from minutes to seconds.

Local file system: periodic rescan in live mode (v1.0.15)

The local file system source in live mode now periodically rescans and recreates the watcher if it becomes stale. OS file-system watchers (inotify, kqueue) have known limitations under high churn, network mounts, and certain editors; periodic rescan provides a safety net that catches any events the watcher missed.

New connector summary

Agent & developer experience

cocoindex show: full-fledged inspection tool (v1.0.12)

cocoindex show was upgraded from a basic state dump into a full inspection tool for pipeline debugging. It provides structured, human-readable output of pipeline state, component configuration, and target schema: the production debugging workflow that previously required querying the LMDB or Postgres state store directly.

Python error messages

Runtime errors crossing the Rust-to-Python boundary now use Display (human-readable) rather than Debug (Rust internals) formatting, making error messages that surface in Python stack traces significantly more actionable.

Multiple GPU and fractional GPU allocation (v1.0.15)

GPUPool was added to support multiple GPUs and fractional GPU allocations within a pipeline. For embedding-heavy pipelines running on multi-GPU infrastructure, this allows work to be distributed across available GPU resources without hand-partitioning the pipeline.

Example walkthroughs

v1.0.13 shipped a major documentation expansion: 17 new example walkthroughs (10 full + 7 variants) covering the breadth of CocoIndex use cases:

• Meeting Notes → Knowledge Graph, with Neo4j and FalkorDB variants
• CSV → Kafka streaming pipeline
• Semantic Search over PDFs with mixed embedding models
• Search Images by Text, a multimodal CLIP walkthrough
• Face Recognition, ported from v0 to v1
• Docs to Knowledge graph construction

Each walkthrough was structured for agent followability, with consistent heading patterns, copyable commands, and explicit prerequisites, so AI coding agents can execute them as step-by-step procedures.

Bug fixes and correctness

Summary

This cycle’s theme is depth and breadth: depth in the engine (stateful components, in-memory intermediate collections, GPU pools, performance batching) and breadth in connectors (completing cloud warehouse coverage with Snowflake, BigQuery, and Azure Blob, plus Valkey and enhanced Qdrant). The through-line is the incremental engine growing new primitives — persistent state, live intermediate collections, cooperative rate limiting, batched writes — that bring a wider class of stateful, production-grade pipelines inside a single declarative CocoIndex app.

For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.

Thanks to the community

Thanks to everyone who shipped this cycle, core maintainers and community alike, including twelve first-time contributors.

@georgeh0

Thanks @georgeh0 for the bulk of this cycle’s engine work: LiveMap, the token-bucket RateLimiter, LiveComponent support inside mount_each, and the shared-parse code infrastructure behind cocoindex-code.

@badmonster0

Thanks @badmonster0 for the documentation updates.

@hakuuww

Thanks @hakuuww for building coco.use_state(), persistent per-component state, then following it with engine-side performance work: prefetching fn-memos and user states in one read transaction and minimizing UserStateCache serialization.

@sdhilip200

Thanks @sdhilip200 for a prolific connector cycle: the Snowflake and BigQuery targets, the Azure Blob source, and Mermaid diagram rendering in the docs.

@Sujit-1509

Thanks @Sujit-1509 for LanceDB vector and FTS indexing, plus correctness fixes: preserving data and NOT NULL constraints during Postgres column-type evolution and reconciling orphaned child-existence rows.

@prrao87

Thanks @prrao87 for deep LanceDB work: tying table maintenance to LanceDB’s own stats, implementing drop_index, and fixing nullable schema-only row rewrites, plus batching leaf target actions.

@lichuang

Thanks @lichuang for GPUPool, supporting multiple and fractional GPU allocations, and for adding type_hint to use_state for typed state deserialization.

@Haleshot

Thanks @Haleshot for the initial zvec target integration and adding FTS field support to it.

@nuthalapativarun

Thanks @nuthalapativarun for switching runtime errors crossing into Python to Display formatting, adding Qdrant and local file system connector unit tests, and fixing broken documentation links.

@daric93

Thanks @daric93 for the Valkey vector store target connector, a fully supported vector index target for teams standardized on Valkey.

@prithvi-moonshot

Thanks @prithvi-moonshot for upgrading cocoindex show into a full-fledged inspection tool for debugging.

@MrAnayDongre

Thanks @MrAnayDongre for preview mode for update actions, letting a run describe the adds, reprocesses, and deletes it would perform before touching a target.

@pyjuan91

Thanks @pyjuan91 for making the OCI source skip its full startup scan when pipeline logic is unchanged, cutting cold-start time on large buckets.

@tomz-alt

Thanks @tomz-alt for native sparse vector support and eager point-ID validation in the Qdrant connector, making it first-class for hybrid retrieval.

@jordigilh

Thanks @jordigilh for adding periodic rescan and watcher recreation to the local file system source in live mode, a safety net for events an OS watcher misses.

@MaxRong

Thanks @MaxRong for making the engine reject rows with missing or null primary keys instead of silently writing corrupt rows.

@slliland

Thanks @slliland for making the state store automatically resize the LMDB map when it fills, removing the manual LMDB_MAP_SIZE tuning operators used to need.

@SuperMarioYL

Thanks @SuperMarioYL for !-prefixed negation in excluded_patterns, so a path matched by an earlier rule can be explicitly un-excluded.

@davidmyriel

Thanks @davidmyriel for documenting Tigris alongside MinIO as a supported S3-compatible service.

@dashitongzhi

Thanks @dashitongzhi for improving the docs CI with proper triggers and build verification.

CocoIndex

Fresh context for long-horizon agents.

Learn the concept → View on GitHub

About the author.

George He Cofounder & CTO of CocoIndex · @georgehe0

Maintainer of CocoIndex, Ex-Google Infra Lead. Writes about incremental data infrastructure, Rust internals, and the engineering decisions behind the engine.