--- title: "CocoIndex Changelog 1.0.8 - 1.0.16" description: "CocoIndex's second post-v1 changelog (1.0.8–1.0.16): persistent per-component state, LiveMap, structural code matching, token-bucket rate limiting, and BigQuery / Snowflake / Azure connectors." last_updated: 2026-07-07 doc_version: "2026-07-07" canonical: https://cocoindex.io/blogs/changelog-108-1016/ --- # CocoIndex Changelog 1.0.8 - 1.0.16 > CocoIndex's second post-v1 changelog (1.0.8–1.0.16): persistent per-component state, LiveMap, structural code matching, token-bucket rate limiting, and BigQuery / Snowflake / Azure connectors. Published: 2026-07-07 · Canonical: https://cocoindex.io/blogs/changelog-108-1016/ This is the second changelog since the v1.0 launch, covering nine releases (1.0.8–1.0.16). The focus shifts from initial production hardening into **new engine primitives for stateful and streaming pipelines**, a wholly new **structural code-matching subsystem**, a major expansion of **cloud and enterprise connectors**, and continued depth on **agent ergonomics**. CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an [incremental engine](https://cocoindex.io/docs/programming_guide/core_concepts/). The source is on [GitHub](https://github.com/cocoindex-io/cocoindex). ## Why this release cycle matters For developers building retrieval, extraction, and knowledge-graph pipelines, the most significant changes are: - **Persistent per-component state.** `coco.use_state()` lets pipeline components carry state that survives across runs, opening the door to accumulators, counters, and stateful deduplication without an external store. - **In-memory intermediate collections.** `LiveMap` holds a mutable keyed collection in memory during a live pipeline run, enabling fan-out aggregation patterns that are impossible in pure declarative pipelines. - **Structural code matching.** The new `code_match` crate and `CodePattern` Python API let pipelines search codebases by syntax structure rather than text strings, matching function calls, containment relationships, and token patterns with O(N·k) performance. - **Token-bucket rate limiting.** `RateLimiter` is now a first-class engine primitive, replacing ad-hoc sleep loops and per-integration retry logic with a consistent, cooperative throttle shared across pipeline components. - **Enterprise cloud connectors.** BigQuery, Snowflake, and Azure Blob Storage land alongside sparse-vector Qdrant support and a new Valkey vector store connector, completing coverage of the major cloud data platforms. ## Engine ### `coco.use_state()`: persistent per-component state `coco.use_state()` gives any pipeline component a persistent, typed state slot that survives across update cycles. The state is stored in the LMDB state store alongside other engine metadata, keyed by a `StableKey` (not just a plain string, as of [**#2085**](https://github.com/cocoindex-io/cocoindex/pull/2085)). A `type_hint` parameter added in v1.0.15 enables typed deserialization so components can round-trip structured state objects cleanly. Why it matters: the incremental engine already knows *what* changed between runs. `use_state` lets components know *what they have already done*. This enables accumulators (running totals, seen-entity sets), watermark trackers, and stateful deduplication, patterns that previously required an external Redis or Postgres sidecar. The net effect is that a wider class of stateful streaming logic can now live inside the pipeline declaration, subject to the same incremental reconciliation semantics as the rest of CocoIndex. ### `LiveMap`: in-memory intermediate collections `LiveMap` is a new in-memory keyed collection that lives inside a live pipeline run. Where `coco.stats_group` provides observability over a data slice, `LiveMap` provides *computation* over a data slice: a mutable intermediate state that components can read and write as events flow through the pipeline. `LiveComponent` classes can now be used inside `mount_each()`, enabling dynamic live component trees where the set of active live components changes based on source data. ### Token-bucket `RateLimiter` `RateLimiter` is now a first-class engine primitive implementing a token-bucket algorithm. It provides a cooperative throttle shared across pipeline components: any component that acquires a token from the same `RateLimiter` instance participates in the same budget. This replaces per-integration sleep loops and ad-hoc retry logic. Combined with the existing retry-on-429 (HTTP rate limit) behavior added in earlier cycles, `RateLimiter` gives pipelines deterministic, budget-aware flow control rather than reactive backpressure alone. ### Preview mode for update actions Update actions can now be previewed before they are applied. A pipeline run can describe the adds, reprocesses, and deletes it *would* perform, without writing to any target, enabling dry-run validation and change auditing before committing to a production store. ### Batched target writes across components Row-level writes from sibling components are now coalesced into a single batched apply. The common shape is `mount_each(..., table_target)`: many child components each declare one row into the same target. Previously every child submitted its own leaf action directly to the sink after precommit, so the connector saw a stream of single-row calls even though its API already accepts a batch. In practice a 50-row ingest into LanceDB produced roughly 50 `merge_insert` commits, and 50 table versions, instead of one ([**#2219**](https://github.com/cocoindex-io/cocoindex/issues/2219)). An app-scoped batcher, keyed by the target sink, now merges concurrent sibling commits that hit the same sink before the connector's apply call ([**#2221**](https://github.com/cocoindex-io/cocoindex/pull/2221)). The design reuses CocoIndex's existing [`Batcher`](https://github.com/cocoindex-io/cocoindex/blob/main/rust/utils/src/batching.rs) primitive rather than a bespoke queue, so leaf-target batching shares the same path as the engine's other batched work and adds no debounce latency to a lone write. Sink lifetime drives cleanup: a keeper owns the batcher one-to-one and drops its registry entry the moment the sink goes away, so a pipeline with an unbounded number of sink functions cannot leak batcher entries. Batching also preserves order, each output mapping back to its input, which let container targets route through the same batched path instead of a separate one. Why it matters: for connectors with real per-operation latency, such as LanceDB and cloud warehouses, write cost is dominated by round-trips and commits, not row count. Collapsing N single-row applies into one batched apply cuts that overhead directly. For a store like LanceDB it also collapses N table versions into one, which keeps version history and compaction manageable under heavy fan-out. ### Engine performance improvements - **Prefetch fn-memos and user-states in one read transaction**: the engine now batches all LMDB reads for function memos and user states into a single read transaction per cycle, replacing N individual reads ([**#2076**](https://github.com/cocoindex-io/cocoindex/pull/2076)). - **Minimize serialization for UserStateCache**: reduces round-trip cost for state reads during hot-path execution ([**#2127**](https://github.com/cocoindex-io/cocoindex/pull/2127)). - **LMDB map auto-resize**: the state store now automatically resizes the LMDB map when it fills, eliminating the manual `LMDB_MAP_SIZE` tuning that previously required operator intervention ([**#2231**](https://github.com/cocoindex-io/cocoindex/pull/2231)). - **LMDB map size rounded to page size**: map sizes are now aligned to the OS page boundary, preventing subtle allocation failures on some Linux kernels ([**#2109**](https://github.com/cocoindex-io/cocoindex/pull/2109)). ### Other engine fixes - **`coco.fn` callable outside a component context**: `coco.fn`-decorated functions can now be called directly in scripts and tests without being inside a mounted pipeline component ([**#2101**](https://github.com/cocoindex-io/cocoindex/pull/2101)). - **`!`-prefixed negation in `excluded_patterns`**: source file filters now support `!pattern` to explicitly un-exclude a path that matched an earlier pattern, following the `.gitignore`-style rule ordering ([**#1901**](https://github.com/cocoindex-io/cocoindex/pull/1901)). - **Retry transient LiteLLM embedding errors**: transient failures (network blips, upstream 5xx) during embedding calls are retried automatically ([**#2090**](https://github.com/cocoindex-io/cocoindex/pull/2090)). - **Reject rows with missing or null primary keys**: the engine now fails fast rather than silently writing corrupt rows ([**#2239**](https://github.com/cocoindex-io/cocoindex/pull/2239)). ## Code AST matching (new subsystem) The most architecturally novel work in this cycle is the `code_match` crate: a structural code-search engine built on [tree-sitter](https://cocoindex.io/blogs/how-tree-sitter-works) that allows CocoIndex pipelines to match source code by *syntax structure*, not just text content. ### What it is `CodePattern` compiles a structural pattern once and reuses it across an entire codebase. A pattern can match: - **Literal tokens**: specific keywords, operators, identifiers - **Anonymous regex matchers**: any token matching a regular expression - **Named regex matchers**: `\(NAME:/re/*\)` binds the match to a named capture - **Quantifiers**: `*` (zero-or-more), `+` (one-or-more) - **Containment**: `\{{ INNER \}}` matches any node that contains INNER as a descendant - **Whole-node boundary**: `\{ P \}` matches a node whose complete token sequence matches P ("is") Patterns are matched via `CodePattern.match_file`, returning all non-overlapping matches per node with source positions. ### Why it matters for codebase indexing Text-based search (grep, BM25) finds patterns anywhere in a file, including comments, strings, and variable names that happen to spell the target. Structural search respects the parse tree: a pattern matching a function call only matches actual function calls in the AST, not the word "function" in a comment. For AI pipelines extracting API usage, dependency graphs, or security-relevant patterns (SQL queries, exec calls, HTTP requests), this distinction is the difference between noisy results and precise extraction. The prefilter system (`index_terms`, `matches_prefiltered`) extracts required-content tokens from a pattern and can drive an inverted index, so structurally-matched search avoids scanning the full tree on documents that can't possibly match. ### Performance The implementation received substantial algorithmic attention in v1.0.13: - Child-run scan pruned by trailing literal token - Child-run tolerance folded into one dynamic-programming pass per candidate: O(N·k) rather than O(N²) - GIL released across all Python-facing entry points (code-matcher is a Rust hot path) - Byte→position index memoized, eliminating per-query full scans ### Shared `CodeSource` (v1.0.16) v1.0.16 introduced `CodeSource`, a shared parse context that performs a single tree-sitter parse and reuses it across all consumers: the splitter, the matcher, and any custom analysis functions. Previously each consumer triggered its own parse. On large codebases where both chunking and structural matching run over the same files, this is a significant throughput improvement. Deprecated compat shims keep existing `CodeAst` surface usage working through the transition. ## Connectors ### Target: Snowflake (v1.0.15) A native Snowflake target connector lands, bringing CocoIndex's declare-target-state model to one of the most widely deployed cloud data warehouses. Snowflake targets participate in the same incremental reconciliation as all other CocoIndex targets: rows for deleted sources are removed automatically, and only changed rows are upserted. ### Target: BigQuery (v1.0.16) A Google BigQuery target connector was added alongside the Snowflake connector in the same cycle. For organizations already running Google Cloud infrastructure, this completes cloud data warehouse coverage across the major platforms. ### Source: Azure Blob Storage (v1.0.16) An Azure Blob Storage source connector mirrors the OCI and S3 source APIs. For teams on Azure infrastructure, this enables the same live-mode, incremental bucket-watching behavior available to AWS and OCI users. ### Target: Valkey vector store (v1.0.12) [Valkey](https://valkey.io), the open-source Redis fork under Linux Foundation governance, gains a target connector with vector store support. For organizations that have standardized on Valkey over Redis, this provides a fully supported vector index target without requiring a separate vector database. ### Qdrant: native sparse vector support (v1.0.16) The Qdrant connector was upgraded with native sparse vector support and eager point-ID validation. Sparse vectors are essential for hybrid search pipelines that combine dense (semantic) and sparse (BM25/TF-IDF) representations, making CocoIndex's Qdrant integration first-class for hybrid retrieval. ### LanceDB: vector and FTS index support (v1.0.11) LanceDB gained explicit vector and full-text search (FTS) index management. The `drop_index` operation was added alongside a dependency bump to LanceDB 0.33.0 and PyArrow 23.0.0. Nullable schema-only row rewrites (a correctness bug that appeared on partial schema changes) were fixed in both the Python and Rust layers. In v1.0.15, LanceDB table maintenance was tied to LanceDB's own stats API, making compaction scheduling driven by actual table metrics rather than a fixed interval. ### zvec: FTS field support (v1.0.14) The `zvec` target connector gained full-text search field support, allowing pipelines to write both vector and FTS fields to zvec in a single target export. ### OCI: skip startup scan on unchanged logic (v1.0.11) The OCI Object Storage source now detects when pipeline logic is unchanged since the last run and skips the expensive full-bucket startup scan, going directly to live-stream event processing. For large buckets, this can reduce cold-start time from minutes to seconds. ### Local file system: periodic rescan in live mode (v1.0.15) The local file system source in live mode now periodically rescans and recreates the watcher if it becomes stale. OS file-system watchers (`inotify`, `kqueue`) have known limitations under high churn, network mounts, and certain editors; periodic rescan provides a safety net that catches any events the watcher missed. ### New connector summary | Connector | Type | Release | |-----------|------|---------| | Snowflake | Data warehouse target | v1.0.15 | | BigQuery | Data warehouse target | v1.0.16 | | Azure Blob Storage | Object store source | v1.0.16 | | Valkey | Vector store target | v1.0.12 | | Qdrant sparse vectors | Enhanced vector target | v1.0.16 | | LanceDB vector + FTS indexes | Enhanced vector target | v1.0.11 | | zvec FTS fields | Enhanced target | v1.0.14 | ## Agent & developer experience ### `cocoindex show`: full-fledged inspection tool (v1.0.12) `cocoindex show` was upgraded from a basic state dump into a full inspection tool for pipeline debugging. It provides structured, human-readable output of pipeline state, component configuration, and target schema: the production debugging workflow that previously required querying the LMDB or Postgres state store directly. ### Python error messages Runtime errors crossing the Rust-to-Python boundary now use `Display` (human-readable) rather than `Debug` (Rust internals) formatting, making error messages that surface in Python stack traces significantly more actionable. ### Multiple GPU and fractional GPU allocation (v1.0.15) `GPUPool` was added to support multiple GPUs and fractional GPU allocations within a pipeline. For embedding-heavy pipelines running on multi-GPU infrastructure, this allows work to be distributed across available GPU resources without hand-partitioning the pipeline. ## Example walkthroughs v1.0.13 shipped a major documentation expansion: 17 new [example walkthroughs](https://cocoindex.io/docs/examples/) (10 full + 7 variants) covering the breadth of CocoIndex use cases: - **[Meeting Notes → Knowledge Graph](/blogs/meeting-notes-graph/)**, with [Neo4j and FalkorDB variants](https://cocoindex.io/docs/examples/meeting-notes-to-knowledge-graph/) - **[CSV → Kafka](/blogs/csv-to-kafka-live/)** [streaming pipeline](https://cocoindex.io/docs/examples/csv-to-kafka/) - **[Semantic Search over PDFs](/blogs/pdf-elements/)** with [mixed embedding models](https://cocoindex.io/docs/examples/pdf-embedding/) - **[Search Images by Text](/blogs/live-image-search/)**, a [multimodal CLIP walkthrough](https://cocoindex.io/docs/examples/image-search/) - **[Face Recognition](/blogs/face-detection/)**, [ported from v0 to v1](https://cocoindex.io/docs/examples/face-recognition/) - **[Docs to Knowledge](/blogs/knowledge-graph-for-docs/)** [graph construction](https://cocoindex.io/docs/examples/docs-to-knowledge-graph/) Each walkthrough was structured for agent followability, with consistent heading patterns, copyable commands, and explicit prerequisites, so AI coding agents can execute them as step-by-step procedures. ## Bug fixes and correctness | Issue | Impact | Release | |-------|--------|---------| | S3 ETag encoded to bytes for content fingerprint | Prevents false cache invalidation on binary ETags | v1.0.10 | | `StableKey::Bytes` serde round-trip via msgpack | Correct key serialization for binary primary keys | v1.0.14 | | Postgres: preserve data and NOT NULL on column type evolution | Schema evolution no longer drops non-null constraints | v1.0.15 | | Postgres: schema-qualify DROP INDEX for vector indexes | Avoids dropping wrong-schema indexes in multi-schema deployments | v1.0.15 | | Logic deps propagated across mount boundary | Dependency tracking correct for nested mounted components | v1.0.12 | | Orphaned child-existence rows when Directory→Component | Prevents stale rows when a directory node becomes a component node | v1.0.15 | | LanceDB: escape single quotes in delete filter predicates | Prevents filter injection on document paths containing apostrophes | v1.0.11 | | LanceDB: nullable schema-only row rewrites | Correct behavior on partial schema updates | v1.0.15 | | Idle batchers cleared to avoid stale accumulation | Prevents write amplification on quiet pipelines | v1.0.8 | | PEP 695 type aliases in numpy 2.5 NDArray | Fixes `analyze_type_info` crash on numpy 2.5+ | v1.0.13 | ## Summary This cycle's theme is **depth and breadth**: depth in the engine (stateful components, structural code search, GPU pools, performance batching) and breadth in connectors (completing cloud warehouse coverage with Snowflake, BigQuery, and Azure Blob, plus Valkey and enhanced Qdrant). The `code_match` subsystem is the most significant new capability. Structural code search is a primitive that unlocks a class of precise, noise-free codebase intelligence pipelines that text-based search cannot provide. For the complete list of changes, see the [GitHub releases](https://github.com/cocoindex-io/cocoindex/releases). If CocoIndex is useful to you, consider starring the [repository](https://github.com/cocoindex-io/cocoindex). ## Thanks to the community Thanks to everyone who shipped this cycle, core maintainers and community alike, including twelve first-time contributors. ### @georgeh0 Thanks [@georgeh0](https://github.com/georgeh0) for the bulk of this cycle's engine work: the [`code_match` structural-search crate](https://github.com/cocoindex-io/cocoindex/pull/2133) and its [shared `CodeSource` parse](https://github.com/cocoindex-io/cocoindex/pull/2253), [`LiveMap`](https://github.com/cocoindex-io/cocoindex/pull/2088), the [token-bucket `RateLimiter`](https://github.com/cocoindex-io/cocoindex/pull/2057), and [`LiveComponent` support inside `mount_each`](https://github.com/cocoindex-io/cocoindex/pull/2091). ### @badmonster0 Thanks [@badmonster0](https://github.com/badmonster0) for the documentation updates. ### @hakuuww Thanks [@hakuuww](https://github.com/hakuuww) for building [`coco.use_state()`](https://github.com/cocoindex-io/cocoindex/pull/2034), persistent per-component state, then following it with engine-side performance work: [prefetching fn-memos and user states in one read transaction](https://github.com/cocoindex-io/cocoindex/pull/2076) and [minimizing UserStateCache serialization](https://github.com/cocoindex-io/cocoindex/pull/2127). ### @sdhilip200 Thanks [@sdhilip200](https://github.com/sdhilip200) for a prolific connector cycle: the [Snowflake](https://github.com/cocoindex-io/cocoindex/pull/2046) and [BigQuery](https://github.com/cocoindex-io/cocoindex/pull/2250) targets, the [Azure Blob source](https://github.com/cocoindex-io/cocoindex/pull/2252), and [Mermaid diagram rendering in the docs](https://github.com/cocoindex-io/cocoindex/pull/2045). ### @Sujit-1509 Thanks [@Sujit-1509](https://github.com/Sujit-1509) for [LanceDB vector and FTS indexing](https://github.com/cocoindex-io/cocoindex/pull/2115), plus correctness fixes: [preserving data and NOT NULL constraints during Postgres column-type evolution](https://github.com/cocoindex-io/cocoindex/pull/2228) and [reconciling orphaned child-existence rows](https://github.com/cocoindex-io/cocoindex/pull/2212). ### @prrao87 Thanks [@prrao87](https://github.com/prrao87) for deep LanceDB work: [tying table maintenance to LanceDB's own stats](https://github.com/cocoindex-io/cocoindex/pull/2245), [implementing `drop_index`](https://github.com/cocoindex-io/cocoindex/pull/2132), and [fixing nullable schema-only row rewrites](https://github.com/cocoindex-io/cocoindex/pull/2235), plus [batching leaf target actions](https://github.com/cocoindex-io/cocoindex/pull/2221). ### @lichuang Thanks [@lichuang](https://github.com/lichuang) for [`GPUPool`](https://github.com/cocoindex-io/cocoindex/pull/2224), supporting multiple and fractional GPU allocations, and for [adding `type_hint` to `use_state`](https://github.com/cocoindex-io/cocoindex/pull/2210) for typed state deserialization. ### @Haleshot Thanks [@Haleshot](https://github.com/Haleshot) for the [initial `zvec` target integration](https://github.com/cocoindex-io/cocoindex/pull/2092) and [adding FTS field support to it](https://github.com/cocoindex-io/cocoindex/pull/2215). ### @nuthalapativarun Thanks [@nuthalapativarun](https://github.com/nuthalapativarun) for [switching runtime errors crossing into Python to `Display` formatting](https://github.com/cocoindex-io/cocoindex/pull/2052), adding [Qdrant](https://github.com/cocoindex-io/cocoindex/pull/1952) and [local file system](https://github.com/cocoindex-io/cocoindex/pull/2083) connector unit tests, and [fixing broken documentation links](https://github.com/cocoindex-io/cocoindex/pull/2081). ### @daric93 Thanks [@daric93](https://github.com/daric93) for the [Valkey vector store target connector](https://github.com/cocoindex-io/cocoindex/pull/2027), a fully supported vector index target for teams standardized on Valkey. ### @prithvi-moonshot Thanks [@prithvi-moonshot](https://github.com/prithvi-moonshot) for upgrading [`cocoindex show` into a full-fledged inspection tool for debugging](https://github.com/cocoindex-io/cocoindex/pull/1920). ### @MrAnayDongre Thanks [@MrAnayDongre](https://github.com/MrAnayDongre) for [preview mode for update actions](https://github.com/cocoindex-io/cocoindex/pull/1945), letting a run describe the adds, reprocesses, and deletes it would perform before touching a target. ### @pyjuan91 Thanks [@pyjuan91](https://github.com/pyjuan91) for making the [OCI source skip its full startup scan when pipeline logic is unchanged](https://github.com/cocoindex-io/cocoindex/pull/2116), cutting cold-start time on large buckets. ### @tomz-alt Thanks [@tomz-alt](https://github.com/tomz-alt) for [native sparse vector support and eager point-ID validation in the Qdrant connector](https://github.com/cocoindex-io/cocoindex/pull/2242), making it first-class for hybrid retrieval. ### @jordigilh Thanks [@jordigilh](https://github.com/jordigilh) for [adding periodic rescan and watcher recreation to the local file system source in live mode](https://github.com/cocoindex-io/cocoindex/pull/2232), a safety net for events an OS watcher misses. ### @MaxRong Thanks [@MaxRong](https://github.com/MaxRong) for making the engine [reject rows with missing or null primary keys](https://github.com/cocoindex-io/cocoindex/pull/2239) instead of silently writing corrupt rows. ### @slliland Thanks [@slliland](https://github.com/slliland) for making the state store [automatically resize the LMDB map when it fills](https://github.com/cocoindex-io/cocoindex/pull/2231), removing the manual `LMDB_MAP_SIZE` tuning operators used to need. ### @SuperMarioYL Thanks [@SuperMarioYL](https://github.com/SuperMarioYL) for [`!`-prefixed negation in `excluded_patterns`](https://github.com/cocoindex-io/cocoindex/pull/1901), so a path matched by an earlier rule can be explicitly un-excluded. ### @davidmyriel Thanks [@davidmyriel](https://github.com/davidmyriel) for [documenting Tigris alongside MinIO as a supported S3-compatible service](https://github.com/cocoindex-io/cocoindex/pull/2063). ### @dashitongzhi Thanks [@dashitongzhi](https://github.com/dashitongzhi) for [improving the docs CI with proper triggers and build verification](https://github.com/cocoindex-io/cocoindex/pull/2009). ## Sitemap - [Blog index](https://cocoindex.io/blogs/) - [Site index (llms.txt)](https://cocoindex.io/llms.txt) - [Full blog corpus](https://cocoindex.io/llms-full.txt) - [Markdown sitemap](https://cocoindex.io/sitemap.md) - [XML sitemap](https://cocoindex.io/sitemap.xml) - [RSS feed](https://cocoindex.io/blogs/rss.xml)