Persistent state, rate limits, Snowflake and BigQuery

CocoIndex 1.0.8–1.0.16 adds per-component state, LiveMap, a shared rate limiter, batched target writes, and Snowflake, BigQuery and Azure Blob connectors.

ChangelogUpdated 18 min read
Four feature cards: use_state keeps a value across runs 1, 2 and 3; LiveMap mounts one component per key; batched writes apply the first row alone, then the 3 rows that arrived meanwhile in one apply; RateLimiter puts three callers under one shared token bucket.
Enlarge cover

Persistent state, rate limits, Snowflake and BigQuery

Zoom in to see details.

Four feature cards: use_state keeps a value across runs 1, 2 and 3; LiveMap mounts one component per key; batched writes apply the first row alone, then the 3 rows that arrived meanwhile in one apply; RateLimiter puts three callers under one shared token bucket.

This is the second changelog since the v1.0 launch, covering 9 releases (1.0.8–1.0.16). The new engine APIs are coco.use_state() for state that persists between runs, LiveMap for passing data between two parts of a live pipeline, and a shared token-bucket RateLimiter. Writes from many components to one target are now batched. New connectors add Snowflake, BigQuery and zvec targets and an Azure Blob Storage source. Debugging tools, examples, and fixes follow.

CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.

Start here

Why this release cycle matters

  • Persistent state. coco.use_state() gives a component a value that it reads at the start of a run and that the next run sees, with no external store.
  • In-memory collections. LiveMap lets one part of a live pipeline declare keyed entries and another part run one component per entry.
  • Shared rate limits. RateLimiter puts every caller that holds the same instance under one token budget, served in arrival order.
  • Batched target writes. Rows that many components write to the same target at the same time go out together in batched applies. A 50-row LanceDB ingest that made about 50 commits now makes fewer, depending on how close together the rows arrive.
  • Connectors. Snowflake and BigQuery targets, an Azure Blob Storage source, Valkey and zvec vector store targets, and sparse vectors in Qdrant.

New engine primitives

coco.use_state(): persistent per-component state (v1.0.8)

The incremental engine already knows what changed between runs. A component did not know what it had already done: a running total, the set of entities it has seen, or a watermark (the position it read up to) needed an external Redis or Postgres. coco.use_state() gives any pipeline component a persistent, typed state slot that survives across update cycles.

The state is stored in the LMDB state store (the embedded key-value database where CocoIndex keeps its own metadata), keyed by a StableKey, such as a string, a number, a UUID, or a tuple of these. Before #2085 the key was a plain string. A type_hint parameter added in v1.0.15 decodes the stored value back into your own type, such as a dataclass.

This counter counts how many times the component has run:

python
import cocoindex as coco

@coco.fn
async def process_feed(feed: Feed) -> None:
    runs = coco.use_state("runs", 0)  # 0 on the first run
    runs.value += 1                   # stored when the component commits

What you see. On the first run runs.value is 0, and 1 is stored. Each later run of the same component reads the value the previous run committed. The value is written once, when the component commits.

Stateful logic of this kind can now live inside the pipeline declaration, under the same incremental reconciliation as the rest of CocoIndex.

use_state · state that survives every runLMDB state store
run 1run 2run 3prefetchmiss 0hit 1hit 2process()+1+1+1commit123
state store
  1. 0initial
  2. 1run 1
  3. 2run 2
  4. 3run 3
Each run reads what the last one committed, so the counter climbs with no external store.
Quick check: can you call coco.use_state() inside a @coco.fn(memo=True) function?

No. On a cache hit the memoized body is skipped, so the key would never be declared and the stored state would be cleaned up as stale. CocoIndex raises RuntimeError instead. Call use_state in the component’s own processor.

LiveMap: in-memory intermediate collections (v1.0.8)

Some live pipelines have two halves: one part reads a stream and finds items, and another part processes each item. LiveMap is a new in-memory keyed collection that connects the two halves of the same live run. Producing components declare entries into it as target states (LiveMap.declare_entry). A target state is something a component declares should exist; CocoIndex adds, updates and removes it to match. The consuming side reads the same in-memory dict as a live source through mount_each, which mounts one component per entry.

python
import cocoindex as coco
from cocoindex.resources.live_map import LiveMap

@coco.fn
async def app_main() -> None:
    lm: LiveMap[str, str] = await LiveMap.create()
    await coco.mount(produce_entries, lm)     # producers call lm.declare_entry(key, value)
    await coco.mount_each(process_entry, lm)  # one component per entry

LiveComponent classes can now be used inside mount_each() as well, so the consuming side can be a tree of live components, one per entry, mounted and dropped as producers add keys and stop declaring them.

What you see. When a producer declares a new key, a component for it mounts. When the value for a key changes, that key’s component runs again. When the producer stops declaring a key, its component is removed. The map lives in process memory and is rebuilt from its producers each time the app starts.

LiveMap · one component per keyin memory, live run
producers · declare_entryProducing components
+ add user:5~ upd user:2− del user:4
LiveMap · key → valueKeyed collection
user:1user:2updateduser:3user:5newuser:4dropped
mount_each · live componentsOne per key
component(1)component(2)re-runcomponent(3)component(5)mountedcomponent(4)dropped
Keys appear and vanish; CocoIndex mounts and drops one component each.
Quick check: a producer stops declaring user
. What happens on the consuming side?

The entry is removed, because deletion follows target-state ownership, and mount_each removes the component for user:4. There is no explicit delete call.

DocumentationConnect two halves of a live pipeline.Create a LiveMap, declare entries from any component, and consume them with mount_each: one component per entry, kept in sync.

RateLimiter: one shared token bucket (v1.0.8)

An external API allows a fixed number of requests per second. Each call site used to pace itself with its own sleep loop, so the sources and targets that call one API had no shared view of that limit. RateLimiter is now an engine primitive that implements a token bucket: tokens refill at a steady rate up to a fixed capacity, and each call takes a token or waits for one. Any component that acquires a token from the same RateLimiter instance draws from the same budget, and concurrent callers are served FIFO (first in, first out), so none starve.

python
from cocoindex.resources.rate_limit import RateLimiter

limiter = RateLimiter(max_rows_per_second=20.0)  # burst_window_secs defaults to 1.0

async def fetch(item: Item) -> Data:
    await limiter.acquire()  # waits until a token is free
    return await call_external_api(item)

What you see. Calls go out at about 20 per second. After an idle period, up to 20 calls (max_rows_per_second × burst_window_secs) can start at once before the rate settles. Set burst_window_secs=0.0 for strict spacing.

The same release added retries for transient LiteLLM embedding errors, including 429 (HTTP “too many requests”). Pipelines can now set their rate up front and keep retries for the requests that still get rejected.

The demo below sends 40 calls from 3 callers through limiters of max_rows_per_second=10. Callers A and B use limiter_1 and caller C uses limiter_2. When limiter_2 is the same instance, the 3 callers share 10 calls per second; when it is a second instance, the API gets 20. Switch the instance and the burst window, then move the time slider to watch each bucket refill:

How many calls per second reach the API

Illustrative: 40 calls queued at 0 s

Interactive figure. Three callers send 40 calls through RateLimiter(max_rows_per_second=10). With one shared instance the calls leave at 10 per second after the initial burst; with two instances, A and B on one and C on the other, they leave at 20 per second after the burst. Controls pick the sharing, burst_window_secs and a moment in time; the figure shows the tokens in each bucket and the calls each caller has sent at that moment.

limiter_1 = RateLimiter(
    max_rows_per_second=10,
    burst_window_secs=1.0,
)
# limiter_2 is the same instance:
limiter_2 = limiter_1
# a second instance, same settings:
limiter_2 = RateLimiter(...)

# A and B: await limiter_1.acquire()
# C:       await limiter_2.acquire()
limiter_2 is
burst_window_secs

With one shared instance, the 3 callers together get 10 calls at 0 s, then 10 per second, served in arrival order. All 40 are sent by 3.0 s.

calls sent to the APIthis setupwith a second instance10 per second

limiter_1 and limiter_2: one instance1 of 10 tokens, 1 more every 0.1 s

caller A7 sent, 3 waiting
caller B6 sent, 4 waiting
caller C6 sent, 14 waiting

Each moment shows the buckets just before the calls due then take their tokens. The calls at 0 s are the burst: a new or idle limiter holds max_rows_per_second × burst_window_secs tokens, and at least 1. A window of 0.0 spaces calls strictly: at 10 per second, one every 0.1 s.

Quick check: two connectors each create RateLimiter(max_rows_per_second=10). What is their combined rate?

Up to 20 per second. The budget lives in the instance, so two instances are two independent budgets. To share one budget, pass the same instance to both, for example through a context key.

DocumentationKeep outbound calls under one budget.RateLimiter parameters, burst behavior, and how to share one limiter across components with a context key.

Preview mode for update actions (v1.0.8)

Before you point a pipeline at a production store, you want to see what it would change. Update actions can now be previewed before they are applied. A run with preview enabled describes the adds, reprocesses, and deletes it would perform, without writing to any target, so you can audit the changes first.

sh
cocoindex update main.py --preview

What you see. The command prints Preview: planned target actions and one line per action, or No target actions planned. Nothing is written to a target. In Python, await app.update(preview=True) returns a handle whose result() is the list of actions; app.update_blocking(preview=True) returns the list directly. Preview cannot be combined with --live or --reset.

preview mode · see it before you writedry run
update actionupdate()preview=True
plan · computed, not appliedWhat this run would do
  • +3adds
  • ~2reprocesses
  • −1deletes
target · no writeProduction storeuntouched, 0 rows written

Fixed in v1.0.26 (after this span). In these releases a preview could still store, delete or refresh memos (cached function results) and write committed state, so the next real update could skip work that the preview never did. A preview also failed with an error on container targets, such as a table with rows under it. Both are fixed; see preview runs on real pipelines and writes nothing.

Performance

Batched target writes across components (v1.0.15)

When many components write rows to the same target at the same time, the rows now reach the connector together in batched applies. The common shape is mount_each(..., table_target), where each child component declares one row into the same table. Before this release, each component sent its rows to the connector on its own after precommit, so the connector got a stream of single-row calls even though its API accepts a batch. A 50-row ingest into LanceDB made about 50 merge_insert commits and 50 table versions (#2219).

What you see. The same 50-row ingest lands in fewer commits than about 50, and LanceDB creates one table version per commit. How many commits depends on how close together the rows reach the sink. You change no pipeline code.

The demo below spreads the arrival of the 50 rows over 0, 1 or 4 s and shows the applies the batcher makes:

50 rows, one LanceDB table

Illustrative: each apply takes 0.2 s

mount_each runs 50 child components, and each declares one row. Pick how far apart the rows reach the sink:

Rows reach the sink over
rows arrive
applies

Rows reach the sink over 0 s: 2 commits. The first carries 1 row; the second carries 49 rows.

rows arrive
applies

Rows reach the sink over 1 s: 6 commits. The first carries 1 row; the other 5 carry 9 to 10 rows each.

rows arrive
applies

Rows reach the sink over 4 s: 21 commits. The first carries 1 row; the other 20 carry 2 to 3 rows each.

Before v1.0.15
about 50 commits
about one per row, about 50 table versions
v1.0.15
2 commits
2 table versions
v1.0.15
6 commits
6 table versions
v1.0.15
21 commits
21 table versions

The first row finds the sink idle and is written at once, alone. Rows that arrive while an apply runs wait and go out together in the next one, so the count depends on timing.

For connectors with real per-operation cost, such as LanceDB and cloud warehouses, write cost depends on the number of write transactions (round-trips and commits) more than on row count. A batched apply that carries N rows is one write transaction instead of N, and for a store like LanceDB one table version instead of N, which keeps version history and compaction manageable under heavy fan-out.

How the batching works

The batching is keyed by the target sink (the connector function that applies writes to the external system), so any components whose writes land on the same sink at the same time are batched together. Each sink owns one batcher, which merges concurrent commits before the connector’s apply call (#2221). It reuses CocoIndex’s existing Batcher primitive, the same one behind the engine’s other batched work.

  • Lone writes go out at once. A write that reaches an idle sink is applied at once, with no debounce delay. Writes that arrive while an apply is running are queued and go out together in the next apply. The number of applies therefore depends on timing.
  • Results go back to each component. Each component gets back the results of its own actions. For a container target, the per-action child handlers that the sink returns are split back to the component that declared each action.
  • Cleanup follows the sink. The batcher lives inside the sink’s keeper, so it is dropped with the sink. No separate registry grows with the number of sink functions.

Engine performance

  • Prefetch fn-memos and user-states in one read transaction: the engine loads a component’s function memos and user states in one LMDB read transaction instead of two, which removes a read-transaction begin per component under mount_each fan-out (#2076).
  • Minimize serialization for UserStateCache: reduces round-trip cost for state reads during hot-path execution (#2127).
  • LMDB map auto-resize: the state store now resizes the LMDB map (the maximum database size that LMDB reserves up front) when it fills, so operators no longer tune COCOINDEX_LMDB_MAP_SIZE by hand (#2231).
  • LMDB map size rounded to page size: a COCOINDEX_LMDB_MAP_SIZE that is not a multiple of the system page size is rounded up, instead of failing with the error “must be a multiple of the system page size” (#2109).

Connectors

The table lists the connectors added or extended in this cycle. Details follow it.

ConnectorTypeRelease
SnowflakeData warehouse targetv1.0.15
BigQueryData warehouse targetv1.0.16
Azure Blob StorageObject store sourcev1.0.16
ValkeyVector store targetv1.0.12
zvecEmbedded vector store targetv1.0.11
Qdrant sparse vectorsEnhanced vector targetv1.0.16
LanceDB vector + FTS indexesEnhanced vector targetv1.0.11
zvec FTS fieldsEnhanced targetv1.0.14

Target: Snowflake (v1.0.15)

A native Snowflake target connector brings CocoIndex’s declare-target-state model to the Snowflake cloud data warehouse. Snowflake targets take part in the same incremental reconciliation as all other CocoIndex targets: rows for deleted sources are removed automatically, and only changed rows are upserted.

Target: BigQuery (v1.0.16)

A Google BigQuery target connector was added in the same cycle as the Snowflake connector. Teams on Google Cloud can now write to their warehouse directly.

Source: Azure Blob Storage (v1.0.16)

An Azure Blob Storage source connector mirrors the S3 source API: list_blobs(), get_blob() and read() over an async Azure ContainerClient. Each update lists the container and reprocesses only the blobs that changed. Like the S3 source, it has no live mode: it does not watch the container for change events, as the OCI source does.

Target: Valkey vector store (v1.0.12)

Valkey, the open-source Redis fork under Linux Foundation governance, gains a target connector with vector store support. Teams that run Valkey instead of Redis get a supported vector index target without a separate vector database.

Qdrant: native sparse vector support (v1.0.16)

The Qdrant connector gained native sparse vector support and eager point-ID validation: declare_point rejects an ID that is not an unsigned 64-bit integer or a UUID before anything is written, instead of failing later in the server. Hybrid search pipelines combine dense (semantic) vectors with sparse vectors (BM25 or TF-IDF term weights), and need both in one store. The Qdrant target now holds both.

LanceDB: vector and FTS index support (v1.0.11)

LanceDB gained explicit vector and full-text search (FTS) index management. In v1.0.12, the drop_index operation was added alongside a dependency bump to LanceDB 0.33.0 and PyArrow 23.0.0. Nullable schema-only row rewrites (a correctness bug that appeared on partial schema changes) were fixed in both the Python and Rust layers. In v1.0.15, LanceDB table maintenance was tied to LanceDB’s own stats API, so compaction runs on actual table metrics rather than a fixed interval.

Target: zvec (v1.0.11), with FTS fields (v1.0.14)

A new zvec target connector writes documents to zvec, an embedded vector database that runs inside the application, with no server, and stores each collection in a directory on disk (#2092). Each document is keyed by a single string id, so the table’s primary key must be exactly one column; composite keys are rejected. In v1.0.14 the connector gained full-text search field support, so a pipeline can write both vector and FTS fields to zvec in a single target export (#2215).

OCI: skip startup scan on unchanged logic (v1.0.11)

The OCI Object Storage source can now skip the full-bucket startup scan when the pipeline logic is unchanged since the last run. The skip is opt-in: pass OCIWalker(logic_version=...) and bump the version whenever the processing logic changes. On a rerun whose recorded version matches, the source goes straight to live-stream event processing and relies on the stream to replay the backlog; without logic_version, it scans as before (#2116).

Local file system: periodic rescan in live mode (v1.0.15)

The local file system source in live mode now periodically rescans and recreates the watcher if it becomes stale. OS file-system watchers (inotify, kqueue) have known limitations under high churn, on network mounts, and with certain editors. The periodic rescan catches events the watcher missed.

Debugging and developer experience

cocoindex show: an inspection tool for debugging (v1.0.12)

When a pipeline misbehaves in production, you need to see its state without querying the LMDB state store directly. cocoindex show was upgraded from a basic state dump into an inspection tool for pipeline debugging. It lists each component’s stable path with its type, version, processor, memoization flag and the target states it owns. -l gives the detail, --tree the hierarchy, and a path argument with -r or -p its children or parents.

Extended in v1.0.18 (after this span). cocoindex show now lists every target state with the component that owns it, and finds entries an interrupted run left behind. See cocoindex show: target-state inspection.

Python error messages: readable Display text instead of Rust Debug (v1.0.8)

Runtime errors crossing the Rust-to-Python boundary now use Display (human-readable) rather than Debug (Rust internals) formatting, so the messages in Python stack traces describe the problem instead of Rust data structures.

GPUPool: multiple and fractional GPUs in one pipeline (v1.0.15)

GPUPool was added to support multiple GPUs and fractional GPU allocations within a pipeline. Embedding pipelines on multi-GPU machines can spread work across the available GPUs without splitting the pipeline by hand.

Examples

v1.0.13 added 17 new example walkthroughs (10 full and 7 variants) across CocoIndex use cases:

Each walkthrough uses consistent headings, copyable commands, and explicit prerequisites, so an AI coding agent can follow it step by step.

DocumentationEvery example, ready to run.Browse all CocoIndex example walkthroughs: knowledge graphs, streaming pipelines, multimodal search, and PDF extraction. Each ships with copyable commands and explicit prerequisites.

Fixes

Engine fixes

  • coco.fn callable outside a component context: coco.fn-decorated functions can now be called directly in scripts and tests without being inside a mounted pipeline component (#2101).
  • !-prefixed negation in excluded_patterns: source file filters now support !pattern to un-exclude a path that matched an earlier pattern, following the .gitignore-style rule ordering (#1901).
  • Retry transient LiteLLM embedding errors: transient failures (network blips, upstream 5xx) during embedding calls are retried automatically (#2090).
  • SQLite target rejects rows with missing or null primary keys: declare_row() raises ValueError instead of declaring a row whose key is None (#2239).

Correctness fixes

IssueImpactRelease
S3 ETag encoded to bytes for content fingerprintIncremental updates on S3 and MinIO sources no longer crash with DeserializationError when decoding memo statev1.0.10
StableKey::Bytes serde round-trip via msgpackCorrect key serialization for binary primary keysv1.0.14
Postgres: preserve data and NOT NULL on column type evolutionSchema evolution no longer drops non-null constraintsv1.0.15
Postgres: schema-qualify DROP INDEX for vector indexesRecreating a vector index outside the default schema no longer fails with DuplicateTableErrorv1.0.15
Logic deps propagated across mount boundaryDependency tracking correct for nested mounted componentsv1.0.12
Orphaned child-existence rows when Directory→ComponentPrevents stale rows when a directory node becomes a component nodev1.0.15
LanceDB: escape single quotes in delete filter predicatesPrevents filter injection on document paths containing apostrophesv1.0.11
LanceDB: nullable schema-only row rewritesCorrect behavior on partial schema updatesv1.0.15
Idle batchers cleared to avoid stale accumulationBatched functions no longer keep a batcher per object or argument set, and a recycled object id no longer reuses a dead object’s batcherv1.0.8
PEP 695 type aliases in numpy 2.5 NDArrayFixes analyze_type_info crash on numpy 2.5+v1.0.13

What to try next

  • Keep a value between runs. Call coco.use_state("key", initial) in a component’s processor and assign to .value, as in the counter example.
  • Put every caller of one API under one budget. Create one RateLimiter(max_rows_per_second=...) and share the instance, as in the rate limiter example.
  • See what an update would change. Run cocoindex update main.py --preview before you point the pipeline at a production store.
  • Inspect a running pipeline. Run cocoindex show to see each component with its processor, memoization and the target states it owns. From v1.0.18 it also lists target states and their owners.

For the complete list of changes, see the GitHub releases.

Thanks to the community

Thanks to everyone who contributed this cycle, core maintainers and community, including 12 first-time contributors.

@georgeh0

@georgeh0

Thanks @georgeh0 for most of this cycle’s engine work: LiveMap, the token-bucket RateLimiter, LiveComponent support inside mount_each, and the shared-parse code infrastructure behind cocoindex-code.

@badmonster0

@badmonster0

Thanks @badmonster0 for the documentation updates.

@hakuuww

@hakuuww

Thanks @hakuuww for building coco.use_state(), persistent per-component state, then following it with engine-side performance work: prefetching fn-memos and user states in one read transaction and minimizing UserStateCache serialization.

@sdhilip200

@Sujit-1509

@prrao87

@lichuang

@lichuang

Thanks @lichuang for GPUPool, supporting multiple and fractional GPU allocations, and for adding type_hint to use_state for typed state deserialization.

@Haleshot

@nuthalapativarun

@daric93

@daric93

Thanks @daric93 for the Valkey vector store target connector, a vector index target for teams that run Valkey.

@prithvi-moonshot

@MrAnayDongre

@MrAnayDongre

Thanks @MrAnayDongre for preview mode for update actions, letting a run describe the adds, reprocesses, and deletes it would perform before touching a target.

@pyjuan91

@pyjuan91

Thanks @pyjuan91 for making the OCI source skip its full startup scan when pipeline logic is unchanged, cutting cold-start time on large buckets.

@tomz-alt

@jordigilh

@MaxRong

@MaxRong

Thanks @MaxRong for making the SQLite target reject rows with missing or null primary keys instead of declaring a row whose key is None.

@slliland

@slliland

Thanks @slliland for making the state store automatically resize the LMDB map when it fills, removing the manual COCOINDEX_LMDB_MAP_SIZE tuning operators used to need.

@SuperMarioYL

@SuperMarioYL

Thanks @SuperMarioYL for !-prefixed negation in excluded_patterns, so a path matched by an earlier rule can be explicitly un-excluded.

@davidmyriel

@dashitongzhi

Build fresh context for your agents.

CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.