Control Processing Concurrency in CocoIndex

How CocoIndex v1 caps concurrent processing components with one app-level limit, lends slots to nested mounts, and adds rate and batch bounds.

FeatureUpdated 8 min read

Control processing concurrency

a.mdOne slot
  1. process_fileComponent
  2. For each chunkcoco.map · Embed
b.mdOne slot
  1. process_fileComponent
  2. For each chunkcoco.map · Embed

max_inflight_components applies across the app.

CocoIndex processes items in parallel. In CocoIndex v1, each processing component (typically one per source item, such as a file) is a unit of concurrent work, and one setting caps how many run at the same time. Together with bounds on batches and outbound requests, it lets you tune throughput for your workload.

Processing too many items simultaneously can cause:

  • Memory exhaustion: every running component holds its own data (a file’s contents, its chunks, its embeddings) in memory.
  • Resource contention: CPU, disk I/O, and network bandwidth get overwhelmed by competing operations.
  • System instability: timeouts, rate-limit errors from external APIs, degraded performance, or outright crashes.

CocoIndex v1 gives you:

  • One cap on processing components per app, max_inflight_components, with a default of 1024.
  • A cap that holds across nested components without deadlocking: a parent that waits for its children lends its slot to them.
  • Bounds for the heavy steps inside components: how many batches a batched function runs at once, a shared token-bucket RateLimiter for outbound calls, and a request cap on LiteLLMEmbedder.
Where each bound applies

CocoIndex v1

app · max_inflight_components
mount_each · 1 slot
process_filea.md
coco.map · same slot
process_chunk× N chunks
mount_each · 1 slot
process_fileb.md
coco.map · same slot
process_chunk× N chunks
batched across componentsmax_batch_size · max_in_flight
embed()
outbound callsRateLimiter · max_inflight_requests · coco.GPU
model or API

Review the concurrency control documentation or browse the CocoIndex source.

One component limit per app

max_inflight_components caps how many processing components execute at the same time in one app. Set it in coco.AppConfig, or with the COCOINDEX_MAX_INFLIGHT_COMPONENTS environment variable; the default is 1024. The AppConfig value takes precedence over the environment variable, and the environment variable over the default:

python
app = coco.App(
    coco.AppConfig(name="MyPipeline", max_inflight_components=4),
    app_main,
    sourcedir=pathlib.Path("./data"),
)
sh
export COCOINDEX_MAX_INFLIGHT_COMPONENTS=4

When every slot is in use, the next component waits until one finishes. The wait starts at the mount(), use_mount() or mount_each() call, so a parent cannot run ahead of its children without bound: a loop that mounts a child per source item advances only as fast as children finish. With max_inflight_components=1, components run one at a time.

Note

max_inflight_components is a concurrency cap, not a memory bound. It limits how many components run, not how many bytes they hold. A few very large items can still use a lot of memory under a small limit.

What counts toward the limit

A component takes a slot. Concurrent work inside one component does not:

  • coco.mount_each(process_file, files.items(), ...) creates one component per file. Each one counts.
  • coco.map(process_chunk, chunks, ...) runs one async task per chunk inside the current component. It creates no components, so the chunks of one file run concurrently under that file’s single slot.

We checked this on CocoIndex 1.1.1 (script) with 40 files mounted by mount_each(), each running 10 chunk calls with coco.map(). With max_inflight_components=4, at most 4 file components ran at a time, and up to 40 chunk calls ran at the same time. With an asyncio.Semaphore(2) around the chunk call, at most 2 ran.

So if a step inside a component is the heavy one, bound that step directly (see Bounds inside a component), or mount it as its own component so the limit covers it.

Slot lending: a cap that can’t deadlock a hierarchy

A parent component usually waits for its children: use_mount() returns the child’s result, and a component’s own target states are committed only after all its children are done. If waiting parents kept their slots, a small limit could fill up with parents and leave no slot for the children they wait for.

Instead, a parent keeps its slot while it waits and lends it to one child at a time:

  • The first child a parent mounts runs on the parent’s slot.
  • Further children take free slots from the pool, and their mount() / use_mount() calls wait while the pool is empty.
  • When the child on the lent slot finishes, the slot passes to the parent’s next waiting child.

So a hierarchy of any depth completes, even with max_inflight_components=1. The figure below replays real runs on CocoIndex 1.1.1, recorded by a trace script: an app mounts 3 file components, and each file mounts 3 chunk components. Each file spends 30 ms before it mounts its chunks, and each chunk runs for 100 ms. A file’s dashed bar ends when its mount_each() call returns. Its last chunk can still be starting or running then, and the file component is complete only when its chunks are. Pick a limit to compare. The last option, “1, no lending”, is not a recording: it shows the deadlock that lending prevents, with the root component app_main holding the only slot while its first file waits for one.

max_inflight_components

The limit bounds the components that execute at the same time. Counting the parents that wait on them, the components in flight are bounded by the limit times the depth of the component hierarchy.

A few related rules:

  • A component that awaits a function call, including a batched one, keeps its slot. Only mounting a child lends it.
  • When several parents mount the same component at once, only the run that executes takes a slot. The others wait for it without a slot and reuse its result.
  • Deleting components (cleaning up a removed subtree, App.drop()) is bounded by the same limit.

Live components share the limit fairly

In live mode, several updates can compete for slots: a full re-index next to a refresh of a few changed files. The limit is shared fairly between independent updates (an App.update(), one cycle of a live component, or one item update of mount_each() over a live map). A component that finds a free slot takes it at once, so an update that runs alone can use the whole limit. When no slot is free, a slot that frees up goes to the waiting update that holds the fewest slots, so a small update never waits for a large one’s whole backlog.

Bounds inside a component

The heavy part of an incremental pipeline is usually one step: running a model locally, or calling a remote API. v1 bounds these steps where they happen:

  • Component limit: max_inflight_components on AppConfig, or COCOINDEX_MAX_INFLIGHT_COMPONENTS. Applies to all processing components of one app.
  • Batches in flight: max_in_flight=n on @coco.fn.as_async(batching=True, ...). At most n batches of that function run at once (counted per batcher: per instance and set of extra arguments).
  • Batch size: max_batch_size=n on the same decorator. At most n inputs per call to the function body.
  • Request rate: a RateLimiter(max_rows_per_second=...), shared through a context key or passed as rate_limiter= to a batched function. Applies to every caller that holds the same limiter.
  • Open requests: max_inflight_requests=n on LiteLLMEmbedder. Applies to one embedder instance.
  • GPU work: runner=coco.GPU on @coco.fn.as_async. Each call (each batch, for a batched function) takes a GPU from a pool that counts whole GPUs and fractions (coco.GPU(0.5) takes half), so on one GPU, coco.GPU calls run one at a time. Batched functions on the runner also share one batch queue.

A RateLimiter (rate limiting docs) is a token bucket, and its budget lives in the object. To give the whole app one budget, provide one instance through a context key:

python
import cocoindex as coco
from cocoindex.resources.rate_limit import RateLimiter

RATE_LIMIT = coco.ContextKey[RateLimiter]("rate_limiter")


@coco.lifespan
async def lifespan(builder: coco.EnvironmentBuilder):
    builder.provide(RATE_LIMIT, RateLimiter(max_rows_per_second=50.0))
    yield


@coco.fn
async def fetch_one(item: Item) -> Data:
    await coco.use_context(RATE_LIMIT).acquire()
    return await call_external_api(item)

For a batched function that sends one request per batch, pass the limiter as rate_limiter. Each batch then takes one token, so a scarce quota is spent on full batches:

python
limiter = RateLimiter(max_rows_per_second=1.0)


@coco.fn.as_async(batching=True, max_batch_size=100, rate_limiter=limiter)
async def fetch_many(ids: list[str]) -> list[Item]:
    return await call_external_api(ids)  # one request per batch, one token each

What v1 does not have

The v0 flow API had three limits that v1 does not carry over:

  • Byte limit. v0 could cap the input bytes in flight. v1 caps components, not bytes. For inputs whose sizes vary widely, lower max_inflight_components, or split a large item into child components so each holds less.
  • Per-source limit. v0 could give each source its own row budget. In v1, the limit is per app: workloads that need different limits can run as separate apps, each with its own max_inflight_components. Inside one app, bound the heavy step of one source with a RateLimiter or an asyncio.Semaphore that only that source’s components use.
  • Per-iteration limit. v0 could cap rows in one nested iteration. In v1, mount nested items as components (mount_each()) to put them under the component limit, or keep them in coco.map() and bound the heavy call they make.

Tune concurrency limits

In an incremental pipeline, the bottleneck is usually a few heavy operations, such as running a model locally or calling a remote API. Keeping more items in memory than the backend can take at once means new work is ready as soon as the backend frees up. The component limit caps how many items are held this way (a count of items, not bytes).

tuning guide

default first

defaultCOCOINDEX_MAX_INFLIGHT_COMPONENTS=1024
works well
OK
unstable?
Decrease limit
very stable but slow?
Increase limit
what to tune
Tunemax_inflight_components
most cases
OK
rate-limited or bursty endpoint?
Share a RateLimiter or cap open requests
highly unbalanced item (file) size?
Lower the component limit (no byte limit)
workloads of unbalanced complexity?
Separate apps, each with its own limit

Start with the default

The default component limit (COCOINDEX_MAX_INFLIGHT_COMPONENTS: 1024) loads more items than the heavy operations can consume at once, within a bound. Change it only when one of the cases below applies.

When to tune

Lower the limit if the pipeline is unstable (memory overuse, or timeouts on some operations). Raise it if the pipeline is stable and needs more throughput.

  • Start with max_inflight_components (or COCOINDEX_MAX_INFLIGHT_COMPONENTS), before the more specific bounds.

  • Use the more specific bounds for unbalanced workloads:

    • On a rate-limited or bursty endpoint, share one RateLimiter across the app (or pass it as rate_limiter to the batched function that calls it), or cap open requests with max_inflight_requests on LiteLLMEmbedder or max_in_flight on your own batched function.
    • When item (file) sizes vary widely, lower the component limit, because v1 has no byte limit.
    • When workloads differ widely in cost (one goes through a compute-intensive model, another only moves data), run them as separate apps with their own limits.

Set the component limit for the app, then bound the heavy calls inside it.

Source

Browse the CocoIndex source on GitHub.

For configuration examples, see the concurrency control documentation and the rate limiting documentation.

Frequently asked questions

How do you control processing concurrency in CocoIndex?

In CocoIndex v1, each processing component (typically one per source item, such as a file) is a unit of concurrent work. max_inflight_components in coco.AppConfig, or the COCOINDEX_MAX_INFLIGHT_COMPONENTS environment variable, caps how many run at once per app. When every slot is in use, the next mount(), use_mount() or mount_each() call waits until one frees up.

See One component limit per app and the concurrency control documentation.

What is the default concurrency limit in CocoIndex?

By default, up to 1024 processing components can run at the same time per app. The AppConfig value takes precedence over COCOINDEX_MAX_INFLIGHT_COMPONENTS, and the environment variable over the default. Start with the default, then lower it if your workload runs out of memory or times out, or raise it if the pipeline is stable and needs more throughput.

See One component limit per app and Start with the default.

Does coco.map count toward max_inflight_components?

No. coco.mount_each() creates one processing component per item, and each one takes a slot. coco.map() runs async tasks inside the current component and creates no components, so they share that component's slot. In our check on CocoIndex 1.1.1 with a limit of 4, at most 4 file components ran at once while up to 40 chunk calls from coco.map() ran at the same time. Bound a heavy call inside a component with a RateLimiter, a batched function's max_in_flight, or an asyncio.Semaphore.

See What counts toward the limit.

Can a small concurrency limit deadlock nested components?

No. A parent that waits for its children keeps its slot and lends it to one child at a time: the first child runs on the parent's slot, further children take free slots from the pool, and when the child on the lent slot finishes, the slot passes to the parent's next waiting child. A hierarchy of any depth completes, even with max_inflight_components=1. Counting waiting parents, the components in flight are bounded by the limit times the depth of the hierarchy.

See Slot lending: a cap that can't deadlock a hierarchy.

How do I rate-limit calls to an external API in CocoIndex?

Use a RateLimiter from cocoindex.resources.rate_limit, a token bucket with a sustained rate (max_rows_per_second) and a burst allowance. Its budget lives in the object, so provide one instance through a context key and call await coco.use_context(RATE_LIMIT).acquire() before each call. For a batched function that sends one request per batch, pass the limiter as rate_limiter=, and each batch takes one token. LiteLLMEmbedder(model, max_inflight_requests=n) caps open requests to an embedding endpoint.

See Bounds inside a component.

Can I limit concurrency by bytes or per source in CocoIndex v1?

No. The v0 flow API could cap input bytes in flight, give each source its own row budget, and cap rows in one nested iteration; v1 has none of these. v1 caps processing components per app. For inputs of very uneven size, lower max_inflight_components or split large items into child components. For workloads that need different limits, run them as separate apps, or bound the heavy step of one source with a RateLimiter or asyncio.Semaphore.

See What v1 does not have.

When should I tune concurrency settings instead of using the defaults?

Decrease the limit if the pipeline isn't stable enough (memory overuse or timeouts), or increase it if the pipeline stays stable and needs more throughput. Start with max_inflight_components, and add the more specific bounds only when needed: a RateLimiter or request cap for a rate-limited or bursty endpoint, a lower component limit for highly unbalanced item sizes, and separate apps for workloads of very different complexity.

See When to tune.

Build fresh context for your agents.

CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.