CocoIndex processes items in parallel. In CocoIndex v1, each processing component (typically one per source item, such as a file) is a unit of concurrent work, and one setting caps how many run at the same time. Together with bounds on batches and outbound requests, it lets you tune throughput for your workload.
Processing too many items simultaneously can cause:
- Memory exhaustion: every running component holds its own data (a file’s contents, its chunks, its embeddings) in memory.
- Resource contention: CPU, disk I/O, and network bandwidth get overwhelmed by competing operations.
- System instability: timeouts, rate-limit errors from external APIs, degraded performance, or outright crashes.
CocoIndex v1 gives you:
- One cap on processing components per app,
max_inflight_components, with a default of 1024. - A cap that holds across nested components without deadlocking: a parent that waits for its children lends its slot to them.
- Bounds for the heavy steps inside components: how many batches a batched function runs at once, a shared token-bucket
RateLimiterfor outbound calls, and a request cap onLiteLLMEmbedder.
Review the concurrency control documentation or browse the CocoIndex source.
One component limit per app
max_inflight_components caps how many processing components execute at the same time in one app. Set it in coco.AppConfig, or with the COCOINDEX_MAX_INFLIGHT_COMPONENTS environment variable; the default is 1024. The AppConfig value takes precedence over the environment variable, and the environment variable over the default:
app = coco.App(
coco.AppConfig(name="MyPipeline", max_inflight_components=4),
app_main,
sourcedir=pathlib.Path("./data"),
)
export COCOINDEX_MAX_INFLIGHT_COMPONENTS=4
When every slot is in use, the next component waits until one finishes. The wait starts at the mount(), use_mount() or mount_each() call, so a parent cannot run ahead of its children without bound: a loop that mounts a child per source item advances only as fast as children finish. With max_inflight_components=1, components run one at a time.
max_inflight_components is a concurrency cap, not a memory bound. It limits how many components run, not how many bytes they hold. A few very large items can still use a lot of memory under a small limit.
What counts toward the limit
A component takes a slot. Concurrent work inside one component does not:
coco.mount_each(process_file, files.items(), ...)creates one component per file. Each one counts.coco.map(process_chunk, chunks, ...)runs one async task per chunk inside the current component. It creates no components, so the chunks of one file run concurrently under that file’s single slot.
We checked this on CocoIndex 1.1.1 (script) with 40 files mounted by mount_each(), each running 10 chunk calls with coco.map(). With max_inflight_components=4, at most 4 file components ran at a time, and up to 40 chunk calls ran at the same time. With an asyncio.Semaphore(2) around the chunk call, at most 2 ran.
So if a step inside a component is the heavy one, bound that step directly (see Bounds inside a component), or mount it as its own component so the limit covers it.
Slot lending: a cap that can’t deadlock a hierarchy
A parent component usually waits for its children: use_mount() returns the child’s result, and a component’s own target states are committed only after all its children are done. If waiting parents kept their slots, a small limit could fill up with parents and leave no slot for the children they wait for.
Instead, a parent keeps its slot while it waits and lends it to one child at a time:
- The first child a parent mounts runs on the parent’s slot.
- Further children take free slots from the pool, and their
mount()/use_mount()calls wait while the pool is empty. - When the child on the lent slot finishes, the slot passes to the parent’s next waiting child.
So a hierarchy of any depth completes, even with max_inflight_components=1. The figure below replays real runs on CocoIndex 1.1.1, recorded by a trace script: an app mounts 3 file components, and each file mounts 3 chunk components. Each file spends 30 ms before it mounts its chunks, and each chunk runs for 100 ms. A file’s dashed bar ends when its mount_each() call returns. Its last chunk can still be starting or running then, and the file component is complete only when its chunks are. Pick a limit to compare. The last option, “1, no lending”, is not a recording: it shows the deadlock that lending prevents, with the root component app_main holding the only slot while its first file waits for one.
Not a recording: CocoIndex lends a waiting parent's slot, so this cannot happen. It shows what the limit would do without lending.
The limit bounds the components that execute at the same time. Counting the parents that wait on them, the components in flight are bounded by the limit times the depth of the component hierarchy.
A few related rules:
- A component that awaits a function call, including a batched one, keeps its slot. Only mounting a child lends it.
- When several parents mount the same component at once, only the run that executes takes a slot. The others wait for it without a slot and reuse its result.
- Deleting components (cleaning up a removed subtree,
App.drop()) is bounded by the same limit.
Live components share the limit fairly
In live mode, several updates can compete for slots: a full re-index next to a refresh of a few changed files. The limit is shared fairly between independent updates (an App.update(), one cycle of a live component, or one item update of mount_each() over a live map). A component that finds a free slot takes it at once, so an update that runs alone can use the whole limit. When no slot is free, a slot that frees up goes to the waiting update that holds the fewest slots, so a small update never waits for a large one’s whole backlog.
Bounds inside a component
The heavy part of an incremental pipeline is usually one step: running a model locally, or calling a remote API. v1 bounds these steps where they happen:
- Component limit:
max_inflight_componentsonAppConfig, orCOCOINDEX_MAX_INFLIGHT_COMPONENTS. Applies to all processing components of one app. - Batches in flight:
max_in_flight=non@coco.fn.as_async(batching=True, ...). At mostnbatches of that function run at once (counted per batcher: per instance and set of extra arguments). - Batch size:
max_batch_size=non the same decorator. At mostninputs per call to the function body. - Request rate: a
RateLimiter(max_rows_per_second=...), shared through a context key or passed asrate_limiter=to a batched function. Applies to every caller that holds the same limiter. - Open requests:
max_inflight_requests=nonLiteLLMEmbedder. Applies to one embedder instance. - GPU work:
runner=coco.GPUon@coco.fn.as_async. Each call (each batch, for a batched function) takes a GPU from a pool that counts whole GPUs and fractions (coco.GPU(0.5)takes half), so on one GPU,coco.GPUcalls run one at a time. Batched functions on the runner also share one batch queue.
A RateLimiter (rate limiting docs) is a token bucket, and its budget lives in the object. To give the whole app one budget, provide one instance through a context key:
import cocoindex as coco
from cocoindex.resources.rate_limit import RateLimiter
RATE_LIMIT = coco.ContextKey[RateLimiter]("rate_limiter")
@coco.lifespan
async def lifespan(builder: coco.EnvironmentBuilder):
builder.provide(RATE_LIMIT, RateLimiter(max_rows_per_second=50.0))
yield
@coco.fn
async def fetch_one(item: Item) -> Data:
await coco.use_context(RATE_LIMIT).acquire()
return await call_external_api(item)
For a batched function that sends one request per batch, pass the limiter as rate_limiter. Each batch then takes one token, so a scarce quota is spent on full batches:
limiter = RateLimiter(max_rows_per_second=1.0)
@coco.fn.as_async(batching=True, max_batch_size=100, rate_limiter=limiter)
async def fetch_many(ids: list[str]) -> list[Item]:
return await call_external_api(ids) # one request per batch, one token each
What v1 does not have
The v0 flow API had three limits that v1 does not carry over:
- Byte limit. v0 could cap the input bytes in flight. v1 caps components, not bytes. For inputs whose sizes vary widely, lower
max_inflight_components, or split a large item into child components so each holds less. - Per-source limit. v0 could give each source its own row budget. In v1, the limit is per app: workloads that need different limits can run as separate apps, each with its own
max_inflight_components. Inside one app, bound the heavy step of one source with aRateLimiteror anasyncio.Semaphorethat only that source’s components use. - Per-iteration limit. v0 could cap rows in one nested iteration. In v1, mount nested items as components (
mount_each()) to put them under the component limit, or keep them incoco.map()and bound the heavy call they make.
Tune concurrency limits
In an incremental pipeline, the bottleneck is usually a few heavy operations, such as running a model locally or calling a remote API. Keeping more items in memory than the backend can take at once means new work is ready as soon as the backend frees up. The component limit caps how many items are held this way (a count of items, not bytes).
COCOINDEX_MAX_INFLIGHT_COMPONENTS=1024 max_inflight_components Start with the default
The default component limit (COCOINDEX_MAX_INFLIGHT_COMPONENTS: 1024) loads more items than the heavy operations can consume at once, within a bound. Change it only when one of the cases below applies.
When to tune
Lower the limit if the pipeline is unstable (memory overuse, or timeouts on some operations). Raise it if the pipeline is stable and needs more throughput.
-
Start with
max_inflight_components(orCOCOINDEX_MAX_INFLIGHT_COMPONENTS), before the more specific bounds. -
Use the more specific bounds for unbalanced workloads:
- On a rate-limited or bursty endpoint, share one
RateLimiteracross the app (or pass it asrate_limiterto the batched function that calls it), or cap open requests withmax_inflight_requestsonLiteLLMEmbedderormax_in_flighton your own batched function. - When item (file) sizes vary widely, lower the component limit, because v1 has no byte limit.
- When workloads differ widely in cost (one goes through a compute-intensive model, another only moves data), run them as separate apps with their own limits.
- On a rate-limited or bursty endpoint, share one
Set the component limit for the app, then bound the heavy calls inside it.
Source
Browse the CocoIndex source on GitHub.
For configuration examples, see the concurrency control documentation and the rate limiting documentation.
Frequently asked questions
How do you control processing concurrency in CocoIndex?
In CocoIndex v1, each processing component (typically one per source item, such as a file) is a unit of concurrent work. max_inflight_components in coco.AppConfig, or the COCOINDEX_MAX_INFLIGHT_COMPONENTS environment variable, caps how many run at once per app. When every slot is in use, the next mount(), use_mount() or mount_each() call waits until one frees up.
See One component limit per app and the concurrency control documentation.
What is the default concurrency limit in CocoIndex?
By default, up to 1024 processing components can run at the same time per app. The AppConfig value takes precedence over COCOINDEX_MAX_INFLIGHT_COMPONENTS, and the environment variable over the default. Start with the default, then lower it if your workload runs out of memory or times out, or raise it if the pipeline is stable and needs more throughput.
Does coco.map count toward max_inflight_components?
No. coco.mount_each() creates one processing component per item, and each one takes a slot. coco.map() runs async tasks inside the current component and creates no components, so they share that component's slot. In our check on CocoIndex 1.1.1 with a limit of 4, at most 4 file components ran at once while up to 40 chunk calls from coco.map() ran at the same time. Bound a heavy call inside a component with a RateLimiter, a batched function's max_in_flight, or an asyncio.Semaphore.
Can a small concurrency limit deadlock nested components?
No. A parent that waits for its children keeps its slot and lends it to one child at a time: the first child runs on the parent's slot, further children take free slots from the pool, and when the child on the lent slot finishes, the slot passes to the parent's next waiting child. A hierarchy of any depth completes, even with max_inflight_components=1. Counting waiting parents, the components in flight are bounded by the limit times the depth of the hierarchy.
How do I rate-limit calls to an external API in CocoIndex?
Use a RateLimiter from cocoindex.resources.rate_limit, a token bucket with a sustained rate (max_rows_per_second) and a burst allowance. Its budget lives in the object, so provide one instance through a context key and call await coco.use_context(RATE_LIMIT).acquire() before each call. For a batched function that sends one request per batch, pass the limiter as rate_limiter=, and each batch takes one token. LiteLLMEmbedder(model, max_inflight_requests=n) caps open requests to an embedding endpoint.
Can I limit concurrency by bytes or per source in CocoIndex v1?
No. The v0 flow API could cap input bytes in flight, give each source its own row budget, and cap rows in one nested iteration; v1 has none of these. v1 caps processing components per app. For inputs of very uneven size, lower max_inflight_components or split large items into child components. For workloads that need different limits, run them as separate apps, or bound the heavy step of one source with a RateLimiter or asyncio.Semaphore.
When should I tune concurrency settings instead of using the defaults?
Decrease the limit if the pipeline isn't stable enough (memory overuse or timeouts), or increase it if the pipeline stays stable and needs more throughput. Start with max_inflight_components, and add the more specific bounds only when needed: a RateLimiter or request cap for a rate-limited or bursty endpoint, a lower component limit for highly unbalanced item sizes, and separate apps for workloads of very different complexity.
See When to tune.
Build fresh context for your agents.
CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.

