Adaptive Batching - 5x throughput on your data pipelines

CocoIndex batches concurrent calls to a function into one call to the model or API behind it. Your code still handles one item at a time. In CocoIndex v1, the sentence-transformers and LiteLLM embedders batch by default, and any function can opt in with @coco.fn.as_async(batching=True). When batching shipped in November 2025, it made the code_embedding example about 5x faster end to end (58.9 s to 12.5 s, 79% less runtime) with the sentence-transformers/all-MiniLM-L6-v2 model. A v1 re-run of the embedding step alone measured 8.2x.

Why batching makes processing fast

When we call a function or remote API, every call has:

• Fixed overhead you pay once per call: GPU kernel launch setup, Python-to-C transitions, scheduling, memory allocator work and framework bookkeeping.

• Data-size-dependent work that scales with input: floating-point operations (FLOPs) for the model, bytes copied, tokens processed.

Doing 1,000 items one by one pays the fixed overhead 1,000 times. Doing them in batches pays that overhead once per batch, and the hardware can run the size-dependent work more efficiently (fewer launches, better cache use, fuller pipelines).

Batching helps in three ways:

• Shared fixed overhead: the batch pays for one set of GPU kernel launches and Python-to-C transitions instead of one per item.

• Larger matrix multiplies: larger batches map to dense matrix multiplies (General Matrix–Matrix Multiplication, or GEMM), which keep more of the GPU busy (higher utilization).

• Fewer data copies: fewer transfers between CPU memory and GPU memory, H2D (Host-to-Device) and D2H (Device-to-Host), so more of the time goes to computing than to moving bytes.

What batching looks like for normal Python code

Non-batching code: simple but less efficient

The most natural way to organize a pipeline is to process data piece-by-piece. For example, a two-layer loop like this:

for file in os.listdir(directory):
    content = file.read()
    chunks = split_into_chunks(content)
    for chunk in chunks:
        vector = model.encode([chunk.text])         # one item at a time
        index.upsert(file_id=file.name, chunk_offset=chunk.offset, vector=vector)

This is easy to read and reason about: each chunk flows straight through multiple steps.

Batching manually: more efficient but complicated

You can speed it up by batching, but even the simplest “just batch everything once” version has to collect the inputs, make one call, and zip the results back to their sources:

# 1) Collect payloads and remember where each came from
batch_texts = []
metadata = []  # (file_id, chunk_id)
for file in os.listdir(directory):
    content = file.read()
    chunks = split_into_chunks(content)
    for chunk in chunks:
        batch_texts.append(chunk.text)
        metadata.append((file.name, chunk.offset))

# 2) One batched call (library will still mini-batch internally)
vectors = model.encode(batch_texts)

# 3) Zip results back to their sources
for (file_name, chunk_offset), vector in zip(metadata, vectors):
    index.upsert(file_id=file_name, chunk_offset=chunk_offset, vector=vector)

Batching everything at once delays downstream steps until the batch finishes.

CocoIndex’s batching support

CocoIndex batches calls in the runtime, so your code can keep its per-item control flow. This excerpt from the v1 code embedding example embeds one chunk per call, and coco.map() runs the calls for all chunks of a file concurrently:

@coco.fn
async def process_chunk(
    chunk: Chunk,
    filename: pathlib.PurePath,
    id_gen: IdGenerator,
    table: postgres.TableTarget[CodeEmbedding],
) -> None:
    embedding = await coco.use_context(EMBEDDER).embed(chunk.text)
    table.declare_row(...)


@coco.fn(memo=True)
async def process_file(
    file: FileLike,
    table: postgres.TableTarget[CodeEmbedding],
) -> None:
    ...
    await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)

EMBEDDER is a context key that holds a SentenceTransformerEmbedder. Its embed() takes one text and returns one vector, and concurrent calls (from the chunks of one file, and from the other files in flight) join one batch.

These built-in embedders batch with no change to your code:

• SentenceTransformerEmbedder.embed() (sentence-transformers docs): up to 64 texts per batch, run on the coco.GPU runner.
• LiteLLMEmbedder.embed() (LiteLLM docs): up to 64 texts per request.

CocoIndex v0 also shipped batched ColPali image and query embedding functions. v1 has no built-in ColPali function: the v1 ColPali example calls the model in its own code, one image per call. To batch that call, wrap it in a batched function as shown below.

Batch your own function

For your own functions, enabling batching takes two changes:

• Decorate the function with @coco.fn.as_async(batching=True). Batching needs an async interface, so @coco.fn(batching=True) raises ValueError.
• Make the body take a list of inputs and return a list of outputs, in the same order.

Callers still pass one input and get one output back. For example, suppose you want to create a function that calls an API to build thumbnails for images:

@coco.fn.as_async(batching=True, max_batch_size=32)
async def make_image_thumbnail(images: list[bytes]) -> list[bytes]:
    return await thumbnail_api.create_many(images)  # one request per batch


thumbnail = await make_image_thumbnail(image_bytes)  # callers pass one image

With a stub API in place of thumbnail_api (script), CocoIndex 1.1.1 ran 100 concurrent calls as 5 batches: 1, 32, 32, 32 and 3 images. The first call found the queue idle and ran alone, each full batch of 32 started as soon as it filled, and the last 3 waited for the running batches to finish.

A few rules come with the batched body:

• Only the first parameter is batched. Other arguments pass through to the body unchanged, and calls share a batch only when those arguments are equal (so they must be hashable) and passed the same way: an explicit default and an omitted one select different batches.
• The body runs outside the calling components. One batch merges calls from many components, so coco.use_context() raises inside the body. Resolve context values in the caller and pass them as extra arguments.
• max_in_flight bounds how many batches run at once, and rate_limiter makes every batch take one token from a RateLimiter (rate limiting docs) before it runs.
• coco.RetryWithSmallerBatch splits a failed batch. Raise it from the body when a failure may depend on the batch’s contents (a provider’s payload limit, one rejected input), and CocoIndex halves the batch and retries each half, down to single items. SentenceTransformerEmbedder raises it on GPU out-of-memory errors.

See the batching documentation for more details.

How CocoIndex batches

Common approaches

To batch calls, you first accumulate them in a queue, then decide when to flush that queue as a batch. Two widely used policies are:

• Time-based (W ms): Flush whatever arrived in the last W milliseconds.

  • Pros: predictable wait bound; simple.
  • Cons: adds idle latency when traffic is sparse; needs tuning per workload.

• Size-based (K items): Flush when at least K items are waiting.

  • Pros: predictable batch size; easy to reason about memory.
  • Cons: under sparse traffic, the first queued call can wait too long; still needs tuning.

Many systems combine them (“flush when W or K triggers first”). That bounds both the wait and the batch size, but W and K still need tuning, and the right values shift with traffic.

CocoIndex’s approach

Framework level: adaptive, no timers

With the defaults (no max_in_flight, no rate_limiter), CocoIndex keeps batching simple:

• When nothing is running, a new call starts at once as a batch of 1.

• While a batch is running, new calls keep queuing.

• When the running batches finish, CocoIndex sends all queued calls as the next batch.

• There is no timer. Each batch holds the calls that arrived while the previous batch ran.

max_batch_size caps a batch; with the defaults, CocoIndex does not wait for a batch to fill. A batch that reaches the cap is sent at once, without waiting for the running batch to finish (on the coco.GPU runner it then waits for the GPU), and new calls start the next batch. max_in_flight=n makes a full batch wait while n batches run, and with a rate_limiter every batch waits for a token, even the first call on an idle queue.

The batch size follows the traffic:

• Sparse traffic: few calls arrive while a batch runs, so batches are small (often 1) and each call runs at about single-call latency.

• Busy traffic: more calls arrive during each batch, so the next batch is larger and the fixed overhead is spread over more items.

• Tuning: there is no wait time W to set. max_batch_size only caps how many inputs one call to the body gets.

The demo below runs both policies on the same calls. It models one backend that runs one batch at a time, as SentenceTransformerEmbedder does on the coco.GPU runner with one GPU, and a batch of n items takes the fixed overhead plus n times the per-item cost. The timer sends the open batch every W ms, or at once when it reaches max_batch_size. The values are illustrative, not measured. Change the call rate, the costs or W and compare the two policies on the shared timeline and in the table. A timer needs a W that suits the traffic: too short and its small batches pile up behind the backend, too long and every call waits for the tick. A well-chosen W can spend less backend time per call than the adaptive policy, because each call waits longer to join a bigger batch; the adaptive policy sends a batch as soon as the backend is free, which keeps latency low and leaves nothing to set.

Function level: each function decides how to run its batch

Each batched function receives a batch (the calls queued at that moment, up to max_batch_size) and decides how to run it on its model or library.

We use SentenceTransformerEmbedder as an example. It passes the batch (up to 64 texts) to the sentence-transformers library in one encode() call. The library splits it into micro-batches (default size: 32) so that each fits in device memory and runs at a batch size the kernels handle well. CocoIndex keeps that default.

Transformer runtimes also pad every sequence in a batch to the length of the longest one, so the GPU can run uniform kernels. Short texts then pay the cost of the longest text in the batch: mixing 64-token and 256-token items makes each 64-token item cost about 4x as much. encode() sorts the texts by length before it forms micro-batches, so each micro-batch holds texts of similar lengths and little compute goes to padding.

Other functions can send the whole batch to the backend in one request, or pack it their own way (e.g., SIMD tiles, merged writes). The framework decides when a batch is sent and which calls it holds; the function decides how to run it.

Performance evaluation

We measured batching when it shipped in November 2025, on the v0 API. We ran all benchmarks on a MacBook Pro (Apple M1 Pro, 16 GB unified memory, 2022). For each configuration:

• We ran two versions: batching on (cocoindex v0.3.1) and batching off (cocoindex v0.2.23).

• We recorded two timings: (1) end-to-end benchmark wall-clock time, and (2) the cumulative time inside the embedding function (where batching happens).

• We executed each test 5 times, discarded the fastest and slowest run to dampen outliers, and reported the mean of the remaining three runs (a 20% trimmed mean).

text_embedding and code_embedding

We ran the 2 basic examples from CocoIndex, with their inputs at the time:

• text_embedding: with 3 input files, 106 chunks in total.

• code_embedding: with 273 input files, 3,383 chunks in total.

Both embed with the sentence-transformers library, using the all-MiniLM-L6-v2 embedding model (with 22.7M parameters).

This is the evaluation outcome:

code_embedding has many more chunks (3,383 against 106), so it has more chances to batch, and its saving is larger (78.76% against 67.84%).

code_embedding on different micro-batch sizes

We varied the micro-batch size in sentence-transformers:

Raising the micro-batch size from 4 to 8 to 16 cuts end-to-end time from 23.254 s to 16.522 s to 12.812 s. Above 16 the gains shrink, because the fixed overhead per item is already low.

Sizes 64 and 128 run slightly faster (11.925 s and 11.939 s end to end), but we keep the sentence-transformers default (currently 32) and rely on the library to keep that default reasonable.

code_embedding on nomic-embed-text-v1.5 model

This run swaps all-MiniLM-L6-v2 (22.7M parameters) for nomic-embed-text-v1.5, a larger model with 0.1B parameters:

The saving drops to 4.18%, against 78.76% with all-MiniLM-L6-v2. In a larger model, the data-size-dependent work usually far outweighs the fixed overhead.

code_embedding on Ollama with all-minilm model

In this experiment, we switched from running sentence-transformers in process to calling an embedding API: Ollama’s, over HTTP. We picked the all-minilm model, which is the same model we used with sentence-transformers above.

For ease of comparison, we copy the data from running sentence-transformers with the same model from above:

Without batching, Ollama is faster than sentence-transformers (44.248 s against 58.931 s). With batching, Ollama saves only 13.35%, so it ends up much slower than sentence-transformers (38.343 s against 12.516 s).

Ollama computes embeddings for different inputs separately (Ollama code), even when they arrive in the same request. So batching saves little with Ollama, likely only the cost of the extra HTTP calls.

In v1, an Ollama model is reached through LiteLLMEmbedder (for example LiteLLMEmbedder("ollama/all-minilm")), which batches up to 64 texts per request. As long as Ollama embeds the inputs of a request one by one, batching saves HTTP calls but little compute.

code_embedding chunks on CocoIndex v1

We re-ran the embedding step on CocoIndex 1.1.1 in November 2026, on a MacBook Pro with an Apple M5 Pro (48 GB), with all-MiniLM-L6-v2 on the MPS device. The benchmark script chunks the CocoIndex repo the way the v1 code_embedding example does and keeps the first 3,383 chunks, the same count as the 2025 run. It issues one embedding call per chunk, all at once:

• on: SentenceTransformerEmbedder.embed(), the built-in batched embedder.
• off: the same model behind @coco.fn.as_async(runner=coco.GPU), one text per call.

The times cover embedding only (no database writes), as a trimmed mean of 5 runs:

The batching-off runs varied more (120.663 s to 201.464 s) than the batching-on runs (18.240 s to 22.003 s); other work was running on the machine during the runs. On v1, batching still removes most of the embedding time on a small model, as it did in 2025.

Conclusion

Batching spreads fixed overhead across multiple items and can reduce GPU launches and data transfers. The measured benefit depends on the model and API. CocoIndex v1 batches the sentence-transformers and LiteLLM embedders automatically, and @coco.fn.as_async(batching=True) batches your own functions.

Batching helps most when fixed overhead is a large share of the total work, as with small models. It also needs a library or API that runs a batch as one operation: Ollama, which embeds the inputs of a request one by one, gained 13.35%.

Source

Browse the CocoIndex source on GitHub.

About the author

George HeCofounder & CTO of CocoIndex · @georgehe0

Maintainer of CocoIndex, Ex-Google Infra Lead. Writes about incremental data infrastructure, Rust internals, and the engineering decisions behind the engine.