Declaring functions with #[cocoindex::function]

The #[cocoindex::function] attribute makes Rust functions participate in CocoIndex's change detection, memoization, and execution pipeline — with batching and GPU-aware runners included.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

It’s common to factor work into helper functions (for parsing, chunking, embedding, formatting, etc.). In CocoIndex, you can mark any function with #[cocoindex::function] when you want to add incremental capabilities to it. The marked function is still a normal function: its signature stays the same, and you can call it normally.

#[cocoindex::function]
async fn process_file(_ctx: &Ctx, file: FileEntry) -> Result<String> {
    Ok(file.content_str()?.to_string())
}

// Called like any other function, with the component's `Ctx`.
let text = process_file(&ctx, file).await?;

How to think about #[cocoindex::function]

Marking a function tells CocoIndex that calls to it are part of the incremental update engine. You still write normal Rust, but CocoIndex can now:

• Detect when inputs or code have changed (change detection)
• Skip work when nothing has changed (memoization)

This is what lets CocoIndex avoid rerunning expensive steps on every app.update(). See Processing Component for how marked functions are mounted at component paths.

If you don’t need any of the above for a helper, keep it as a plain Rust function.

Change detection and memoization

Every #[cocoindex::function] function participates in CocoIndex’s change detection system. With memo, the function’s results are cached and reused when nothing has changed. These two mechanisms (detecting changes and acting on them) work together to enable incremental updates. For the big-picture view (how memoization skips unchanged work across both input and code changes) see the Function memoization section in Core Concepts.

Change detection

CocoIndex detects three kinds of changes:

Logic changes — a function’s logic is its own body (or, when you set version, that integer instead of the body), plus any deps value. Tracked by #[cocoindex::function]. Logic fingerprints propagate transitively up the call chain, following runtime invocations rather than the static call graph: every #[cocoindex::function] reached during a memoized call — directly, or through plain unmarked helpers that pass the Ctx on — contributes its logic to that call. So if foo (memoized) calls bar (also #[cocoindex::function], with or without memo) and bar’s logic changes, foo’s memo is invalidated; unmarked helpers in between don’t break the chain. What the attribute buys is having a function’s own logic tracked: an unmarked helper is invisible to change detection, so editing it invalidates nothing. This is why #[cocoindex::function] matters for any function in the call chain, not just memoized ones — see Common patterns and the FAQ. To stop propagation deliberately, set logic_tracking to "self" or "none".

Input changes — the function’s arguments. Tracked by #[cocoindex::function]. When you call a function with different arguments, the fingerprints change. Input fingerprints do not propagate transitively.

Context changes — context values whose key is declared with detect_change. Tracked by ctx.get_key() at the call site — independent of #[cocoindex::function]. Context reads propagate transitively: if a memoized foo calls bar and bar reads a change-detected context value, then changing that value invalidates foo’s memo too — even though foo itself never called get_key(). What matters is whether the key was read anywhere during the memoized call, not where in the call chain.

Memoization

With memo, the function’s result is cached. On subsequent calls, if no logic, input, or read context values have changed, the cached result is reused without executing the function body — it carries over target states declared during the function’s previous invocation and returns its previous return value.

#[cocoindex::function(memo)]
async fn parse_file(_ctx: &Ctx, file: FileEntry) -> Result<Vec<Section>> {
    // Skipped when the file, the logic and the context values are unchanged
    // ...
}

Every argument of a memo function is part of the key: through its MemoInput implementation, or by its serialized value for a type that is only Serialize (such as a target handle). The return type must implement serde::Serialize and Deserialize. See Serialization.

Cache-hit semantics

When a memoized function’s cache hits, its body does not run — and neither do any nested #[cocoindex::function] calls inside it. The cached output is replayed directly: the previous return value is returned, and any target states the function declared on its previous run are carried over.

static OUTER_CALLS: AtomicUsize = AtomicUsize::new(0);
static INNER_CALLS: AtomicUsize = AtomicUsize::new(0);

#[cocoindex::function(memo)]
async fn read_version(ctx: &Ctx) -> Result<String> {
    INNER_CALLS.fetch_add(1, Ordering::SeqCst);
    Ok(ctx.get_key(version_key())?.clone())
}

#[cocoindex::function(memo)]
async fn outer_read_version(ctx: &Ctx) -> Result<String> {
    OUTER_CALLS.fetch_add(1, Ordering::SeqCst);
    read_version(ctx).await
}

// First update: both bodies run (OUTER_CALLS == 1, INNER_CALLS == 1).
// Second update with the same context value: neither counter moves —
// outer's cached value is returned directly, and read_version is not invoked.

Propagation (logic, deps, context) is recorded during the previous invocation and replayed on cache lookup. You don’t need to model the internals — change anything that affects behavior and the cache invalidates correctly.

When to memoize

Cost: A directly-called memoized function’s cache entry must store everything needed to serve a hit — the return value, plus the path of every target state it declared during the call (the state values themselves stay in the enclosing component’s tracking; the recorded paths are what keep those states alive on a hit). Large return values, or a call that declares many target states, mean large entries. (A function mounted as a component avoids the per-state paths entirely: a memoized component’s entry holds fingerprints plus its return value — () for a typical mounted declarer.)

Benefit: Memoization saves more when:

• The computation is expensive
• The function’s caller is reprocessed frequently (due to logic or input changes)

Placement: memoize at a boundary only if its key can hit while every enclosing memo boundary misses. Stacked levels pay off exactly when the outer boundary is keyed more coarsely than the inner one’s inputs. A per-file component plus a per-chunk embedding memo composes well: an edited file re-runs its component while its unchanged chunks still skip embedding.

If an enclosing memoized component’s identity already commits to this function’s inputs (for example, the component path is keyed by a hash of the content the function transforms), the inner memo can never hit on a content-driven rerun. The component memo already covers those cases. The inner memo can still help when the enclosing component re-runs because its code changed or its version was bumped. Each entry permanently stores a value, often one the target also stores, for those reruns.

A 1536-dim float32 vector is ~6 KB. An embedding memo under content-hash-keyed components therefore stores ~600 MB per 100k chunks to cover those rare reruns. Consider that storage cost when deciding whether to keep the inner memo.

Examples:

• ✅ Embedding functions — good to memoize when callers can re-run with the same text (e.g., per-file components re-running on edits). Computation is heavy; return value is fixed-size and not too large.
• ❌ Splitting text into fixed-size chunks — usually not worth memoizing. Computation is light; return value can be large.
• ✅ Processing component for files that are mostly stable between runs — memoization skips unchanged files, saving the cost of reading and processing their contents.
• 🤔 Chunk embedding when file-level memoization is already enabled — still beneficial, but less so for stable files. The benefit increases for files that change frequently, or when your code evolves (e.g., adding more features per file triggers file-level reprocessing, but unchanged chunks can still skip embedding).
• ❌ Chunk embedding inside a memoized component keyed by the chunk’s content hash — usually redundant by the placement rule: the component memo already commits to the text, so the inner memo serves only reruns caused by the component’s own code or version changes — at the cost of storing every vector twice.

Controlling change detection scope

Three arguments of #[cocoindex::function] let you customize how logic changes are detected:

• logic_tracking — controls the scope of automatic logic change detection
• version — provides explicit manual control over when dependent memos are invalidated
• deps — declares external values (e.g. a prompt const) as part of the function’s logic, so changing them invalidates dependent memos

These arguments control logic fingerprinting. Data fingerprinting (for arguments, deps values, and context values) is controlled by the objects themselves (see Memoization Keys & States). Three more tools cover what an attribute argument cannot: LogicDep / ctx.use_logic_dep for a dependency decided per call, DataDep / ctx.use_data_dep for data a body reads that is not among its arguments, and ctx.logic_propagated_to_callers for callbacks that cross a "self" boundary.

logic_tracking

The logic_tracking argument controls whether and how logic changes are detected:

• "full" (default): Track this function’s logic AND all transitively called #[cocoindex::function] functions’ logic. A change anywhere in the call chain invalidates dependent memos.
• "self": Track only this function’s own logic — its fingerprint still propagates to its callers, but the logic of functions it calls does not propagate through it.
• "none": Don’t track logic through this function at all — neither its own logic nor its callees’ reaches its callers.

These settings govern nested function calls. A mounted child component is tracked through the component tree instead, so its logic reaches the parent regardless of the parent’s logic_tracking.

version

The version argument lets you explicitly invalidate dependent memos by bumping an integer:

#[cocoindex::function(memo, version = 2)]
async fn my_fn(ctx: &Ctx, arg: &String) -> Result<String> { /* ... */ }

deps

The deps argument declares external value(s) the function logic depends on but that aren’t visible in its body — for example a prompt string or a model identifier defined at item level. When the value changes, the function’s logic fingerprint changes and dependent memos are invalidated, exactly as if the function body had been edited.

const SYSTEM_PROMPT: &str = "Summarize the document in three sentences.";

#[cocoindex::function(memo, deps = SYSTEM_PROMPT)]
async fn summarize(_ctx: &Ctx, text: &str) -> Result<String> {
    llm_call(SYSTEM_PROMPT, text).await
}

For multiple dependencies, pass a tuple:

#[cocoindex::function(memo, deps = (PROMPT, MODEL))]
async fn summarize(_ctx: &Ctx, text: &str) -> Result<String> { /* ... */ }

deps = <expr> takes one expression whose type implements MemoInput; a tuple groups several values.

The value is fingerprinted through its MemoInput implementation, as an argument is.

deps requires logic_tracking to be enabled; combining deps = <expr> with logic_tracking = "none" is a compile error.

Per-call dependencies: LogicDep and use_logic_dep

deps describes the whole function: every call depends on the value. Some dependencies are decided per call. A tree walk’s visit_file depends on the parser of that file’s language, and a change to one parser should re-run the files it parsed, not every file the walk saw. LogicDep::new declares such a value once, before the app runs; ctx.use_logic_dep records it as part of the current call’s logic:

static RUST_PARSER: LazyLock<LogicDep> =
    LazyLock::new(|| LogicDep::new(&("rust-parser", 2)).unwrap());

#[cocoindex::function]
async fn visit_file(ctx: &Ctx, file: FileEntry) -> Result<Vec<Symbol>> {
    if file.path.extension() == Some("rs".as_ref()) {
        ctx.use_logic_dep(&RUST_PARSER)?;
    }
    parse_symbols(&file)
}

LogicDep::new takes any MemoInput value. Record the dependency per call with ctx.use_logic_dep(&DEP)?.

A recorded dependency travels exactly like the function’s own fingerprint: into the entry of a memoized caller, onward through "full" callers, hidden by a "self" caller above unless the call is carried by logic_propagated_to_callers, and from a mounted component into every ancestor component’s entry. (A memoized callee that hits re-offers its stored entry to the caller whole, and that rides to the enclosing component regardless of modes, so a "self" caller may record more after a hit than after a miss, never less.) When a later release constructs the LogicDep with a new value, only the entries that recorded the old one are invalidated: in the example, a parser’s new version re-runs its files and the directories above them, and every other subtree is reused. It is recorded whatever the function’s own logic_tracking is: it describes the call, not the function’s code.

Adopting use_logic_dep in a function that memoized callers already stored entries for does not re-store those entries (the caution under logic_tracking applies): ship the adoption with a version bump or a memoization-key change on those callers.

Data dependencies: DataDep and use_data_dep

Logic, arguments and context values are all known before a body runs. Some bodies read things decided only while they run: an agent session opens the files the model chose, a test runner touches the files the tests import. DataDep::new declares a kind of such a read once, before the app runs, with a validator; ctx.use_data_dep records one read into the current call, as a key and the state observed. The next probe validates each record from the stored key and state alone, through the kind’s validator, so the entry can be reused even though nothing in its arguments covers the read.

static FILE: LazyLock<DataDep<String, (u64, Fingerprint)>> = LazyLock::new(|| {
    DataDep::new("file", |path: String, (mtime, digest): (u64, Fingerprint)| async move {
        let now = modified_nanos(&path)?;
        if now == mtime {
            return MemoStateDecision::new(&(mtime, digest), true);
        }
        let current = Fingerprint::from_bytes(&std::fs::read(&path)?);
        MemoStateDecision::new(&(now, current), current == digest)
    })
    .unwrap()
});

#[cocoindex::function(memo)]
async fn agent_session(ctx: &Ctx, task: &str) -> Result<String> {
    let path = pick_file(task);
    let mtime = modified_nanos(&path)?;
    let data = std::fs::read(&path)?;
    ctx.use_data_dep(&FILE, &path, &(mtime, Fingerprint::from_bytes(&data)))?;
    Ok(run_agent(task, data).await?)
}

DataDep<K, S> is typed by its key and state, so the validator takes them already deserialized and returns a MemoStateDecision (cocoindex::memo::MemoStateDecision). As with LogicDep, force the static before the app runs.

The validator takes the recorded key and the stored state and returns a future of Result<MemoStateDecision>: the state now observed (stored in place of the old one when it differs) and whether the entry is still valid. Renaming a kind invalidates every entry that recorded it, which doubles as its version bump. A second live kind with the same name replaces the registration and logs a warning.

Key and state are any Serialize values and are snapshotted at the call: identity and equality are fingerprints of the serialized bytes, and MemoInput is not consulted, so use plain values (the resolved path string, a tuple of mtime and digest). Recording the same kind and key again with an equal state records once; recording it with a different state makes the run store no entry, because output that used two states of one key has no single state to validate against. A body that writes what it then reads back should not record the read-back.

The rules for an entry that recorded data dependencies:

• The entry is reused only when its arguments, logic and context match and every record validates. A record whose kind is no longer registered, or whose stored key or state no longer deserializes, is a logged miss; a validator’s error propagates. A changed-but-valid state is refreshed without a run. On a miss the body’s new records replace the stored set.
• A record rides up through every enclosing function frame regardless of logic_tracking, into every enclosing memoized function’s entry and the component’s entry, where it stops. Records are not copied into ancestor components’ entries. A component reports one bit upward instead, and its parent’s entry lists the children that reported it: a component is skipped only if its own entry validates and every descendant that recorded data dependencies validates too, which the parent’s probe checks from those descendants’ own stored entries, by path, without mounting them. A changed-but-valid descendant record is refreshed in place. The probe stops, and the body re-runs, at a listed descendant with no entry (a non-memoized one, whose records are dropped, or one whose last run observed a key in two states), a record that no longer decodes, a kind no longer registered, or a validator answering invalid. Closed descendants are carried over by the reuse unvalidated, as they always were. A component that records nothing is unaffected.
• Because a parent’s probe may validate a record while the component that recorded it is not mounted, a validator runs with no component context and must depend on its key and stored state alone.
• ctx.use_data_dep returns an error outside a pipeline context, and when the function that owns the current frame has already returned (a spawned task recording after it). Await readers inside the function that started them.

Callbacks: logic_propagated_to_callers

logic_tracking is per function, so "self" hides the logic of everything a function calls — including callbacks it received as arguments. A library whose traverse is logic_tracking = "self" therefore hides the user’s visit: an edit to visit invalidates nothing, even though it is the user’s input, not the library’s implementation detail.

ctx.logic_propagated_to_callers(f) closes this gap. It is library-side; the user only puts #[cocoindex::function] on the callback. It does not make f tracked: the #[cocoindex::function] it calls and that function’s logic_tracking decide what it offers, exactly as for any function. The wrapper only carries that up to the frame that called it, through layers that would otherwise drop it.

Call it at the entry point that hands a callback down. The function that calls logic_propagated_to_callers is the callback’s owner. The returned wrapper runs f as usual; afterwards the logic f offered (by its own logic_tracking) is recorded in every memo entry between the call site and the owner’s frame (the owner’s own entry included) whatever their logic_tracking. The owner’s mode only decides what goes further up: with "full" (the default) the owner offers the callback’s logic to its callers like any other dependency; with "self" or "none" it stops there.

type Visit = Box<dyn Fn(Ctx, String) -> VisitFuture + Send + Sync>;
type VisitFuture = std::pin::Pin<Box<dyn Future<Output = Result<String>> + Send>>;

// Library: a memoized "self" function calling a callback it received.
#[cocoindex::function(memo, logic_tracking = "self", version = 1, memo_key(callback = skip))]
async fn traverse(
    ctx: &Ctx,
    value: &str,
    callback: &Arc<PropagatedCallback<Visit>>,
) -> Result<String> {
    callback.call(ctx, value.to_string()).await
}

// User: a tracked callback.
#[cocoindex::function(logic_tracking = "self", version = 1)]
async fn visit(_ctx: &Ctx, value: &str) -> Result<String> {
    Ok(format!("visited {value}"))
}

// The owner wraps the callback once and hands it down.
let callback: Visit = Box::new(|ctx: Ctx, value: String| {
    Box::pin(async move { visit(&ctx, &value).await })
});
let wrapped = Arc::new(ctx.logic_propagated_to_callers(callback)?);
traverse(&ctx, "a.rs", &wrapped).await?;

ctx.logic_propagated_to_callers(f) returns a PropagatedCallback<F>. The library function takes it as an argument, skips it in the memo key, and calls it with callback.call(ctx, ..).

The user edits visit; every traverse memo entry misses and runs again. The user never sees logic_propagated_to_callers. traverse needs no code of its own: the wrapper it receives already carries its owner.

When a library does not do this, wrap the callback in your own code before you hand it to the library: call ctx.logic_propagated_to_callers in your own function, which becomes the owner.

Rules, for whoever calls logic_propagated_to_callers:

• Mark the function the callback calls with #[cocoindex::function] for its code to be tracked. A closure that calls no marked function contributes nothing.
• The wrapper is valid only while the owner’s call is running. A call after the owner has returned returns an error instead of silently running untracked. Mount the components that call it with use_mount!, or wait for their handles’ ready() inside the owner.
• PropagatedCallback does not implement MemoInput, so a memoized function that takes one skips it with memo_key(callback = skip), as traverse does.
• Adopting the wrapper does not re-store existing memo entries (see the caution under logic_tracking): entries stored before the wrap recorded no callback logic and stay valid until their key or the memoized caller’s version moves. Ship the adoption with one such change.

Common patterns

These arguments can be set on any #[cocoindex::function] — not just memoized ones.

Fully automatic (default) — use logic_tracking = "full" (or omit it) without setting version. Any logic change in the function or its callees invalidates dependent memos.

#[cocoindex::function]
async fn full_default(_ctx: &Ctx, x: &i64) -> Result<i64> { /* ... */ }

#[cocoindex::function(logic_tracking = "full")]
async fn full_explicit(_ctx: &Ctx, x: &i64) -> Result<i64> { /* ... */ }

Manual, precise control — use logic_tracking = "self" with version. You decide what counts as a behavior change by bumping version, without being affected by implementation detail changes (performance optimizations, logging tweaks, refactoring, etc.).

#[cocoindex::function(memo, logic_tracking = "self", version = 1)]
async fn caller_self(ctx: &Ctx, test: &str, value: &str) -> Result<String> {
    records(ctx, &test, "rust", &value).await
}

Opt out of tracking — use logic_tracking = "none" for functions with a stable contract (where logic changes don’t affect output), or functions whose changes don’t affect behavior (e.g., logging, performance hints). This prevents unnecessary reprocessing when only internals change.

#[cocoindex::function(logic_tracking = "none")]
async fn untracked(_ctx: &Ctx, x: &i64) -> Result<i64> { /* ... */ }

Debugging unexpected re-runs

If a memoized function is re-running when you expect a cache hit, walk through the inputs in order:

1. Logic — did the function’s source code change? Did any #[cocoindex::function] it transitively calls change? Was a version bumped? Was a deps value edited?
2. Inputs — are the arguments byte-identical to the previous call (after canonicalization)? Custom types with unstable equality are a common source of spurious invalidation — see Memoization Keys & States.
3. Context — did any detect_change context value read during the previous invocation change? Remember this propagates transitively through nested #[cocoindex::function] calls.

If none of those changed and the function still re-runs, the most common cause is a non-stable fingerprint on a custom-typed argument or context value — derive MemoInput, or implement it by hand over the fields that identify the value, to make it deterministic.

Conversely, if a memo is hitting when you expect invalidation, common causes are:

• A logic change in a helper that is not marked with #[cocoindex::function]. Add the attribute to it (no memo needed) so its logic participates in propagation — see Common patterns.
• A deps value that you changed at runtime: deps is fingerprinted once per process, so per-request values must be passed as regular arguments instead.
• The function uses logic_tracking = "none", which opts it out of code-change detection entirely.
• The change is in a callback that passes through a logic_tracking = "self" function and nobody wrapped it with ctx.logic_propagated_to_callers.

Customizing data fingerprinting

Arguments, deps values and change-detected context values are fingerprinted through the MemoInput trait: primitives, String, containers, tuples and Option implement it, and #[derive(MemoInput)] covers your own types. For custom types, or when you need multi-level validation (e.g., check mtime first, then content hash), see Memoization Keys & States.

The per-function override is memo_key(..) on the attribute: a transform function replaces one argument’s identity, and skip leaves it out. A target handle is keyed by its serialized stable key, or left out with memo_key(table = skip); a client that is neither MemoInput nor Serialize must be skipped:

#[cocoindex::function(memo, memo_key(table = skip))]
async fn process_file(ctx: &Ctx, file: &FileEntry, table: &postgres::TableTarget) -> Result<()> { /* ... */ }

#[cocoindex::function(memo, memo_key(entry = entry_identity, client = skip))]
async fn fetch(_ctx: &Ctx, entry: &Entry, client: &ApiClient) -> Result<String> { /* ... */ }

See Override at the call site with memo_key=.

Execution capabilities

The following capabilities control how the function executes, independent of change detection and memoization.

Batching

With batching, multiple concurrent calls to the function are automatically batched together. This is useful for operations that are more efficient when processing multiple inputs at once, such as embedding models.

When batching is enabled:

• The function body receives a Vec<T> and returns a Vec<R>
• The generated function takes one T and returns one R (single input, single output)
• Concurrent calls are collected and processed together
• With memo, a cache hit is served per item and only misses enter a batch

#[cocoindex::function(memo, batching, max_batch_size = 64)]
async fn embed_batch(
    ctx: &Ctx,
    texts: Vec<String>,
    model: String,
) -> Result<Vec<Vec<f32>>> {
    ctx.get_key(&EMBEDDING_CLIENT)?.embed(texts, &model).await
}

// External usage: one input, one output.
let embedding: Vec<f32> = embed_batch(&ctx, text, model).await?;

// Concurrent calls are batched: `ctx.map` runs one call per item.
let embeddings = ctx
    .map(texts, {
        let ctx = ctx.clone();
        move |text| {
            let ctx = ctx.clone();
            let model = model.clone();
            async move { embed_batch(&ctx, text, model).await }
        }
    })
    .await?;

The max_batch_size argument limits how many inputs can be processed in a single batch.

A batching function must not call itself from its own body: that waits on the batcher which is already running the body.

Pacing batches with a rate limiter

When each batch is one request against a quota, take a token from a shared RateLimiter at the top of the batching body, with limiter.acquire(1).await?. One token then pays for one batch rather than one call.

Context values and extra parameters in a batched body

A batching body receives a Ctx, so ctx.get_key(&KEY)? works inside it, as in the embed_batch example above. The engine joins the dependencies the body reads into every caller’s memo entry.

The parameter after the &Ctx is the batched one. Every other parameter is passed through to the body unchanged and must implement Serialize: its serialized value selects the batcher, so calls share a batch only when these values are equal. With memo, each must also implement MemoInput, unless memo_key(..) transforms or skips it.

Retrying a failed batch as smaller batches

Some batch failures depend on the batch’s composition rather than any single input — e.g. a provider’s per-request token or payload limit, or one input the provider rejects. No fixed max_batch_size can rule these out. When the function body hits such a failure, return Error::retry_with_smaller_batch(cause) from it: CocoIndex halves the batch and retries each half, recursing down to single items. At size 1, cause becomes that item’s own failure — so one bad input fails only its own caller, while the rest of the batch still succeeds. There is no need to check the batch size before returning it: at size 1 the engine simply unwraps the signal and fails that caller with cause.

#[cocoindex::function(memo, batching, max_batch_size = 64)]
async fn embed_batch(ctx: &Ctx, texts: Vec<String>) -> Result<Vec<Vec<f32>>> {
    let client = ctx.get_key(&EMBEDDING_CLIENT)?;
    client
        .embed(&texts)
        .await
        .map_err(|err| Error::retry_with_smaller_batch(Error::user(err)))
}

Returning it is safe for any deterministic batch failure, even when you can’t tell whether the batch size was the cause: if the error was actually global (it would fail at any size), the recursion terminates at single items and every caller receives the underlying error, at the bounded cost of extra attempts. Only propagate directly for errors where splitting provably can’t help — global errors like invalid credentials, and transient errors (rate limits, network blips) that haven’t yet been retried at the same size. Once same-size retries are exhausted, splitting is a reasonable last resort: a smaller request may pass where the large one timed out.

Runner

The runner argument allows functions to execute in a specific context, such as a dedicated GPU runner that serializes GPU workloads.

#[cocoindex::function(runner = cocoindex::GPU)]
async fn whole_gpu(_ctx: &Ctx, item: i64) -> Result<i64> {
    // Runs while holding one whole GPU; `current_gpu()` names it.
    let device = cocoindex::current_gpu();
    // ...
}

#[cocoindex::function(memo, runner = GPURunner::fraction(0.5))]
async fn half_gpu(_ctx: &Ctx, item: i64) -> Result<i64> {
    // Two calls share one GPU.
    // ...
}

// External usage: async
let result = whole_gpu(&ctx, item).await?;

runner = <expr> takes a value of any type that implements cocoindex::Runner; it is built on the function’s first call and shared by every call. The GPU runners draw from a process-wide GPU pool:

• cocoindex::GPU takes a whole GPU per call (or per physical batch, with batching), so bodies on one GPU run one at a time
• GPURunner::fraction(f) takes a share, so several calls run at once on one GPU
• Inside the body, current_gpu(), current_gpus() and current_gpu_fraction() report the assignment
• The pool size comes from COCOINDEX_NUM_GPUS, CUDA_VISIBLE_DEVICES or nvidia-smi (one GPU by default); configure_gpu_pool(n) overrides it before the first GPU function runs

A runner of your own implements cocoindex::Runner: it receives each body as a future and decides when to run it.

struct AtMostTwo(tokio::sync::Semaphore);

#[async_trait::async_trait]
impl cocoindex::Runner for AtMostTwo {
    async fn run(&self, body: cocoindex::RunnerBody<'_>) -> Result<cocoindex::RunnerOutput> {
        let _permit = self.0.acquire().await.map_err(|e| Error::engine(e.to_string()))?;
        body.await
    }
}

#[cocoindex::function(memo, runner = AtMostTwo(tokio::sync::Semaphore::new(2)))]
async fn summarize(_ctx: &Ctx, text: String) -> Result<String> { /* ... */ }

You can combine batching with a runner:

#[cocoindex::function(batching, max_batch_size = 2, runner = GPU)]
async fn gpu_batch(_ctx: &Ctx, items: Vec<i64>) -> Result<Vec<i64>> {
    // Each physical batch, and each retried half, holds the GPU.
    // ...
}

// Concurrent calls
let calls = [5, -1, 6, 7].map(|item| gpu_batch(&ctx, item));
let outputs = futures::future::join_all(calls)
    .await
    .into_iter()
    .collect::<Result<Vec<_>>>()?;