# CocoIndex Blog · llms-full.txt
Plain-text mirror of every published blog post on cocoindex.io.
Generated at build time; canonical URLs are below each entry.
Source: https://cocoindex.io/blogs/ · feed: https://cocoindex.io/blogs/rss.xml
---
# CocoIndex Changelog 1.0.17 - 1.0.25
URL: https://cocoindex.io/blogs/changelog-1017-1025/
Date: 2026-10-07
Summary: Changelog for CocoIndex 1.0.17-1.0.25: coco.timeout() deadlines, RetryWithSmallerBatch, target-state inspection in cocoindex show, cheaper large target updates, and fixes led by one for a Linux hang.
This is the third changelog since the v1.0 launch, covering nine releases (1.0.17–1.0.25). It leads with new APIs, headed by coco.timeout() for cooperative deadlines, coco.RetryWithSmallerBatch for splitting a failed batch, and target-state inspection in cocoindex show. A much cheaper commit path for large target updates follows, then fixes, led by one for a permanent hang on Linux, then examples and internal improvements.
CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.
## Why this release cycle matters
- **Cooperative deadlines.** coco.timeout() puts a time budget around one file or one LLM call, and every call beneath it inherits the deadline.
- **Batch splitting on failure.** A batched function can raise coco.RetryWithSmallerBatch to halve a failed batch until the bad input stands alone.
- **Target-state inspection.** cocoindex show lists every tracked target state with its owning component and reads a database directly with --db.
- **More new APIs.** LiteLLM request timeouts and concurrency caps, whole-GPU reservations, Environment.close() and recollect_after_run, child slots for connector authors, and declarative batching and schemas in the Rust SDK.
- **Cheaper large target updates.** Rows held encoded, native fingerprints, skipped reconcile() for unchanged rows, and faster Postgres statements. Engine memory per declared Postgres row drops by two thirds.
- **Fixes.** Long updates no longer hang on Linux, a failure in a merged sink batch or write transaction stays with the component that caused it, and memoized components run once under overlapping updates.
## Upgrade notes
Check these before upgrading. Each item names the release it applies from:
- **Linux, before 1.0.24.** A long update could hang permanently on LMDB's writer lock under sustained load. Upgrade to 1.0.24 or later; see the fix.
- **LiteLLM timeout bounds the whole request** (v1.0.22). LiteLLMEmbedder(timeout=30) used to forward a per-attempt value to litellm and retry for up to ten minutes; it now means thirty seconds end to end, not retried. LiteLLMTranscriber(timeout=...) means the same, and a per-call timeout to transcribe() raises TypeError. Callers who passed timeout before re-embed once, because it has left the memo key.
- **Exception handlers receive the original typed exception** (v1.0.23), not a RuntimeError wrapping a formatted string. The traceback is no longer inside str(exc); use traceback.format_exception(exc) or logging(..., exc_info=exc).
- **Composite-key deletes on user-managed Postgres tables cast to the declared key types** (v1.0.25). A managed_by="user" table whose key column is declared text but is really uuid or an enum now fails on multi-column-key deletes. Declare the real type.
- **Target action sinks** (v1.0.23). Returning a ChildTargetDef list from a sink callback still works with a DeprecationWarning; use from_fn_with_children. Equal sink callbacks must also share a type and be weak-referenceable, so a NamedTuple callback now raises TypeError.
- **Valkey** needs valkey-glide 2.5.2 or later (v1.0.25).
## New APIs and capabilities
### coco.timeout(): cooperative deadlines (v1.0.17)
coco.timeout() takes a timedelta and sets an absolute deadline for everything inside the with block (**#2251**, addressing **#2054**). You set the budget once, typically in the component that owns a piece of work, and the whole call tree beneath it inherits it: @coco.fn calls, the item tasks coco.map() fans out to, and coco.use_mount() children with subtrees of their own, with no timeout parameter threaded through any of them.
```python
from datetime import timedelta
import cocoindex as coco
from cocoindex.connectors import postgres
from cocoindex.resources.file import FileLike
@coco.fn(batching=True)
async def embed(texts: list[str]) -> list[list[float]]:
... # batched: each call inherits its caller's deadline; the shared batch body does not
@coco.fn
async def summarize(section: str) -> str:
... # inherits the caller's deadline; nothing configured here
@coco.fn
async def parse(file: FileLike) -> ParsedDoc:
... # inherits it too, through use_mount()
@coco.fn(memo=True)
async def process_file(file: FileLike, target: postgres.TableTarget[DocRow]) -> None:
# 30-second budget for this file; every call below checks it.
with coco.timeout(timedelta(seconds=30)):
parsed = await coco.use_mount(coco.component_subpath("parse"), parse, file)
summaries = await coco.map(summarize, parsed.sections)
summary = "\n".join(summaries)
vector = await embed(summary)
target.declare_row(row=DocRow(name=out_name(file), summary=summary, embedding=vector))
```
A per-item budget like this stays meaningful as the source grows, where a whole-run limit does not, and it applies however the app runs, including through the cocoindex update CLI. The budget is cooperative rather than preemptive: CocoIndex checks the deadline at its own checkpoints (entering a @coco.fn, coco.use_mount(), coco.check_cancellation()) and raises coco.DeadlineExceededError, a subclass of the built-in TimeoutError, once it has passed. An await of your own, such as an HTTP request or a database query, is generally not interrupted (the built-in retry helpers are the exception and cancel an in-flight attempt at the deadline), so keep the client library's own timeout on individual I/O calls; coco.timeout() bounds the CocoIndex work between checkpoints.
One deadline set at a component, inherited by its whole call tree. process_file(file) opens with coco.timeout(30s); solid edges carry that deadline down to use_mount(parse), coco.map(summarize) and a direct @coco.fn embed call, and a nested coco.timeout(10s) inside parse only narrows it. Dashed edges with stop marks show where the chain ends: a mount_each() child (background, never awaited), a batched function body (one call serving many callers), and target-sink writes (shared, after processing succeeded).
#### What inherits the deadline
- **@coco.fn calls**, at every depth of the tree.
- **coco.map() item tasks**, so a fan-out to a thousand items is a thousand tasks under one budget. Items are not cancelled mid-flight; each observes the deadline at its next checkpoint.
- **coco.use_mount() children**, because the caller awaits them. The child runs under the same deadline and raises at its own checkpoints; the parent sees DeadlineExceededError only after the child has completed or failed, so there is never a half-done child.
- **Nested coco.timeout() scopes** take the shorter remaining budget and restore the outer one on exit.
#### What does not
- **coco.mount() and coco.mount_each() children, and live components.** These run in the background and the caller never waits for them, so the caller's clock must not rush them; the parent's timeout bounds only the parent's own wait on the handle mount() returns. To time-box each mounted child, set the timeout inside the child's processor, as process_file does above, rather than around mount_each() in the parent.
- **Batched function bodies.** One call serves many callers with different budgets, so the body runs without any of them; the calls into it still check their own clocks.
- **Target writes (submit and sinks).** Once processing has succeeded, target synchronization is allowed to finish rather than being torn in half. A timeout ends only the component's own work.
#### Guarantees
- **No partial writes.** A processor run that times out applies no target writes and memoizes nothing; the next update retries it.
- **Completed work stays completed.** Committed components and in-progress target synchronization are never rolled back, and coco.map() items that already finished are not cancelled.
- **Long loops need a checkpoint.** A loop inside a processor should call coco.check_cancellation() so it stops near the deadline rather than at the function's return.
That also makes a whole-update budget useful for callers that are themselves time-boxed, such as a CI job or a shared cron window: with coco.timeout(timedelta(minutes=10)): app.update_blocking() indexes as much as it can in ten minutes and then raises DeadlineExceededError; everything that finished stays committed, and the next update resumes where this one left off.
#### Retries
Retry loops used to stop after a fixed number of attempts, an ad-hoc cap that does not correspond to time. The built-in retry logic now treats its own time limit as a nested coco.timeout() scope, so it merges with any ambient deadline and the shorter wins; no attempt starts past the limit and backoff sleeps never exceed the remaining time. Without an ambient deadline a helper's own limit still applies: LlmPairResolver, the LLM-based entity resolver, keeps its attempt cap. The LiteLLM embedder and transcriber expose that limit as a timeout parameter; see LiteLLM below.
### coco.RetryWithSmallerBatch: batch splitting on failure (v1.0.17)
Some batch failures depend on the batch's composition rather than on any single input: a provider's per-request token or payload cap, one input the provider rejects, a GPU running out of memory. No fixed max_batch_size rules these out, because the threshold depends on the content. A batched function body can raise coco.RetryWithSmallerBatch to have the engine halve the batch and retry the halves, recursing down to single items (**#2270**).
```python
@coco.fn(batching=True, max_batch_size=64)
async def embed(texts: list[str]) -> list[list[float]]:
try:
return await provider.embed(texts)
except ProviderError as e:
if is_transient(e) or is_auth_error(e) or is_timeout(e):
raise # global, transient and not yet retried, or a timeout: splitting can't help
raise coco.RetryWithSmallerBatch() from e
```
How RetryWithSmallerBatch isolates a failing input. A batch of eight items containing one bad input fails, so the engine splits it into two halves of four. The half without the bad input succeeds; the other fails and splits again into two pairs, then into single items. At size one the engine unwraps the signal and raises the original provider error for the bad input only, while the other seven items return results: 4 + 2 + 1, in 7 attempts.
What that gives you:
- **No size check needed.** At size 1 the engine unwraps the signal and raises the original error as that item's own failure, so raise coco.RetryWithSmallerBatch() from e is correct at every batch size. An error that would fail at any size therefore bottoms out at single items and reaches every caller, at the cost of up to about twice the batch size in extra attempts.
- **Errors land per item.** A failed sub-batch fails only the callers whose inputs were in it. One poison input no longer takes the other 63 down with it.
- **Works across subprocess runners.** The signal and its cause survive the trip through a subprocess boundary.
Re-raise, rather than wrap, errors where splitting cannot help: global ones such as bad credentials or a missing model, transient ones such as rate limits and network blips that have not yet been retried at the same size, and timeouts, since every sub-batch would start a fresh full budget against a backend that is already too slow.
Two built-in embedders do this for you. LiteLLMEmbedder raises it for errors that are neither transient, global, nor a timeout, such as a per-batch token cap: voyage-code-3 rejects batches over 120k tokens with TOO_MANY_TOKENS_IN_BATCH, which 64 long chunks can exceed. SentenceTransformerEmbedder raises it on OOM and frees the accelerator cache before retrying.
The engine now confines failures the same way on its own, with no code, in merged target-sink batches and write transactions; see the fixes.
### cocoindex show: target-state inspection (v1.0.17 to v1.0.18)
A target state is what a component declares should exist in an external system: a file in a directory, a row in a table, or an embedding in a vector store. CocoIndex records every declared target state under a stable path together with the component that owns it, and reconciles the declarations from run to run to apply only the necessary creates, updates, and deletes. That record is the ground truth for what CocoIndex has written where, and this cycle makes it readable from the CLI.
**Readable paths in the long listing.** cocoindex show -l lists each component with the target states it owns. Paths now render as the target's name and key instead of raw fingerprints (output trimmed):
```bash
$ cocoindex show main -l
Stable paths:
...
/@process_file/"bizarre_animals.md"
type:component version:1 processor:process_file
has_memoization:true target_state_count:1
Target states:
- path:/@cocoindex/localfs/[null,"output_html/data__bizarre_animals.md.html"]
```
**The inverted view.** --target-states starts from the target side instead: every tracked target state, with the component that owns it. This answers "what does CocoIndex believe exists in this target, and which component put it there?", and it is the only view that can surface an entry nothing owns anymore, for example after an interrupted run; the listing marks those [dangling].
```bash
$ cocoindex show main --target-states
Target states:
/@cocoindex/localfs/[null,"output_html/data__bizarre_animals.md.html"]
owner:/@process_file/"bizarre_animals.md"
/@cocoindex/localfs/[null,"output_html/data__chuck_norris.md.html"]
owner:/@process_file/"chuck_norris.md"
```
**Flags that compose.** --tree nests entries under their parent containers, so a table's rows sit beneath the table. --fingerprints shows the stored fingerprints instead of readable keys, for correlating with logs or the raw store.
**Without loading the app.** Pass --db --app-name (the name from your coco.AppConfig) instead of an app target to read a database directly, from any process, without importing the app module or installing its dependencies. This is how you look at a database copied out of a deployment:
```bash
$ cocoindex show --db ./cocoindex.db --app-name FilesTransform --target-states --tree
Target states:
- @cocoindex/localfs
- [null,"output_html/data__bizarre_animals.md.html"] owner:/@process_file/"bizarre_animals.md"
- [null,"output_html/data__chuck_norris.md.html"] owner:/@process_file/"chuck_norris.md"
```
When to reach for it:
- After an update, to confirm what was written and by which component.
- To find [dangling] entries after an interrupted run.
- To inspect a database on a machine that does not have the app's dependencies.
Under the hood, this took five changes from @junzh0u (**#2299**, **#2301**, **#2303**, **#2306**, **#2307**), including a fix without which --db inspection failed across processes, and a detail view about 2× faster at 10,000 target states.
### LiteLLM: request timeouts, concurrency caps, and transcription retries (v1.0.22)
Three changes affect how the LiteLLM embedder and transcriber talk to a provider. Both now share one retry and timeout implementation.
```python
from datetime import timedelta
from cocoindex.ops.litellm import LiteLLMEmbedder
embedder = LiteLLMEmbedder(
"openai/bge-m3",
api_base="http://localhost:7997",
timeout=timedelta(minutes=30), # one request end to end, retries included
max_inflight_requests=8, # open requests against this backend
)
```
**One clock per request.** LiteLLMEmbedder(timeout=...) and LiteLLMTranscriber(timeout=...) take a timedelta or a number of seconds and bound one request end to end, defaulting to ten minutes (**#2394**, **#2395**). Within that bound, fast failures such as a 429, a 5xx, or a reset connection are retried with backoff. A timeout is terminal: neither retried nor split into smaller batches. A timeout has already spent its duration, so retrying it only re-spends what is left, and splitting is worse, because every sub-batch would start a fresh full budget against a backend that is already too slow. Before this change, one slow self-hosted endpoint could turn a batch of 64 into a 64 → 32 → … → 1 split tree with a fresh ten minutes at every node. An HTTP 408 response stays retryable, since it arrives fast and spends nothing. See the upgrade notes if you already pass timeout.
**Bounded concurrent requests.** LiteLLMEmbedder(max_inflight_requests=N) caps how many requests one embedder keeps open against its backend (**#2392**, v1.0.22). Batching bounds how many texts ride in one request, never how many requests are open: 2,334 chunks from one file at max_batch_size=64 formed 38 batches, 37 of them concurrent, and a batch that splits fans out further. The permit covers one backend request only, so it is not held during a backoff sleep or across a split; the default is no cap, and the cap is per instance. See limiting concurrent requests.
**Transcription retries.** Transcription requests are retried under the same contract; they were not retried at all before. Each attempt rebuilds the audio buffer, because litellm reads it to the end and a replayed buffer would transcribe silence.
### GPU pool: whole-GPU reservations and arrival-order queueing (v1.0.25)
Since v1.0.15, a function run with runner=coco.GPU takes a whole GPU from a pool for the duration of the call, and coco.GPU(0.5) takes half of one, so two such calls share a card while a third waits. The pool size comes from COCOINDEX_NUM_GPUS, then CUDA_VISIBLE_DEVICES, then nvidia-smi, and defaults to one; coco.configure_gpu_pool(n) overrides it. Inside the function, coco.current_gpu() and coco.current_gpu_fraction() report what was assigned. See the runner section of the function guide.
v1.0.25 rewrites the pool behind that runner in Rust and exposes it as coco.GPUPool (**#2277**, closing **#2243** and **#2433**). Two capabilities are new:
- **Requests are served in arrival order.** A request that arrives while others are queued cannot take the capacity those are waiting for, so a whole-GPU request is not starved by a stream of fractional ones. Releasing capacity serves the queue from the front.
- **Whole-GPU reservations.** acquire_full(n) returns the ids of n GPUs that are completely free. It holds the ones it has as the rest free up and returns only when all n are held, so a caller never ends up with a partial set; asking for more GPUs than the pool has is an error. The coco.GPU runner still requests a fraction of one GPU per call, so this is the building block for functions that need several cards at once, such as a model sharded across GPUs, rather than a new runner form.
The rewrite also fixes waiters that were never woken when capacity was released from another event loop. The Python pool rebound one asyncio.Condition to whichever event loop touched it last; the Rust pool signals each waiter through its own channel. It was reported from a cocoindex-code deployment on 1.0.22, with a reproduction that needs no GPU; see the other fixes table. Two limits are unchanged: subprocess mode (COCOINDEX_RUN_GPU_IN_SUBPROCESS=1) does not yet support more than one GPU (**#2382**), and a function with both batching=True and the GPU runner still serializes its batches through one queue.
### Smaller additions (v1.0.18 to v1.0.25)
- **Environment.close()** releases the LMDB environment explicitly, so the same database path can be reopened without waiting for every Python reference to be collected (**#2481**, v1.0.25). On free-threaded Python 3.14 a thread started inside a component inherited the component's context for life, which kept the environment open and made reopening fail with EnvAlreadyOpened. LazyEnvironment.stop() now closes its environment. See closing an environment.
- **recollect_after_run for memo states.** A memo state function can return recollect_after_run=True in its MemoStateOutcome to have the state collected again after the function body re-runs, for states that depend on what the body read (**#2474**, v1.0.25). Without the flag the stored state is the one the check returned before the run, which is right for states read from outside, such as a modification time. Functions only for now; a memoized component raises ValueError for it. See state that depends on what the function did.
- **SurrealDB: declare_fields and flexible record IDs** (**#2439**, v1.0.25). TableTarget.declare_fields lets more than one component own fields of the same record, and record IDs and relation endpoints may be strings, integers, or arrays. The same change fixes statement errors that were silently swallowed; see the correctness fixes.
- **Syntax-aware splitting for Dart and Lua.** RecursiveSplitter gained tree-sitter grammars for both (**#2316**, **#2370**), bringing grammar-backed splitting to 38 languages; .dart and .lua files previously fell back to separator-based splitting. The supported-languages table was audited and corrected at the same time (**#2367**). Unknown language= values fall back to separator splitting rather than raising, so if chunk quality looks off, check the value against the table ("csharp", not "c_sharp").
### For connector authors (v1.0.21 to v1.0.25)
Several contracts in the custom target connector API changed this cycle. Shipped connectors are all migrated. The container-deletion contract is now documented too; see documentation.
- **Child slots replace aligned child lists** (**#2407**, v1.0.23). A container sink used to return a list of child handler definitions index-aligned with the batch; only the length was checked, so a wrong index attached a child to the wrong target state silently, and a leaf-only batch through a shared sink had to return a None per row. Container sinks are now built with TargetActionSink.from_fn_with_children or from_async_fn_with_children, whose callback receives child_slots, a mapping with an entry only for actions that carry a child provider, and fulfills each one. TargetActionSink takes one type argument, so one shared sink serves a container level and its leaf level without casts. The old shape keeps working with a DeprecationWarning; see the migration section.
- **Value-based sink identity, now type-keyed** (**#2368**, v1.0.21; **#2404**, v1.0.23). Sinks built from callbacks that compare equal share one batching identity, so writes to the same resource from different components are batched and, for transactional targets, committed together, without a hand-kept registry of sinks. Equality alone was not safe, since a NamedTuple equals any same-shaped tuple and two connectors' sinks could merge. Canonicalization now also requires the same type, and a callback that cannot be weak-referenced raises TypeError at construction; use a frozen dataclass, with weakref_slot=True if it uses slots=True.
- **Handlers and sinks see equal copies of plain-data values** (**#2469**, v1.0.25). The copy is taken at declaration, so mutating a value after declaring it is not observed.
- **tracks_value_fingerprint = True** (**#2472**, v1.0.25). A handler whose tracking record is exactly fingerprint_object(value) can set this class attribute to let the engine skip reconcile() for unchanged values, as described under performance. A handler that opts in but tracks something else never matches, so it never skips.
- **reconcile() may run more than once per commit** (**#2475**, v1.0.25). It runs inside the pre-commit callback, which is re-invoked when an LMDB batch re-runs after a map grow or after a body ahead of it fails. Only the attempt that commits is applied, so reconcile() must be free of side effects.
### Rust SDK: declarative batching, context keys, and row-type schemas (v1.0.25)
The Rust SDK needed more hand wiring than the Python SDK for batching, context resources, and connector schemas. **#2287** makes those three declarative:
- **#[cocoindex::function(memo, batching)]** gives a Rust function item-shaped batching with per-item memoization and body-hash invalidation, so cache misses are coalesced into batches while hits are served item by item.
- **context_key!** declares a context key in one line, replacing hand-written LazyLock> definitions.
- **SchemaFields and TableSchema::from_row** derive a connector schema from a Rust row type, with the vector dimension resolved at runtime from the embedder. Duplicate columns and unresolved dimensions fail early.
```rust
cocoindex::context_key!(static DB: postgres::Database);
cocoindex::context_key!(static EMBEDDER: SentenceTransformerEmbedder, detect_change);
#[derive(Clone, Serialize, Deserialize, SchemaFields)]
struct DocEmbedding {
id: i64,
filename: String,
text: String,
#[coco(vector)]
embedding: Vec,
}
async fn app_main(ctx: Ctx) -> Result<()> {
let dim = ctx.get_key(&EMBEDDER)?.dimension();
let schema = postgres::TableSchema::from_row::(["id"])?
.with_vector_dim("embedding", dim)?;
// ...
}
```
The Rust quickstart and the Rust examples teach the new forms across Postgres, Qdrant, Turbopuffer, and LanceDB.
## Performance
### Large target updates: less memory, fewer reconciles (v1.0.25)
A component that declares a million rows used to cost the engine about a kilobyte per row in Python objects until it committed, and every update called reconcile() once per row whether or not the row had changed. Four changes in v1.0.25 take most of that out. None of them changes a stored fingerprint or tracking record, so nothing re-runs after the upgrade.
The commit path for one declared row in v1.0.25. declare_row(value) in the component, then encode (plain data to bytes, held encoded and released after pre-commit), then fingerprint (a native walk, same value as before and about nine times faster), then compare with the tracking record. An unchanged row takes the green path: skip reconcile(), no per-row Python at commit. A new, changed, or removed row is decoded, reconciled into an action, and sent to the sink, which upserts through INSERT ... SELECT FROM unnest(...) with one array parameter per column and 2,000 rows per statement.
#### Rows are held encoded and released early
The engine holds every declared target state from declaration until the component commits, and every action reconcile() returns until the sink has applied it. In the Python profile both were the Python objects themselves: a row dict of fifteen short strings and ints costs about 1.2 KB, so a component declaring 1.14 million rows held about 1.33 GiB of row dicts, and the map of declared states stayed alive through the final commit although nothing read it after pre-commit (**#2469**).
Plain-data values are now encoded at declaration: dicts with string keys, lists, tuples, scalars, and the standard library and numpy types all round-trip exactly, and a dict row stores its values plus one interned reference to its key tuple, so column names cost nothing per row. The declared-state map is dropped as soon as pre-commit is done. Dataclasses, pydantic and msgspec models, handles, and encodings over 64 KiB stay as objects.
#### Smaller costs on the commit path
- **Native fingerprints.** fingerprint_object, which connectors call once per declared row at commit, and memo fingerprints now walk plain data natively instead of canonicalizing it in Python first (**#2470**). The result is identical to before, pinned by golden values in the tests, so existing tracking records and memo keys stay valid. Anything that is not plain data falls back to the Python canonicalizer as a whole.
- **Postgres upserts through unnest.** Tables whose columns are all builtin scalars, or arrays of them, are upserted with one array parameter per column, INSERT ... SELECT ... FROM unnest($1::t1[], ...) ON CONFLICT ..., 2,000 rows per statement, instead of a multi-row VALUES list whose statement text grew with the row count (**#2471**). Column types come from the catalog, read once per table, so a managed_by="user" table whose declared types differ from the real ones keeps working. Tables with pgvector, enum, or domain columns stay on VALUES; parsing the vectors dominates either statement.
- **Postgres composite-key deletes.** Rows with multi-column keys were deleted with an OR chain of per-key conjunctions, which Postgres plans with time that grows quadratically in the key count; a full chunk took about a minute, so a pass deleting many rows could stall for hours. The delete now matches (c1, c2, ...) IN (VALUES ...), one join against the key list (**#2458**). The caveat from the upgrade notes applies: placeholders are cast to the declared key types.
#### Unchanged rows skip reconcile() in the table targets
The row handlers of Postgres, SQLite, Doris, Snowflake, BigQuery, LanceDB, Neo4j, and FalkorDB now tell the engine that their tracking record is exactly the value's fingerprint (**#2472**). For those targets the engine fingerprints the declared value natively and calls reconcile() only for rows that are new, changed, removed, or possibly missing. An update where most rows are unchanged runs no per-row Python at commit. Qdrant, Valkey, SurrealDB, Turbopuffer, and zvec fingerprint a derived value rather than the row, so they do not opt in yet, nor do index and attachment handlers; they behave exactly as before. full_reprocess, a provider schema change, and a value that is not plain data still go through reconcile().
| Measurement | Before | After |
|---|---:|---:|
| Engine memory per declared Postgres row | 1,449 B | 487 B |
| Fingerprint, 15-column row with two list columns | 17.8 µs | 1.95 µs |
| 200,000 unchanged rows at commit, no-op sink | 2.34 s, 200,000 reconcile() calls | 1.34 s, 0 calls |
| Postgres upsert, 631,119 rows, ten connections | 5.1–5.6 s | 2.8–3.2 s |
| Postgres delete, 8,000 four-column keys | 59.1 s | 0.26 s |
## Fixes
### Long updates no longer hang on Linux (v1.0.24)
Under sustained load, a long App.update() on Linux could stop making progress and never return, with every later write blocked behind it (**#2426**, fixing **#2424**). LMDB ties a write transaction to the OS thread that began it. The engine opened the transaction and then awaited the batched bodies on the multi-thread runtime, so a body that suspended could resume on another worker thread. The commit then released the writer lock from the wrong thread, the release failed silently, and the next write waited forever.
Triggering it needed a body that actually suspended inside the transaction, which took a large coalesced batch or a component with more than about 125 target states, so it surfaced in production rather than in tests. It never reproduced on macOS, where LMDB uses a different lock. Each write transaction now runs on one OS thread from open to commit, whatever a body awaits. One side effect: a batch whose callers have all been cancelled now always commits, where before it committed or rolled back depending on timing. If you run on Linux, upgrade to 1.0.24 or later.
### Failures stay with the component that caused them (v1.0.23, v1.0.25)
The engine merges work in two places that user code never sees: actions from every component that finishes while a sink call is in flight go into the next call, and concurrent write-transaction bodies share one LMDB transaction. Both merges are pure optimizations, and a failure inside a merged unit used to fail everyone in it. Reviewing a shared Postgres sink made the cost concrete: a mount_each fan-out of 60 components produced one sink call with 59 rows, one poisoned row rolled back the other 58 components' writes, and app.update() returned without raising, because mounted children fail independently.
Both layers now follow, automatically, the rule that coco.RetryWithSmallerBatch gives batched functions: a failure reaches only the callers whose inputs caused it.
#### Target sink batches (v1.0.23)
A sink receives one flat action list and cannot see component boundaries, and the runner behind it was all-or-nothing, so a failed merged call fanned one error out to every component in it. The runner now returns one result per component (**#2406**). A batch spanning several components is applied whole first. If that fails, it is bisected along component boundaries, never inside a component, whose actions stay one atomic unit, and each half is re-applied, recursively. An error reaches only the components in a failing sub-batch, and a component that still fails alone gets its own error.
A merged sink batch of eight components, A through H, fails as one call. The engine splits it between components: A to D commit, E to H fail; then G and H commit, E and F fail; then E commits and F fails alone with its own error. Seven of eight components commit in seven sink calls. The happy path is still one sink call and one transaction; the split never crosses a component boundary; at most 2n minus 1 sink calls for n components.
The happy path is unchanged: one sink call, one transaction. The failure path costs at most 2n − 1 sink calls for n components, about 2·log₂(n) when a single component is bad. Cancellation and deadline errors are not retried, since no particular component caused them, and connector code is untouched.
#### Write transactions (v1.0.25)
reconcile() runs inside pre-commit, which runs inside a batched write-transaction body. A connector rejecting one component's declared value therefore failed the pre-commit of every component batched with it, including components of other apps sharing the same Environment, each reporting a copy of an unrelated error (**#2451**). The runner now reports one result per body. When a body fails, the transaction is aborted, since LMDB has no savepoints, and the error goes to that body's caller only. The bodies ahead of it ran cleanly, so they run again and commit in a transaction of their own; the bodies behind it continue in the next transaction, still in call order. Each body runs at most twice however many of its batch mates fail, and nothing changes on the happy path.
#### Where a failure is delivered
Component exception handlers now receive the component's original Python exception instead of a RuntimeError wrapping a formatted string, and an exception a handler raises keeps its type through await handle.ready() (**#2383**, v1.0.23). A handler installed with coco.exception_handler around coco.mount_each(fn, items) now receives the items' failures when items is a LiveMap, as it already did for a list; coco.auto_refresh items are covered the same way (**#2478**, v1.0.25).
### Memoization under concurrency and redefinition (v1.0.23 to v1.0.24)
Two classes of spurious re-execution are fixed. Neither changes a memo key or a stored entry.
#### Concurrent same-key component runs execute once
Two runs of one memoized component with the same key that both started before either had stored a memo, for example two overlapping App.update() calls on an app whose main function is memoized, both executed the body (**#2412**). The engine now holds the component's build permit through the memo store, and a run queued on the permit re-checks the memo a same-key run just stored and reuses it. A failed run stores nothing, so the queued run executes as before. The same reuse applies under full_reprocess, scoped to the current update: a memo stored earlier in the same update is reused, while memos left by previous updates are still ignored (**#2414**).
#### Redefinitions and copies no longer drop a memo
A @coco.fn(memo=True) function registers its logic fingerprint in a process-global registry and releases it when the object is collected. Two live objects can share one fingerprint whenever identical code is defined twice: a re-run notebook cell, importlib.reload, or a factory that defines the function on each call. Dropping the old object removed the fingerprint for both, and the surviving function's memo entries then failed the logic check and re-ran, paid embedding calls included. The registry is now reference-counted (**#2405**), and copy.copy and copy.deepcopy of a coco function return the function itself rather than an unregistered copy whose deletion had the same effect (**#2409**).
### Correctness fixes
- **SurrealDB statement errors are raised** (**#2439**, v1.0.25). A statement error inside a batch was silently swallowed by query(); it is now raised. The same change sends values as typed CBOR parameters, writes records with UPSERT ... MERGE so fields no longer declared are unset, bounds transactions at 500 actions with a retry on conflict, makes DDL idempotent, and keeps vector indexes HNSW only, since MTREE was removed in SurrealDB 3. Its new APIs are listed under smaller additions.
- **Replacing a component with a deeper one no longer deletes the deeper component's rows** (**#2379**, v1.0.22). When a component at a/b from a previous run was replaced by a/b/c, the demotion of a/b was meant to remove only its own target states; it removed every child row under it as well, and the next sweep then deleted target states a/b/c had written in the same update.
- **Live mode terminates even when a child's exception is retained** (**#2385**, v1.0.22). A component counted as active while any strong reference to it existed, and an exception's traceback holds such a reference, so appending a child's exception to a list in a handler kept update_blocking(live=True) from ever returning. Activity is now an explicit count of in-flight tasks.
- **LiteLLM embeddings are aligned to inputs by index** (**#2381**, v1.0.22). A provider or proxy that returned items out of order attached embeddings to the wrong texts with no error. The memo version is deliberately unchanged, so vectors cached while misaligned are not invalidated automatically; if you suspect a proxy reordered responses, run a full_reprocess.
- **Schema changes under full_reprocess, in the table and graph targets.** These connectors silently skipped a schema change (a table column, or a node or relationship property in the graph targets) that arrived in an update run with full_reprocess=True. Row writes then failed on the missing column, yet later updates reported success, and the target stayed out of sync. The gap was in the connectors' reconcile and apply paths rather than in the engine, and it is fixed in Postgres, SQLite, Doris, Snowflake, BigQuery, and SurrealDB (**#2346**), LanceDB (**#2352**), and Neo4j and FalkorDB (**#2362**).
- **Failed DDL is reported, not recorded as applied.** SQLite no longer treats a genuine ADD COLUMN failure as an idempotent no-op (**#2328**), and Doris does the same for ADD COLUMN and DROP COLUMN (**#2359**), so the tracked schema cannot drift from the real table.
- **Memoization preserves target invalidation** (the same **#2346**). Before, if a memoized function declared rows and the target's provider changed in a way that should rewrite them, a cache hit could skip the rewrite. Provider changes are now part of what a cache hit validates.
- **Recreated container target states no longer replay stale deletes** (**#2365**). Recreating a container, for example a partition after its parent table's primary key changed, could replay deletes from the old layout against its children, failing the component or silently dropping fresh rows.
- **Memoization keys with Fingerprint values are order-independent** (**#2334**). Dictionaries and sets of Fingerprint keys built in different orders now produce the same key instead of a spurious cache miss.
#### Other fixes
| Fix | Impact | Release |
|---|---|---|
| Recursively strip NUL from Postgres array and composite bindings | Strings containing U+0000 inside text[] or composite values no longer fail with ValueError: string cannot contain NUL | v1.0.17 |
| Clear the inverted target-owner index on component deletion | No permanently dangling owner rows after a component is dropped | v1.0.19 |
| Avoid blocking the event loop during iterator cleanup | Async source iterators no longer stall the event loop while their producer thread exits | v1.0.21 |
| Parse Windows app target paths | C:\project\main.py:app2 is no longer parsed as module C | v1.0.21 |
| Preserve tuple and bytes path segments through state-store decoding | Components keyed by a tuple or by bytes no longer leak or misattribute target-state ownership; rows already leaked are not repaired | v1.0.23 |
| Collect preview actions once when a write batch re-runs | Preview no longer reports an action once per LMDB map grow | v1.0.25 |
| Purge Valkey document hashes when an index is deleted | Un-declaring an index or running App.drop() no longer leaves the index's document hashes behind | v1.0.25 |
| Tag Valkey connections with LIB-NAME cocoindex | CocoIndex clients are identifiable in CLIENT LIST; requires valkey-glide ≥ 2.5.2 | v1.0.25 |
| Stop recursive localfs walks at symlink cycles | A directory containing a symlink back to itself or a parent no longer yields an endless stream of keys | v1.0.25 |
| Return empty bytes for zero-length and empty-object S3 reads | read(0) and partial reads of an empty object no longer send a range request S3 rejects | v1.0.25 |
| Wake GPU pool waiters across event loops | A function waiting for a GPU on one event loop no longer sleeps forever when capacity is released on another, which could leave a process idle with embeddings pending | v1.0.25 |
## Examples
### Slides-to-speech, local narration and per-slide incrementality
The slides-to-speech example now narrates locally on the CPU with no TTS API key: Kyutai's Pocket TTS replaces Piper, and a typed DSPy signature drives the vision model, which still needs LLM credentials (Gemini by default) (**#2262**). It is also incremental per slide rather than per deck (**#2267**): editing one slide re-runs the vision call, synthesis, and embedding for that slide only, and the chosen TTS voice is a tracked input, so switching voices re-narrates every slide.
### Meeting notes to a knowledge graph on SurrealDB
A new example builds a knowledge graph from meeting notes into SurrealDB (**#2329**), joining the Neo4j and FalkorDB versions of the same example (the walkthrough covers the Neo4j one).
## Internal improvements
### Documentation
- Memoization placement: where memo=True belongs relative to component structure, what a cache entry costs, and that full_reprocess re-runs every memoized function and component, paid LLM and embedding calls included (**#2363**).
- LMDB map auto-resize and logic-fingerprint propagation, including the gotcha that setting version= on a @coco.fn replaces automatic source-change tracking, so later code edits stop invalidating until you bump it (**#2364**).
- A dedicated rate limiting page (**#2265**), mount_each() return semantics (**#2327**), connection resolution at action time in the custom target connector guide (**#2291**), and the LiteLLM shared behavior section (**#2396**).
- **Container deletion, destroy versus abandon** (**#2418**, v1.0.24). The docs now spell out an existing contract that connector authors depend on. Once a container is no longer declared, its own removal runs and its children are not visited individually. A container handler therefore either destroys, for containers CocoIndex owns (DROP TABLE, rmtree), or abandons, for user-managed ones such as Kafka and Iggy topics and managed_by="user" tables, which CocoIndex stops writing to and leaves untouched, with no tombstones or deletion messages. App.drop() follows the same rule; the target-state guide and the connector guide now say so.
### Security and dependencies
- **pyo3 0.27 → 0.29** plus lockfile dependency bumps, resolving open security advisories (**#2292**, **#2293**).
- **surrealdb ≥ 3.2.3** (**#2315**), raising the floor for a quinn-proto denial of service (CVSS 7.5) and two ammonia XSS advisories in the optional SurrealDB feature; the lockfile bump that moved quinn-proto to 0.11.16 and ammonia to 4.1.4 shipped in v1.0.20 with **#2355**.
- **Dependency upgrade sweep** clearing the remaining high-severity alerts; two low-severity, upstream-blocked ones remain (**#2355**).
## Summary
This cycle adds a time budget a pipeline sets once and every foreground call inherits, a way to turn a batch failure into a split-and-retry, and a CLI view of what CocoIndex has written where, along with new APIs in LiteLLM, the GPU pool, the connector API, and the Rust SDK. It takes most of the per-row cost out of committing a large table. Underneath, long updates no longer hang on Linux, failures in merged sink batches and write transactions stay with the component that caused them, memoized components run once under overlapping updates, and the table and graph targets apply schema changes correctly under full_reprocess, which is what keeps a pipeline converging rather than reporting success while drifting.
For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.
## Thanks to the community
Thanks to everyone who contributed this cycle, including eleven first-time contributors.
### @junzh0u
Thanks @junzh0u, a first-time contributor, for making target state legible: fixing cross-process show --db, rendering tracked paths readably, adding the --target-states listing, persisting names for provider-only segments, and a faster detail view.
### @Sujit-1509
Thanks @Sujit-1509 for carrying the schema-evolution fix across connectors: LanceDB and then Neo4j and FalkorDB, plus deterministic Fingerprint canonicalization, recursive NUL stripping in Postgres array and composite bindings, stopping localfs walks at symlink cycles, and zero-length and empty-object S3 reads.
### @hardness1020
Thanks @hardness1020, a first-time contributor, for wiring the tree-sitter Lua grammar, then for three correctness fixes: aligning LiteLLM embeddings to inputs by index, preserving typed exceptions in component exception handlers, and preserving tuple and bytes path segments through state-store decoding.
### @tomz-alt
Thanks @tomz-alt for cooperative timeout support: coco.timeout(), DeadlineExceededError, and the inheritance rule that separates foreground work from background and shared work; and for the Rust SDK ergonomics release: declarative batching with per-item memoization, context_key!, and schemas derived from row types.
### @prrao87
Thanks @prrao87 for rebuilding slides-to-speech: local CPU TTS with a DSPy vision signature, then moving the incremental boundary to the slide with the TTS voice as a tracked context value.
### @martinschaer
Thanks @martinschaer, a first-time contributor, for the SurrealDB connector overhaul: raised statement errors, typed CBOR values, flexible record IDs, UPSERT ... MERGE writes with declare_fields, bounded transactions, and HNSW-only indexes for SurrealDB 3.
### @YaxinCheng
Thanks @YaxinCheng, a first-time contributor, for rewriting the GPU pool in Rust: a fair arrival-order queue, whole-GPU reservations through acquire_full(n), and a fix for waiters never woken when capacity was released from another event loop.
### @suju-droid
Thanks @suju-droid, a first-time contributor, for clearing the inverted target-owner index on component deletion and for preserving SQLite schema consistency on failed column migrations.
### @ca-ke
Thanks @ca-ke, a first-time contributor, for wiring the tree-sitter Dart grammar, so .dart files get syntax-aware splitting instead of the separator fallback.
### @AmirF194
Thanks @AmirF194, a first-time contributor, for making the Doris connector re-raise genuine column DDL failures instead of swallowing them, including tests that exercise the real control flow without a live cluster.
### @AbhayMahalle
Thanks @AbhayMahalle, a first-time contributor, for tagging Valkey connections with the CocoIndex library name, so they can be told apart in the server's CLIENT LIST.
### @aeonframework
Thanks @aeonframework, a first-time contributor, for raising the surrealdb floor to ≥ 3.2.3 for two security advisories.
### @justavibedev
Thanks @justavibedev, a first-time contributor, for keeping the event loop unblocked during async iterator cleanup.
### @yhz5613813
Thanks @yhz5613813, a first-time contributor, for parsing Windows app target paths, so a drive-letter colon is no longer read as the app-name separator.
---
# CocoIndex Changelog 1.0.8 - 1.0.16
URL: https://cocoindex.io/blogs/changelog-108-1016/
Date: 2026-07-07
Summary: Changelog for CocoIndex 1.0.8-1.0.16: persistent per-component state, LiveMap, rate limiting, batched target writes, BigQuery and Snowflake connectors.
This is the second changelog since the v1.0 launch, covering nine releases (1.0.8–1.0.16). The focus shifts from initial hardening into **new engine primitives for stateful, production-grade pipelines**, a major expansion of **cloud and enterprise connectors**, and continued depth on **agent ergonomics**.
CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.
## Why this release cycle matters
For developers building retrieval, extraction, and knowledge-graph pipelines, the most significant changes are:
- **Persistent per-component state.** coco.use_state() lets pipeline components carry state that survives across runs, opening the door to accumulators, counters, and stateful deduplication without an external store.
- **In-memory intermediate collections.** LiveMap is a live, in-memory keyed collection that bridges a producing and a consuming stage of the same run: one side declares entries, the other consumes them through mount_each, which grows and shrinks one live component per entry as the data changes — a producer/consumer hand-off that pure declarative pipelines can't express.
- **Batched target writes.** Row-level writes to the same target from different components can now be coalesced into one apply, collapsing N write transactions into one — a direct win for connectors like LanceDB and cloud warehouses where per-transaction cost dominates.
- **Token-bucket rate limiting.** RateLimiter is now a first-class engine primitive: a consistent, cooperative throttle shared across pipeline components, replacing hand-rolled sleep pacing at each call site with one fair, budgeted rate.
- **Enterprise cloud connectors.** BigQuery, Snowflake, and Azure Blob Storage land alongside sparse-vector Qdrant support and a new Valkey vector store connector, completing coverage of the major cloud data platforms.
## Engine
### coco.use_state(): persistent per-component state
coco.use_state() gives any pipeline component a persistent, typed state slot that survives across update cycles. The state is stored in the LMDB state store alongside other engine metadata, keyed by a StableKey (not just a plain string, as of **#2085**). A type_hint parameter added in v1.0.15 enables typed deserialization so components can round-trip structured state objects cleanly.
Why it matters: the incremental engine already knows *what* changed between runs. use_state lets components know *what they have already done*. This enables accumulators (running totals, seen-entity sets), watermark trackers, and stateful deduplication, patterns that previously required an external Redis or Postgres sidecar. The net effect is that a wider class of stateful logic can now live inside the pipeline declaration, subject to the same incremental reconciliation semantics as the rest of CocoIndex.
Two update cycles of a component calling use_state. In cycle 1 the state prefetch misses and returns the initial value 0, then commits 1 to the LMDB state store. In cycle 2 the prefetch hits and returns 1, then commits 2. The counter value persists across cycles in the store, so the component remembers what it already did.
### LiveMap: in-memory intermediate collections
LiveMap is a new in-memory keyed collection that bridges a producing and a consuming half of the same live run. Producing components *declare* entries into it as target states (LiveMap.declare_entry), and the consuming side reads that same in-memory dict as a live source through mount_each. Where coco.stats_group provides observability over a data slice, LiveMap provides *computation* over one: a mutable intermediate collection the pipeline both writes and reads, decoupled through the shared map.
Because LiveComponent classes can now be used inside mount_each(), the consuming side is a dynamic live component tree — one live component per entry, mounted and dropped as producers add and stop declaring keys.
How LiveMap bridges producing and consuming components. On the left, producing components declare entries into LiveMap — adding and updating keys, and dropping a key when its producer stops declaring it. LiveMap, a keyed collection held in memory for the live run, exposes those entries as a live source. On the right, each key drives one live component through mount_each, so the component tree grows and shrinks with the data: components are mounted, re-run, or dropped as keys appear, change, and vanish.
### Token-bucket RateLimiter
RateLimiter is now a first-class engine primitive implementing a token-bucket algorithm. It provides a cooperative throttle shared across pipeline components: any component that acquires a token from the same RateLimiter instance participates in the same budget, and concurrent callers are served FIFO so none starve. This replaces hand-rolled sleep pacing at each call site with one budgeted, coordinated rate.
Combined with the existing retry-on-429 (HTTP rate limit) behavior added in earlier cycles, RateLimiter gives pipelines deterministic, budget-aware flow control rather than reactive backpressure alone.
Before and after the token-bucket RateLimiter. Before, each integration hand-rolls its own sleep loops and 429 retries, producing uncoordinated bursts and reactive backpressure against the external API. After, every component calls acquire() on one shared token bucket that refills at max_rows_per_second up to a burst_window capacity, so requests leave at a steady, budgeted rate; components wait when the budget is empty, and retry-on-429 remains only for the unexpected.
### Preview mode for update actions
Update actions can now be previewed before they are applied. A pipeline run can describe the adds, reprocesses, and deletes it *would* perform, without writing to any target, enabling dry-run validation and change auditing before committing to a production store.
Preview mode as a dry run. An update() call with preview enabled produces a plan listing 3 adds, 2 reprocesses, and 1 delete, computed but not applied. A blocked, dashed arrow marked "no write" leads to a dashed, untouched target store, so a run reports what it would change without writing a single row to production.
### Batched target writes across components
Row-level writes to the same target from different components can now be coalesced into a single batched apply. The common shape is mount_each(..., table_target), where many child components each declare one row into the same target — but the coalescing is keyed by the target *sink*, not by any sibling relationship, so any components whose writes land on the same sink concurrently are batched together. Previously each component submitted its leaf actions directly to the sink after precommit, so the connector saw a stream of single-row calls even though its API already accepts a batch. In practice a 50-row ingest into LanceDB produced roughly 50 merge_insert commits — and 50 table versions — instead of one (**#2219**).
An app-scoped batcher, keyed by the target sink, merges concurrent commits that hit the same sink before the connector's apply call (**#2221**). It reuses CocoIndex's existing Batcher primitive rather than a bespoke queue, so leaf-target batching shares the same path as the engine's other batched work and adds no debounce latency to a lone write. Batching preserves order, each output mapping back to its input; container targets, which return per-action child handlers, stay on the direct apply path, while the batched path handles leaf writes. Sink lifetime drives cleanup: a keeper owns the batcher one-to-one and drops its registry entry the moment the sink goes away, so a pipeline with an unbounded number of sink functions cannot leak batcher entries.
Why it matters: for connectors with real per-operation cost, such as LanceDB and cloud warehouses, write cost is dominated by the number of write transactions — round-trips and commits — not by row count. Coalescing N single-row applies into one batched apply collapses N write transactions into one, cutting that overhead directly. For a store like LanceDB it also collapses N table versions into one, which keeps version history and compaction manageable under heavy fan-out.
Batched target writes across mount_each children. Before, three child components each apply a single row to LanceDB, producing three commits and three table versions. After, the same three rows pass through a batcher and land as one apply, producing a single commit and one table version.
### Engine performance improvements
- **Prefetch fn-memos and user-states in one read transaction**: the engine now batches all LMDB reads for function memos and user states into a single read transaction per cycle, replacing N individual reads (**#2076**).
- **Minimize serialization for UserStateCache**: reduces round-trip cost for state reads during hot-path execution (**#2127**).
- **LMDB map auto-resize**: the state store now automatically resizes the LMDB map when it fills, eliminating the manual LMDB_MAP_SIZE tuning that previously required operator intervention (**#2231**).
- **LMDB map size rounded to page size**: map sizes are now aligned to the OS page boundary, preventing subtle allocation failures on some Linux kernels (**#2109**).
### Other engine fixes
- **coco.fn callable outside a component context**: coco.fn-decorated functions can now be called directly in scripts and tests without being inside a mounted pipeline component (**#2101**).
- **!-prefixed negation in excluded_patterns**: source file filters now support !pattern to explicitly un-exclude a path that matched an earlier pattern, following the .gitignore-style rule ordering (**#1901**).
- **Retry transient LiteLLM embedding errors**: transient failures (network blips, upstream 5xx) during embedding calls are retried automatically (**#2090**).
- **Reject rows with missing or null primary keys**: the engine now fails fast rather than silently writing corrupt rows (**#2239**).
## Connectors
### Target: Snowflake (v1.0.15)
A native Snowflake target connector lands, bringing CocoIndex's declare-target-state model to one of the most widely deployed cloud data warehouses. Snowflake targets participate in the same incremental reconciliation as all other CocoIndex targets: rows for deleted sources are removed automatically, and only changed rows are upserted.
### Target: BigQuery (v1.0.16)
A Google BigQuery target connector was added alongside the Snowflake connector in the same cycle. For organizations already running Google Cloud infrastructure, this completes cloud data warehouse coverage across the major platforms.
### Source: Azure Blob Storage (v1.0.16)
An Azure Blob Storage source connector mirrors the OCI and S3 source APIs. For teams on Azure infrastructure, this enables the same live-mode, incremental bucket-watching behavior available to AWS and OCI users.
### Target: Valkey vector store (v1.0.12)
Valkey, the open-source Redis fork under Linux Foundation governance, gains a target connector with vector store support. For organizations that have standardized on Valkey over Redis, this provides a fully supported vector index target without requiring a separate vector database.
### Qdrant: native sparse vector support (v1.0.16)
The Qdrant connector was upgraded with native sparse vector support and eager point-ID validation. Sparse vectors are essential for hybrid search pipelines that combine dense (semantic) and sparse (BM25/TF-IDF) representations, making CocoIndex's Qdrant integration first-class for hybrid retrieval.
### LanceDB: vector and FTS index support (v1.0.11)
LanceDB gained explicit vector and full-text search (FTS) index management. The drop_index operation was added alongside a dependency bump to LanceDB 0.33.0 and PyArrow 23.0.0. Nullable schema-only row rewrites (a correctness bug that appeared on partial schema changes) were fixed in both the Python and Rust layers. In v1.0.15, LanceDB table maintenance was tied to LanceDB's own stats API, making compaction scheduling driven by actual table metrics rather than a fixed interval.
### zvec: FTS field support (v1.0.14)
The zvec target connector gained full-text search field support, allowing pipelines to write both vector and FTS fields to zvec in a single target export.
### OCI: skip startup scan on unchanged logic (v1.0.11)
The OCI Object Storage source now detects when pipeline logic is unchanged since the last run and skips the expensive full-bucket startup scan, going directly to live-stream event processing. For large buckets, this can reduce cold-start time from minutes to seconds.
### Local file system: periodic rescan in live mode (v1.0.15)
The local file system source in live mode now periodically rescans and recreates the watcher if it becomes stale. OS file-system watchers (inotify, kqueue) have known limitations under high churn, network mounts, and certain editors; periodic rescan provides a safety net that catches any events the watcher missed.
### New connector summary
| Connector | Type | Release |
|-----------|------|---------|
| Snowflake | Data warehouse target | v1.0.15 |
| BigQuery | Data warehouse target | v1.0.16 |
| Azure Blob Storage | Object store source | v1.0.16 |
| Valkey | Vector store target | v1.0.12 |
| Qdrant sparse vectors | Enhanced vector target | v1.0.16 |
| LanceDB vector + FTS indexes | Enhanced vector target | v1.0.11 |
| zvec FTS fields | Enhanced target | v1.0.14 |
## Agent & developer experience
### cocoindex show: full-fledged inspection tool (v1.0.12)
cocoindex show was upgraded from a basic state dump into a full inspection tool for pipeline debugging. It provides structured, human-readable output of pipeline state, component configuration, and target schema: the production debugging workflow that previously required querying the LMDB or Postgres state store directly.
### Python error messages
Runtime errors crossing the Rust-to-Python boundary now use Display (human-readable) rather than Debug (Rust internals) formatting, making error messages that surface in Python stack traces significantly more actionable.
### Multiple GPU and fractional GPU allocation (v1.0.15)
GPUPool was added to support multiple GPUs and fractional GPU allocations within a pipeline. For embedding-heavy pipelines running on multi-GPU infrastructure, this allows work to be distributed across available GPU resources without hand-partitioning the pipeline.
## Example walkthroughs
v1.0.13 shipped a major documentation expansion: 17 new example walkthroughs (10 full + 7 variants) covering the breadth of CocoIndex use cases:
- **Meeting Notes → Knowledge Graph**, with Neo4j and FalkorDB variants
- **CSV → Kafka** streaming pipeline
- **Semantic Search over PDFs** with mixed embedding models
- **Search Images by Text**, a multimodal CLIP walkthrough
- **Face Recognition**, ported from v0 to v1
- **Docs to Knowledge** graph construction
Each walkthrough was structured for agent followability, with consistent heading patterns, copyable commands, and explicit prerequisites, so AI coding agents can execute them as step-by-step procedures.
## Bug fixes and correctness
| Issue | Impact | Release |
|-------|--------|---------|
| S3 ETag encoded to bytes for content fingerprint | Prevents false cache invalidation on binary ETags | v1.0.10 |
| StableKey::Bytes serde round-trip via msgpack | Correct key serialization for binary primary keys | v1.0.14 |
| Postgres: preserve data and NOT NULL on column type evolution | Schema evolution no longer drops non-null constraints | v1.0.15 |
| Postgres: schema-qualify DROP INDEX for vector indexes | Avoids dropping wrong-schema indexes in multi-schema deployments | v1.0.15 |
| Logic deps propagated across mount boundary | Dependency tracking correct for nested mounted components | v1.0.12 |
| Orphaned child-existence rows when Directory→Component | Prevents stale rows when a directory node becomes a component node | v1.0.15 |
| LanceDB: escape single quotes in delete filter predicates | Prevents filter injection on document paths containing apostrophes | v1.0.11 |
| LanceDB: nullable schema-only row rewrites | Correct behavior on partial schema updates | v1.0.15 |
| Idle batchers cleared to avoid stale accumulation | Prevents write amplification on quiet pipelines | v1.0.8 |
| PEP 695 type aliases in numpy 2.5 NDArray | Fixes analyze_type_info crash on numpy 2.5+ | v1.0.13 |
## Summary
This cycle's theme is **depth and breadth**: depth in the engine (stateful components, in-memory intermediate collections, GPU pools, performance batching) and breadth in connectors (completing cloud warehouse coverage with Snowflake, BigQuery, and Azure Blob, plus Valkey and enhanced Qdrant). The through-line is the incremental engine growing new primitives — persistent state, live intermediate collections, cooperative rate limiting, batched writes — that bring a wider class of stateful, production-grade pipelines inside a single declarative CocoIndex app.
For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.
## Thanks to the community
Thanks to everyone who shipped this cycle, core maintainers and community alike, including twelve first-time contributors.
### @georgeh0
Thanks @georgeh0 for the bulk of this cycle's engine work: LiveMap, the token-bucket RateLimiter, LiveComponent support inside mount_each, and the shared-parse code infrastructure behind cocoindex-code.
### @badmonster0
Thanks @badmonster0 for the documentation updates.
### @hakuuww
Thanks @hakuuww for building coco.use_state(), persistent per-component state, then following it with engine-side performance work: prefetching fn-memos and user states in one read transaction and minimizing UserStateCache serialization.
### @sdhilip200
Thanks @sdhilip200 for a prolific connector cycle: the Snowflake and BigQuery targets, the Azure Blob source, and Mermaid diagram rendering in the docs.
### @Sujit-1509
Thanks @Sujit-1509 for LanceDB vector and FTS indexing, plus correctness fixes: preserving data and NOT NULL constraints during Postgres column-type evolution and reconciling orphaned child-existence rows.
### @prrao87
Thanks @prrao87 for deep LanceDB work: tying table maintenance to LanceDB's own stats, implementing drop_index, and fixing nullable schema-only row rewrites, plus batching leaf target actions.
### @lichuang
Thanks @lichuang for GPUPool, supporting multiple and fractional GPU allocations, and for adding type_hint to use_state for typed state deserialization.
### @Haleshot
Thanks @Haleshot for the initial zvec target integration and adding FTS field support to it.
### @nuthalapativarun
Thanks @nuthalapativarun for switching runtime errors crossing into Python to Display formatting, adding Qdrant and local file system connector unit tests, and fixing broken documentation links.
### @daric93
Thanks @daric93 for the Valkey vector store target connector, a fully supported vector index target for teams standardized on Valkey.
### @prithvi-moonshot
Thanks @prithvi-moonshot for upgrading cocoindex show into a full-fledged inspection tool for debugging.
### @MrAnayDongre
Thanks @MrAnayDongre for preview mode for update actions, letting a run describe the adds, reprocesses, and deletes it would perform before touching a target.
### @pyjuan91
Thanks @pyjuan91 for making the OCI source skip its full startup scan when pipeline logic is unchanged, cutting cold-start time on large buckets.
### @tomz-alt
Thanks @tomz-alt for native sparse vector support and eager point-ID validation in the Qdrant connector, making it first-class for hybrid retrieval.
### @jordigilh
Thanks @jordigilh for adding periodic rescan and watcher recreation to the local file system source in live mode, a safety net for events an OS watcher misses.
### @MaxRong
Thanks @MaxRong for making the engine reject rows with missing or null primary keys instead of silently writing corrupt rows.
### @slliland
Thanks @slliland for making the state store automatically resize the LMDB map when it fills, removing the manual LMDB_MAP_SIZE tuning operators used to need.
### @SuperMarioYL
Thanks @SuperMarioYL for !-prefixed negation in excluded_patterns, so a path matched by an earlier rule can be explicitly un-excluded.
### @davidmyriel
Thanks @davidmyriel for documenting Tigris alongside MinIO as a supported S3-compatible service.
### @dashitongzhi
Thanks @dashitongzhi for improving the docs CI with proper triggers and build verification.
---
# Index Your Codebase for AI Agents with CocoIndex V1
URL: https://cocoindex.io/blogs/index-codebase-v1/
Date: 2026-06-10
Summary: Index a codebase for RAG and AI coding agents with CocoIndex V1 and Tree-sitter: language-aware chunking, embedding, and a live vector index in async Python.
In this blog, we'll build a live semantic index over a codebase with CocoIndex V1. Point it at a repo, and you get a vector index you can search in natural language ("where do we embed chunks?") that updates itself as you edit. It's the kind of fresh, low-latency code context a coding agent needs, and it's about 100 lines of plain async Python.
CocoIndex has built-in support for codebase chunking, with native Tree-sitter parsing and live updates.
The heavy lifting (incremental processing, change tracking, managed targets) runs in a Rust engine underneath. With a source that watches for changes, like the local filesystem in live mode, the index tracks your edits as they happen: save a file, and only the chunks that changed get re-embedded and re-upserted.
## Use case
- Code context for agents: semantic context for Claude, Codex, OpenCode, Factory.
- Code search: natural-language and semantic search over your repo.
- Review & refactor agents: context for code review, security analysis, and large-scale refactoring.
CocoIndex Use Case
## Why CocoIndex for codebase indexing
A codebase is hard to keep indexed well, and it exercises most of what CocoIndex was built for:
Why CocoIndex for codebase indexing: syntax-aware chunking, incremental updates, live updates, a Rust core, plain Python, and a consistent embedder across indexing and query.
- Chunks follow real code structure (functions, classes, blocks) instead of arbitrary line windows. Tree-sitter parsing is built in for every major language.
- Code changes constantly: every commit, save, and rebase mutates a handful of files. CocoIndex re-embeds only the chunks that changed, and with live=True plus cocoindex update -L it keeps watching and applies each edit as it happens.
- The engine is Rust. The run below indexes 622 files in 43 seconds cold; after that, a save costs one file.
- The pipeline is ordinary async Python and your own types. Pick your embedding model, chunking strategy, and vector database; if your agent can write Python, it can extend this flow.
## Overview
CocoIndex Flow for Code Embedding
From a high level, these are the steps:
1. Read code files from a local directory.
2. Split each file into syntax-aware chunks with Tree-sitter, then embed every chunk.
3. Store the chunks and their embeddings in Postgres (as **target states**).
With CocoIndex, you declare the transformation logic with native Python, without worrying about handling updates.
Think: **target_state = transformation(source_state)**.
When your codebase changes, or your processing logic changes (a different embedding model, a new chunk size), only the difference is reprocessed. The Incremental updates section walks through exactly what gets recomputed when.
## Setup
In this example we store the indexed rows in Postgres with the pgvector extension. Your metadata and the vector search index both live there. CocoIndex supports many targets, so you can pick the one that works best for you.
1. If you don't have Postgres running, use the Docker Compose file in the cocoindex repo to start one.
```sh
docker compose -f dev/postgres.yaml up -d
```
2. Point CocoIndex at it
```sh
export POSTGRES_URL="postgres://cocoindex:cocoindex@localhost/cocoindex"
```
3. Install CocoIndex and the dependencies
```sh
pip install -U "cocoindex[postgres,sentence_transformers]" asyncpg pgvector numpy python-dotenv
```
## Define the App
Apps are the top-level runnable unit in CocoIndex. An app names your pipeline and binds a main function with its parameters.
### Define the data and shared resources
This block sets up two things the rest of the code builds on. CodeEmbedding defines one row of the output table: each chunk of code becomes one row, with its text, location, and embedding vector. coco_lifespan provides the shared resources every step needs, the Postgres connection pool and the embedding model, built once at startup instead of per file.
```python
import os
import pathlib
from dataclasses import dataclass
from typing import AsyncIterator, Annotated
import asyncpg
from numpy.typing import NDArray
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter, detect_code_language
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.chunk import Chunk
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.id import IdGenerator
DATABASE_URL = os.getenv("POSTGRES_URL", "postgres://cocoindex:cocoindex@localhost/cocoindex")
TABLE_NAME = "code_embeddings"
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
PG_DB = coco.ContextKey[asyncpg.Pool]("code_embedding_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
_splitter = RecursiveSplitter()
@dataclass
class CodeEmbedding:
id: int
filename: str
code: str
embedding: Annotated[NDArray, EMBEDDER]
start_line: int
end_line: int
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
```
### Define file processing
With the row schema and shared resources in place, we can write the actual work: turning a file into rows.
#### Process file
```python
@coco.fn(memo=True)
async def process_file(
file: FileLike,
table: postgres.TableTarget[CodeEmbedding],
) -> None:
text = await file.read_text()
language = detect_code_language(filename=str(file.file_path.path.name))
chunks = _splitter.split(
text,
chunk_size=1000,
min_chunk_size=300,
chunk_overlap=300,
language=language,
)
id_gen = IdGenerator()
await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)
```
process_file runs once per file. It reads the file, detects the language so Tree-sitter can parse it, splits the code along the syntax tree, and maps each chunk to process_chunk.
Each chunk is a coherent syntactic unit, so retrieval returns whole functions or blocks, never a fragment cut mid-statement. All major languages are supported (Python, Rust, JavaScript, TypeScript, Java, C++, and more); unknown types fall back to plain text.
One processing component per file: each file is chunked with Tree-sitter and embedded, producing CodeEmbedding rows.
Note that @coco.fn with memo=True is what makes this incremental: if a file's content and this function's code are both unchanged, the whole file is skipped on the next run. coco.map fans out to one process_chunk call per chunk.
Here is what chunking produces: each file is split into syntactic chunks, each with its own location and text.
Each file split into chunks, with the location and text of every chunk
#### Process chunk
Next, we embed the chunk with the shared embedder and declare the target row.
```python
@coco.fn
async def process_chunk(
chunk: Chunk,
filename: pathlib.PurePath,
id_gen: IdGenerator,
table: postgres.TableTarget[CodeEmbedding],
) -> None:
embedding = await coco.use_context(EMBEDDER).embed(chunk.text)
table.declare_row(
row=CodeEmbedding(
id=await id_gen.next_id(chunk.text),
filename=str(filename),
code=chunk.text,
embedding=embedding,
start_line=chunk.start.line,
end_line=chunk.end.line,
),
)
```
We use SentenceTransformerEmbedder with all-MiniLM-L6-v2; there are 12k+ sentence-transformer models on Hugging Face, so swap in whichever you prefer. Note chunk.start.line and chunk.end.line: V1 carries line numbers through, so search results point straight at the lines that matched.
### Define the main function
mount_each fans out one processing component per file, from the codebase source to the Postgres target.
app_main wires the source to the target. It mounts the Postgres table (with a vector index), walks the codebase, and mounts one processing component per file.
```python
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
target_table = await postgres.mount_table_target(
PG_DB,
table_name=TABLE_NAME,
table_schema=await postgres.TableSchema.from_class(
CodeEmbedding, primary_key=["id"],
),
)
target_table.declare_vector_index(column="embedding")
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(
included_patterns=["**/*.py", "**/*.rs", "**/*.toml", "**/*.md", "**/*.mdx"],
excluded_patterns=["**/.*", "**/target", "**/node_modules"],
),
live=True, # watch for changes; pass -L to `cocoindex update` to run live
)
await coco.mount_each(process_file, files.items(), target_table)
```
We walk the codebase from sourcedir (point this at the repository you want to index). We index files with the extensions .py, .rs, .toml, .md, .mdx, and skip dot-directories, target, and node_modules.
mount_table_target creates and manages the Postgres table for you: schema, the pgvector index, idempotent upserts, and orphan cleanup when a file disappears. live=True makes the filesystem source watch for changes, and mount_each runs one component per file so the engine can track and update them independently.
### Create the App
A CocoIndex App binds source, transform, and target state into one runnable unit.
Bind app_main into a coco.App and point it at the codebase root.
```python
app = coco.App(
coco.AppConfig(name="CodeEmbeddingV1"),
app_main,
sourcedir=pathlib.Path(__file__).parent / ".." / "..", # index from repo root
)
```
That is the entire indexing path.
## Run the pipeline
Run the cocoindex CLI to set up and update the index. Choose catch-up (scan, sync, exit) or live (catch up, then keep watching):
```sh
# Catch-up run
cocoindex update main
# Live run: keep watching for file changes
cocoindex update -L main
```
You'll see the index update state in the terminal:
```text
✅ app_main: 1 total | 1 added
✅ declare_target_state_with_child: 1 total | 1 added
✅ _MountEachLiveComponent.process: 1 total | 1 added
✅ process_file: 622 total | 622 added
⏳ Elapsed: 43.4s
```
## Query the index
We match user text against the index with a plain SQL query, reusing the *same* embedder from the indexing flow so indexing and querying stay consistent.
```python
async def query_once(pool, embedder, query: str, *, top_k: int = 5) -> None:
query_vec = await embedder.embed(query)
async with pool.acquire() as conn:
rows = await conn.fetch(
f"""
SELECT filename, code, embedding <=> $1 AS distance, start_line, end_line
FROM "{TABLE_NAME}"
ORDER BY distance ASC
LIMIT $2
""",
query_vec, top_k,
)
for r in rows:
score = 1.0 - float(r["distance"])
print(f"[{score:.3f}] {r['filename']} (L{r['start_line']}-L{r['end_line']})")
print(f" {r['code']}")
print("---")
```
The operator is pgvector's cosine distance. We turn it into a similarity score and print the filename, the matched line range, and the code snippet. Run it against your index:
```bash
python main.py "embedding"
```
The search results print in the terminal:
Search results in terminal
## Incremental updates
You never compute a diff or write update logic: you change something, and CocoIndex works out what to embed, upsert, and delete. @coco.fn(memo=True) decides what to *recompute*: a file is skipped when its content and the function's code are both unchanged, and an embedding is reused when its chunk text is unchanged. mount_table_target decides what to *write*: each row's id is derived from its chunk's content, so only rows that actually changed are upserted, and rows whose source is gone are deleted.
The same machinery covers two kinds of change: changes to your **data** (the code being indexed) and changes to your **logic** (the pipeline itself).
**Data changes.**
- **A file is added**: only that file is chunked and embedded, and its rows are inserted. The rest of the repo is untouched.
- **A file is deleted**: its rows are removed from the target.
- **A file is edited**: the file is re-chunked, and the new chunks usually overlap the old ones. Chunks whose text is unchanged keep their id and embedding, so they are left as-is; genuinely new chunks are embedded and inserted; chunks that no longer exist are deleted. Edit one function and only that function's chunks are re-embedded, even though the whole file was re-read.
A file edited and re-chunked: unchanged chunks are reused with no re-embedding, a removed chunk's row is deleted, and a new chunk is embedded and inserted.
**Logic changes.** A pipeline change is reconciled the same way: CocoIndex compares the new output against what is already in Postgres and applies only the difference.
- **Change the file patterns** (included_patterns / excluded_patterns): files that now match are added automatically; files that no longer match have their rows deleted automatically.
- **Tune the chunking** (chunk size, overlap): files are re-chunked, producing the same partial-overlap diff shown above: unchanged chunks are no-ops, new chunks are embedded and inserted, dropped chunks are deleted.
- **Swap the embedding model**: the vectors genuinely change, so all embeddings are recomputed, but row identity is stable: it is an in-place update of the embedding column, with no rows added or removed.
A catch-up run (cocoindex update main) does this once and exits; live mode (cocoindex update -L main) keeps watching and applies each change as you code.
## CocoIndex Code
If you'd rather not wire the pipeline yourself, CocoIndex Code is an end-to-end implementation of exactly this indexing, packaged as a CLI and an MCP server. It does the same thing the example above does: AST-aware chunking, incremental re-index on file changes, local embeddings by default.
CocoIndex Code: semantic code search for coding agents, as a CLI and MCP server
You can plug it straight into your coding agent or code-review agent:
- **Claude Code skill:** npx skills add cocoindex-io/cocoindex-code, then invoke /ccc.
- **MCP server:** claude mcp add cocoindex-code -- ccc mcp (Codex, OpenCode, Cursor, and any MCP client work the same way).
- **CLI:** ccc index to build the index, ccc search "where we embed chunks" to query it.
The agent gets the whole repository as semantic context instead of reading one file at a time.
## Next steps
The full, runnable example is in the CocoIndex repo: examples/code_embedding, with a walkthrough in the index-codebase example.
- Point sourcedir at your own repository.
- Swap the embedding model or the target store; the pipeline is plain Python.
- Read the CocoIndex V1 docs for more on incremental processing, components, and targets.
Issues and contributions are welcome on GitHub.
---
# CocoIndex Changelog 1.0.1 - 1.0.7
URL: https://cocoindex.io/blogs/changelog-101-107/
Date: 2026-06-01
Summary: CocoIndex's first post-v1 releases: stable memoization keys, scheduled live refresh, scoped stats, safer SQL connectors, and more integrations.
CocoIndex reached v1.0 at the start of this cycle.
This is the first changelog since the v1 launch, covering seven releases (1.0.1–1.0.7). The focus is on making the engine easier to operate in production pipelines, then connecting it to more of the systems where teams already store data.
CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.
## The short version
The engine work is the headline this cycle. If you run long-lived, expensive, or shared pipelines:
- Per-argument memoization keys let you exclude clients, loggers, and debug flags from the cache key, so recomputation depends only on the inputs that matter.
- coco.auto_refresh turns "poll this source every few minutes" into a live component with consistent error handling and target-state reconciliation.
- coco.stats_group(...) breaks the adds / reprocesses / deletes counts down by data slice (per tenant, project, or folder) instead of one aggregate per processor.
- New connectors: OCI Object Storage and Apache Iggy as sources; Turbopuffer, Neo4j, and FalkorDB as targets; LanceDB gained background compaction and in-place schema evolution.
- Security and correctness fixes, including SQL identifier injection hardening and a concurrency race that only appeared under real database latency.
## Engine
### Per-argument memoization control
@coco.fn now accepts a per-argument **memo_key**, giving you fine-grained control over which arguments participate in the memoization cache key.
Production functions take more than data: clients, handles, config objects, loggers, tracing context, debug flags. Those values change across runs even when the meaningful input is identical, and each change could invalidate the cache and rerun expensive transforms, embeddings, or LLM calls for the wrong reason. memo_key keeps the cache keyed to the semantic input.
The API maps parameter names to either a callable (transform the value before fingerprinting) or None (exclude the parameter entirely):
```python
@coco.fn(memo=True, memo_key={"entry": lambda e: (e.name, e.version), "extra": None})
def transform(entry: SourceDataEntry, extra: str) -> str:
...
```
- **Callable** → applied to the argument; its return value is fingerprinted in place of the original.
- **None** → the parameter is excluded from the memo key; changing it never invalidates the cache.
- **Not listed** → fingerprinted normally.
How memo_key builds the cache key: the entry argument is fingerprinted through a lambda, extra is excluded with None, and any unlisted argument is fingerprinted as-is, so changing a handle like extra does not invalidate the cache
Read more in Memoization keys & states → Override at the call site (**#1888**, **#2000**).
### coco.auto_refresh for live components
A common live-mode pattern is "run this processor on a fixed schedule." coco.auto_refresh now makes that pattern first-class.
Without an engine-level primitive, every pipeline hand-rolls the same loop and re-decides how errors propagate and how missing rows get reconciled. coco.auto_refresh wraps any processor function as a live component that re-runs on an interval and routes cycle failures through one error-handling channel.
```python
import datetime
import cocoindex as coco
@coco.fn
async def app_main(db, target) -> None:
await coco.mount(
coco.auto_refresh(sync_users, interval=datetime.timedelta(minutes=5)),
db, target,
)
```
Each cycle reconciles target states against the previous cycle. If sync_users stops declaring a row, CocoIndex deletes the corresponding target automatically. See Live components → periodic refresh (**#1995**).
### Scoped stats reports
By default, CocoIndex already breaks stats down **by processor**: each entry function (process_doc, process_code, …) reports its own counts: adds, reprocesses, deletes. What it can't do by default is break those counts down across **data slices**. coco.stats_group(title) opens a scope where everything mounted inside aggregates into a **separate** report under title, split out of the parent (no double counting).
Seeing that one tenant did 600 reprocesses while another did 1,150 fresh adds tells you who is churning, who is growing, and where an unexpected reprocess storm came from. One aggregate per processor hides all of that. The natural slice is the data: per tenant, per project, or per source folder.
```python
@coco.fn
async def app_main(tenants, target):
for tenant in tenants:
# same processor, but stats scoped per tenant
with coco.stats_group(f"tenant:{tenant.id}", report_to_stdout=True):
files = localfs.walk_dir(tenant.docs_dir, ...)
await coco.mount_each(process_doc, files.items(), target)
```
What coco.stats_group changes in the report: by default every processor reports one bucket of adds, reprocesses, and deletes that mixes all tenants together, while stats_group emits a separate report per data slice (tenant:acme vs tenant:globex) so you can see each slice's adds, reprocesses, and deletes (acme growing, globex churning) split out of the parent with no double counting
The block also yields a handle with the same stats() and watch() methods as UpdateHandle, so you can stream a single slice's counts to a dashboard. See Progress monitoring → Scoped reports (**#2042**).
### Other engine improvements
- **Configurable logging**: set the Python logger level directly via the COCOINDEX_LOG_LEVEL environment variable (**#2035**).
- **Cleaner Rust internals**: the workspace was unified and the core SDK isolated from PyO3, separating the engine from its Python bindings and laying the groundwork for native Rust consumers of the engine (**#1973**).
## Built-in operations
### LiteLLM speech-to-text
CocoIndex added **speech-to-text (STT)** support through LiteLLM (**#1889**), so audio sources can be transcribed inside a pipeline using any LiteLLM-backed STT provider. A LiteLLMTranscriber wraps the transcription API:
```python
from cocoindex.ops.litellm import LiteLLMTranscriber
transcriber = LiteLLMTranscriber("whisper-1")
transcript = await transcriber.transcribe(audio_file)
```
### Code splitter: eight new languages
The RecursiveSplitter gained tree-sitter support for **eight new languages** this cycle, so syntax-aware chunking now covers a much wider slice of real codebases and config:
- **Svelte** and **Vue**: component-aware chunking for frontend code (**#1937**)
- **Julia**: for scientific and numerical codebases (**#1942**)
- **Elm** (**#1955**) and **Astro** (**#1984**)
- **Bash, CMake, and HCL**: shell scripts, build files, and Terraform/infra configs (**#1954**)
Pass the language name to language= (or let detect_code_language() infer it from a filename):
```python
from cocoindex.ops.text import RecursiveSplitter
splitter = RecursiveSplitter()
chunks = splitter.split(source_code, chunk_size=2000, chunk_overlap=200, language="svelte")
```
### Entity resolution: parallel by default
Entity resolution (dedup-and-canonicalize a set of extracted entity names) now resolves independent connected components of the candidate graph in parallel, speeding up LLM-backed deduplication on large entity sets (**#2006**).
## Connectors
### Source: OCI Object Storage with live bucket watching
A new **Oracle Cloud Infrastructure (OCI) Object Storage** source mirrors the S3 source's API for scanning a bucket, and adds an optional live mode.
It's a good illustration of the **stream–state duality** in CocoIndex's source model. A bucket is *state*: list_objects() scans it and establishes the full set of objects. A change feed is a *stream*: when you supply a live_stream, CocoIndex does the initial scan once and then keeps watching, applying createobject / updateobject / deleteobject events incrementally instead of rescanning. For OCI those events flow through OCI Streaming, consumed over the Kafka protocol; an event-time cutoff with a 5-second clock-skew tolerance decides which streamed events predate the scan. Under the hood it's built on a new lower-level LiveStream abstraction, a keyless stream of messages with in-memory watermark tracking and ack-on-completion (**#1905**).
Stream-state duality for sources: an initial scan establishes the full state, a change feed streams create, update, and delete events, and CocoIndex merges them into a LiveMapView that processes only what changed; the local file system gets its change stream from OS file-system events, OCI from OCI Streaming over the Kafka protocol, with more sources such as S3 to come
This is the same model CocoIndex already uses for the local file system, where the change stream comes from OS file-system events rather than Kafka. More sources, such as S3, will gain live mode over time.
### Source: Apache Iggy
CocoIndex now reads from **Apache Iggy**, a persistent message-streaming platform (**#1969**).
### Target: Turbopuffer
A new **Turbopuffer** target connector landed, along with a runnable text-embedding example (**#1934**).
### Target: Neo4j
A native **Neo4j** property-graph target landed alongside a meeting_notes_graph_neo4j example, mapping nodes and relationships into Neo4j without writing Cypher by hand (**#1932**).
### Target: FalkorDB
The **FalkorDB** property-graph target was brought forward into the v1 connector set, with support for nodes, relationships, and vector/FTS indexes (**#1908**).
### Target: LanceDB
LanceDB got two upgrades for production workloads:
- The v1 target now **optimizes (compacts) tables periodically** in the background, keeping query performance steady as data churns (**#2008**, with a scheduling fix in **#2013**).
- Table targets can **add new columns in place**, so schema evolution doesn't require a rebuild (**#1951**).
## Bug fixes
### Postgres correctness
- halfvec op classes are now used for indexes on half-precision vectors (**#2029**).
- U+0000 (NUL) bytes are stripped when writing text/jsonb, so Postgres no longer rejects otherwise-valid payloads (**#2032**).
- The pgvector extension now installs into the default schema, avoiding search_path surprises (**#1979**).
### Other fixes
- **SQL identifier validation**: the Postgres and SQLite connectors now validate table and column identifiers before interpolating them into statements, closing a class of SQL-injection vectors at the connector boundary (**#1947**, **#1965**).
- **Ownership-transfer race**: a pending-state protocol closes a preempt race so target-state ownership transfer stays correct under Postgres I/O latency, not just on microsecond-fast LMDB (**#1994**).
- **Clean cancellation**: cancellation propagates through task spawn boundaries, tearing work down cleanly instead of leaking orphaned tasks (**#1902**).
- **on_error cascade**: errors cascade through the Build-mode GC sweep (**#1999**).
- **numpy serde**: _frombuffer is registered under both numpy 1.x and 2.x paths (**#2012**).
- **Responsive progress display**: the PTY reader moved onto a dedicated OS thread, keeping the async runtime responsive under heavy logging (**#2033**, **#2040**).
## Build with CocoIndex
### Meeting notes → knowledge graph, now on Neo4j
Meeting Notes Knowledge Graph
The popular **meeting-notes-to-knowledge-graph** example now ships in a Neo4j flavor (meeting_notes_graph_neo4j). It watches a folder of meeting notes, uses LLM extraction to pull out structured entities (Meetings, People, Tasks) and the relationships between them, then maps everything into Neo4j as nodes and edges. Change one note and CocoIndex reprocesses only that note, keeping the graph continuously in sync.
### Turbopuffer and OCI text-embedding examples
Two more end-to-end examples landed alongside the new connectors: text_embedding_turbopuffer shows a complete embed-and-query pipeline against Turbopuffer, and oci_object_storage_embedding demonstrates ingesting from OCI Object Storage with live bucket watching. Both are ready to clone and run.
### Watch: build it end to end on FalkorDB
Follow along with the Build with CocoIndex walkthrough, which builds the meeting-notes-to-knowledge-graph pipeline end to end on FalkorDB, then clone the meeting_notes_graph_falkordb example to run it yourself.
For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.
## Thanks to the community
Thanks to the contributors who shipped changes this cycle.
### @prrao87
Thanks @prrao87 for fixing LanceDB optimize scheduling.
### @zherendong
Thanks @zherendong for parallelizing entity resolution via candidate-graph components.
### @countradooku
Thanks @countradooku for adding the Apache Iggy connector.
### @Haleshot
Thanks @Haleshot for keeping the docs in shape after the v1 launch: removing the stale v1 mention from the install command, fixing a broken URL, and updating stale v1 branch links to main.
### @galshubeli
Thanks @galshubeli for bringing the FalkorDB target connector into the v1 connector set.
### @Gohlub
Thanks @Gohlub for adding LiteLLM speech-to-text support, CocoIndex's first audio operation.
### @MrAnayDongre
Thanks @MrAnayDongre for adding per-argument memo_key support to @coco.fn.
### @aaronjmars
Thanks @aaronjmars for validating SQL identifiers in the Postgres and SQLite connectors.
### @tuanaiseo
Thanks @tuanaiseo for reporting and fixing a potential SQL injection in the Postgres connector, responsibly disclosed.
### @nuthalapativarun
Thanks @nuthalapativarun for a prolific cycle of splitter work: adding Elm, Astro, and Bash, CMake, and HCL tree-sitter support, plus adding context to Rust→Python error messages and expanding the supported-languages docs.
### @qWaitCrypto
Thanks @qWaitCrypto for making the LanceDB v1 target optimize tables periodically and for adding columns in place.
### @shaiar
Thanks @shaiar for fixing serde _frombuffer registration across numpy 1.x and 2.x.
### @octo-patch
Thanks @octo-patch for adding Python formatting and linting commands to CLAUDE.md.
### @phuctoan123
Thanks @phuctoan123 for documenting gws for Google Drive setup.
---
# Live CSV → Kafka with CocoIndex's New Kafka Target Connector
URL: https://cocoindex.io/blogs/csv-to-kafka-live/
Date: 2026-04-28
Summary: Walk through a live CocoIndex pipeline that watches a folder of CSV files and publishes each row as JSON to a Kafka topic incrementally, with no glue code.
CocoIndex just got a **Kafka target connector**. You can now declare Kafka topics as a target of your pipeline the same way you'd declare a Postgres table or a vector index, and CocoIndex will incrementally produce messages as your source data changes: no producer loop or bookkeeping, and no "did I already publish this row?" logic.
In this post we walk through a small but complete example: watch a local directory of CSV files, turn each row into a JSON message keyed by the row's primary key, and publish to a Kafka topic hosted on StreamNative. The whole pipeline is about 60 lines of Python, and it runs in **live mode**: edit a CSV, and within a second only the changed rows show up on the topic.
Getting data into Kafka has always been a pile of glue: producers, dedup state, deletes, schema re-bootstraps, file-watcher debouncing. Easy to get subtly wrong, never interesting to maintain.
The new connector lets you skip it. Hand CocoIndex an AIOProducer, a topic, and a function mapping each source row to a key and value. It handles the rest, producing messages only for rows that *actually changed*.
## From static knowledge to streaming signals
Many agent stacks today are built around periodic snapshots of their knowledge sources. The wiki is re-indexed overnight, the codebase is re-embedded on a cron, the CRM is re-pulled on a schedule, and agents read those snapshots over and over, hoping to detect when something has changed. For long-running agents that may execute for hours at a time, a snapshot captured at the start of the run quickly drifts out of sync with the underlying data.
A common next step is to wire sources directly to consumers via point-to-point webhooks, but this approach has well-known limitations once more than one consumer is involved. There is no shared path for replay or backfill, no buffer to absorb bursts when many changes arrive at once, and no common schema describing what a "change" looks like across systems. Teams that go down this path tend to end up reimplementing pieces of a durable log, separately, in each integration.
One source fanning out to four consumers via point-to-point webhooks: each pair carries a different payload schema and each duplicates the same retry, dedup, queue, and ack machinery, while a late-arriving consumer has no way to replay history
The combination of CocoIndex and Kafka takes a different approach: **it treats the knowledge layer the same way operational data has been handled for years: as a stream of change events rather than a snapshot to be re-read.** Drives, repos, design files, wikis, PDFs, and file shares (the unstructured data that has traditionally lived outside the streaming world) can be published to the same event backbone that already carries orders, clicks, and CDC traffic. The benefits show up in several places:
- **More efficient AI workloads.** Embeddings, retrievals, and agent context are refreshed only when something has actually changed, which reduces redundant work and improves freshness at the same time.
- **A single change reaches every consumer.** A commit, a renamed Drive document, or a Notion edit can update the vector index, notify an agent, update search, feed a Flink job, and land in a BI tile, without any of those systems needing to know about each other.
- **Easier extensibility.** A new agent, a rebuilt RAG layer, or a compliance tool can be added as another subscriber to the topic, with the log providing replay so it sees historical changes the same way it sees new ones.
- **Better auditability.** Each change consumed by an agent is durably recorded with offsets and timestamps, which makes it possible to answer questions like "did the agent see the updated policy before it acted?" with concrete evidence.
- **A stable contract over time.** The change-event schema on the topic provides a stable interface between sources and consumers. Detectors, sources, and models can evolve independently while the wire format stays consistent.
Unstructured sources (Drives, repos, design files, wikis, PDFs, file shares) flow through CocoIndex's diff stage onto a single durable Kafka topic with offset and timestamp markers, fanning out to vector index, agent runtime, search, Flink, BI, and compliance consumers. A new subscriber joining later replays history from offset zero. Below the diagram, five numbered cards summarize the benefits: efficient AI, fan-out, extensibility, auditability, and a stable schema contract
It is worth stating the contrast directly. A **static unstructured knowledge graph** is rebuilt on a schedule, drifts between rebuilds, and pushes freshness logic into each consumer. A **stream of unstructured change events** stays current by construction and gives every consumer (agents, indexes, analytics, auditors) a shared view of what has changed. Most AI stacks today are built around the former; production agents tend to benefit more from the latter.
## Kafka: message in, message out — now from the unstructured world
Kafka's contract is famously simple: **a message goes in, a message comes out.** That single shape is what makes the rest of the streaming ecosystem composable: Flink, ksqlDB, vector stores, OLAP sinks, search backends, microservices, agent runtimes. Anything downstream gets a real-time, replayable, fan-out-by-default view of the world by just reading a topic.
The catch is that the messages going *in* have historically been the structured kind: orders, clicks, IoT telemetry, Debezium CDC off Postgres or MySQL. The other half of the business (meeting notes, codebases, design files, PDFs, file shares, wikis) has lived in a parallel universe of vendor-specific webhooks, nightly batch jobs, and one-off ETL scripts no one wants to own.
Two lanes: structured sources flow cleanly into a Kafka topic and out to Flink, vector stores, search, and agents; unstructured sources (meeting notes, codebases, PDFs, wikis) sit in a parallel universe of webhooks, batch jobs, and ETL scripts that never reaches Kafka cleanly
CocoIndex closes that gap. It treats dynamically-changing unstructured assets as **first-class CDC sources** and emits clean key/value change events into Kafka, with upsert / delete / no-op semantics described later in this post.
Four unstructured sources (meeting notes, codebase, files/S3, wikis/docs), each shown as an incremental-processing card with row-level cached/changed status, feed CocoIndex and land on a Kafka topic as (key, value) upsert/delete/no-op events
The point is not "another connector." It is that Kafka stays Kafka (**message in, message out**) and CocoIndex makes the *in* side cover the dynamic, unstructured majority of the data that previously couldn't get there cleanly. Every team that already knows how to consume from Kafka now gets the unstructured world for free.
## Why choose CocoIndex with Kafka
Kafka is great at durable, high-throughput event streaming, but it expects you to solve data shaping, schema evolution, and incremental change detection yourself. CocoIndex turns that into a declarative problem:
- Define transformations like formulas: “from this table or API, derive these messages and fields.” CocoIndex computes and re-computes derived data whenever sources change, similar to spreadsheet formulas reacting to edits.
- Built-in incremental processing: CocoIndex keeps local state (checksums, primary keys, or version markers) so subsequent runs push only changed or deleted rows into Kafka topics.
- Schema-aware updates: when schemas change upstream, CocoIndex re-derives downstream messages instead of forcing you to manually re-bootstrap topics or write ad-hoc migration scripts.
- Pluggable sinks beyond Kafka: the same flow that feeds Kafka can also populate vector databases, OLTP/OLAP stores, or search indexes, so Kafka events and your AI/search backends stay in sync from a single definition.
## Partnership with StreamNative
We're partnering with StreamNative to bring real-time data infrastructure to AI workloads. The two halves fit cleanly because they answer different questions: CocoIndex looks at noisy, dynamic, unstructured sources and produces a precise answer to **"what's different since last time?"** StreamNative's Ursa-powered Kafka service makes sure that answer reaches every system that cares, reliably, at scale, with history intact. With Kafka in the middle, sources don't have to behave like streams and agents don't have to behave like database clients; the topic absorbs the impedance mismatch and lets each side evolve on its own schedule.
## The example: CSV files → JSON messages
To make all of this concrete, we'll build the smallest end-to-end pipeline that exercises the new connector: a local data/ folder of CSV files, watched in real time, with each row published as a JSON message to a Kafka topic on StreamNative. Edit a cell, and within a second exactly one message (for that one row) appears on the topic. Add a row, get one new message. Delete a file, and every row from it is tombstoned. The whole thing is around sixty lines of Python.
A local data/ folder of CSV files (products.csv, employees.csv) is watched live by CocoIndex, which publishes each row as a JSON message keyed by the row's primary key to a Kafka topic on StreamNative
CSV is a deliberately humble starting point. It's the format that shows up everywhere and gets respect nowhere: exports from BI tools, dumps from vendors who won't give you a webhook, spreadsheets parked in a shared drive by a colleague, output from a nightly script no one wants to maintain. CSV files look structured but live like unstructured assets: dropped into a folder, edited at random, with no notifications and no schema contract. If we can turn a directory of them into a clean, row-keyed, live Kafka stream with diff-only writes, the same pattern carries over to the noisier sources mentioned earlier: PDFs, codebases, wikis, design files. CSV just lets us focus on the new connector and the live-update loop without getting tangled in a richer parser.
The full example lives in the CocoIndex repo. Here's the shape of it.
We have a data/ folder with a couple of CSV files:
```csv
# data/products.csv
sku,name,category,price
SKU001,Wireless Mouse,Electronics,29.99
SKU002,Mechanical Keyboard,Electronics,89.99
SKU003,USB-C Hub,Accessories,45.00
```
```csv
# data/employees.csv
emp_id,first_name,last_name,department,email
E101,Alice,Chen,Engineering,alice.chen@example.com
E102,Bob,Smith,Marketing,bob.smith@example.com
```
The goal: every row becomes a JSON message on a Kafka topic, keyed by the value of the row's first column (the primary key). When a CSV file is edited, only the rows that actually changed should be re-published.
## The pipeline
First, the Kafka producer is set up once at app startup using a lifespan hook, and stashed in a ContextKey so the rest of the pipeline can grab it without passing it around:
```python
import cocoindex as coco
from cocoindex.connectors import kafka, localfs
from confluent_kafka.aio import AIOProducer
KAFKA_PRODUCER = coco.ContextKey[AIOProducer]("kafka_producer", tracked=False)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder):
config = {
"bootstrap.servers": KAFKA_BOOTSTRAP_SERVERS,
"sasl.mechanism": "PLAIN",
"security.protocol": "SASL_SSL",
"sasl.username": KAFKA_SASL_USERNAME,
"sasl.password": KAFKA_SASL_PASSWORD,
}
producer = AIOProducer(config)
builder.provide(KAFKA_PRODUCER, producer)
yield
```
ContextKey is how CocoIndex shares the producer across components without threading it through every function call; we'll come back to it later. The SASL block is what StreamNative (or any production broker) wants. For a local broker you can drop those four lines and just point bootstrap.servers at localhost:9092.
Next, the per-file processor. This is where the new Kafka API shows up:
```python
@coco.fn(memo=True)
async def process_csv(file: FileLike, topic_target: kafka.KafkaTopicTarget) -> None:
text = await file.read_text()
reader = csv.DictReader(io.StringIO(text))
headers = reader.fieldnames
if not headers:
return
first_col = headers[0]
for row in reader:
key_value = row.get(first_col)
if key_value is not None:
value = json.dumps(row)
topic_target.declare_target_state(key=key_value, value=value)
```
The @coco.fn(memo=True) decorator means the per-file work itself is memoized too: if a file's contents haven't changed, process_csv doesn't even run.
### Declare states, not messages
```python
topic_target.declare_target_state(key=key, value=value)
```
It's deliberately *not* called send_message() or produce() or declare_message(). It's declare_target_state.
CocoIndex is a **state-driven** data framework. The mental model is the same one you'd use for a spreadsheet, a React component tree, or a SQL materialized view: **you describe what the target should look like as a function of the source, and the framework figures out the transitions.** You don't compute deltas. You don't track "what did I send last time." You don't handle insert vs. update vs. delete as separate code paths. You just say, "given this CSV row, the target state for key SKU001 is this JSON blob," and that's it.
Kafka makes that distinction unusually visible because Kafka's wire model is the opposite: a topic is a *log of events*, not a snapshot. Producers send change events; consumers (or compacted topics) reconstruct state from the log. So the question is: who's responsible for the gap between "I have desired states" and "the broker needs to receive change events"?
In CocoIndex, the framework owns that gap. When you call declare_target_state(key=k, value=v):
- If k is new or v differs from the last value the framework remembers for k, it emits an **upsert** message: (k, v) on the wire.
- If k was previously declared but isn't declared this time, it emits a **delete** message: (k, None), or (k, deletion_value_fn(k)) if you supplied a tombstone constructor.
- If k was previously declared with the same v, **nothing is sent** to the broker, and no consumer wakes up.
Declared states above the line, derived Kafka messages below: five declared states produce four wire-level messages because the unchanged one is silent
Messages are derived from state transitions. You only ever talk about states. This is exactly the same pattern as the Postgres target (declare_target_state → INSERT / UPDATE / DELETE) and the vector index targets: the wire-level operations differ, but the user-facing API is the same shape, because the *semantics* are the same.
The reason this matters in practice: it means the same process_csv function works correctly on the first run and every run after, whether a row is edited or removed, a file is deleted, or the whole pipeline crashes and restarts. There is no separate "initial load" code path versus "incremental update" code path. There's just "given the source, here's what the target should look like," and that statement is true whether the target is empty, half-populated, or already in sync.
Finally, wire it all together:
```python
@coco.fn
async def app_main() -> None:
topic_target = await kafka.mount_kafka_topic_target(KAFKA_PRODUCER, KAFKA_TOPIC)
files = localfs.walk_dir(
localfs.FilePath(path="./data"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.csv"]),
live=True,
)
await coco.mount_each(process_csv, files.items(), topic_target)
app = coco.App(coco.AppConfig(name="CsvToKafka"), app_main)
```
Two things to notice here:
1. mount_kafka_topic_target(...) resolves the producer from the context key and gives back a handle. The topic itself is user-managed: CocoIndex never creates or deletes topics; it just produces into one you already own.
2. localfs.walk_dir(..., live=True) turns the directory walker into a live source: it does an initial scan, then keeps watching the filesystem and pushes incremental updates downstream. Combined with cocoindex update -L, the whole pipeline runs continuously instead of one-shot.
## Live mode: one flag, everything else is the same
So far we've described what happens on a single run. But in reality, source files get edited, rows get added and removed, and you usually want the topic to keep up. The same process_csv runs as a **catch-up run** (scan once, reconcile everything that's changed since last time, exit) *or* as a continuously-running pipeline that keeps watching for changes. The diff between the two is one keyword argument and one CLI flag:
**Catch-up run:**
```python
files = localfs.walk_dir(
localfs.FilePath(path="./data"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.csv"]),
)
await coco.mount_each(process_csv, files.items(), topic_target)
```
```sh
cocoindex update main.py
```
**Live:**
```python
files = localfs.walk_dir(
localfs.FilePath(path="./data"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.csv"]),
live=True, # ← +1 line
)
await coco.mount_each(process_csv, files.items(), topic_target)
```
```sh
cocoindex update -L main.py # ← +1 flag
```
That's the entire diff. process_csv doesn't change. The Kafka target doesn't change. There's no separate "streaming" code path to maintain.
Run the live version:
```sh
cocoindex update -L main.py
```
CocoIndex does a full scan first, publishes one message per row, then sits there watching data/:
- **Edit one cell** in products.csv: exactly one Kafka message is produced for that one row (modulo broker retries; the producer is at-least-once by default). The other four rows are silent.
- **Add a new row**: one new message.
- **Delete a row**: one delete message (no value, since this example doesn't supply a deletion_value_fn).
- **Add a brand new CSV file**: process_csv runs once for that file, publishing its rows.
- **Delete a CSV file**: every row from that file gets a delete message.
### What the flag actually does (and doesn't do)
It's worth being precise about this, because it's easy to assume the flag is doing more magic than it is.
**Reconciliation happens in both modes.** Whether you run cocoindex update or cocoindex update -L, you declare target states and CocoIndex computes deltas against the previous run, emitting only the upsert and delete messages needed to bring the topic in line. To do that, CocoIndex maintains an internal state store that remembers, for every declared key, the fingerprint of the value last sent. That store survives restarts, so when you stop and restart the pipeline it picks up where it left off: no re-broadcast of unchanged rows.
This is also where the ContextKey from earlier earns its keep. The state store doesn't identify "this Kafka topic" by SASL credentials or bootstrap address. It identifies it by the ContextKey it was anchored to (here, KAFKA_PRODUCER) plus the topic name. If you later rotate the SASL password, swap the broker endpoint, or replace the AIOProducer with a differently configured one, the same ContextKey is enough for CocoIndex to recognize the backend as the same one, and the state store carries over.
**The flag controls when reconciliation runs.** That's it. In a catch-up run, CocoIndex scans sources once, reconciles up to the moment, and exits. In live mode, it does the same initial catch-up, *then* keeps watching the sources for changes and reconciles incrementally as they arrive. The code path, reconciliation logic, and target API are identical: just catch-up-and-exit vs. catch-up-and-keep-watching.
The "watching" half comes from the source itself, not the flag. localfs.walk_dir(..., live=True) returns a live source backed by watchfiles; it knows how to deliver filesystem events. A non-live source (or live=False) just enumerates current state and stops. The CLI -L flag tells the runtime to keep the app alive so the watcher's events have something to drive.
One Kafka detail worth mentioning since the connector doesn't change it: each declared key is hashed to a partition by Kafka's default partitioner, so the same key always lands on the same partition. Log compaction and key-based consumers behave exactly the way they'd behave with any hand-rolled producer.
## What do the messages look like on the topic?
We pointed this at a Kafka cluster on StreamNative Cloud. It gave us a real SASL_SSL endpoint in one click, with a hosted console for inspecting messages without writing a consumer. (Plain localhost:9092 works too if you skip the SASL fields in the producer config.)
Here's what shows up in the console for the cocoindex-csv-rows topic after running the example:
Messages on the StreamNative-hosted Kafka topic after running the CSV → Kafka pipeline
Keys are the row's primary-key column (SKU001, E101, …); values are the JSON-encoded rows. Edit a CSV locally, refresh, and a new message with the same key appears: same key, latest value wins, log-compacted consumers get exactly the current state.
## Try it
The Kafka connector ships as an optional dependency group: pip install cocoindex[kafka] if you're pulling it into an existing project. To run the example directly:
```sh
git clone https://github.com/cocoindex-io/cocoindex
cd cocoindex/examples/csv_to_kafka
cp .env.example .env # fill in your Kafka bootstrap + SASL creds
pip install -e .
cocoindex update -L main.py
```
No cloud broker handy? Point KAFKA_BOOTSTRAP_SERVERS at localhost:9092 and leave the SASL fields empty. The example works against any Kafka the confluent_kafka client can connect to.
Then go edit data/products.csv and watch the messages land.
## Support our work
If this project helps you, we'd appreciate a GitHub star at the CocoIndex project.
---
# CocoIndex V1 is Live!
URL: https://cocoindex.io/blogs/cocoindex-v1/
Date: 2026-04-22
Summary: CocoIndex V1 is live: a ground-up redesign of incremental data pipelines, built for AI engineers and agent builders shipping RAG, memory, and knowledge graphs.
CocoIndex is an incremental engine for long-horizon agents. It continuously transforms data from any source (codebase, meeting notes) and builds the context for agents deployed in production.
CocoIndex: the incremental engine for long-horizon agents. Continuously transforms data from any source (codebase, meeting notes, and more) and builds the context for agents deployed in production.
**CocoIndex V1 is now live**. It is a fundamental redesign of how you write incremental data pipelines, built from a year of watching what people actually wanted to do with CocoIndex. CocoIndex V1 is built for AI engineers and agent builders: people building context, RAG, memory, knowledge graphs that live agents depend on. The design choice that matters most to that audience is deliberately small: the framework meets you in the language you already write, with the data types you already use. If Claude or Cursor can already fluently write it, it works here.
The headline change is simple: **there is no DSL anymore**. Your whole pipeline is now regular async Python, growing organically as functions call each other. The engine still does what CocoIndex has always done (managed targets, incremental processing, change tracking in Rust), but it now lives behind a Python-native surface, not behind a DSL with its own type system and its own rules about what you can and can't do.
The mental model is unchanged, and it's the clearest way to describe what CocoIndex does: you declare what the *target* should look like as a function of the *source*, and the engine figures out the transitions. CocoIndex calls this **state-driven programming**, the same shape as React, spreadsheets, or SQL materialized views. Declare the state; Coco handles the rest.
## Why CocoIndex: the incremental engine for long-horizon agents
Agents run roughly 50× faster than humans, but the tools they rely on were built for *human speed*.
At GTC 2026, Jeff Dean and Bill Dally named a bottleneck that's about to reshape every piece of infrastructure around AI. Amdahl's law takes over: no matter how fast the model gets, end-to-end throughput is capped by the slowest thing in the loop. Jeff's specific emphasis was on ultra-low-latency inference for agents "operating in the background," running tasks that "take hours or perhaps even days, independently doing a bunch of things, correcting themselves, doing some more things."
Data infrastructure is one of those tools, and it matters beyond inference. An agent reasoning over a codebase, a conversation graph, a document corpus, or a stream of events needs that data **fresh, organized, and cheap to query**, not just on the first call, but throughout the run, because:
- **Agents write code.** The artifacts they produce become source data for the next reasoning step.
- **Agents make decisions and record them.** Traces, plans, audit logs: written by agents, read by other agents.
- **Agents update data while they run.** A long-horizon agent isn't a one-shot query; it's a process that edits its own context as it proceeds.
- **Source data arrives faster, too.** Codebases, logs, Slack, PRs, tickets, sensor feeds: all updating in the background, all expected to be visible to whatever the agent is doing next.
The agent data loop: source streams (code, Slack, PRs, tickets, logs, IoT) feed an incremental Δ engine that keeps a derived context (vector, graph, SQL, search) fresh and in sync. Long-horizon agents read the context and write back code, traces, plans, and decisions, which flow in as new source data. A crossed-out nightly-rebuild clock in the corner marks the batch cadence as broken: the loop never stops.
The default pattern for indexing that data ("batch-rebuild overnight") was built for humans who check a dashboard every morning. For agents running 50x faster, that cadence *is* the bottleneck. What agents need is an engine that treats **derived data as a function of source data** and keeps it in sync **incrementally**: only what actually changed gets re-embedded, re-upserted, or re-published, with no full rebuild and no waiting for the next batch window.
That has always been CocoIndex's job. V1 makes it the *right shape* for agent-era workloads: the same incremental, state-driven guarantees, but now expressive enough to cover the pipeline shapes agents actually produce (entity resolution, clustering, multi-phase reduction, per-tenant topologies, conditional targets) instead of just "chunk → embed → upsert." Every pattern in the examples gallery is something a long-horizon agent might want to run itself, and have its outputs become fresh source data for the next agent, without a human babysitting the refresh job.
## What is an incremental engine, and why it's hard to build for production
Incremental engine for long-horizon agents: Python-native transformation flows connect eight source categories through a shared pipeline to six target stores. Only the Δ is reprocessed. The persistent data-pipeline control plane runs eight always-on subsystems: live caching, pipeline catalog, version tracking, continuously learning, lineage, task scheduling, metrics collection, and failure management.
An incremental engine does one job: keep derived data in sync with source data without redoing work that's already been done. The source data is changing and the processing code is dynamic.
To do it right, you need: reliable change detection across heterogeneous sources (filesystems, databases, queues, APIs), content fingerprints stable enough to diff but cheap enough to compute every run, persistent state that records what the last run produced so this run can compare against it, memoization keyed on *both* data and code fingerprints so that editing a helper function invalidates only the callers that actually depended on it, managed target lifecycles (schema evolution, orphan cleanup when a source disappears, idempotent upserts), transactional behavior when you're writing to multiple targets at once, and recovery logic for every partial failure in between, and many more.
Teams that take this on seriously typically allocate 10 – 20 engineers for at least six months to land the first production-worthy version, and then keep paying for maintenance indefinitely as sources, targets, and schemas evolve. CocoIndex ships all of this in the engine, so the code you write is the pipeline itself, not the scaffolding around it.
Custom incremental solution: a small user-code box plus a large grid of scaffolding concerns: CDC event handling, key lineage tracking, CRUD, change propagation, stale data cleanup, schema evolution, data backfill, partial reprocessing
Notion has recently published their engineering work around the tip of the iceberg on maintaining an in-house data pipeline with incremental processing on simpler data transformation (e.g., without clustering, etc.) for live agents.
## The V1 mental model: declare states, not steps
Target = F(Source): declare the target state as a function of the source; the engine keeps it in sync
The mental model didn't change. The execution model did.
CocoIndex has always been a **state-driven** framework. You don't write code that says "compute this delta and apply it"; you describe what the *target* should look like as a function of the *source*, and the engine works out the transitions. If you've used React, spreadsheets, or SQL materialized views, you already know the shape of this:
- **React**: declare UI as a function of state → React re-renders what changed.
- **Spreadsheets**: declare formulas → cells recompute when inputs change.
- **CocoIndex**: declare target states as a function of source → engine syncs what changed.
What's new in V1 is *how* you express that declaration. The whole pipeline is now plain async Python, without FlowBuilder, DataScope, or a two-phase "define, then run" lifecycle. Functions call other functions. Loops are loops. if statements are if statements. Behind the scenes, CocoIndex tracks every target state you declare and every memoized intermediate value, so the next run only does what changed.
V1 is deliberately not intrusive. You can start with a single @coco.fn on a plain Python function and opt into more of the engine as the pipeline grows: flip on memo=True when re-running starts to cost real time, add declare_* targets when you want the engine to own the output lifecycle, move to live mode when you want the pipeline to keep watching for changes. There's no "set up the framework first" step and no ceremony you have to write before you get value. You turn the incremental knobs on when you actually need them.
That model rests on a small vocabulary: an **App** is the top-level entity you run; inside it, each **processing component** groups one item's work with its **target states** and runs independently; **incremental processing** means only what changed gets reapplied to the target; and **function memoization** skips any function whose input and code haven't changed since the last run.
What that means in practice is code you don't have to write: a bookkeeping table tracking which file was last processed, insert / update / delete branches, "did I already embed this chunk?" checks, migration scripts when your schema changes. The engine owns the target state, diffs against the previous run, and only does what changed. V0 made that tractable. V1 makes it feel like writing native Python.
A complete V1 app (a PDF-to-Markdown converter) is short enough to read at a glance:
```python
import pathlib
import cocoindex as coco
from cocoindex.connectors import localfs
from cocoindex.resources.file import PatternFilePathMatcher
from docling.document_converter import DocumentConverter
_converter = DocumentConverter()
@coco.fn(memo=True)
def process_file(file: localfs.File, outdir: pathlib.Path) -> None:
markdown = _converter.convert(file.file_path.resolve()).document.export_to_markdown()
outname = file.file_path.path.stem + ".md"
localfs.declare_file(outdir / outname, markdown, create_parent_dirs=True)
@coco.fn
async def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:
files = localfs.walk_dir(
sourcedir, path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
)
await coco.mount_each(process_file, files.items(), outdir)
app = coco.App("PdfToMarkdown", app_main,
sourcedir=pathlib.Path("./pdf_files"),
outdir=pathlib.Path("./out"))
app.update_blocking()
```
Three things to notice:
1. process_file is just a function: you can set a breakpoint in it, call it directly, run it under pytest, whatever. The @coco.fn(memo=True) decorator hooks it into the engine's change-detection and memoization, but doesn't change its calling convention.
2. localfs.declare_file(...) is a *declaration*, not a write. It tells CocoIndex "the target state for this path is this content." If the target state hasn't changed since the last run, nothing happens on disk. If the source file disappears, the declaration disappears too, and CocoIndex deletes the corresponding output file automatically.
3. coco.mount_each(...) mounts one **processing component** per file. Each component is its own unit of incremental change detection: when one file's content changes, only that component re-runs.
That's the whole programming model. Everything below is consequences.
## Why V1: the V0 ceiling
V0 worked (developers shipped real pipelines in production), but four constraints kept coming back, all consequences of the DSL-first design:
- **Two worlds, one program.** FlowBuilder / DataScope / DataSlice for topology, regular Python for leaves. The DSL world couldn't if its way into a different shape, couldn't pass values into normal Python and back. Clustering, entity resolution, reduction across items, conditional sources: every one of them hit the seam.
- **A separate type system.** Like a database, the engine had its own types distinct from Python's, so every Python type you wanted to use needed a bi-directional conversion wired in. dataclass and NDArray were covered; a PIL Image or a pyarrow array was not: if no conversion existed, you couldn't hand it to a function, and what did cross the boundary got marshaled on the way in and out.
- **Postgres as a hard dependency.** The engine used Postgres for its own bookkeeping, so pip install cocoindex && python script.py wasn't a real path: you stood up a database first, even for a tiny local pipeline.
- **Static topology.** add_source() and .export() were declared before execution, so multi-tenancy, config-driven topologies, and runtime target selection all required scaffolding *around* CocoIndex rather than being expressible *in* it.
Fixing any one of these meant changing the design.
## Four shifts that change what you can build
Four shifts from V0 to V1: side-by-side comparison. (1) No more DSL: V0's FlowBuilder graph was frozen at build time; V1's flow grows at runtime as await coco.mount() calls execute. (2) Native Python types: V0 supported a bounded set of bindings like dataclass and NDArray; V1 accepts any Python type: PIL.Image, pyarrow, torch.Tensor, or anything else. (3) Embedded LMDB: V0 required a Postgres server for engine bookkeeping; V1 keeps state in a single local LMDB file, so pip install is the whole setup. (4) Dynamic I/O: V0 declared topology up front; V1 creates sources and targets during execution, one per tenant or per config row.
### 1. No more DSL — the flow grows during execution
In V0, the flow had to be fully described before any data moved through it. FlowBuilder returned a frozen graph; the engine then interpreted that graph at execution time. In V1, the flow *is* the execution. app_main runs as the root processing component. Every await coco.mount(...) it does adds a child component. Every declare_* call records a target state. The component tree is built by running your code, not by parsing a graph beforehand.
This unlocks shapes that were awkward or impossible in V0:
- **Reduction patterns**: "for each project, look at all its files, then aggregate." The multi-codebase summarization example does exactly this:
```python
@coco.fn(memo=True)
async def process_project(project_name, files, output_dir):
file_infos = await coco.map(extract_file_info, files) # fan out
project_info = await aggregate_project_info(project_name, file_infos) # reduce
markdown = generate_markdown(project_name, project_info, file_infos)
localfs.declare_file(output_dir / f"{project_name}.md", markdown)
```
- **Multi-phase pipelines with cross-item logic**: entity resolution across sessions, clustering, deduplication. The conversation-to-knowledge example uses three phases (per-session extraction, cross-session entity resolution, knowledge-base assembly) all expressed as function calls in a single app_main.
- **Conditional topology**: if config.enable_kafka: await mount_kafka_target(...). No flag-pattern, no separate flow definitions.
And because everything is real Python code, debugging is real Python debugging. Set a breakpoint. Step through. Print things. The engine's not interpreting an opaque graph; it's calling your functions.
### 2. Native Python types — no separate type system
V1 doesn't have a separate type system: it uses Python's directly. The types v0 supported through bindings (dataclass, Pydantic, NDArray, dict, list, tuple) still work with nothing to wire up, and types v0 couldn't express (a PIL Image, a pyarrow array, an arbitrary class from a library you just pulled in) can be passed as function arguments the same way:
```python
from PIL import Image
@coco.fn(memo=True)
async def caption_image(img: Image) -> str:
return await vlm.describe(img)
```
That Image is just PIL.Image: no wrapper, no conversion. Pass the function a pyarrow Table, a torch.Tensor, or any class from a library you happen to have imported; the signature is whatever you'd write in plain Python.
Targets, function arguments, return types, memoization keys: everything is described in the type hints you'd write anyway. For memoized return values that persist between runs, the same type hint drives serialization; we wrote a long post on how the type-guided serializer works underneath.
### 3. Embedded LMDB — pip install and go
V0's "Postgres for engine bookkeeping" requirement is gone. V1 stores its internal state in LMDB, an embedded key-value store that lives in a single local file. There's no server process to run, schema to migrate, or port to expose.
```bash
pip install cocoindex
cocoindex update main.py
```
That's the whole setup. The LMDB file holds the target-state ledger, the memoization cache, and the component-path tree across runs. On a 64-bit system the default 4 GiB map size is virtual address space, not physical memory. You can bump it to 8 GiB or more without paying the cost upfront.
Postgres is still a first-class **target** (with pgvector support); it's just no longer required for the engine to function.
### 4. Dynamic sources and targets
In V0, sources and targets were declared in the flow definition, before execution. In V1, they're created during execution, by regular function calls, and the component-path tree tracks their identity across runs. So you can:
- **Mount a target conditionally** based on a config row, an environment variable, or a feature flag.
- **Create different targets per item**: e.g., one Postgres schema per tenant, one Kafka topic per category, one S3 prefix per dataset.
- **Build the topology from a database**: query the list of active tenants at startup and mount one component per tenant, each with its own set of targets. When a tenant disappears from the list, CocoIndex automatically cleans up their target states (because the path is no longer mounted).
The conversation-to-knowledge example does this for entity types: it iterates over a list of entity configurations and mounts one SurrealDB target per type, all from inside a normal for loop in app_main.
```python
entity_tables = {
cfg.name: await surrealdb.mount_table_target(SURREAL_DB, cfg.name, entity_schema)
for cfg in ENTITY_TYPES
}
```
Add a new entity type to ENTITY_TYPES, restart, and a new table appears, managed by CocoIndex from then on. Remove a type, and its table goes away. No migration, no orphaned data.
The real leap over v0 is when that list isn't a constant at all (a changing file, a database row, a live tenant registry), since v0 could already generate topology from static inputs at flow-creation time, but couldn't react when those inputs changed.
## What we kept: the core promise
The redesign was the means, not the goal. The reason V1 exists is to make the things V0 was already good at *more* available. Those things are still here, and now they apply across a much wider set of pipeline shapes.
### Fully managed targets
You declare what your target should look like; CocoIndex makes sure it's in that state, on *any* change, whether to source data or to your code. This is the lifecycle, top to bottom:
| Layer | Upsert (first or changed declaration) | Delete (no longer declared) |
|---|---|---|
| **Container**: schema (tables · directories · other containers) | Create or alter | Drop |
| **Leaf**: data (rows · files · embeddings · other leaves) | Insert or update | Delete |
No manual migrations. No orphaned data. No "did I already write this row?" code paths. The target is fully owned by CocoIndex: if you stop declaring something, it goes away. This is the same contract whether the target is a Postgres table, a LanceDB collection, a Neo4j graph node, a Kafka topic, an S3 prefix, or a directory of files on disk.
### Incremental processing, two levels deep
Incremental processing is still the core of the engine, and V1 exposes it through two concepts:
- **Components own target states.** When a component finishes, CocoIndex diffs its declared target states against the last run and applies only the changes (create / alter / drop for containers, insert / update / delete for contents), including recursive cleanup of sub-paths no longer mounted. When a component's inputs and code haven't changed, the whole component is skipped.
- **@coco.fn functions participate in change detection.** Every decorated function's code is fingerprinted, and that fingerprint propagates up the call chain. Add memo=True and the return value is also cached, keyed on the function's arguments plus its code fingerprint, so the body is skipped when neither has changed.
Edit a helper function and only the components and memoized callers that actually depend on it are invalidated.
### Rust-powered engine
The change-detection, fingerprinting, target-state diffing, and persistence are all written in Rust on top of Tokio. That's still true: Python is the user-facing API, but the hot paths run in Rust. You get pythonic ergonomics without giving up the performance characteristics that made V0 feel snappy.
## What it looks like end-to-end
Here's a complete text-embedding pipeline that walks markdown files, chunks them, embeds the chunks with a SentenceTransformer model, and writes everything (with a vector index) to Postgres. It's the text_embedding example trimmed down: every line of pipeline logic is below.
Text-embedding app overview: a local markdown_files directory feeds the CocoIndex TextEmbedding App, which walks the list of .md files, splits and embeds each file per-file (memoized), and declares DocEmbedding rows (filename, chunk, vector) that sync to a Postgres doc_embeddings table with a vector index.
```python
import pathlib
from dataclasses import dataclass
from typing import AsyncIterator, Annotated
import asyncpg
from numpy.typing import NDArray
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.chunk import Chunk
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.id import IdGenerator
PG_DB = coco.ContextKey[asyncpg.Pool]("text_embedding_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
_splitter = RecursiveSplitter()
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with await postgres.create_pool("postgres://...") as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder("sentence-transformers/all-MiniLM-L6-v2"))
yield
@dataclass
class DocEmbedding:
id: int
filename: str
chunk_start: int
chunk_end: int
text: str
embedding: Annotated[NDArray, EMBEDDER]
@coco.fn
async def process_chunk(chunk, filename, id_gen, table):
table.declare_row(row=DocEmbedding(
id=await id_gen.next_id(chunk.text),
filename=str(filename),
chunk_start=chunk.start.char_offset,
chunk_end=chunk.end.char_offset,
text=chunk.text,
embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
))
@coco.fn(memo=True)
async def process_file(file: FileLike, table) -> None:
text = await file.read_text()
chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language="markdown")
id_gen = IdGenerator()
await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
table = await postgres.mount_table_target(
PG_DB, table_name="doc_embeddings",
table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=["id"]),
)
table.declare_vector_index(column="embedding")
files = localfs.walk_dir(
sourcedir, recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
)
await coco.mount_each(process_file, files.items(), table)
app = coco.App(coco.AppConfig(name="TextEmbedding"), app_main,
sourcedir=pathlib.Path("./markdown_files"))
```
Worth pausing on what's *not* in there:
- No flow definition phase. app_main runs and the topology emerges.
- No custom column types. embedding: Annotated[NDArray, EMBEDDER] is the schema; from_class reads it.
- No insert / update / delete branches. declare_row(...) describes the desired state; the engine diffs.
- No "did this chunk already get embedded?" check. Memoization handles it.
- No bookkeeping table for which file was last processed. LMDB has the previous run's fingerprints.
- No manual migration if you change DocEmbedding. Schema evolution is part of the managed-target contract.
Add a markdown file → its chunks get embedded and inserted. Edit one paragraph → only the affected chunks re-embed; the others reuse cached embeddings. Delete a file → its rows vanish from Postgres. Change the splitter's chunk_size → every file re-chunks, but embed(chunk) calls whose chunk text didn't change still hit the memo cache. Change the embedder model → because EMBEDDER is detect_change=True, every embedding gets recomputed and the column re-syncs.
That's the whole thing. None of those scenarios need a separate code path.
## Living shapes V0 couldn't reach
Three pipeline shapes V1 makes natural: multi-codebase summarization (fan out per project × fan out per file, reduce to a markdown summary per project), conversation-to-knowledge (three mount phases: per-session extraction, cross-session entity resolution, final knowledge-base assembly), and CSV → Kafka live (file-level source, row-level change events published as Kafka messages).
A short tour of pipeline shapes the V1 model makes natural, drawn from the example gallery:
- **Multi-codebase summarization**: fan out per project, fan out per file with coco.map, reduce back up to a project summary, write a markdown file per project. Pure Python control flow; the engine handles which projects need re-summarizing.
- **Conversation-to-knowledge**: three phases in one app_main: per-session extraction (mount per YouTube ID), cross-session entity resolution (mount per entity type), and knowledge-base assembly (one final mount). Each phase reads the outputs of the prior one through normal function returns.
- **CSV → Kafka live**: a live filesystem watcher feeding a Kafka topic target with at-most-once-per-change semantics, in about 60 lines. File-level source, row-level change events: one Kafka message per changed CSV row.
- **HN trending topics**, **image search**, **PDF embedding**, **paper metadata extraction**, **Amazon S3 + embeddings**, **Google Drive + embeddings**, **code embedding to LanceDB**: the connector ecosystem that landed during V0 came over to V1, and a few new ones (SurrealDB, SQLite, Kafka, S3) joined it.
## Try it
```bash
pip install cocoindex
```
Then either follow the Quickstart (PDF → Markdown in about thirty lines), or clone the repo and run any of the examples directly:
```bash
git clone https://github.com/cocoindex-io/cocoindex
cd cocoindex/examples/text_embedding
pip install -e .
cocoindex update main.py
```
The docs for V1 live at cocoindex.io/docs. The pieces most worth reading first:
- **Core Concepts**: state-driven processing, target states, processing components, memoization
- **App**: top-level entry point, lifespan, CLI
- **Processing Component**: mount, use_mount, mount_each, component paths
- **Function**: @coco.fn, memoization, change detection
- **Live Mode**: pipelines that keep watching for changes
Bug reports, feature requests, and "this section of the docs is unclear" notes are all very welcome on GitHub issues, and we hang out in Discord. ⭐ us on GitHub if you want to follow where this goes. There's a lot more shipping over the next few months.
## Support us
If you found this useful, starring CocoIndex on GitHub is the most direct way to help. It's how other developers find the project :)
---
# Turn Podcasts into a Knowledge Graph with LLM and CocoIndex
URL: https://cocoindex.io/blogs/podcast-to-knowledge-graph/
Date: 2026-04-02
Summary: Build a pipeline that turns YouTube podcasts into a knowledge graph: extract speakers, statements, and entities with an LLM, then dedupe them with embeddings.
Podcasts are one of the richest sources of expert knowledge on the internet. A single Lex Fridman or Dwarkesh Patel episode can contain dozens of substantive claims about people, technologies, and organizations, but it's all locked inside hours of audio. You can't query any of it. You can't cross-reference what two different guests said about the same topic.
In this post, we'll build a CocoIndex pipeline that turns YouTube podcast episodes into a queryable knowledge graph. The pipeline downloads audio, transcribes with speaker diarization, uses an LLM to extract structured statements and entities, resolves duplicates across episodes, and stores everything in SurrealDB as a graph.
Pipeline diagram showing podcast episodes flowing through CocoIndex with Pydantic transformations into a SurrealDB knowledge graph
We use CocoIndex to build the pipeline. CocoIndex is a data indexing framework for building incremental data transformation pipelines: it tracks what's been processed, so re-running the pipeline only processes new or changed episodes. It also made it simple to take out any unwanted podcast content that gets merged into the knowledge graph.
The full source code is available at CocoIndex Examples: podcast_to_knowledge.
## What We're Building
Here's the knowledge graph schema, five node types connected by four relationship types:
Knowledge graph schema showing session, statement, person, tech, and org nodes with their relationships
A **session** is one podcast episode. A **statement** is a thematic claim extracted from the conversation, e.g., "Scaling laws suggest that larger models will continue to improve." Each statement is linked to who said it and what entities it mentions.
**Person**, **tech**, and **org** are named entities. The tricky part is that the same entity can appear under different names across episodes ("GPT-4", "GPT4", "OpenAI's GPT-4"). We handle this with entity resolution, more on that later.
## Pipeline Overview
The pipeline runs in three phases:
Pipeline overview showing three phases: per-session processing, entity resolution, and knowledge base creation
**Phase 1** processes each episode independently: download audio, transcribe, and use an LLM to extract metadata, speakers, and statements. Sessions and statements are written to the database immediately since they don't need cross-episode deduplication.
**Phase 2** collects all raw entity names (persons, techs, orgs) from every episode and deduplicates them using embedding similarity + LLM confirmation.
**Phase 3** writes the deduplicated entities and all relationships to the database.
Detailed pipeline overview: podcast URLs are fetched, transcribed, extracted via Pydantic into raw entities and statements, then resolved into canonical Tech, Person, and Org nodes stored in SurrealDB
## Phase 1: Per-Session Processing
Phase 1 (per-session processing): iterate over podcast URLs, fetch audio, transcribe, and extract entities and statements via Pydantic
Each session goes through a multi-step pipeline, starting from a YouTube URL.
### Fetch Transcript
We download the audio with yt-dlp and transcribe it with AssemblyAI, which gives us speaker-diarized utterances (labeled "Speaker A", "Speaker B", etc.) along with YouTube metadata (channel name, title, description, upload date):
```python title="conv_knowledge/fetch.py"
@coco.fn(memo=True)
async def fetch_transcript(youtube_id: str) -> SessionTranscript:
"""Download audio via yt-dlp, transcribe with speaker diarization via AssemblyAI."""
url = f"https://www.youtube.com/watch?v={youtube_id}"
with tempfile.TemporaryDirectory() as tmpdir:
# 1. Download audio via yt-dlp, convert to mp3
audio_path = os.path.join(tmpdir, "audio.mp3")
ydl_opts = {"format": "bestaudio/best", "outtmpl": audio_path, "quiet": True,
"postprocessors": [{"key": "FFmpegExtractAudio",
"preferredcodec": "mp3"}]}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
info = ydl.extract_info(url, download=True)
# 2. Transcribe with AssemblyAI (speaker diarization)
config = aai.TranscriptionConfig(speaker_labels=True)
transcript = aai.Transcriber().transcribe(audio_path, config)
# 3. Convert diarized output to structured utterances
utterances = [Utterance(speaker=u.speaker, text=u.text)
for u in transcript.utterances]
return SessionTranscript(
utterances=utterances,
yt_channel=info["channel"],
yt_title=info["title"],
yt_description=info.get("description"),
yt_upload_date=info.get("upload_date"),
)
```
The @coco.fn(memo=True) decorator is a CocoIndex feature that **memoizes** the function: if you've already fetched and transcribed a video, re-running the pipeline skips it entirely. This is essential when you're iterating on the downstream extraction logic and don't want to re-download hours of audio every time.
### Two-Step LLM Extraction
Incremental relationship extraction: podcast episodes are processed through CocoIndex with Pydantic to extract persons, sessions, statements, techs, and orgs into a knowledge graph
Here's where it gets interesting. We can't extract everything in a single LLM call because of a bootstrapping problem: to extract good statements, the LLM needs to know who the speakers are (so it can attribute statements correctly). But the raw transcript only has generic labels like "Speaker A". We need to figure out who's who first.
So we split extraction into two passes. Both steps use a shared format_transcript() function that replaces raw diarization labels with names: Step 1 passes an empty map (no names known yet, so all speakers show as "Speaker A", "Speaker B"), and Step 2 passes the mapping from Step 1.
**Step 1: Identify speakers and extract metadata.** We format the transcript with generic labels and give the LLM the YouTube metadata (channel name, title, description) as context. The LLM returns who each speaker is, plus session metadata:
```python title="conv_knowledge/extract.py"
@coco.fn(memo=True)
async def extract_metadata(
reformatted_transcript: str, transcript: SessionTranscript
) -> SessionMetadata:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
return await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=SessionMetadata,
messages=[
{"role": "system", "content": METADATA_PROMPT},
{"role": "user", "content":
f"YouTube channel: {transcript.yt_channel}\n"
f"Video title: {transcript.yt_title}\n"
f"Description: {transcript.yt_description or 'N/A'}\n\n"
f"Transcript:\n{reformatted_transcript}"},
],
)
```
The LLM output is a Pydantic model, enforced by instructor:
```python title="conv_knowledge/models.py"
class SpeakerIdentification(pydantic.BaseModel):
label: str # e.g. "A", "B"
name: str # e.g. "Lex Fridman" — unidentifiable speakers are excluded
class SessionMetadata(pydantic.BaseModel):
name: str
description: str | None
date: str | None
speakers: list[SpeakerIdentification]
```
**Step 2: Extract statements with real names.** Now that we know "Speaker A" is "Lex Fridman", we reformat the transcript with real names and ask the LLM to extract thematic statements:
```python
# Reformat: replace "Speaker A" with "Lex Fridman", etc.
speaker_map = {s.label: s.name for s in metadata.speakers}
step2_text = format_transcript(transcript.utterances, speaker_map)
# Extract statements from the named transcript
stmt_extraction = await extract_statements(step2_text)
```
Each extracted statement includes the speakers who made it and the entities it mentions:
```python title="conv_knowledge/models.py"
class RawStatement(pydantic.BaseModel):
statement: str # "Scaling laws suggest larger models will improve"
speakers: list[str] # ["Lex Fridman"]
mentioned_person: list[str] # ["Ilya Sutskever"]
mentioned_tech: list[str] # ["Large language model"]
mentioned_org: list[str] # ["OpenAI"]
```
All entity names must be **self-contained**. The extraction prompt explicitly instructs the LLM: *"Never use pronouns, speaker labels, or contextual references. Every name must be a clear, unambiguous identifier that stands on its own."* This matters because statements from different episodes will later be cross-referenced: an entity name like "he" or "the host" would be meaningless outside its original transcript.
### Declaring Results with CocoIndex
After extraction, we declare the session and its statements as records in SurrealDB. CocoIndex uses a **declarative** model: you describe what the target state should look like, and the framework handles inserts, updates, and deletes.
IDs are generated using CocoIndex's IdGenerator, which produces stable IDs: the same inputs always yield the same ID, so re-running the pipeline doesn't create duplicates. Calling next_id() without arguments generates a sequential ID; calling next_id(content) incorporates the content into the ID so it remains stable even if the order of statements changes:
```python title="conv_knowledge/app.py"
id_gen = IdGenerator(youtube_id)
session_id = await id_gen.next_id()
session_table.declare_record(row=Session(
id=session_id,
youtube_id=youtube_id,
name=metadata.name,
transcript=step2_text,
# ... description, date
))
for stmt in stmt_extraction.statements:
stmt_id = await id_gen.next_id(stmt.statement)
statement_table.declare_record(row=Statement(id=stmt_id, statement=stmt.statement))
session_statement_rel.declare_relation(from_id=session_id, to_id=stmt_id)
```
### Mounted Components
Each session is processed as an independent **mounted component** via coco.use_mount(). A mounted component is a self-contained unit of work that CocoIndex tracks independently: each YouTube video ID gets its own component, so adding a new episode only processes that episode:
```python title="conv_knowledge/app.py"
for youtube_id in video_ids:
raw = await coco.use_mount(
coco.component_subpath("session", youtube_id),
process_session, youtube_id,
session_table, statement_table, session_statement_rel,
)
all_session_raw.append(raw)
```
Each process_session call returns a SessionRawEntities, the raw entity names and statement linkages that Phases 2 and 3 will need:
```python title="conv_knowledge/models.py"
@dataclass
class SessionRawEntities:
session_id: int
raw_entities: dict[str, list[str]] # e.g. {"person": ["Lex Fridman", ...]}
statements: list[IdentifiedStatement] # statements with generated IDs
```
Sessions and statements are already written to SurrealDB at this point. The raw entities are carried forward for cross-session deduplication.
## Phase 2: Entity Resolution
Phase 2 (entity resolution): raw Tech, Person, and Org names are deduplicated via embedding similarity and LLM confirmation into canonical entities stored in SurrealDB
After processing all sessions, we have a pile of raw entity names from every episode. The same entity often appears under different names: "GPT-4" vs "GPT4", "Google" vs "Google LLC", "Sam Altman" vs "Samuel Altman". We need to collapse these into canonical names.
The approach uses **embedding similarity to find candidates, then an LLM to confirm**:
1. Compute an embedding for each entity name (using a small sentence transformer model).
2. For each entity, search a FAISS index for the most similar names already processed.
3. If any are close enough (cosine distance < 0.3, i.e. similarity > 0.7), ask an LLM: "Is GPT4 the same thing as GPT-4?"
4. The LLM picks the canonical name. Build a deduplication map.
```python
async def resolve_entities(all_raw_entities: set[str]) -> dict[str, str | None]:
index = faiss.IndexFlatIP(dim)
for entity in entity_list:
vec = embeddings[entity].reshape(1, -1).copy()
faiss.normalize_L2(vec)
# Find similar entities already in the index
candidates = []
if index.ntotal > 0:
sims, idxs = index.search(vec, k=min(TOP_N, index.ntotal))
for sim, idx in zip(sims[0], idxs[0]):
if sim >= 1.0 - MAX_DISTANCE and idx >= 0:
canonical = resolve_canonical(index_names[idx], dedup)
if canonical != entity:
candidates.append(canonical)
# Deduplicate candidate list
unique_candidates = list(dict.fromkeys(candidates))
if unique_candidates:
resolution = await resolve_entity_pair(entity, unique_candidates)
# Update dedup map based on LLM decision
...
index.add(vec)
index_names.append(entity)
return dedup # e.g. {"Apple Inc.": None, "Apple": "Apple Inc.", "AAPL": "Apple Inc."}
```
The embedding step is fast and cheap: it filters out the vast majority of entity pairs that are obviously different. The LLM call only happens for the small set of near-matches, keeping the cost down. We use a smaller, cheaper model for these resolution calls since the task is simple (configurable via the RESOLUTION_LLM_MODEL environment variable).
Entity resolution runs independently per entity type (person, tech, org), so CocoIndex processes them concurrently:
```python title="conv_knowledge/app.py"
entity_dedup = dict(zip(
[cfg.name for cfg in ENTITY_TYPES],
await asyncio.gather(*(
coco.use_mount(
coco.component_subpath("resolve", cfg.name),
resolve_entities,
collect_all_raw(all_session_raw, cfg.name),
)
for cfg in ENTITY_TYPES
)),
))
```
## Phase 3: Knowledge Base Creation
With the deduplication maps ready, we write the final knowledge graph. Canonical entities become nodes, and all relationships use the resolved canonical names. A helper resolve_canonical(name, dedup) chases the dedup chain to find the root, e.g., resolve_canonical("AAPL", dedup) → "Apple Inc.":
```python title="conv_knowledge/app.py"
@coco.fn
async def create_knowledge_base(all_session_raw, entity_dedup, ...):
# Declare canonical entity nodes (name IS the id)
for cfg in ENTITY_TYPES:
dedup = entity_dedup[cfg.name]
table = entity_tables[cfg.name]
for name, upstream in dedup.items():
if upstream is None: # this name is canonical
table.declare_record(row=Entity(id=name, name=name))
# Declare relationships using canonical names
person_dedup = entity_dedup["person"]
for session_raw in all_session_raw:
for person_name in session_raw.raw_entities.get("person", []):
canonical = resolve_canonical(person_name, person_dedup)
person_session_rel.declare_relation(from_id=canonical, to_id=session_raw.session_id)
for stmt in session_raw.statements:
# person_statement: who made the statement
for speaker in stmt.raw.speakers:
canonical = resolve_canonical(speaker, person_dedup)
person_statement_rel.declare_relation(from_id=canonical, to_id=stmt.id)
# statement_mentions: what the statement is about (polymorphic)
for cfg in ENTITY_TYPES:
dedup = entity_dedup[cfg.name]
for entity_name in getattr(stmt.raw, f"mentioned_{cfg.name}"):
canonical = resolve_canonical(entity_name, dedup)
statement_mentions_rel.declare_relation(
from_id=stmt.id, to_id=canonical,
to_table=entity_tables[cfg.name]) # polymorphic target
```
The statement_mentions relationship is **polymorphic**: the target can be a person, tech, or org table. The to_table parameter tells CocoIndex which table the target ID belongs to.
## Incremental Updates
A pipeline like this isn't a one-shot job. You'll add new podcast episodes over time, and you'll want to evolve the schema, maybe adding Product, Event, or Place as new entity types, or refining the extraction prompts. CocoIndex's memoization and component model make both scenarios efficient.
### Adding new episodes
When you add a new YouTube URL and re-run the pipeline, only the new episode is processed. Existing episodes are untouched:
- **Fetch transcript**: Skipped for existing episodes (memo=True on fetch_transcript).
- **LLM extraction** (Step 1 + Step 2): Skipped for existing episodes (both are memoized).
- **Entity resolution**: Re-runs to incorporate new entity names, but compute_entity_embedding and resolve_entity_pair are memoized: existing entities' embeddings and resolution decisions are reused. Only the new names trigger fresh LLM calls.
- **Knowledge base**: CocoIndex's declarative targets handle the diff: new records are inserted, removed episodes are cleaned up.
Removing an episode works the same way: its mounted component is gone, so CocoIndex deletes its session, statements, and relationships from SurrealDB.
### Evolving the schema
Say you want to add a new entity type, like Product. You'd add it to ENTITY_TYPES, update the extraction prompt, and re-run. Here's what happens:
| Pipeline step | What happens | Why |
|---|---|---|
| Fetch transcript | **Reused** for all episodes | Memoized, no input changed |
| Step 1: Speaker identification | **Reused** for all episodes | Memoized, prompt unchanged |
| Step 2: Statement extraction | **Re-runs** for all episodes | Extraction prompt changed (new entity type) |
| Entity resolution (person, tech, org) | **Reused** | Raw entities unchanged, memoized |
| Entity resolution (product) | **Runs fresh** | New entity type |
| Knowledge base creation | **Re-declared** | New entities and relationships added |
The expensive operations (downloading audio, transcribing, and identifying speakers) are fully reused. Only the statement extraction LLM calls re-run (because the prompt changed), plus entity resolution for the new type. If you have 50 episodes and add one entity type, you save most of the compute cost.
This is the payoff of CocoIndex's memo and use_mount design: each function is memoized by its inputs, and each session is an independent component. Changes propagate only to what's actually affected.
## Running the Pipeline
### Prerequisites
- Python 3.11+, FFmpeg, Docker
- An AssemblyAI API key (for transcription)
- An OpenAI API key (for LLM extraction)
### Setup
Start SurrealDB:
```bash
docker run -d --name surrealdb --user root -p 8787:8000 \
-v surrealdb-data:/data surrealdb/surrealdb:latest \
start --user root --pass root surrealkv:/data/database
```
Set environment variables and install:
```bash
export ASSEMBLYAI_API_KEY="..."
export OPENAI_API_KEY="sk-..."
pip install -e .
```
Add YouTube URLs to input/sample.txt:
```text
# AI podcasts
https://www.youtube.com/watch?v=VIDEO_ID_1
https://www.youtube.com/watch?v=VIDEO_ID_2
```
### Build the Knowledge Graph
```bash
cocoindex update podcast_knowledge.app
```
This is incremental: re-running skips episodes that have already been processed.
## Exploring the Results
Once the pipeline completes, you can explore the knowledge graph in SurrealDB's built-in explorer (Surrealist). Connect to ws://localhost:8787, namespace cocoindex, database yt_conversations.
The graph view shows the relationships between persons and statements: each pink node is a statement, each blue node is a person:
SurrealDB graph visualization showing persons and statements with person_statement and statement_mentions edges
You can also run analytical queries. For example, finding which technologies are mentioned by the most people across all episodes:
```surql
SELECT
name,
array::distinct(
<-statement_mentions<-statement<-person_statement<-person.name
) AS persons,
array::len(array::distinct(
<-statement_mentions<-statement<-person_statement<-person.id
)) AS person_count
FROM tech
ORDER BY person_count DESC
LIMIT 15;
```
SurrealDB table showing top technologies mentioned by most people: Artificial intelligence (12), Language model (9), Machine learning (7)
Other useful queries:
```surql
-- All statements a person made
SELECT <-person_statement<-person.name AS speaker, statement FROM statement;
-- Full graph around a person
SELECT name,
->person_session->session.name AS sessions,
->person_statement->statement.statement AS statements
FROM person;
-- All entities involved in statements
SELECT statement,
->statement_mentions->person.name AS persons,
->statement_mentions->tech.name AS techs,
->statement_mentions->org.name AS orgs
FROM statement;
```
## Summary
We built a pipeline that turns YouTube podcasts into a queryable knowledge graph:
1. **Fetch**: Download audio and transcribe with speaker diarization.
2. **Extract** (two-step): First identify speakers using YouTube metadata, then extract statements with real names, solving the bootstrapping problem of needing to know who's speaking before you can attribute what they said.
3. **Resolve**: Deduplicate entities across episodes using embedding similarity + LLM confirmation: cheap embeddings filter candidates, expensive LLM calls only happen for near-matches.
4. **Store**: Write the knowledge graph to SurrealDB with CocoIndex's declarative target model.
Because every expensive step is memoized and each session is an independent component, the pipeline is fully incremental, whether you're adding episodes or evolving the schema. See Incremental Updates for details.
The full source code is at CocoIndex Examples: podcast_to_knowledge. To stay updated on CocoIndex, star us on GitHub.
## Support us ❤️
If this walkthrough was useful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub. It's how other developers discover the project, and every star keeps us motivated to ship better tools for building knowledge graphs, RAG pipelines, and live data infrastructure for AI.
Got a podcast, a meeting archive, or any other corpus you want to turn into a graph? We'd love to hear what you're building: drop by our Discord or open an issue on GitHub.
---
# From pickle to type-guided, safer Python serialization
URL: https://cocoindex.io/blogs/type-guided-serde/
Date: 2026-03-27
Summary: How CocoIndex moved from pickle to type-guided serialization that uses Python type hints to pick the right serializer, no decorators or registration needed.
CocoIndex is a framework for building incremental data pipelines. It has a Rust core for performance and exposes a Python SDK for users to define their pipelines. Under the hood, the engine needs to serialize and deserialize Python objects constantly, caching function results, persisting pipeline state, tracking records for change detection, with the serialized data crossing the Rust/Python boundary and stored by the Rust core.
In CocoIndex v0, we had a closed type system: a fixed set of supported data types, with serialization handled entirely in Rust. Safe and fast, but too rigid. Users couldn't use their own data types without converting them into our type system first.
When we started building v1, we wanted to natively support any Python data type. Pickle was the obvious first choice.
## Pickle: Fast to Prototype, Painful to Live With
Pickle is Python's built-in serialization. It handles virtually any Python object out of the box: dataclasses, NamedTuples, Pydantic models, nested containers, custom classes. For early prototyping under prerelease, it was perfect: one line of code, and everything just worked.
But pickle has two fundamental problems that make it unsuitable for production.
**Security.** Pickle executes arbitrary code during deserialization. A crafted payload can run any Python code on your machine. In CocoIndex's case, the serialized data lives in internal storage (a database managed by the framework), so the risk requires an attacker to have access to that storage, which is unlikely for most deployments. Still, defense in depth matters: if the storage is ever exposed through a misconfiguration or a compromised database, pickle becomes an escalation path from data access to code execution. We didn't want that risk in the foundation of our persistence layer.
Pickle escalation path: a crafted blob in storage becomes arbitrary code execution when CocoIndex calls pickle.loads
**Overhead.** Pickle encodes type information into every value: each value is a sequence of opcodes that tells the deserializer what type to reconstruct and how. This makes pickle self-describing, but at a cost. The integer 1024 takes 15 bytes in pickle, while msgpack encodes it in just 3. Msgpack can be this compact because it relies on the deserializer already knowing the expected type:
| Value | pickle | msgpack | Ratio |
|---|---|---|---|
| "" (empty string) | 15 bytes | 1 byte | 15x |
| 1024 | 15 bytes | 3 bytes | 5x |
| "hello" | 20 bytes | 6 bytes | 3.3x |
| {"key": [1,2,3]} | 31 bytes | 9 bytes | 3.4x |
| Point(42, "origin") | 63 bytes | 16 bytes | 3.9x |
For a pipeline that caches thousands of intermediate results, this overhead adds up.
## Whitelisted Pickle: Secure but Tedious
Our first attempt to fix the security problem was a restricted unpickler. We maintained an allowlist of types that were safe to deserialize, and rejected everything else. Users could opt their types into the allowlist with a @coco.unpickle_safe decorator:
```python
@coco.unpickle_safe
@dataclass
class DocumentChunk:
text: str
embedding: list[float]
```
This solved the security problem. But it introduced a new one: annotation burden.
Every type that appeared anywhere in a serialized object graph needed the decorator, not just the top-level type, but every type nested inside it. Miss one, and you get a runtime error.
We experienced this firsthand when building the conversation_to_knowledge example. It needed **9** @coco.unpickle_safe annotations across the codebase. Getting there took **4 rounds** of run-fail-annotate-repeat: run the pipeline, hit an error about an unregistered type, add the decorator, run again, hit the next error. Four times.
For our own example. Imagine what this is like for users building real pipelines with dozens of model types.
## Type-Guided Deserialization: Letting Type Hints Do the Work
We stepped back and asked: what do we actually have that describes the shape of users' data? The answer was already in their code: **type hints**.
In CocoIndex, there are three places where data gets serialized and deserialized: memoized function return values (the most common case), memo state values for change detection, and target tracking records. For memoized return values, the type hint is right there in the function signature:
```python
@coco.fn(memo=True)
async def embed_chunk(chunk: str) -> list[float]:
return await embedding_model.embed(chunk)
```
The return type list[float] tells us exactly how to deserialize the cached result. No extra annotation needed: the type hint that users already write for readability and IDE support doubles as the deserialization schema.
This led us to a type-guided serialization system built on msgspec, a fast msgpack-based serialization library for Python. Msgspec natively handles dataclasses, NamedTuples, and standard types (str, int, list, dict, datetime, UUID, etc.), and it does so by relying on type information at deserialization time.
### The routing byte
Different types need different serialization engines. We use a single routing byte at the start of each payload to select the engine:
| Byte | Engine | Used for |
|---|---|---|
| 0x01 | msgspec (msgpack) | Dataclasses, NamedTuples, primitives, collections |
| 0x02 | Pydantic | BaseModel subclasses |
| 0x80 | Pickle | Opted-in types, legacy data |
Serialization checks types in priority order: explicit pickle opt-in first (the user specifically requested it), then Pydantic models, then msgspec as the default. Deserialization reads the routing byte and dispatches accordingly.
For **Pydantic models**, we serialize via model_dump(mode="json") into msgpack, and deserialize via TypeAdapter.validate_python(). This avoids pickle entirely while preserving Pydantic's validation semantics.
For types that msgspec can't handle natively, like numpy arrays or pathlib paths, users can opt in to pickle serialization with @coco.serialize_by_pickle. These types get the pickle routing byte (0x80), still through a restricted unpickler for safety.
### What changed for users
The conversation_to_knowledge example? All 9 @coco.unpickle_safe annotations, gone. Users define their data types with standard Python type hints, and serialization just works:
```python
@dataclass
class DocumentChunk:
text: str
embedding: list[float]
metadata: dict[str, str]
```
No decorators. No registration. No run-fail-annotate-repeat cycle. The framework reads the type hint and picks the right serializer and deserializer automatically.
## The Edge Cases: When Round-Trip Isn't Perfect
We'd be lying if we said this approach is perfect for every type. There are edge cases with **union types** that users should be aware of, and these aren't specific to CocoIndex. They're fundamental to how msgspec and Pydantic handle ambiguous types.
### msgspec: loud failure
msgspec rejects ambiguous unions at construction time:
```python
import msgspec.msgpack
# These all raise TypeError immediately:
msgspec.msgpack.Decoder(str | datetime.date) # two str-like types
msgspec.msgpack.Decoder(list[int] | tuple[int, ...]) # two array-like types
msgspec.msgpack.Decoder(bytes | bytearray) # two bytes-like types
```
This is actually a feature. msgspec tells you upfront that these types can't be distinguished on the wire, rather than silently corrupting your data.
### Pydantic: silent surprise
Pydantic is more permissive: it accepts date | str, but the behavior after a serialization-deserialization round-trip can be surprising.
Consider a Pydantic model with a date | str field:
```python
class Event(BaseModel):
ts: datetime.date | str
e = Event(ts=datetime.date(2024, 1, 15))
```
If you serialize this to JSON (or msgpack), date becomes the string "2024-01-15". On deserialization, Pydantic sees a string input and needs to decide: is it a str or a date?
With Pydantic v2's default **smart mode**, the answer is always str. Smart mode scores candidates by exactness: a string input is an *exact match* for str but only a *lax match* (requires coercion) for date. Exact wins. Ordering doesn't matter: date | str and str | date both produce str.
**Left-to-right mode** behaves differently. With date | str ordering, Pydantic tries date first, the lax coercion from "2024-01-15" succeeds, and you get your date back:
```python
from pydantic import Field
from typing import Annotated
class Event(BaseModel):
# Left-to-right mode: more specific type first
ts: Annotated[datetime.date | str, Field(union_mode='left_to_right')]
```
But this requires users to explicitly opt in to left-to-right mode AND put the more specific type first.
### The fundamental issue
This isn't a bug in any library. It's inherent to the wire format. Msgpack (like JSON) has no date type: dates are serialized as strings. Once the type information is erased on the wire, no amount of clever decoding can reliably recover it when multiple types map to the same wire representation.
The practical advice: **avoid union types where members share a wire representation** in data objects that go through serialization. Use unambiguous types, or use Pydantic's discriminated unions with a literal tag field.
## What We Learned
Four lessons: type hints as a source of truth, fail loudly on ambiguity, design with the wire format, and stage simple-to-production with a plan
**Type hints are an underutilized source of truth.** Python developers already annotate their data types. Using those annotations for serialization is a natural extension: no new concepts, decorators, or registration APIs.
**Make ambiguity loud, not silent.** msgspec's approach of rejecting ambiguous unions at construction time is better than silently producing wrong results. When building a serialization layer, fail early and clearly.
**Don't fight the wire format.** If the wire format can't distinguish two types, your serialization layer can't either. Design your data types around what the serialization format can express, not the other way around.
**Start simple, but have a plan.** Pickle was right for prototyping. The whitelisted pickle was right for the security fix. The type-guided approach was right for production. Each step taught us what the next step needed to be.
## Learn More
- CocoIndex on GitHub: the open-source data indexing framework
- CocoIndex v1 Documentation: getting started guides and API reference
- Building an Invisible Daemon: how we built the local daemon architecture behind CocoIndex
## Support us
⭐ Star CocoIndex on GitHub and share with your community if you find it useful!
---
# Invisible Daemon: architecture patterns for local dev tools
URL: https://cocoindex.io/blogs/building-an-invisible-daemon/
Date: 2026-03-24
Summary: Five patterns for a Python CLI background daemon that auto-starts, upgrades transparently, and shuts down fast, from the daemon behind cocoindex-code.
Building a Python CLI tool with a background daemon? The five patterns that eliminate manual start/restart, connection leaks, and 15-second shutdown times are:
1. **Auto-start on first use** via a Unix socket probe (no daemon start command).
2. **Version handshake** on every connection for transparent upgrades after pipx upgrade.
3. **Per-request connections** instead of persistent sessions: connection leaks become impossible by construction.
4. **Resource closure** instead of thread signaling: close the listener; don't poll a threading.Event.
5. **PID file removal** as the exit signal: os.kill(pid, 0) returns True for zombies.
This is how we built the daemon behind cocoindex-code (ccc), an embedded AST-based semantic code search tool for Claude Code, Codex, and Cursor.
**Why this post exists.** cocoindex-code is a local MCP server and CLI that gives coding agents semantic understanding of your whole codebase via AST-based chunking: Tree-sitter parses code by syntax (functions, classes, methods), and only changed chunks are re-embedded on update. The daemon described here is the background process that makes that possible: it loads the ML model once, watches for file changes, and serves searches in under a second. If you're building something similar (a local AI context layer, a code-aware MCP server, any dev tool with a heavy background process), the patterns below are what we landed on after shipping to production.
cocoindex-code
We moved cocoindex-code from a monolithic per-session process (load model, index, search, exit) to a persistent daemon with a thin CLI (ccc) and a lightweight MCP client as frontends. This unlocked a benefit beyond performance: humans can call ccc search "auth logic" directly from the terminal and pipe it into other tools.
The journey from "obvious daemon" to production-ready was full of design decisions, and the best ones were usually about what to *remove*. Here's what the user sees:
```bash
ccc init # one-time setup: creates settings files (no daemon)
ccc index # build the search index (daemon auto-starts here)
ccc search "auth logic" # semantic search (daemon handles it)
ccc search --refresh "auth" # re-index then search (two daemon requests)
```
ccc init is purely local: it creates configuration files and a .gitignore entry. Every other command talks to the daemon. But the user never starts, stops, or restarts the daemon manually. It just works.
The initial daemon was largely built by Claude Code: persistent connections, async handler loops, multi-step shutdown. It worked, but it was complex in ways we didn't appreciate until we looked closely. The simplifications came from stepping back and questioning the design rather than fixing individual bugs.
This post covers the patterns we landed on, in the hope that they're useful to anyone building a local daemon for developer tools.
## How do you make a local daemon invisible to the user?
The most important design constraint: **users should never think about the daemon**. No ccc daemon start in the setup guide. No "please restart the daemon after upgrading." No stale config surprises.
### How do you auto-start a Python daemon on first use without a manual command?
The first time a user runs ccc index or ccc search, the client tries to connect to the daemon's Unix socket. If it fails, it starts the daemon as a detached background process and waits for the socket to appear.
This logic is built into the lowest-level connection function, not into a separate "ensure daemon" step:
```python
_daemon_ensured = False
def _connect_and_handshake() -> Connection:
global _daemon_ensured
if _daemon_ensured:
return _raw_connect_and_handshake()
# First connection — auto-start/restart as needed.
try:
conn = _raw_connect_and_handshake()
_daemon_ensured = True
return conn
except DaemonVersionError:
stop_daemon()
except (ConnectionRefusedError, OSError):
pass
start_daemon()
_wait_for_daemon()
# ... retry and return
```
A module-level _daemon_ensured flag means the first call in a CLI session pays the startup cost. Every subsequent call connects directly, with no separate "ensure" step or throwaway probe connection.
### How do you auto-restart a daemon when the CLI tool upgrades?
When a user runs pipx upgrade cocoindex-code, the next CLI command should just work, using the new version. No manual restart.
Every connection starts with a version handshake. The client sends its version; the daemon responds with its version:
```python
# In the client
conn.send_bytes(encode_request(HandshakeRequest(version=__version__)))
resp = decode_response(conn.recv_bytes())
if not resp.ok or _needs_restart(resp):
raise DaemonVersionError(resp)
```
On the first call (_daemon_ensured = False), a version mismatch triggers a restart: stop the old daemon, start a new one. On subsequent calls, a mismatch raises an error (the daemon was replaced mid-session, which is unusual and worth surfacing).
### How should a local daemon handle config — restart, reload, or re-read fresh?
Two-Level Settings: global settings (stateful, mtime-checked, restart on change) vs project settings (stateless, re-read fresh on every indexing pass)
We have two levels of configuration:
**Global settings** (~/.cocoindex_code/global_settings.yml): embedding model, API keys, environment variables. These affect daemon-wide state: the ML model is loaded once at startup, environment variables are set once. When these change, the daemon *must* restart.
**Project settings** ($PROJECT/.cocoindex_code/settings.yml): include/exclude patterns, language overrides. These only affect per-operation behavior: which files to index, how to parse them.
Both files are created automatically by ccc init with sensible defaults. Users only edit them if they need to customize.
For global settings, the daemon records the file's mtime (as integer microseconds) at startup and reports it in the handshake response. The client compares it with the current mtime. Different? Restart.
For project settings, we do nothing special. Each indexing pass reads the settings file fresh from disk, so there is no staleness check or cache to invalidate.
The principle: **minimize what the daemon holds as state**. Global settings are unavoidable state (the loaded model), so we detect staleness. Project settings can be stateless, so we make them stateless.
## Per-request vs. persistent connections — which is better for local IPC?
**Per-request.** Each request opens a fresh Unix socket connection, sends, receives, closes. Setup overhead is ~0.1ms, negligible for human-initiated CLI commands. Persistent connections require idle handler tasks, shutdown signaling across async/thread boundaries, and connection lifecycle management at every call site. We tried persistent connections first. We removed them. This was the single biggest simplification, and it came from a bug.
### Why we tried persistent connections first
Our first design followed the obvious pattern: the client connects, handshakes, then sends multiple requests on the same connection. The daemon handler loops, reading and dispatching requests until the client disconnects:
```python
# The original handler — persistent connection with recv loop
async def handle_connection(conn, registry, ...):
loop = asyncio.get_event_loop()
def _recv():
while not shutdown_event.is_set():
if conn.poll(0.5): # wake every 0.5s to check shutdown
return conn.recv_bytes()
raise EOFError("shutdown")
try:
while not shutdown_event.is_set():
data = await loop.run_in_executor(None, _recv)
req = decode_request(data)
# ... handshake, dispatch, respond ...
except (EOFError, OSError, asyncio.CancelledError):
pass
finally:
conn.close()
```
This handler had to deal with a lot:
- A _recv helper that polls every 0.5 seconds, checking a threading.Event for shutdown
- An executor thread per connection, blocked in conn.poll()
- A while loop waiting for the next request that may never come
- Cancellation handling for shutdown
Two problems emerged from this design:
**Leaked connections.** CLI commands didn't always close their connections. Each leaked connection meant a handler task stuck in _recv, waiting forever. We added __enter__/__exit__, with blocks, .close() calls, but it was whack-a-mole.
**Shutdown was broken.** Our test suite took 15 seconds per test teardown. The stop_daemon() function had a 5-step escalation (StopRequest, wait 5 seconds, SIGTERM, wait 2 seconds, SIGKILL) and always fell through to SIGKILL. Three bugs were stacked on top of each other:
1. asyncio.Event.is_set() called from executor threads, but asyncio.Event is not thread-safe. The _recv threads never saw the shutdown signal.
2. asyncio.run() cleanup hanging: it calls shutdown_default_executor(wait=True), which waits for the _recv threads that would never exit (because of bug #1).
3. os.kill(pid, 0) returning True for zombie processes, so the client kept waiting for a process that had already exited.
We fixed each bug individually. But the real insight came when we stepped back: **all three bugs existed because of persistent connections**. Persistent connections required idle handler tasks. Idle tasks required a shutdown signal. The signal had to cross async-to-thread boundaries. And every layer of that chain had a bug.
> We fixed three stacked shutdown bugs with careful threading, event types, and process liveness checks. Then we removed persistent connections and all three bugs disappeared. The debugging was correct. The design was wrong.
### The per-request model: read two messages, respond, close
Instead of fixing the plumbing, we removed the need for it. What if each request just opens a fresh connection?
```text
connect → handshake → one request → response(s) → close
```
The handler becomes straight-line code:
```python
# The current handler — per-request, no loop, no polling
async def handle_connection(conn, registry, ...):
loop = asyncio.get_event_loop()
try:
# 1. Handshake
data = await loop.run_in_executor(None, conn.recv_bytes)
req = decode_request(data)
# ... validate handshake, send response ...
# 2. Single request
data = await loop.run_in_executor(None, conn.recv_bytes)
req = decode_request(data)
result = await _dispatch(req, registry, ...)
# 3. Send response(s) and done
if isinstance(result, AsyncIterator):
async for resp in result:
conn.send_bytes(encode_response(resp))
else:
conn.send_bytes(encode_response(result))
except (EOFError, OSError, asyncio.CancelledError):
pass
finally:
conn.close()
```
Read two messages. Respond. Close. That's it.
### What per-request connections eliminated
The diff wasn't just shorter. Entire concepts disappeared:
- The _recv polling loop (0.5-second wakeups checking shutdown_event): **gone**
- shutdown_event (threading.Event): **gone entirely**
- The DaemonClient class, __enter__/__exit__, .close(): **gone**
- with client: blocks in every CLI command: **gone**
- Connection leak bugs: **impossible by construction**
- All three shutdown bugs: **gone** (no idle handler tasks means nothing to signal)
The client module went from a stateful class to module-level functions:
```python
# No class, no state, no close() to forget
def index(project_root, on_progress=None, on_waiting=None):
conn = _connect_and_handshake()
try:
conn.send_bytes(encode_request(IndexRequest(project_root=project_root)))
# ... read streaming responses ...
finally:
conn.close()
```
### Compound operations — open a second connection, not a second protocol
What about operations that need multiple requests? ccc search --refresh needs to index first, then search. Simple. Two connections:
```python
def search_cmd(project_root, refresh=False):
if refresh:
client.index(project_root) # connection 1
client.search(project_root, query) # connection 2
```
MCP background indexing? Just call client.index() from a thread. It opens its own connection, with no shared state or connection-isolation bugs to reason about.
The overhead is negligible: Unix socket setup is ~0.1ms, the handshake is a tiny msgspec payload, and CLI commands are human-initiated. Streaming responses (like indexing progress) still work: the connection lives for one request's full response stream. It just doesn't linger afterwards.
> Per-request connections can't leak by construction. The overhead of Unix socket setup is ~0.1ms, negligible for developer tools.
## How do you cleanly shut down a Python daemon without falling through to SIGKILL?
Per-request connections solved the hardest shutdown problems (no idle tasks to signal). The remaining lesson: **design shutdown around closing resources, not coordinating threads**.
The accept loop runs in a background thread, blocking on listener.accept(). To stop it, we don't set a flag and hope the thread notices: we close the listener. accept() raises OSError, the thread exits immediately:
```python
def _accept_loop():
while True:
try:
conn = listener.accept()
loop.call_soon_threadsafe(_spawn_handler, conn)
except OSError:
break # listener was closed — exit
```
The same principle applies to the full shutdown sequence:
```python
try:
loop.run_forever()
finally:
listener.close() # 1. stop accepting
for task in tasks: task.cancel() # 2. cancel handlers
if tasks:
loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
registry.close_all() # 3. release resources
loop.close()
# 4. remove socket and PID file ...
os._exit(0) # 5. skip slow Python teardown
```
A few details worth noting:
**PID file as the exit signal.** The client needs to know when the daemon has finished cleaning up.
:::note
os.kill(pid, 0) returns True for zombie processes: they're still in the process table. The fix: the daemon removes its PID file as the last cleanup step, and the client polls for its absence instead.
:::
**os._exit(0) for fast teardown.**
:::note
Python's normal exit takes seconds when torch is loaded (atexit handlers, GC finalizers, module teardown). os._exit(0) skips all of that, safe because we've already cleaned up explicitly.
:::
(One caveat: os._exit(0) terminates the *entire process*, so we guard it with threading.current_thread() is threading.main_thread() for tests that run the daemon in a thread.)
**Event loop on the main thread.** The accept loop runs in a background thread, but the asyncio event loop stays on the main thread. This is required for loop.add_signal_handler() to work: SIGTERM and SIGINT call loop.stop(), which causes run_forever() to return into the finally block above.
## The Result
```text
per-request connections
CLI / MCP ──────────────────────────────► Daemon
(thin) connect → handshake → (persistent)
request → response → - event loop (main thread)
close - accept loop (background thread)
- project registry
- ML model (loaded once)
```
Before and after:
| | Before | After |
|---|---|---|
| Connection handler | ~70 lines, recv loop, polling, shutdown event | ~40 lines, straight-line read-respond-close |
| Client | DaemonClient class, __enter__/__exit__, .close() | Module-level functions, no state |
| Shutdown concepts | threading.Event, executor polling, 5-step escalation | listener.close(), loop.stop(), os._exit(0) |
| Daemon stop time | ~15 seconds (fell through to SIGKILL) | <1 second |
| Impact for coding-agent users | Per-session ML model reload on every ccc search | Model loaded once; search returns in <1 second |
## Patterns for Local Daemon Architecture
Five patterns at a glance: make the daemon invisible, prefer stateless IPC, categorize state, close don't signal, and question the design when debugging gets complex
If you're building a local daemon for developer tools, here's what we'd suggest:
**Make the daemon invisible.** Auto-start on first use. Auto-restart on version mismatch. Auto-reload config where possible. The user's mental model should be "I run CLI commands". The daemon is an implementation detail.
**Prefer stateless IPC.** Per-request connections can't leak by construction. The overhead of Unix socket setup (~0.1ms) is negligible for developer tools. The handshake on every connection catches staleness faster.
**Categorize your state.** What *must* live in the daemon (loaded ML model, open databases) vs what can be read fresh (config files, file lists)? Only restart for the former. The less state the daemon holds, the fewer staleness bugs you'll have.
**Design shutdown around resource closure, not signaling.** Close the listener to interrupt accept(). Remove the PID file to signal completion. Don't poll threads to check if they've noticed an event. Close the thing they're waiting on.
**When debugging gets complex, question the design.** We fixed three stacked shutdown bugs with careful threading, event types, and process liveness checks. Then we removed persistent connections and all three bugs disappeared. The debugging was correct. The design was wrong.
## Support the Project
If you find this article helpful, please check out cocoindex-code for the implementation and star the project if you like it!
---
# CocoIndex Changelog 0.3.27 - 0.3.34
URL: https://cocoindex.io/blogs/changelog-0327-0334/
Date: 2026-03-10
Summary: Featuring five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.
Since the last release, CocoIndex shipped **8 releases (0.3.27–0.3.34)**, and this cycle was focused on one clear goal: making CocoIndex the **context engine** that adapts to the knowledge ecosystems (vector stores, graph databases, or analytics backends your agents need) while getting even smarter about keeping itself healthy underneath.
We are super excited to build with the amazing infrastructure projects together on the path of agents going to production in 2026.
Agents don't just need fresh data: they need it **wherever it lives**. Different teams pick different databases by their use cases. Some want LanceDB for multi-modal capability. Some want ChromaDB for local-first development. Graph-heavy workloads need FalkorDB or Ladybug. Analytics & Search workloads want Apache Doris. And… a lot more great projects in the ecosystems coming soon. This cycle, we made sure CocoIndex speaks all of them, so you pick the best target for your use case, and CocoIndex handles the rest.
**CocoIndex helps you ace the data for agents without heavy data engineering work.** Continuous fresh, structured and programmable context for AI, from PDFs, Codebase, Emails, Screenshots, Meeting Notes, ... Always up-to-date, at any scale, ultra-performant, with a smart incremental engine. **⭐ Star the repo**
Here is what we shipped since the last release.
**0.3.27–0.3.34 was about target reach and engine hardening**: five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.
## Core capability
CocoIndex's core engine got sharper this cycle. The runtime now automatically cleans up stale derived metadata and data when you rename or remove a source: no manual intervention, no orphaned rows. Tighter validation catches bad table and column names before they hit your database. A new print_stats option surfaces pipeline statistics directly in the update() API. And for the first time, CocoIndex supports **Python 3.14 free-threaded mode**, running without the GIL for true parallelism in Python-heavy pipelines.
### Automatic stale source cleanup
When CocoIndex detects that a source has been renamed or removed, it now automatically streams through tracking metadata, deletes corresponding target rows, and removes stale state records, all during flow setup with controlled parallelism to avoid overloading your database (**#1468**).
### Python 3.14 free-threaded support
CocoIndex now runs natively in Python 3.14's free-threaded mode (no GIL). CI builds ship free-threaded wheels on manylinux, so you can leverage true multi-threaded Python without any config changes (**#1357**, **#1609**, **#1638**).
### Tighter target validation
Table names and column names are now validated before target setup, catching naming issues early instead of surfacing cryptic database errors downstream (**#1714**).
### Execution plan initialization fix
Fixed a race condition where the execution plan could be used before target setup completed, ensuring correct pipeline initialization order (**#1715**).
### Pipeline stats in update() API
Added a print_stats option to the update() API, giving you runtime statistics directly at the call site, useful for monitoring, benchmarking, and debugging pipeline performance (**#1525**).
### Custom database schema for internal tables
CocoIndex's internal metadata and tracking tables now support custom Postgres schemas via explicit schema.table qualification. This means cleaner isolation in shared databases: no more search_path side effects (**#1459**).
### Slow HTTP request warnings
The runtime now warns when HTTP requests (e.g., to LLM providers or external APIs) are running slow, making it easier to spot bottlenecks in your pipeline (**#1617**).
### Enhanced tracing
Improved tracing coverage across the engine for better observability and debugging (**#1618**).
### ColPali/ColQwen organization prefix fix
Fixed a crash when using ColPali or ColQwen models with organization-prefixed model names (e.g., org/model-name) (**#1624**).
Together, these changes make the core engine more self-healing, more observable, and ready for the next generation of Python runtimes.
## Integrations
This cycle is a **massive expansion of where CocoIndex can send data**: five brand-new target connectors, filesystem-level change detection for local files, and deeper LanceDB indexing support.
### Sources
#### Local file change notifications
The LocalFile source now supports **native filesystem change notifications** via the notify crate. Enable the new watch_changes option and CocoIndex will detect file changes through OS-level events instead of polling: faster reaction times, less CPU overhead, and your include/exclude patterns are respected automatically (**#1669**).
### Targets
#### Apache Doris
Full-featured native support for **Apache Doris 4.0** (and VeloDB Cloud) as a vector database target. Includes HNSW and IVF vector indexes, inverted full-text search indexes with Unicode/English/Chinese parsers, Stream Load ingestion with batching and exponential backoff, and upsert/delete support. A complete text embedding example ships alongside (**#1475**, **#1642**).
#### ChromaDB
Initial ChromaDB integration with support for PersistentClient, CloudClient (with SPANN+SPFresh), and HttpClient. Includes HNSW configuration options, explicit document_field specification, and early validation for vector fields (**#1548**, **#1663**).
#### FalkorDB
New **property graph target** powered by FalkorDB, a high-performance graph database built on Redis. Full support for nodes, relationships, vector and FTS indexes, and batch operations. Ships with a complete docs-to-knowledge-graph example using LLM extraction (**#1481**).
#### Ladybug (successor to Kuzu)
Kuzu was archived in October 2025. **Ladybug** is the active community fork and a drop-in replacement with an identical Cypher API. CocoIndex now targets Ladybug natively, with backward-compatible Kuzu* type aliases so existing pipelines migrate seamlessly (**#1487**).
#### LanceDB
Added VectorIndexMethod support for **HNSW and IVF Flat indexes** in LanceDB, giving you fine-grained control over index configuration (**#1409**).
#### Pinecone
Pinecone is now a built-in target connector: bring your managed Pinecone indexes directly into CocoIndex pipelines with a text embedding example included out of the box (**#1556**).
These integrations bring CocoIndex's supported target ecosystem to **10+ databases**, covering vector stores, graph databases, relational systems, and analytics engines.
## Build with CocoIndex
### SEC EDGAR financial analytics with CocoIndex + Apache Doris
SEC EDGAR Financial Analytics with CocoIndex + Apache Doris
A full **multi-source ETL pipeline** demonstrating how to build a financial analytics system from SEC EDGAR data using CocoIndex and Apache Doris.
The example ingests three data sources: TXT filings (10-K/10-Q risk factors), JSON company facts from the SEC XBRL API, and PDF proxy statements. It then indexes them into Apache Doris with **hybrid search** (vector + keyword via Reciprocal Rank Fusion), **topic filtering** with JSON arrays, and **portfolio-level search** with per-company aggregation.
It also demonstrates PII scrubbing before indexing and incremental updates with caching, a production-ready pattern for financial data pipelines.
Check out the **tutorial** for more details.
### Slides-to-Speech: Turn slide decks into searchable, narrated knowledge
Slides-to-Speech: Turn slide decks into searchable, narrated knowledge
Static slide decks sitting in Google Drive are one of the most underutilized knowledge sources in any organization. We published a full walkthrough showing how to build a **continuously updating slides-to-speech pipeline** using CocoIndex and LanceDB.
The pipeline watches a Drive folder for new or modified PDF decks, converts each slide to an image with **PyMuPDF**, extracts structured speaker notes using **BAML + Gemini Vision**, synthesizes narration with **Piper-TTS**, generates embeddings, and indexes everything into **LanceDB**: images, transcripts, audio bytes, and metadata all in one place.
**What's cool about this example:**
- **Incremental by design**: Only changed slides get reprocessed. Update one slide in a 200-page deck? CocoIndex touches only that slide, regenerating its audio, transcript, and embedding while leaving the rest untouched.
- **True multimodal search**: Semantic queries return the top matching slides with images, speaker notes, and synced audio, not just filenames.
- **Local-first TTS**: Neural text-to-speech runs locally via Piper, so there are no API costs for audio generation.
- **Living knowledge base**: What used to be a static PDF becomes a searchable, listenable, always-fresh dataset ready for agents, accessibility tools, or video generation.
Check out the **blog post** for more details.
## Thanks to the community 🤗🎉
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @tomz-alt
Thanks @tomz-alt for adding the Apache Doris 4.0 vector database connector (a full-featured target with vector indexes, full-text search, and Stream Load ingestion) and for building the SEC EDGAR financial analytics example demonstrating multi-source ETL with hybrid search, plus improving the Doris text embedding example and cleaning up the SEC EDGAR tutorial for an end-to-end demo experience.
### @prrao87
Thanks @prrao87 for adding Ladybug (the Kuzu successor) as a target, ensuring CocoIndex users have a smooth migration path to the actively maintained graph database fork.
### @galshubeli
Thanks @galshubeli for adding the FalkorDB target, a complete property graph integration with nodes, relationships, vector/FTS indexes, and a docs-to-knowledge-graph example.
### @Geoff-Robin
Thanks @Geoff-Robin for adding custom database schema support for CocoIndex's internal tables, enabling cleaner isolation in shared Postgres databases with explicit schema.table qualification.
### @wuxiaobai24
Thanks @wuxiaobai24 for adding Python 3.14 free-threaded support, configuring PyO3 for GIL-free operation and bringing CocoIndex to the cutting edge of Python parallelism.
### @ceshine
Thanks @ceshine for fixing the ColPali/ColQwen crash when using models with organization prefixes, unblocking multimodal embedding workflows.
### @LED-0102
Thanks @LED-0102 for adding filesystem change notifications to the LocalFile source using the notify crate, enabling OS-level file watching for faster live updates.
### @prithvi-moonshot
Thanks @prithvi-moonshot for adding Pinecone as a target connector with a text embedding example, bringing managed vector search into the CocoIndex ecosystem.
### @Haleshot
Thanks @Haleshot for building the initial ChromaDB target integration with support for PersistentClient, CloudClient, and HttpClient, fixing ChromaDB collection creation on first run, adding ChromaDB documentation, adding llms.txt to docs, adding short flag aliases for common CLI options, clarifying .env usage across examples, and migrating CI from pre-commit to prek.
### @thisisharsh7
Thanks @thisisharsh7 for adding automatic cleanup of stale source data, ensuring renamed or removed sources don't leave orphaned tracking metadata or target rows.
### @majiayu000
Thanks @majiayu000 for adding VectorIndexMethod support for HNSW and IVF Flat indexes in LanceDB, expanding vector index configuration options.
## Summary
CocoIndex continues evolving into a universal, resilient data engine for the modern AI stack.
This cycle focuses on target ecosystem breadth, engine self-healing, and Python 3.14 readiness, ensuring your pipelines can write to any database your agents need, clean up after themselves, and run on the latest Python runtime without compromise.
If you like this project, please support us with a star ⭐ at https://github.com/cocoindex-io/cocoindex !
🔥🔥 We are cooking something big! v1 is coming soon. Stay tuned! If you read all the way here, you are a true fan!
V1 is Coming
---
# CocoIndex joins the GitHub Secure Open Source Fund
URL: https://cocoindex.io/blogs/cocoindex-joins-security-github-secure-open-source-fund/
Date: 2026-02-17
Summary: CocoIndex joined the GitHub Secure Open Source Fund, hardening the AI data infrastructure developers depend on with threat modeling, CodeQL, and audits.
At CocoIndex, we've always believed that the strength of open source lies in its community and in building transparent, reliable infrastructure together.
CocoIndex is a data transformation framework that sits between raw data and the AI systems that depend on it, and that position makes security a core requirement.
When developers trust CocoIndex to process sensitive documents, codebases, and enterprise data, we owe them the highest standards.
We are honored to be selected to join the GitHub Secure Open Source Fund, to formalize our commitment and accelerate our security practices.
## What is GitHub's Secure Open Source Fund?
GitHub Secure Open Source Fund
When the Log4j "Log4Shell" vulnerability hit in December 2021, it showed that a bug in one small, underfunded open-source project can ripple through the entire global tech stack.
Most apps today rely on countless dependencies, many maintained by unpaid volunteers, and a weakness in any one of them reaches everyone downstream.
Recognizing this, GitHub launched the Secure Open Source (SOS) Fund in late 2024 to strengthen the foundation of open-source security.
The program pairs direct funding with a focused three-week security sprint that combines expert guidance, better tooling, and a
community of security-conscious maintainers.
In its first two cohorts, 125 maintainers from 71 projects participated, collectively fixing over 1,100 vulnerabilities, disclosing 50+ new CVEs, and resolving nearly 270 secret exposures.
Results from the first two cohorts of the GitHub Secure Open Source Fund: 71 projects, 125 maintainers, over 1,100 vulnerabilities fixed, more than 50 new CVEs disclosed, and nearly 270 secret exposures resolved.
Participants also exchanged knowledge, leveraged AI-assisted tools like GitHub Copilot for security tasks, and developed incident response plans that many have since published for others to reuse.
## What is CocoIndex?
CocoIndex
Dynamic context engineering for AI.
As AI systems evolve from chatbots to autonomous agents, one hard problem keeps coming back: keeping context accurate and up to date as the underlying data changes.
Enterprises sit on massive, dynamic datasets (documents, codebases, emails, APIs), yet most data infra is still batch-oriented and blind to change.
Every shift in data or logic forces engineers to rebuild from scratch, breaking the feedback loop between real-time world state and AI decision-making.
Dynamic context engineering addresses this: maintaining an AI system's "mental model" of the world, continuously, reliably, and incrementally.
CocoIndex is our take on making this simple, with a **declarative Python-native data transformation engine designed for AI workloads**, and you will never need to handle change.
Think of it as **React for data processing**.
## The Cohort & Thanks to our batchmates
We were in the program alongside Pandas, Apache Airflow, Fabric.js, PyPI, and others.
We heard from teams dealing with similar challenges: how they handle dependency updates, structure security reviews, and think about trust boundaries.
Open source can be isolating. You're deep in your own codebase, solving your own problems. Hearing how other maintainers answer the same security questions was one of the most useful parts of the program.
## Strengthening CocoIndex: Security Improvements Implemented
The four workstreams of CocoIndex's GitHub Secure Open Source Fund sprint: threat modeling and AI-specific risk assessment, automated checks with CodeQL and secret scanning on every PR, dependency and supply-chain hardening with the OpenSSF Scorecard and SBOMs, and a formal vulnerability disclosure process.
During the program, the CocoIndex team ran a focused security sprint to harden both our codebase and the way we build and ship it.
With guidance from GitHub Security Lab and community experts, we concentrated on four areas that matter most for a framework sitting directly in AI data paths.
### Threat modeling & AI-specific risk assessment
We systematically threat-modeled the CocoIndex framework, examining how an attacker might exploit a data transformation pipeline that feeds directly into AI systems.
We mapped out attack vectors including adversarial document content that could poison embeddings, prompt injection through source data that could manipulate LLM-based extraction, and
compromised dependencies that could silently corrupt downstream indexes. Resources like the Adversarial AI Reading List and Embrace The Red were particularly useful in shaping this analysis.
By identifying these failure points upfront, we developed mitigation strategies before they could be exploited in production pipelines.
### Secure coding & automated checks
We wired GitHub’s security tooling directly into CocoIndex’s development loop.
CodeQL now runs on every pull request, scanning both our Rust core and Python SDK for vulnerability patterns.
For a system where data flows from sources through transformations to targets,
CodeQL’s data flow analysis maps naturally onto our architecture, tracing untrusted input from ingestion all
the way to exported artifacts. We also enabled secret scanning to keep credentials and API keys out of the repository,
and we started using GitHub Copilot as a guardrail during development to surface potential issues before code is even committed.
Together, these checks act as a continuous security layer in our CI pipeline, catching problems early instead of after a release.
How CodeQL data-flow analysis maps onto the CocoIndex architecture: untrusted input enters through sources such as documents, codebases, and APIs, is traced through the Rust core and Python SDK transformations, and exits to targets like indexes and vector stores. On every pull request, CodeQL scanning, secret scanning, and Copilot review gate the merge.
### Dependency & supply chain hardening
CocoIndex integrates with embedding models, vector databases, document parsers, and cloud storage backends, each one pulling in its own dependency chain.
We audited our dependencies across both the Rust core and Python SDK, going beyond direct dependencies to examine the
transitive tree underneath. We tightened our dependency review process with more systematic tracking of what we depend on and why,
and adopted the OpenSSF Scorecard to continuously benchmark our project's security posture across branch protection, dependency updates,
CI/CD practices, and more. Following OpenSSF standards,
we're working toward a comprehensive software bill of materials (SBOM) for each release,
giving users visibility into exactly what ships with CocoIndex.
### Policy, documentation & vulnerability response
We formalized our SECURITY.md with clear instructions for responsible disclosure and defined response commitments.
Vulnerabilities should be reported to security@cocoindex.io, not filed as public issues.
We also established an internal process for triaging and patching reported vulnerabilities: who gets involved,
how we communicate, and how quickly we ship fixes. Every new connector or transformation function now gets
reviewed through the lens of: what's the worst-case input, and what happens if a dependency in this path is compromised?
CodeQL is already catching issues that manual review missed, and the disclosure process gives users a clear channel when something does slip through. Because CocoIndex sits in the data path of the AI systems built on it, these improvements carry downstream to every application that depends on those pipelines.
## Thank You to Our Contributors
None of this would have been possible without the people who build CocoIndex with us every day. Everyone who has submitted a PR, reported a bug, reviewed code, or helped another user in Discord: you're the reason this project exists and the reason it was recognized.
We especially want to thank those doing the work that doesn't show up in commit logs: writing documentation, answering questions, and building examples that help new users get started.
## Looking Forward
The sprint is over, but the checks it added run on every pull request, and the review habits stay with us for every new connector and integration.
The GitHub Secure Open Source Fund gave us dedicated time, expert guidance, and a community of peers to learn from. The improvements we've made will benefit every developer who builds an AI pipeline with CocoIndex.
We're grateful to GitHub and the entire cohort for this experience, and we plan to keep sharing what we've learned with the broader AI infrastructure community.
---
CocoIndex is open source on GitHub. Questions or feedback: join our Discord.
---
# SEC EDGAR financial analytics with Apache Doris
URL: https://cocoindex.io/blogs/sec-edgar-analytics/
Date: 2026-02-09
Summary: A multi-format CocoIndex v1 pipeline that ingests SEC filings (TXT, JSON), scrubs PII, extracts topics, and powers hybrid search with Apache Doris.
SEC filings are the backbone of financial transparency. Every public company in the United States files 10-Ks, 10-Qs, proxy statements, and exhibits with the SEC: thousands of documents each quarter across text, structured data, and PDF formats.
Searching across all of these effectively requires more than keyword matching. You need semantic understanding, structured metadata filtering, and the ability to combine multiple document formats into a single searchable index.
In this post, we walk through the **SEC EDGAR Financial Analytics** example: a CocoIndex v1 pipeline that ingests two source formats (TXT 10-K filings and XBRL company-facts JSON), scrubs PII, extracts topic tags, generates embeddings, and loads everything into Apache Doris for hybrid search combining vector similarity with full-text matching using Reciprocal Rank Fusion (RRF). Both formats fan into one shared path, so adding another format like PDF is the same handful of lines.
The project is open-sourced in the CocoIndex v1 repo.
## Why CocoIndex?
CocoIndex is a Rust-based, open-source data transformation framework for AI workloads, combining high performance with flexibility.
It supports incremental processing, data lineage, and customizable logic, allowing teams to build efficient and intelligent data pipelines.
CocoIndex makes transformation pipelines modular, transparent, and easy to maintain.
## Why Apache Doris?
Apache Doris is an open-source, Apache-licensed real-time MPP data warehouse built for lightning-fast analytics.
It supports high-concurrency workloads with real-time data ingestion and querying, handles both structured and semi-structured (VARIANT) data,
includes full-text inverted indexes, and native vector storage with approximate nearest neighbor (ANN) search.
Together, CocoIndex and Apache Doris form a powerful agentic data infrastructure stack.
CocoIndex transforms and indexes unstructured data through modular, lineage-tracked pipelines with built-in incremental processing, while Apache Doris delivers real-time analytics at scale with
sub-second ingestion latency, sub-100ms query response, and 10k QPS concurrency: capabilities purpose-built for AI agents that need to make fast, data-driven decisions.
The combination bridges the gap from raw, unstructured data to ultra-performant real-time search and analytics, while CocoIndex's data lineage and transparency ensure the auditability and
compliance that regulated industries demand.
## Architecture overview
The pipeline follows a multi-format, shared-path pattern:
```text
┌─────────────────────────────────────────────────────────────────────────┐
│ CocoIndex Multi-Format Pipeline │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ TXT 10-K Filings │ │ XBRL JSON Facts │ │
│ │ (risk factors) │ │ (company facts) │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────┐ │
│ │ Shared _index_text path │ │
│ │ Scrub PII → Chunk → Embed → Tag → declare_row │ │
│ └────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Apache Doris │
│ (vector ANN index + inverted full-text index) │
└─────────────────────────────────────────────────────────────────────────┘
```
Two file formats feed into a single table.
Each source extracts its own metadata, then converges on one shared function that scrubs PII, splits, embeds, and tags.
The output lands in a single Doris table with both a vector index and a full-text index.
## Define the pipeline
Two SEC filing formats, 10-K text and XBRL JSON, flow through one shared CocoIndex path that scrubs PII, chunks, embeds, and tags, into a single Apache Doris table with a vector and a full-text index, queried by hybrid RRF search.
CocoIndex v1 is declarative and Python-native: you write target_state = transformation(source_state) in plain async Python, and the Rust engine handles incremental processing. The whole example lives in one main.py with no DSL and no separate flow-definition file.
The entry point is an app_main function wrapped in a coco.App:
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import doris, localfs
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
app = coco.App(coco.AppConfig(name="SECFilingAnalytics"), app_main)
```
### The row schema
Each chunk becomes one typed row. In v1 you declare the output as a plain dataclass; the embedding field is annotated with the embedder context so the engine knows its vector shape:
```python title=main.py
@dataclass
class FilingChunk:
chunk_id: str # primary key: uuid5 of (filename, chunk offsets)
source_type: str # "filing" | "facts"
doc_filename: str
cik: str
filing_date: str
form_type: str
text: str
topics: list[str]
embedding: Annotated[NDArray, EMBEDDER]
```
### The dual-index target
app_main mounts one Doris table target that carries both a vector index and a full-text inverted index, then walks each source directory and mounts a per-file processor:
```python title=main.py
@coco.fn
async def app_main() -> None:
table = await doris.mount_table_target(
DORIS_DB,
TABLE,
await doris.TableSchema.from_class(FilingChunk, primary_key=["chunk_id"]),
vector_indexes=[
doris.VectorIndexDef(field_name="embedding", metric_type="l2_distance")
],
inverted_indexes=[doris.InvertedIndexDef(field_name="text", parser="unicode")],
)
txt = localfs.walk_dir(
localfs.FilePath(path="./data/filings"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.txt"]),
)
await coco.mount_each(process_filing, txt.items(), table)
facts = localfs.walk_dir(
localfs.FilePath(path="./data/company_facts"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.json"]),
)
await coco.mount_each(process_facts, facts.items(), table)
```
A single mount_table_target declares two indexes on one table:
- **Vector (ANN) index** on the embedding field for l2_distance semantic similarity search
- **Inverted index** on the text field for keyword matching with MATCH_ANY
This dual-index setup is what makes hybrid search possible without maintaining separate stores.
CocoInsight data view with all sources and transformation outputs
Each source uses localfs.walk_dir with pattern filtering, and coco.mount_each runs the per-file processor once per matched file.
CocoInsight showing the text filing source: filename and content fields
### Process text filings
process_filing parses the structured filename convention {CIK}_{date}_{form}.txt to pull out the company identifier (CIK), filing date, and form type (10-K, 10-Q, etc.), metadata needed for filtering and aggregation at query time. It then hands the text to the shared indexing path:
```python title=main.py
@coco.fn(memo=True)
async def process_filing(
file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
"""10-K text filing: metadata from the {CIK}_{date}_{form}.txt filename."""
name = file.file_path.path.name
parts = name.rsplit(".", 1)[0].split("_")
cik = parts[0] if parts else "unknown"
filing_date = parts[1] if len(parts) > 1 else "2024-01-01"
form_type = parts[2] if len(parts) > 2 else "10-K"
await _index_text(
await file.read_text(), "filing", name, cik, filing_date, form_type, table
)
```
memo=True memoizes the function: re-running skips files whose content and code are unchanged, so only new or edited filings are reprocessed.
We intentionally parse metadata with a deterministic string split rather than an LLM. When you know the file format upfront, a regex or string parser is faster, cheaper, and more reliable than calling an LLM on every document. CocoIndex still supports native LLM extraction for formats you don't know in advance, and tools like CocoInsight help validate the output either way.
CocoInsight showing metadata extraction: filename → CIK, filing date, form type
### The shared indexing path
Both sources converge on one _index_text helper that does the real work: scrub PII, split into overlapping chunks, embed, tag, and declare one row per chunk.
```python title=main.py
async def _index_text(text, source_type, filename, cik, filing_date, form_type, table):
embedder = coco.use_context(EMBEDDER)
for chunk in _splitter.split(
_scrub_pii(text), chunk_size=1000, chunk_overlap=200, language="markdown"
):
table.declare_row(row=FilingChunk(
chunk_id=_chunk_id(filename, chunk.start.char_offset, chunk.end.char_offset),
source_type=source_type,
doc_filename=filename,
cik=cik,
filing_date=filing_date,
form_type=form_type,
text=chunk.text,
topics=_extract_topics(chunk.text),
embedding=await embedder.embed(chunk.text),
))
```
**1. Scrub PII before chunking.** PII (Personally Identifiable Information) includes things like Social Security numbers, phone numbers, and email addresses. SEC filings sometimes contain these inadvertently (there are even SEC rules about this: Reg S-T Rule 83). Scrubbing happens on the full document *before* chunking, so a phone number or SSN split across a chunk boundary can't slip through into the search index.
CocoInsight showing PII scrubbing: before and after comparison with emails, phones, SSNs removed
**2. Split into chunks.** Chunk size is a tradeoff: too large and you lose search resolution (a 10,000-character chunk matches broadly but vaguely); too small and each chunk lacks enough context to be meaningful on its own. We use 1,000 characters with 200-character overlap as a practical middle ground for SEC filings. RecursiveSplitter splits text by respecting document structure: it tries to break at markdown headings, then paragraphs, then sentences, before falling back to character boundaries. A flat splitter would just cut every N characters regardless of where a sentence or section ends, often splitting mid-thought.
CocoInsight showing chunking: scrubbed content split into structured chunks
**3. Embed, tag, and declare a row.** For each chunk we generate an embedding vector (for semantic search) and extract topic tags (for structured filtering), then declare_row assembles one FilingChunk combining the chunk's text, embedding, and topics with the document-level metadata (CIK, filing date, form type). chunk_id is a stable uuid5 of the filename and chunk offsets, so re-running reconciles rows in place instead of duplicating.
CocoInsight showing embedding generation: text → 384-dimensional vector
Topics are extracted as string arrays, enabling Doris array filtering:
```python title=main.py
def _extract_topics(text: str) -> list[str]:
low = text.lower()
return [t for t, kws in _TOPIC_KEYWORDS.items() if any(k in low for k in kws)]
```
_TOPIC_KEYWORDS maps labels like RISK:CYBER, RISK:CLIMATE, TOPIC:AI, and TOPIC:FINANCIAL to keyword lists. This produces arrays like ["RISK:CYBER", "TOPIC:AI"], stored as a JSON column you can filter in Doris with json_contains(topics, '"RISK:CYBER"').
CocoInsight showing topic extraction: text → topic tags like RISK:CYBER, RISK:REGULATORY
### Process XBRL company facts
The JSON facts path is the same shape: process_facts reads the file, renders the XBRL metrics to searchable natural-language text, then calls the same _index_text.
```python title=main.py
@coco.fn(memo=True)
async def process_facts(
file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
"""XBRL company-facts JSON: render to text, then index."""
name = file.file_path.path.name
content = await file.read_text()
cik = name.replace("CIK", "").replace(".json", "")
filing_date = (
json.loads(content).get("filingDate", "2024-01-01") if content else "2024-01-01"
)
await _index_text(
_company_facts_to_text(content), "facts", name, cik, filing_date, "FACTS", table
)
```
_company_facts_to_text converts structured financial metrics (revenue, net income, R&D expense) into natural-language text so they're discoverable via semantic search. Because both sources end at the same _index_text and declare_row into the same table, adding a third format such as PDF is just another process_* that converts to text and calls _index_text (see Manuals to Structured Data for the docling PDF path).
### Providing shared resources
The Doris connection and the embedder are provided once in the app lifespan and retrieved anywhere with coco.use_context:
```python title=main.py
DORIS_DB = coco.ContextKey[doris.ManagedConnection]("sec_doris")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.provide(DORIS_DB, doris.connect(doris.DorisConnectionConfig(...)))
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
```
## Hybrid search with RRF
search.py combines semantic and lexical ranking using Reciprocal Rank Fusion, all in one Doris SQL query over the MySQL protocol:
```python title=search.py
def search(query: str, source_type: str | None = None, limit: int = 5) -> None:
vec = asyncio.run(_embedder.embed(query))
vstr = "[" + ",".join(f"{x:.6f}" for x in vec) + "]"
keywords = " ".join(re.findall(r"[A-Za-z]{3,}", query.lower()))
where = f"source_type = '{source_type}'" if source_type else "1 = 1"
sql = f"""
WITH semantic AS (
SELECT chunk_id, doc_filename, source_type, topics, text,
ROW_NUMBER() OVER (ORDER BY l2_distance(embedding, {vstr})) AS rk
FROM filing_chunks WHERE {where}
),
lexical AS (
SELECT chunk_id,
ROW_NUMBER() OVER (ORDER BY CASE WHEN text MATCH_ANY '{keywords}'
THEN 0 ELSE 1 END) AS rk
FROM filing_chunks WHERE {where}
)
SELECT s.doc_filename, s.source_type, s.topics, s.text,
1.0/(60 + s.rk) + 1.0/(60 + l.rk) AS rrf
FROM semantic s JOIN lexical l USING (chunk_id)
ORDER BY rrf DESC LIMIT {limit}
"""
```
The RRF formula 1/(k + rank) with k=60 is a standard approach for combining rankings from different signals without needing to normalize scores. A chunk that ranks #1 in both semantic and lexical search gets 1/61 + 1/61 = 0.0328. A chunk that's #1 semantically but #100 lexically gets 1/61 + 1/160 = 0.0226. The formula naturally balances both signals.
Pass --source filing or --source facts to restrict the search to one document format.
## Running the example
### Prerequisites
- Python 3.11+
- Docker and Docker Compose (for Apache Doris 4.0+, which is required for vector-index support)
### Quick start
```bash
cd examples/sec_edgar_analytics
# 1. Start Doris (FE + BE)
docker compose up -d fe be
# Wait ~90 seconds for Doris to initialize
# 2. Fetch sample data, configure, and install
python download.py # writes data/filings/*.txt + data/company_facts/*.json
cp .env.example .env # Doris host/ports
pip install -e .
# 3. Build the index
cocoindex update main
```
cocoindex update main loads 4 chunks (2 filings + 2 company-facts) into Doris, creating both the vector ANN index and the inverted full-text index. Topics come out as you'd expect: Apple tagged RISK:CYBER, RISK:CLIMATE, RISK:SUPPLY, RISK:REGULATORY, TOPIC:AI; Microsoft RISK:CYBER, RISK:REGULATORY, TOPIC:AI, TOPIC:CLOUD.
### Hybrid search
```bash
python search.py "cloud computing and AI risk"
```
```text
Hybrid search: "cloud computing and AI risk"
[0.0328] 0000789019_2025-10-15_10-K.txt (filing) topics=["RISK:CYBER","RISK:REGULATORY","TOPIC:AI","TOPIC:CLOUD"]
MICROSOFT CORPORATION FORM 10-K ANNUAL REPORT. Our cloud computing business (Azure) faces intense competition...
[0.0320] 0000320193_2025-11-01_10-K.txt (filing) topics=["RISK:CYBER","RISK:CLIMATE","RISK:SUPPLY","RISK:REGULATORY","TOPIC:AI"]
APPLE INC. FORM 10-K ANNUAL REPORT. Cybersecurity threats are a material risk...
```
On the sample data that ranks Microsoft's cloud-and-AI filing first (it carries both TOPIC:CLOUD and TOPIC:AI), Apple's second, and the company-facts rows below. Restrict to one format with python search.py "cloud revenue" --source facts.
## Direct Doris queries
You can also query the index directly via MySQL protocol:
```bash
mysql -h localhost -P 9030 -u root
```
### Array field filtering
```sql
-- Find all chunks tagged with cybersecurity risk
SELECT doc_filename, text, topics
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"');
-- Chunks matching any of multiple topics
SELECT doc_filename, text
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"')
OR json_contains(topics, '"RISK:CLIMATE"');
```
### Portfolio aggregation
```sql
-- Top 3 relevant chunks per company
WITH ranked AS (
SELECT cik, doc_filename, text,
l2_distance(embedding, [...]) AS score,
ROW_NUMBER() OVER (PARTITION BY cik ORDER BY score ASC) AS rank
FROM sec_analytics.filing_chunks
WHERE cik IN ('0000320193', '0000789019')
)
SELECT * FROM ranked WHERE rank <= 3;
```
### Temporal trends
```sql
-- Cybersecurity mentions by filing year
SELECT LEFT(filing_date, 4) AS filing_year,
COUNT(DISTINCT cik) AS num_companies,
COUNT(*) AS total_mentions
FROM sec_analytics.filing_chunks
WHERE text MATCH_ANY 'cybersecurity risk'
GROUP BY filing_year
ORDER BY filing_year DESC;
```
## What's next
The techniques in this example (multi-format ingestion, array field filtering, hybrid search, PII scrubbing) generalize beyond financial documents. The same patterns apply to healthcare records, legal documents, or any domain where you need to search across heterogeneous document formats with structured metadata filtering.
Check out the full source code in the CocoIndex v1 repo, including the sample-data generator and the hybrid-search CLI.
---
# Build a Self-Updating Wiki for Your Codebases with an LLM
URL: https://cocoindex.io/blogs/multi-codebase-summarization/
Date: 2026-02-05
Summary: Auto-generate documentation for every project in your codebase: a CocoIndex pipeline writes a wiki page per repo with an LLM, kept fresh as code changes.
Documentation tends to drift out of sync with the code it describes. In this tutorial, we build a pipeline that generates a wiki page for each project in your codebase and keeps it current with incremental processing.
Each repository is summarized by an LLM, then rolled up into one org-wide wiki that refreshes on every push.
The full source code is available at CocoIndex Examples - multi_codebase_summarization.
The source is on GitHub.
Here's an example of the generated documentation:
Preview of a generated project wiki page with an Overview, a Components section listing classes and functions, and an auto-generated Mermaid pipeline diagram.
## The Problem
Documentation drifts. A module gets refactored, and the wiki describing it is now wrong. It usually stays wrong until someone new asks whether the docs can still be trusted.
The alternative is to derive the documentation from the code itself: a pipeline that reads your source, extracts structure from it, and produces documentation that updates when the code changes.
## What We'll Build
:::note
This project uses CocoIndex v1.
:::
A pipeline that:
1. **Scans subdirectories**, treating each as a separate project
2. **Extracts structured information** from each Python file using an LLM (classes, functions, relationships)
3. **Aggregates** file-level data into project-level summaries
4. **Generates Markdown documentation** with Mermaid diagrams
```mermaid
flowchart LR
subgraph Input
A[projects/] --> B[project_1/]
A --> C[project_2/]
A --> D[project_N/]
end
subgraph "Process (per project)"
B --> E["extract_file_info\n(LLM + Pydantic)"]
E --> F["aggregate_project_info"]
F --> G["generate_markdown"]
end
subgraph Output
G --> H["project_1.md"]
end
classDef extract fill:#FBE7DA,stroke:#BE5133,stroke-width:1.5px,color:#7A2E1A;
classDef aggregate fill:#FCF3D8,stroke:#8F3B24,stroke-width:1.5px,color:#5A2417;
classDef generate fill:#DDF5DE,stroke:#16A534,stroke-width:1.5px,color:#11472A;
class E extract;
class F aggregate;
class G generate;
```
The model behind this is a single relationship:
target_state = transformation(source_state)
You declare the transformation. CocoIndex determines when to run it and which inputs need reprocessing.
### The reality of modern codebases
Codebases change constantly. Teams merge many pull requests a day, and AI coding agents add and modify files continuously. A documentation pipeline has to keep up with that rate of change.
Batch regeneration handles this poorly. A nightly job is already stale by morning. An hourly job spends most of its time rebuilding files that did not change. The cost of a full rebuild scales with the size of the codebase, not with the size of the change.
### Why not just write a script?
A script that loops through files, calls an LLM, and writes Markdown works for a one-time run. Keeping it correct over time raises several problems:
- **File changes**: after editing one file, re-running the whole pipeline is slow and expensive.
- **Tracking state**: to process only what changed, you have to track timestamps, checksums, or diffs yourself.
- **LLM costs**: re-analyzing unchanged files wastes API calls, which adds up at scale.
- **Logic changes**: changing the extraction prompt invalidates results for that transformation across every file.
These are caching, invalidation, and orchestration concerns, separate from the extraction logic you actually want to write.
### The declarative difference
CocoIndex separates the transformation logic from the change tracking. You provide:
1. **Functions** that extract information from a file, aggregate it, and format it.
2. **Dependencies** between those functions, which CocoIndex records.
CocoIndex then handles incremental processing. When a source file is edited, or the processing logic changes (a different model or an updated prompt), only the affected outputs are recomputed, and the output stays consistent with the source.
## Prerequisites
Install dependencies:
```bash
pip install --pre 'cocoindex>=1.0.0a6' instructor litellm pydantic
```
Set up environment:
```bash
export GEMINI_API_KEY="your-api-key"
export LLM_MODEL="gemini/gemini-2.5-flash"
echo "COCOINDEX_DB=./cocoindex.db" > .env
```
Create a projects/ directory with subdirectories for each Python project:
```bash
projects/
├── my_project_1/
│ ├── main.py
│ └── utils.py
├── my_project_2/
│ └── app.py
└── ...
```
## Define the app
Define a CocoIndex App, the top-level runnable unit in CocoIndex.
Diagram of a CocoIndex app: a source system flows through a declared source, a transform f(x), and a target state, into a target system.
The snippets below focus on the pipeline logic; the full import list (localfs, instructor, the Pydantic models, and the rest) is in main.py.
```python title="main.py"
import os
import pathlib
import cocoindex as coco
LLM_MODEL = os.environ.get("LLM_MODEL", "gemini/gemini-2.5-flash")
app = coco.App(
"MultiCodebaseSummarization",
app_main,
root_dir=pathlib.Path("./projects"),
output_dir=pathlib.Path("./output"),
)
```
- The app scans projects/ and outputs documentation to output/
## Define the main function
Diagram of the app_main function walking each project subdirectory and mounting a processing component per project, each emitting a Markdown wiki page to the output folder.
In the main function, we walk through each project in the subdirectories and process it.
It is up to you to declare the process granularity. It can be
- at a directory level per project. For example, code_embedding is a project, containing multiple files,
- or at file level,
- or at even smaller units (e.g., page level, or semantic unit level).
In this example, we have a projects folder containing 20+ projects. It is natural to pick granularity at the directory level for each project, because we want to create a wiki page per project.
```python title="main.py"
@coco.function
def app_main(
root_dir: pathlib.Path,
output_dir: pathlib.Path,
) -> None:
"""Scan subdirectories and generate documentation for each project."""
for entry in root_dir.resolve().iterdir():
if not entry.is_dir() or entry.name.startswith("."):
continue
project_name = entry.name
files = list(
localfs.walk_dir(
entry,
recursive=True,
path_matcher=PatternFilePathMatcher(
included_patterns=["*.py"],
excluded_patterns=[".*", "__pycache__"],
),
)
)
if files:
coco.mount(
coco.component_subpath("project", project_name),
process_project,
project_name,
files,
output_dir,
)
```
The main function does two things:
1. **Find all projects.** Loop through each subdirectory in root_dir, treating each as a separate project.
2. **Mount a processing component for each project.** For each project with Python files, coco.mount() sets up a processing component. CocoIndex handles execution and tracks dependencies.
A processing component groups an item's processing together with its target states. Each component runs independently and in parallel, so when project_a finishes, its results are applied to the external system without waiting for project_b or any other project.
To learn more about processing components, you can read the documentation.
## Process each project
For each project, we will
1. use an LLM to extract info
2. aggregate all the extraction into a project-level summary
3. output the extraction to nice documentation with a Mermaid diagram.
Per-project pipeline: extract file info with an LLM, aggregate into a project summary, render to a Markdown wiki page, and write it to the output folder.
```python title="main.py"
@coco.function(memo=True)
async def process_project(
project_name: str,
files: Collection[localfs.File],
output_dir: pathlib.Path,
) -> None:
"""Process a project: extract, aggregate, and output markdown."""
# Extract info from each file concurrently using asyncio.gather
file_infos = await asyncio.gather(*[extract_file_info(f) for f in files])
# Aggregate into project-level summary
project_info = await aggregate_project_info(project_name, file_infos)
# Generate and output markdown
markdown = generate_markdown(project_name, project_info, file_infos)
localfs.declare_file(
output_dir / f"{project_name}.md", markdown, create_parent_dirs=True
)
```
**Concurrent extraction.** asyncio.gather() runs the file extractions concurrently, which is faster than sequential processing when each call waits on an LLM API response.
## Extract file information with LLM
Now let's take a look at the details for each transformation.
For file extraction, we define a structure using Pydantic and use Instructor to extract with LLMs.
Extraction step highlighted: every Python file in a project is summarized by the LLM into a structured CodebaseInfo object, then aggregated and rendered to Markdown.
### Define the data models
The key to structured LLM outputs is defining clear Pydantic models.
Pydantic data models for structured LLM output: CodebaseInfo holds lists of ClassInfo and FunctionInfo, one repeated per class and function the LLM extracts.
```python title="models.py"
class FunctionInfo(BaseModel):
"""Information about a public function."""
name: str = Field(description="Function name")
signature: str = Field(
description="Function signature, e.g. 'async def foo(x: int) -> str'"
)
is_coco_function: bool = Field(
description="Whether decorated with @coco.function"
)
summary: str = Field(description="Brief summary of what the function does")
class ClassInfo(BaseModel):
"""Information about a public class."""
name: str = Field(description="Class name")
summary: str = Field(description="Brief summary of what the class represents")
class CodebaseInfo(BaseModel):
"""Extracted information from Python code."""
name: str = Field(description="File path or project name")
summary: str = Field(description="Brief summary of purpose and functionality")
public_classes: list[ClassInfo] = Field(default_factory=list)
public_functions: list[FunctionInfo] = Field(default_factory=list)
mermaid_graphs: list[str] = Field(
default_factory=list,
description="Mermaid graphs showing function relationships"
)
```
### Extract file info
The core extraction function uses memoization to cache LLM results:
````python title="main.py"
_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)
@coco.function(memo=True)
async def extract_file_info(file: FileLike) -> CodebaseInfo:
"""Extract structured information from a single Python file using LLM."""
content = file.read_text()
file_path = str(file.file_path.path)
prompt = f"""Analyze the following Python file and extract structured information...""" # see full prompt below
result = await _instructor_client.chat.completions.create(
model=LLM_MODEL,
response_model=CodebaseInfo,
messages=[{"role": "user", "content": prompt}],
)
return CodebaseInfo.model_validate(result.model_dump())
````
:::note
See the full prompt in the repo.
:::
**memo=True** caches the result by the function's inputs. When the file content and the code are unchanged, CocoIndex returns the previous result instead of calling the LLM again, so unchanged files skip the remote call on later runs.
## Aggregate project information
For projects with multiple files, we aggregate into a unified summary:
Aggregation step highlighted: per-file CodebaseInfo extractions are rolled up into one project-level summary before rendering.
```python title="main.py"
@coco.function
async def aggregate_project_info(
project_name: str,
file_infos: list[CodebaseInfo],
) -> CodebaseInfo:
"""Aggregate multiple file extractions into a project-level summary."""
if not file_infos:
return CodebaseInfo(
name=project_name, summary="Empty project with no Python files."
)
# Single file - just update the name
if len(file_infos) == 1:
info = file_infos[0]
return CodebaseInfo(
name=project_name,
summary=info.summary,
public_classes=info.public_classes,
public_functions=info.public_functions,
mermaid_graphs=info.mermaid_graphs,
)
# Multiple files - use LLM to create unified summary
files_text = "\n\n".join(
f"### {info.name}\n"
f"Summary: {info.summary}\n"
f"Classes: {', '.join(c.name for c in info.public_classes) or 'None'}\n"
f"Functions: {', '.join(f.name for f in info.public_functions) or 'None'}"
for info in file_infos
)
# Collect all mermaid graphs from files
all_graphs = [g for info in file_infos for g in info.mermaid_graphs]
prompt = f"""Aggregate the following Python files into a project-level summary...""" # see full prompt in repo
result = await _instructor_client.chat.completions.create(
model=LLM_MODEL,
response_model=CodebaseInfo,
messages=[{"role": "user", "content": prompt}],
)
result = CodebaseInfo.model_validate(result.model_dump())
# Keep original file-level graphs if LLM didn't generate a unified one
if not result.mermaid_graphs and all_graphs:
result.mermaid_graphs = all_graphs
return result
```
:::note
See the full prompt in the repo.
:::
This function combines file-level extractions into a single project summary:
- **Single-file project**: use that file's info directly, with no extra LLM call.
- **Multi-file project**: ask the LLM to synthesize the file summaries into one project overview.
The result is a unified CodebaseInfo that represents the entire project rather than individual files.
## Generate the Markdown documentation
Create output Markdown for each project, including a Mermaid pipeline diagram.
Markdown generation step highlighted: the aggregated project summary is rendered into a wiki page with Overview, Components, and a Mermaid pipeline diagram.
```python title="main.py"
@coco.function
def generate_markdown(
project_name: str, info: CodebaseInfo, file_infos: list[CodebaseInfo]
) -> str:
"""Generate markdown documentation from project info."""
lines = [
f"# {project_name}",
"",
"## Overview",
"",
info.summary,
"",
]
if info.public_classes or info.public_functions:
lines.extend(["## Components", ""])
if info.public_classes:
lines.append("**Classes:**")
for cls in info.public_classes:
lines.append(f"- `{cls.name}`: {cls.summary}")
lines.append("")
if info.public_functions:
lines.append("**Functions:**")
for fn in info.public_functions:
marker = " ★" if fn.is_coco_function else ""
lines.append(f"- `{fn.signature}`{marker}: {fn.summary}")
lines.append("")
if info.mermaid_graphs:
lines.extend(["## CocoIndex Pipeline", ""])
for graph in info.mermaid_graphs:
graph_content = graph.strip()
if not graph_content.startswith("```"):
lines.append("```mermaid")
lines.append(graph_content)
lines.append("```")
else:
lines.append(graph_content)
lines.append("")
if len(file_infos) > 1:
lines.extend(["## File Details", ""])
for fi in file_infos:
lines.extend([f"### {fi.name}", "", fi.summary, ""])
lines.extend(["---", "", "*★ = CocoIndex function*"])
return "\n".join(lines)
```
This function converts the structured CodebaseInfo into readable documentation:
- **Overview**: the project summary at the top.
- **Components**: classes and functions with descriptions (★ marks CocoIndex functions).
- **Pipeline diagram**: Mermaid graphs showing how functions connect.
- **File details**: per-file summaries, for multi-file projects.
## Run the pipeline
```bash
cocoindex update main.py
```
CocoIndex will:
1. Scan each subdirectory in projects/
2. Extract structured information from Python files using the LLM
3. Aggregate file summaries into project summaries
4. Generate Markdown files in output/
Check the output:
```bash
ls output/
# project1.md project2.md ...
```
## Incremental Updates
The real power shows when you make changes:
**Modify a file:**
```bash
# Edit a Python file in one of your projects
cocoindex update main.py
# Only the modified file is re-analyzed
```
**Add a new project:**
```bash
mkdir projects/new_project
# Add .py files
cocoindex update main.py
# Only the new project is processed
```
## Output Example
Example generated wiki page showing the Overview, Components (classes and functions), and the auto-generated Mermaid pipeline diagram.
Each generated Markdown file includes:
- **Overview**: what the project does.
- **Components**: public classes and functions with descriptions.
- **Pipeline diagram**: a Mermaid graph showing how functions connect.
- **File details**: per-file summaries for multi-file projects.
```mermaid
graph TD
app_main[app_main] ==> process_project[process_project]
process_project ==> extract_file_info[extract_file_info]
process_project ==> aggregate_project_info[aggregate_project_info]
process_project --> generate_markdown[generate_markdown]
classDef extract fill:#FBE7DA,stroke:#BE5133,stroke-width:1.5px,color:#7A2E1A;
classDef aggregate fill:#FCF3D8,stroke:#8F3B24,stroke-width:1.5px,color:#5A2417;
classDef generate fill:#DDF5DE,stroke:#16A534,stroke-width:1.5px,color:#11472A;
class extract_file_info extract;
class aggregate_project_info aggregate;
class generate_markdown generate;
```
## Key Patterns
This example combines several patterns:
1. **Structured LLM outputs**: Pydantic models with Instructor return validated, typed data.
2. **Memoized LLM calls**: identical inputs return a cached result, which avoids most LLM calls on incremental runs.
3. **Async concurrent processing**: asyncio.gather() runs file extractions in parallel.
4. **Hierarchical aggregation**: extract at the file level, then aggregate to the project level.
5. **Incremental processing**: only changed inputs are reprocessed.
## Beyond documentation: keeping derived knowledge current
The same pipeline applies wherever derived knowledge has to track a changing source, including the context that coding agents read. It is the same incremental pattern behind indexing a codebase for RAG, text embeddings, and knowledge graphs: declare the transformation once, and only changed inputs are reprocessed.
### Long-horizon agents need current context
AI agents increasingly operate over long sessions, planning and acting across many steps. Their decisions depend on the state of the codebase as it is now: what changed recently, how modules interact, and the current structure of the system. Documentation that was accurate last week can lead an agent to act on assumptions that no longer hold.
### Treating change as the unit of work
Processing change rather than reprocessing the whole corpus is what makes this practical:
- **Efficiency**: only changed inputs are processed, not the entire corpus.
- **Latency**: new information appears in minutes rather than after a full rebuild.
- **Cost**: unchanged content does not incur repeated LLM calls.
- **Scalability**: codebases too large to reprocess from scratch can still be kept current.
Declaring the transformation, rather than scripting the update logic by hand, is what lets the pipeline keep derived knowledge in sync with the source as the source changes.
## Thanks to the community
### @prrao87
Thanks @prrao87 for reviewing the example and providing detailed feedback on terminology, style, and conceptual clarity, which helped improve the developer experience.
If you have ideas, questions, or want to contribute, join us on Discord.
## Next Steps
- Try the full example, or read the example walkthrough in the docs
- Join the CocoIndex Discord for questions
The source is on GitHub.
---
# Slides-to-speech: Bring your slide decks to life with narrated audio
URL: https://cocoindex.io/blogs/slides-to-speech/
Date: 2026-01-22
Summary: Build a CocoIndex v1 pipeline that turns PDF slides into vision-generated speaker notes, local Pocket TTS narration, and searchable LanceDB records.
Presentation slide decks are information-rich, dense sources of company knowledge, containing everything from quarterly updates, technical specs, sales reports, and training materials. These decks are locked away in PDF files that get revised constantly and are generally hard to consume in their static form.
What if we could turn those decks into a continuously updated, searchable, and narrated (audio) knowledge base? Doing that would result in a few immediate benefits:
- **Listen to any slide deck** instead of reading every slide (useful for busy executives on the go, and for people with visual impairments).
- **Extract insights from slide content** as structured speaker notes by using a VLM (vision language model).
- **Search slide content semantically**, not just by keywords or filenames.
Producing those artifacts once is only half the task. When a deck changes, its speaker notes, audio, and embeddings also have to be kept fresh. Once everything is in sync, it can lay the foundation for more advanced user experiences later, including video generation from text.
In this post, we’ll show how to turn the slide deck's assets into a multimodal knowledge base stored in LanceDB: the same dataset contains the slide content, speaker notes, MP3 audio narration, and embeddings for vector search, all of which are orchestrated and kept fresh by CocoIndex.
## Introducing CocoIndex
CocoIndex is an ultra-performant framework that keeps context continuously fresh for AI Agents. Think “React for data processing.” You declare state transformations, and CocoIndex handles the processing logic: it continuously watches for changes, recomputes only what’s necessary, and applies minimal updates downstream for end-to-end freshness. As AI systems become increasingly autonomous, the bottleneck is no longer model capability: it’s keeping context fresh and relevant.
CocoIndex follows an incremental update philosophy:
- New slide decks appear in the source folder
- Updated files get reprocessed automatically
- Only affected rows get new embeddings and regenerated audio, if needed
## Introducing LanceDB
LanceDB is a multimodal lakehouse for AI, available as an open-source, embedded, developer-friendly retrieval library that pairs well with CocoIndex. Under the hood, it’s powered by the Lance columnar format, which is designed for fast random access (great for search workloads) without sacrificing scan performance (great for large-scale aggregation queries).
For this Slides-to-Speech pipeline, that translates into a few concrete benefits:
- Multimodal in one table: store speaker notes, MP3 audio bytes, embeddings, and metadata side by side as a single dataset.
- Indexes are first-class: vector search, full-text search (FTS), and secondary indexes are stored alongside your dataset, so you don’t need to manage those using separate systems.
- Data evolution: add or refine columns over time (backfill new embeddings, quality tags, moderation fields) without touching unmodified columns, which is great for feature engineering and experimentation.
- Local-first simplicity: run it as an embedded database with minimal operational overhead.
CocoIndex's LanceDB connector automatically appends and updates rows as new results arrive via the CocoIndex pipeline, so you don’t have to rewrite your table (or rebuild your indexes) from scratch each time content changes.
## What we are building: one slide, three derived modalities
To begin building our pipeline, let's first get the shape of the data clear. A PDF slide deck doesn’t just become one large database record. We first fan it out page by page, then turn each slide into three related outputs: speaker notes, MP3 narration (audio), and a text embedding for semantic search.
The speaker notes sit at the center of the pipeline, but they first need to be generated by a vision model that reasons over the slide content. Once they exist, audio generation and embedding can run on them concurrently. The completed outputs are then declared together as one SlideRecord, identified by its filename and page number.
Each tool has one focused job:
| Component | Role in the pipeline |
| --- | --- |
| **CocoIndex** | Discovers PDFs, runs the transformations, and reconciles the declared slide rows with the source files. |
| **PyMuPDF** | Renders every PDF page as a PNG for the vision model. |
| **DSPy + Gemini** | DSPy lets us call the VLM via a concise signature, and the VLM (gemini/gemini-2.5-flash) reads each rendered slide PNG and writes the speaker notes. |
| **Pocket TTS** | Synthesizes those notes as MP3 audio locally on the CPU. |
| **Sentence Transformers** | Embeds the speaker notes locally with sentence-transformers/all-MiniLM-L6-v2. |
| **LanceDB** | Stores text, binary audio, metadata, and vectors together, then searches the notes by meaning. |
Most of the pipeline runs locally on CPU: PyMuPDF renders the pages as images for the VLM, Pocket TTS synthesizes audio from the speaker notes, sentence-transformers generates text embeddings, and LanceDB runs as an embedded, local retrieval library.
## Start backwards from the target
CocoIndex pipelines are easiest to reason about _backwards_ from the target, i.e., where the data sits in its final form. Put simply, target_state = transformation(source_state), where transformation can be any function, or a set of functions. For this pipeline, the target state is one clean LanceDB row for **each slide**.
```python
# Declare the LanceDB table shape via a strictly typed schema
# Pydantic or PyArrow schemas work well, but dataclasses are simpler for this example
@dataclass
class SlideRecord:
id: str
filename: str
page: int
speaker_notes: str
voice: bytes
embedding: Annotated[NDArray, EMBEDDER]
# Mount a LanceDB table as the target
table = await lancedb.mount_table_target(
LANCE_DB,
table_name="slides_to_speech",
table_schema=await lancedb.TableSchema.from_class(
SlideRecord,
primary_key=["id"],
),
)
```
**Multimodal data is a first-class citizen in LanceDB.** Each row can hold the slide’s metadata, generated notes, MP3 narration bytes, and vector embedding side by side. SlideRecord is the typed contract that keeps those values together, via a dataclass.
Each slide needs a stable primary key so CocoIndex can match it to the same LanceDB row across runs, updating or removing the correct record as the source changes. This example derives that key from the filename and page number: page 3 of quarterly_review.pdf becomes quarterly_review.pdf#3.
## Functions vs. processing components
Incremental processing is only useful when its boundaries match the unit of change. We want our pipeline to run every time a change is detected in the source files, without wastefully recomputing what didn't change.
In CocoIndex, functions define the work, while processing components group that work into independently synchronized units. Where we draw those component boundaries determines how much of the pipeline runs each time a change is detected, and which target states CocoIndex reconciles afterward.
- **Functions** are callable Python functions decorated with @coco.fn. Setting memo=True lets CocoIndex reuse results for unchanged inputs and code. Several functions can run inside the same processing component.
- **Processing components** are created by grouping functions and _mounting_ them to the target. They manage the necessary function calls and the target states they declare. Processing components help decide the granularity at which CocoIndex reruns work and synchronizes the result.
In this example, we define the processing component at the **per-slide level**, through the process_file function. It takes a PDF file, renders each slide as an image, and mounts one process_slide function for each slide.
```python
@coco.fn(memo=True)
async def process_file(
file: FileLike,
table: lancedb.TableTarget[SlideRecord],
) -> None:
slides = await pdf_to_slides(await file.read())
await coco.mount_each(
process_slide,
((slide.page_number, slide) for slide in slides),
str(file.file_path.path),
table,
)
```
process_file is memoized too, so CocoIndex can skip reading and rendering an unchanged PDF. If the file does change, process_slide provides the finer-grained memoization boundary: slide images with unchanged inputs reuse their prior results, while changed slides go through note generation, TTS, embedding, and the declaration of SlideRecord in the LanceDB target based on its stable key.
Changing a slide in a PDF still reruns page rendering for that entire file (still, PyMuPDF is a cheap CPU-based operation, so no VLM calls happen yet). Once the images are available, CocoIndex fingerprints files with a content-based hash, so it processes _only_ slide images whose contents changed from the previous state. This is the power of memoization and the reason we mount the component at the slide level.
If a PDF is deleted from the source folder, CocoIndex unmounts its per-file component and recursively cleans up the target states it owns. In this pipeline, that means deleting the PDF's slide rows from LanceDB.
Let's now go over each stage of the pipeline below.
## Fan a PDF out into slides
The function pdf_to_slides turns the PDF bytes into typed SlidePage values, each carrying the page number and the rendered PNG that the vision model will inspect.
```python
@dataclass
class SlidePage:
page_number: int
image: bytes
@coco.fn.as_async(runner=coco.GPU, memo=True)
def pdf_to_slides(content: bytes) -> list[SlidePage]:
import pymupdf
slides: list[SlidePage] = []
doc = pymupdf.open(stream=content, filetype="pdf")
for i, page in enumerate(doc):
pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
slides.append(
SlidePage(page_number=i + 1, image=pix.tobytes("png"))
)
doc.close()
return slides
```
PyMuPDF renders each page at 2× scale, so the vision model sees a legible PNG rather than extracted PDF text. The image is an intermediate input: Gemini uses it to generate notes, but the pipeline discards it afterward. If you want to keep the rendered image for reuse, you can add an image field and store its bytes in the same LanceDB table.
## Turn a slide image into narration
Extracting speaker notes from the slide images gives the Pocket TTS model the text it needs to turn the script into an audio file that narrates the insights from that slide. Text embeddings are also generated from the speaker notes, so they can support semantic search, allowing you to easily find slides based on their meaning rather than just keywords in their titles or filenames.
```python
class SlideNotes(dspy.Signature):
"""Write natural spoken narration for a slide, as a presenter would say it aloud."""
slide: dspy.Image = dspy.InputField(desc="the rendered slide image")
speaker_notes: str = dspy.OutputField(
desc="a few sentences of spoken narration — no markdown or bullet symbols"
)
_speaker_notes = dspy.Predict(SlideNotes)
@functools.cache
def _get_lm(model: str) -> dspy.LM:
return dspy.LM(model, max_tokens=8192)
@coco.fn(memo=True)
async def extract_speaker_notes(image: bytes) -> str:
data_url = "data:image/png;base64," + base64.b64encode(image).decode()
with dspy.context(lm=_get_lm(coco.use_context(LLM_MODEL))):
result = await _speaker_notes.acall(slide=dspy.Image(url=data_url))
return result.speaker_notes
```
SlideNotes is a DSPy signature, a typed contract that tells the VLM what we need. When a slide image comes in, speaker notes come out. The CocoIndex function extract_speaker_notes does the actual work of turning the PNG bytes into a text output.
This extraction function is memoized because speaker-note generation is an expensive dependency: it calls a capable VLM to reason over the image. Its memoization key includes the rendered image and the change-detected LLM_MODEL context, so an unchanged slide can reuse its notes if the image bytes and the model remain the same. If either of them changes, it's tracked by CocoIndex and the work is rerun for that slide.
## Turn speaker notes into audio and embeddings
We can now inspect the function process_slide and the work it does, by reading it top to bottom. The speaker notes must first exist, so they are computed and awaited via extract_speaker_notes. Once they exist, audio synthesis and embedding have no dependency on each other, so we call asyncio.gather to run them concurrently. CocoIndex commits them to the LanceDB table only after both are ready.
```python
@coco.fn(memo=True)
async def process_slide(
slide: SlidePage,
filename: str,
table: lancedb.TableTarget[SlideRecord],
) -> None:
notes = await extract_speaker_notes(slide.image)
voice, embedding = await asyncio.gather(
text_to_speech(notes, coco.use_context(TTS_VOICE)),
coco.use_context(EMBEDDER).embed(notes),
)
table.declare_row(
row=SlideRecord(
id=f"{filename}#{slide.page_number}",
filename=filename,
page=slide.page_number,
speaker_notes=notes,
voice=voice,
embedding=embedding,
)
)
```
The target state is declared as a SlideRecord dataclass (per the LanceDB schema) keyed by its unique ID that combines the filename and page number. This is how CocoIndex reconciles the target state with the source files, updating or removing rows as needed.
The expensive nested operations are memoized as well. If only the voice changes, CocoIndex can reuse the notes and embedding while regenerating the audio. If the embedding model changes, it can reuse the notes and audio while computing a new vector. This gives us reuse at the file, slide, and individual transformation levels.
## Define the app
Apps are the top-level runnable unit in CocoIndex. They name your pipeline and bind a main function with its parameters. When you call app.update(), CocoIndex runs that main function as the root processing component that can mount processing components (like our process_file and its child component process_slide) that do work and declare target states.
app_main mounts the SlideRecord table we defined earlier, uses localfs.walk_dir in live mode to discover PDFs, then calls the outer mount_each. The app is configured with a name, the main function, and the source directory to watch for changes.
```python
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
table = await lancedb.mount_table_target(
LANCE_DB,
table_name="slides_to_speech",
table_schema=await lancedb.TableSchema.from_class(
SlideRecord,
primary_key=["id"],
),
)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(
included_patterns=["**/*.pdf"]
),
live=True,
)
await coco.mount_each(process_file, files.items(), table)
app = coco.App(
coco.AppConfig(name="SlidesToSpeech"),
app_main,
sourcedir=pathlib.Path("./slides"),
)
```
## Run the pipeline and run semantic search
### Prerequisites
- Python 3.11+
- A Gemini API key for the default vision model
- ffmpeg for MP3 export
Pocket TTS is open source and runs locally on the CPU. Its weights download on the first run, so it needs neither a TTS API key nor a GPU.
### Quick start
From the example directory, copy the environment file, set GEMINI_API_KEY, and ensure LLM_MODEL=gemini/gemini-2.5-flash to match the example’s default configuration. Then install and run:
```sh
cp .env.example .env
uv sync
cocoindex update main
# Keep watching slides/ for changes instead:
cocoindex update -L main
```
### Search the deck
```sh
python main.py "reducing latency and reliability"
```
The query helper embeds the question with the same sentence-transformer, then asks LanceDB for its nearest neighbors. The results already arrive ranked, so we can print the matching slide and its notes directly:
```python
for row in table.search(vec).limit(5).to_list():
print(f"{row['filename']} slide {row['page']}")
```
The slide deck we use here is an example "quarterly-review" deck, so the query about "reducing latency and reliability" brings the slide about "Engineering Priorities" to the top because its notes discuss "reducing p95 latency and multi-region resilience".
The returned row can also fetch the voice bytes, so a downstream application can use those bytes as audio/mpeg, write them to an MP3 file for playback, and play them on demand if needed.
Here is the narration the pipeline generated for the sample deck, one MP3 per slide:
Slide 1
Your browser doesn't support MP3 playback.
Slide 2: Engineering Priorities
Your browser doesn't support MP3 playback.
Slide 3
Your browser doesn't support MP3 playback.
## How CocoIndex tracks changes
In this demo, CocoIndex watches the ./slides directory for changed PDFs. It then identifies the changed slides, and only those slides regenerate notes, audio, and embeddings. As the table above shows, it can track much more than file changes.
However, source files changing is only one such scenario. The table below shows the other changes that CocoIndex tracks, and what they mean for other pipelines like this one.
| Change | What it means in this pipeline |
| --- | --- |
| Nothing | Memoized calls reuse their prior results and retain their rows. |
| One slide changes inside a PDF | The PDF renders again, then only the slide with different PNG bytes reruns notes, TTS, embedding, and its row declaration. |
| A PDF is added or deleted | CocoIndex creates or removes the file component and all of its owned slide rows. |
| LLM model, embedder, or voice changes | The change-detected context refreshes the functions that consume it; unrelated memoized calls can be reused. |
| Decorated function logic changes | A changed @coco.fn fingerprint invalidates memoized calls that depend on that function, then CocoIndex reconciles the resulting rows. |
| SlideRecord fields or primary key change | The target contract has changed. Treat compatibility, backfill, and identity changes as a deliberate migration decision. |
## Bring your slides to life (and beyond)
The pipeline shown here is a concrete example of how to bring slide decks to life via narration. But don’t just think of organizational data as structured tables and flat text. A large amount of company intelligence is locked away in richer artifacts like PDF reports, dashboards, and video recordings.
With CocoIndex and LanceDB, we can turn _all_ of that raw multimodal content into a living, breathing, contextually rich knowledge base that agents can retrieve from:
- **Unify multimodal content in one storage layer:** text, audio, vectors, and metadata in the same table.
- **Add structure that supports more questions:** speaker notes, extracted fields, embeddings, tags, and other derived signals.
- **Keep everything synchronized over time:** incremental recomputation when source files change, with updated rows in LanceDB.
- **Build user-facing experiences on top:** semantic search, narrated slide playback, assistants, and video-generation pipelines.
Try the complete slides-to-speech example to bring your own deck to life! Then, star CocoIndex and LanceDB on GitHub.
---
# CocoIndex Changelog 0.3.11 - 0.3.26
URL: https://cocoindex.io/blogs/changelog-0311-0326/
Date: 2026-01-18
Summary: CocoIndex updates: production-ready resilience, a structured error system, expanded integrations, and always-fresh context for agents.
CocoIndex is moving at insane speed! 🚀
Since the last release, CocoIndex shipped **15 releases (0.3.11–0.3.26)** and crossed a major milestone: **#1 GitHub Global Trending** across all languages and #1 on Rust. This cycle was focused on one clear goal: making **fresh, structured, programmable context** reliable enough for **agents running in production**.
GitHub Trending - All Languages
Agents are only as good as the data they reason over. Thinking about codebases, meeting notes, emails, relationships, commerce ... anything that may be subject to change. If you have ever done index-swap or had to keep multiple copies of the target to serve live agents on data or code change, you'll never have to do it again.
**CocoIndex helps you ace the data for agents without heavy data engineering work.** Continuous fresh, structured and programmable context for AI, from PDFs, Codebase, Emails, Screenshots, Meeting Notes, ... Always up-to-date, at any scale, ultra-performant, with a smart incremental engine. ⭐ Star the repo
Glad this project resonates with the community! 2026 will see more autonomous agents going into production, and we're excited to see CocoIndex help in that frontier.
Here's what we shipped since the last release. **0.3.11–0.3.26 was about production trust**: resilient pipelines, structured errors, better observability, and always-fresh structured context for agents operating in the real world.
## Core capability
CocoIndex just got a significant upgrade in runtime resilience and developer ergonomics. The core engine now also features a fully structured error system, enabling unified error tracing from Rust to Python so you can finally see human‑readable stack traces across boundaries. Combined with smarter runtime warnings for slow live updates, simplified progress bar rendering, and cleaner internal API boundaries, CocoIndex continues to evolve into the most transparent and trustworthy data‑processing core in its class.
### Failure-tolerant target deletion
Target drop failures no longer block pipeline setup updates: an environment toggle **COCOINDEX_IGNORE_TARGET_DROP_FAILURES** lets you log and continue gracefully (#1454).
### Simplified progress experience
Streamlined progress bar updates for a more predictable runtime visualization experience (#1467).
### Structured error system overhaul
A new foundation for error definition and propagation:
- Introduced **structured error framework** with host exception tunneling
- End-to-end **Python stack trace propagation** now supported
- New unified **error types** improve exception messaging and logging at both Rust and Python layers (#1380, #1413).
### Smart runtime alerts
CocoIndex now warns when live updates exceed the configured refresh interval, ideal for maintaining freshness guarantees (#1408).
### Cleaner internal API boundaries
Refactored **datatype.py** with a dedicated **_internal** module, decluttering the public API surface (#1402).
### Force reprocessing option for updates
Added explicit control for full recomputation during incremental updates for safer state resets. Run cocoindex update --full-reprocess ... to force a reprocessing (#1385).
### Flow correctness improvements
- Fixed target diffing logic for user-managed setup states
- General FTS fixes for LanceDB (#1369, #1403)
Together, these changes strengthen the core runtime's fault tolerance, improve observability, and lay the groundwork for a richer developer debugging experience.
## Integrations
This release massively expands what CocoIndex can plug into.
### Sources & targets
#### Qdrant
Full support for **VectorIndexMethod** HNSW configuration in the Qdrant connector (#1410).
#### Postgres
Native support for **pgvector** types in Postgres source (#1387).
#### Local files
Automatic symlink resolution for easier graph ingestion (#1463).
#### LanceDB
- Added **optimize()** for post-ingest table compaction
- Extended full-text search integration in the LanceDB connector (#1466, #1358)
### Functions & LLM providers
#### Unified OpenAI/Azure architecture
Shared logic between **openai.rs** and **azureopenai.rs** for cleaner code reuse (#1376).
#### Azure OpenAI integration
Bring your Azure LLM accounts seamlessly into CocoIndex pipelines (#1362).
#### OpenRouter embeddings
Added embedding model support across OpenRouter's expanding provider network (#1359).
#### Embedding dimension validation
Introduced **expected_output_dimension** param to enforce vector consistency (#1373).
#### Direct JSON output in functions
**GeneratedOutput** enum now supports structured JSON returns for downstream consumption (#1395).
#### Gemini API key fixes
Cleaner, more consistent propagation of credentials across function calls (#1375).
These integrations make CocoIndex a more open, extensible, and multi-LLM-ready data processing cortex.
## Build with CocoIndex
We are seeing more projects built with CocoIndex.
### Build fresh data with LanceDB and CocoIndex
LanceDB and CocoIndex
🚀 If agents are going to search, reason, and act over long periods of time, we need infrastructure that treats **data freshness and multimodal as a first-class primitive**.
Our friends at LanceDB just published a full walkthrough with us showing how to build a **live-updating multimodal search index** using LanceDB and CocoIndex, where text, images, embeddings, and LLM-derived features stay continuously in sync as data and logic evolve.
Check out the tutorial for more details.
🔥 **What's cool about this example:**
- **Incremental-by-design processing**: Only new or changed records are reprocessed. Add one recipe? Update one image? Fix one ingredient? CocoIndex touches only what changed, not the other 100k rows.
- **True multimodal indexing**: Text, images, embeddings, and metadata live together in LanceDB, optimized for both search and analytics, without splitting data across systems.
- **Schema evolution without rewrites**: We add new LLM-extracted feature columns (vegetarian, dairy, category, etc.) after data is already live. CocoIndex updates the schema and backfills safely, with no table rebuilds.
- **Structured LLM extraction (no brittle prompts)**: DSPY signatures define input/output types, so LLMs behave like programs, not JSON-generating slot machines.
- **Upserts + deletes handled automatically**: Primary keys + incremental flows mean no duplicates or stale rows, and no "did this record change?" logic to maintain.
- **End-to-end lineage & observability**: With CocoInsight, you can see exactly why a record changed and which transformation produced each field.
- **Live mode for agent-ready systems**: Run the pipeline continuously. As data arrives or logic changes, your search layer updates in near real time.
Think PDFs, slides, screenshots, images, videos, and support tickets across massive enterprise data lakes, all kept fresh, indexed, searchable, and generating real insights.
### Building a real-time HackerNews trending topics detector with CocoIndex
HackerNews Trending Topics
In the age of information overload, understanding what's trending (and why) is crucial for developers, researchers, and data engineers. HackerNews is one of the most influential tech communities, but manually tracking emerging topics across thousands of threads and comments is practically impossible.
In this post, we'll explore how to build a **production-ready, real-time trending topics pipeline** on top of HackerNews using **CocoIndex**. It shows how unstructured, high-volume community data can be incrementally ingested, enriched with LLMs, and queried as structured, up-to-date knowledge.
Each HackerNews thread and comment is then processed through an **LLM-powered extraction step** to identify normalized topics (technologies, products, companies, people, etc.). These topics, along with the original messages, are indexed into **Postgres** as queryable tables. Weighted scoring distinguishes primary topics in threads from secondary mentions in comments, enabling higher-quality trend rankings.
The result is a continuously updating system that turns raw HackerNews activity into **structured, queryable intelligence**, ready for dashboards, analytics, or AI agents.
Check out the blog for more details.
### Building a knowledge graph from meeting notes that automatically updates
Meeting Notes Knowledge Graph
Most companies sit on an ocean of meeting notes and treat them like static text files. But inside those documents are decisions, tasks, owners, and relationships: basically an untapped knowledge graph that is constantly changing. We just published a full walkthrough showing how to turn meeting notes in Drive into a **live-updating knowledge graph** using CocoIndex + LLM extraction.
Check out the tutorial for more details.
**What's cool about this example:**
- **Incremental processing**: Only changed documents get reprocessed. If you have thousands of meeting notes, but only 1% change each day, CocoIndex only touches that 1%, saving 99% of LLM cost and compute.
- **Structured extraction with LLMs**: We use a typed Python dataclass as the schema, so the LLM returns real structured objects, not brittle JSON prompts.
- **Graph-native export**: CocoIndex maps nodes (Meeting, Person, Task) and relationships (ATTENDED, DECIDED, ASSIGNED_TO) without writing Cypher, directly into Neo4j.
- **Real-time updates**: If a meeting note changes (task reassigned, typo fixed, new discussion added), the graph updates automatically.
### Extracting structured data from patient intake forms with DSPy and CocoIndex
DSPy and CocoIndex Patient Form Extraction
Patient intake forms are indeed a rich source of structured clinical data, but traditional OCR + regex pipelines fail to reliably capture their nested, conditional, and variable structure, leaving most of that value locked in unstructured text or manual entry.
This project demonstrates a production-ready approach to extracting **clean, validated, structured patient data directly from PDF intake forms** using **DSPy** and **CocoIndex**, without brittle OCR pipelines, regex rules, or Markdown intermediates.
The result is a scalable, explainable, and compliant pipeline well-suited for healthcare and other regulated domains, turning unstructured intake forms into always-fresh, queryable patient models ready for downstream systems, analytics, or AI agents.
Check out the tutorial for more details.
## Thanks to the community 🤗🎉
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @1fanwang
Thanks @1fanwang for adding Azure OpenAI support as an LLM provider in #1362, enabling enterprise users to leverage CocoIndex with their Azure deployments.
### @vipulaSD
Thanks @vipulaSD for adding field-level descriptions for data classes in the Product Recommendation example in #1327, making the example more informative and easier to understand.
### @prrao87
Thanks @prrao87 for adding support for embedding models in OpenRouter in #1359, expanding LLM provider options, and fixing typos and consistency issues in the docs in #1360, making the documentation clearer and more polished.
### @Haleshot
Thanks @Haleshot for improving the README note for better GitHub rendering in #1288, shifting to the tracing crate for logging in #1363, adding Postgres server reminder to quickstart docs in #1368, fixing the broken workflow badge in README in #1370, introducing a structured error system with host exception tunneling in #1380, adding llms.txt file to docs in #1455, adding short flag aliases for common CLI options in #1458, and adding env file sourcing tip to meeting_notes_graph example in #1470.
### @thisisharsh7
Thanks @thisisharsh7 for making openai.rs generic to share logic with azureopenai.rs in #1376, including mypy type checking for the examples directory in #1383, and adding the GeneratedOutput enum for direct JSON returns in #1395, enhancing type safety and code reusability.
### @prithvi-moonshot
Thanks @prithvi-moonshot for adding onBrokenAnchors config to report documentation errors in #1325, providing an option to force reprocessing during update in #1385, and making target deletion (drop) tolerate failures in #1454, improving robustness of the data pipeline.
### @skprasadu
Thanks @skprasadu for wrapping cargo test to avoid embedded Python stdlib discovery crash in #1398, moving Query Support above Built-in Sources in the docs for better organization in #1397, and adding a warning when live update pass exceeds refresh_interval in #1408.
### @Gohlub
Thanks @Gohlub for adding with_context to user‑configured operations in #1275, enabling more powerful context‑aware flows for sources, functions, and targets.
### @wxh06
Thanks @wxh06 for adding expected_output_dimension parameter to embed_text in #1373, providing more control over embedding output dimensions.
### @samojavo
Thanks @samojavo for fixing mypy errors in example entry points and adding a local type-check script in #1248, improving code quality and developer experience.
### @aryasoni98
Thanks @aryasoni98 for enabling programmatic API key passing besides reading from environment variables in #1134, providing more flexibility in configuration.
### @shinobiwanshin
Thanks @shinobiwanshin for enhancing the Button component with hover effects and styling improvements in #1393, making the UI more polished and interactive.
### @majiayu000
Thanks @majiayu000 for adding VectorIndexMethod support for HNSW configuration in Qdrant in #1410, enabling more fine-grained control over vector indexing parameters.
## Summary
CocoIndex continues evolving into a **resilient, intelligent data engine** for the modern AI stack.
This cycle focuses on **error transparency, LLM integration depth, and frictionless developer operations**, ensuring your pipelines stay alive, observable, and blazing fast even under adverse conditions.
If you like this project, please support us with a star ⭐ at https://github.com/cocoindex-io/cocoindex !
🔥🔥 We are cooking something big! v1 is coming soon. Stay tuned! If you read all the way here, you are a true fan!
V1 is Coming
---
# Extract patient intake forms with DSPy and CocoIndex
URL: https://cocoindex.io/blogs/extraction-dspy/
Date: 2025-12-15
Summary: Extract Pydantic-typed structured data from patient intake forms using DSPy and CocoIndex: OCR vision models with incremental processing.
Patient intake forms are indeed a rich source of structured clinical data, but traditional OCR + regex pipelines fail to reliably capture their nested, conditional, and variable structure, leaving most of that value locked in unstructured text or manual entry.
In today's example, we are going to show how to extract clean, typed, Pydantic-validated structured data directly from PDFs, using:
- **DSPy** (for multimodal structured extraction with Gemini 2.5 Flash vision)
- **CocoIndex** (for incremental processing, caching, and database storage)
No manual text extraction, no brittle markdown conversion: just connect to the source, transform the PDFs, and get validated patient models out, ready to go in production.
The entire code is open sourced with Apache 2.0 license. To see more examples built with CocoIndex, you could refer to the examples page.
**⭐ Star** the project if you find it helpful! GitHub
## Why DSPy + CocoIndex?
Before jumping in, here’s what each component contributes:
### DSPy: A programming framework for LLMs
Traditional LLM apps rely on prompt engineering: you write a prompt with instructions, few‑shot examples, and formatting, then call the model and parse the raw text. This approach is fragile:
- Small changes in the prompt, model, or data can break the output format or quality.
- Logic is buried in strings, making it hard to test, compose, or version.
DSPy replaces this with a programming model: you define what each LLM step should do (inputs, outputs, constraints), and the framework figures out how to prompt the model to satisfy that spec.
### CocoIndex: An ultra performant data processing engine for AI workloads
CocoIndex is an ultra performant compute framework for AI workloads, with incremental processing. Users write simple in-memory computations in Python and coco runs them as a resilient, scalable data pipeline (with Rust Engine) – with fresh data always ready for serving, built on a small set of core concepts. The same flow definition you use in a notebook can be lifted easily into production.
With CocoIndex, changes in sources or transformation logic only trigger minimal recompute, cutting cold-start “backfill” latencies from hours to seconds while reducing GPU/API spend. In production, this manifests as always-fresh targets: you run in “live” mode with change data capture or polling, and CocoIndex keeps derived stores in sync with complex unstructured sources like codebases, PDFs, and multi-hop API compositions.
Because every transformation step is observable with lineage, teams get auditability and explainability out of the box, which helps with regulated scenarios like healthcare extraction or financial workflows.
### DSPy & CocoIndex synergy
The synergy shows up most clearly in end-to-end AI data products: DSPy defines robust, typed extractors or decision modules, and CocoIndex wires them into a resilient, incremental pipeline that can meet SLOs and compliance needs. Any change in documents, code, or business rules is reflected quickly and explainably in the targets and features those agents consume.
## Flow overview
### Prerequisites
Before getting started, make sure you have the following set up:
1. Install Postgres if you don't have one, and ensure you can connect to it from your development environment.
2. Python dependencies
```bash
pip install -U cocoindex dspy-ai pydantic pymupdf
```
3. Create a .env file:
```bash
# Postgres database address for cocoindex
COCOINDEX_DATABASE_URL=postgres://cocoindex:cocoindex@localhost/cocoindex
# Gemini API key
GEMINI_API_KEY=YOUR_GEMINI_API_KEY
```
## Pydantic models: Define the structured schema
We defined Pydantic-style classes (Contact, Address, Insurance, etc.) to match a *FHIR-inspired patient schema*, enabling structured and validated representations of patient data. Each model corresponds to a key aspect of a patient's record, ensuring both type safety and nested relationships.
Patient schema
### 1. Contact model
```python
class Contact(BaseModel):
name: str
phone: str
relationship: str
```
- Represents an **emergency or personal contact** for the patient.
- Fields:
- name: Contact's full name.
- phone: Contact phone number.
- relationship: Relation to the patient (e.g., parent, spouse, friend).
### 2. Address model
```python
class Address(BaseModel):
street: str
city: str
state: str
zip_code: str
```
- Represents a **postal address**.
- Fields:
- street, city, state, zip_code: Standard address fields.
### 3. Pharmacy model
```python
class Pharmacy(BaseModel):
name: str
phone: str
address: Address
```
- Represents the **patient’s preferred pharmacy**.
- Fields:
- name: Pharmacy name.
- phone: Pharmacy contact number.
- address: Uses the Address model for structured address information.
### 4. Insurance model
```python
class Insurance(BaseModel):
provider: str
policy_number: str
group_number: str | None = None
policyholder_name: str
relationship_to_patient: str
```
- Represents the patient’s **insurance information**.
- Fields:
- provider: Insurance company name.
- policy_number: Unique policy number.
- group_number: Optional group number.
- policyholder_name: Name of the person covered under the insurance.
- relationship_to_patient: Relationship to patient (e.g., self, parent).
### 5. Condition model
```python
class Condition(BaseModel):
name: str
diagnosed: bool
```
- Represents a **medical condition**.
- Fields:
- name: Condition name (e.g., Diabetes).
- diagnosed: Boolean indicating whether it has been officially diagnosed.
### 6. Medication model
```python
class Medication(BaseModel):
name: str
dosage: str
```
- Represents a **current medication** the patient is taking.
- Fields:
- name: Medication name.
- dosage: Dosage information (e.g., "10mg daily").
### 7. Allergy model
```python
class Allergy(BaseModel):
name: str
```
- Represents a **known allergy**.
- Fields:
- name: Name of the allergen (e.g., peanuts, penicillin).
### 8. Surgery model
```python
class Surgery(BaseModel):
name: str
date: str
```
- Represents a **surgery or procedure** the patient has undergone.
- Fields:
- name: Surgery name (e.g., Appendectomy).
- date: Surgery date (as a string, ideally ISO format).
### 9. Patient model
```python
class Patient(BaseModel):
name: str
dob: datetime.date
gender: str
address: Address
phone: str
email: str
preferred_contact_method: str
emergency_contact: Contact
insurance: Insurance | None = None
reason_for_visit: str
symptoms_duration: str
past_conditions: list[Condition] = Field(default_factory=list)
current_medications: list[Medication] = Field(default_factory=list)
allergies: list[Allergy] = Field(default_factory=list)
surgeries: list[Surgery] = Field(default_factory=list)
occupation: str | None = None
pharmacy: Pharmacy | None = None
consent_given: bool
consent_date: str | None = None
```
- Represents a **complete patient record** with personal, medical, and administrative information.
- Key fields:
- name, dob, gender: Basic personal info.
- address, phone, email: Contact info.
- preferred_contact_method: How the patient prefers to be reached.
- emergency_contact: Nested Contact model.
- insurance: Optional nested Insurance model.
- reason_for_visit, symptoms_duration: Visit details.
- past_conditions, current_medications, allergies, surgeries: Lists of nested models for comprehensive medical history.
- occupation: Optional job info.
- pharmacy: Optional nested Pharmacy model.
- consent_given, consent_date: Legal/administrative consent info.
### Why use Pydantic here?
1. **Validation:** Ensures all fields are the correct type (e.g., dob is a date).
2. **Structured Nested Models:** Patient has nested objects like Address, Contact, and Insurance.
3. **Default Values & Optional Fields:** Handles optional fields and defaults (Field(default_factory=list) ensures empty lists if no data).
4. **Serialization:** Easily convert models to JSON for APIs or databases.
5. **Error Checking:** Automatically raises errors if invalid data is provided.
## DSPy vision extractor
### DSPy signature
Let’s define PatientExtractionSignature. A **Signature** describes what data your module expects and what it will produce. Think of it as a **schema for an AI task**.
PatientExtractionSignature is a dspy.Signature, which is DSPy's way of declaring *what* the model should do, not *how* it does it.
```python
# DSPy Signature for patient information extraction from images
class PatientExtractionSignature(dspy.Signature):
"""Extract structured patient information from a medical intake form image."""
form_images: list[dspy.Image] = dspy.InputField(
desc="Images of the patient intake form pages"
)
patient: Patient = dspy.OutputField(
desc="Extracted patient information with all available fields filled"
)
```
This signature defines **the task contract** for patient information extraction.
- Inputs: form_images – a list of images of the intake form.
- Outputs: patient – a structured Patient object.
From DSPy's point of view, this Signature is a "spec": a mapping from an image-based context to a structured, Pydantic-backed semantic object that can later be optimized, trained, and composed with other modules.
### PatientExtractor module
PatientExtractor is a dspy.Module, which in DSPy is a composable, potentially trainable building block that implements the Signature.
```python
class PatientExtractor(dspy.Module):
"""DSPy module for extracting patient information from intake form images."""
def __init__(self) -> None:
super().__init__()
self.extract = dspy.ChainOfThought(PatientExtractionSignature)
def forward(self, form_images: list[dspy.Image]) -> Patient:
"""Extract patient information from form images and return as a Pydantic model."""
result = self.extract(form_images=form_images)
return result.patient # type: ignore
```
- In __init__, ChainOfThought is a DSPy *primitive module* that knows how to call an LLM with reasoning-style prompting to satisfy the given Signature. In other words, it is a default "strategy" for solving the "extract patient from images" task.
- The forward method is DSPy's standard interface for executing a module. You pass form_images into self.extract(). DSPy then handles converting this call into an LLM interaction (or a trained program) that produces a patient field as declared in the Signature.
Conceptually, PatientExtractor is an *ETL operator*: the Signature describes the input/output types, and the internal ChainOfThought module is the function that fills that contract.
### Single-step extraction
Now let’s wire the DSPy Module to extract from a single PDF. From a high level,
- The extractor receives PDF bytes directly
- Internally converts PDF pages to DSPy Image objects using PyMuPDF
- Processes images with vision model
- Returns Pydantic model directly
```python
@cocoindex.op.function(cache=True, behavior_version=1)
def extract_patient(pdf_content: bytes) -> Patient:
"""Extract patient information from PDF content."""
# Convert PDF pages to DSPy Image objects
pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")
form_images = []
for page in pdf_doc:
# Render page to pixmap (image) at 2x resolution for better quality
pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
# Convert to PNG bytes
img_bytes = pix.tobytes("png")
# Create DSPy Image from bytes
form_images.append(dspy.Image(img_bytes))
pdf_doc.close()
# Extract patient information using DSPy with vision
extractor = PatientExtractor()
patient = extractor(form_images=form_images)
return patient # type: ignore
```
This function is a **CocoIndex custom function** (decorated with @cocoindex.op.function) that takes **PDF bytes** as input and returns a fully structured Patient Pydantic object.
- cache=True allows repeated calls with the same PDF to **reuse results**.
- behavior_version=1 ensures versioning of the function for reproducibility.
### Create DSPy image objects
We open the PDF from bytes using **PyMuPDF** (pymupdf), then we iterate over each page.
- Useful trick: Render the page as a **high-resolution image** (2x) for better OCR/vision performance.
- Convert the rendered page to **PNG bytes**.
- Wrap the PNG bytes in a **DSPy Image object**.
### DSPy extraction
The list of form_images is passed to the DSPy module:
1. **ChainOfThought reasoning** interprets each image.
2. **Vision + NLP** extract relevant text fields.
3. **Populate Pydantic Patient object** with structured patient info.
## CocoIndex flow
CocoIndex Flow
- Loads PDFs from local directory as binary
- For each document, applies single transform: PDF bytes → Patient data
- Exports the results to a PostgreSQL table
### Declare flow
Declare a CocoIndex flow, connect to the source, add a data collector to collect processed data.
```python
@cocoindex.flow_def(name="PatientIntakeExtractionDSPy")
def patient_intake_extraction_dspy_flow(
flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="data/patient_forms", binary=True)
)
patients_index = data_scope.add_collector()
```
- @cocoindex.flow_def tells CocoIndex that this function is a flow definition, not regular runtime code.
- add_source() registers a LocalFile source that traverses the data/patient_forms directory and creates a logical table named documents.
Ingesting Data
You can connect to various sources, or even custom sources with CocoIndex if native connectors are not available. CocoIndex is designed to keep your indexes synchronized with your data sources. This is achieved through a feature called **live updates**, which automatically detects changes in your sources and updates your indexes accordingly. This ensures that your search results and data analysis are always based on the most current information. You can read more in the live updates documentation.
### Process documents
```python
with data_scope["documents"].row() as doc:
# Extract patient information directly from PDF using DSPy with vision
# (PDF->Image conversion happens inside the extractor)
doc["patient_info"] = doc["content"].transform(extract_patient)
# Collect the extracted patient information
patients_index.collect(
filename=doc["filename"],
patient_info=doc["patient_info"],
)
```
This iterates over each document. We transform doc["content"] (the bytes) by our extract_patient function. The result is stored in a new field patient_info.
Then we collect a row with the filename and extracted patient_info.
Transforming Data
Nested Data
### Export to Postgres
```python
patients_index.export(
"patients",
cocoindex.storages.Postgres(table_name="patients_info_dspy"),
primary_key_fields=["filename"],
)
```
We export the collected index to Postgres. This will create/maintain a table patients keyed by filename, automatically deleting or updating rows if inputs change.
Because CocoIndex tracks data lineage, it will handle updates/deletions of source files incrementally.
### Configure CocoIndex settings
Define a **CocoIndex settings function** that configures the AI model for DSPy:
```python
@cocoindex.settings
def cocoindex_settings() -> cocoindex.Settings:
# Configure the model used in DSPy
lm = dspy.LM("gemini/gemini-2.5-flash")
dspy.configure(lm=lm)
return cocoindex.Settings.from_env()
```
It returns a cocoindex.Settings object initialized from environment variables, enabling the system to use the configured model and environment settings for all DSPy operations.
## Running the pipeline
Update the index:
```bash
cocoindex update main
```
### CocoInsight
I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with zero pipeline data retention.
```bash
cocoindex server -ci main
```
## Scalable open ecosystem, not a closed box
CocoIndex is intentionally "composable by default": it gives you a fast, incremental data engine and clean flow abstraction, but never locks you into a specific model, vector DB, processing module or orchestration stack.
CocoIndex treats everything (sources, ops, and storages) as pluggable interfaces rather than proprietary primitives. You can read from local files, S3, APIs, or custom sources, call any data transformation logic (beyond SQL, DSPy modules, any complex Python transformations, generated parsers, etc.), and export to relational databases, vector databases, search engines, or custom sinks through its storage layer.
### Why DSPy + CocoIndex fits this philosophy
DSPy is itself a compositional framework for LLMs: you define typed Signatures and Modules, and it learns how to implement them, making the LLM layer programmable, testable, and optimizable.
CocoIndex treats these modules as first-class operators in the flow, so you get a clean separation of concerns: DSPy owns “how the model thinks,” while CocoIndex owns “how data moves, is cached, and is served” across changing PDFs, code, or APIs.
This pairing is powerful because neither system tries to be the entire stack: CocoIndex does not prescribe a prompt framework, and DSPy does not prescribe a data pipeline engine. Instead, they interlock: DSPy modules become composable building blocks inside CocoIndex flows, and CocoIndex gives those modules a production context with retries, batching, caching, and live updates.
## Comparison of DSPy vs BAML examples
For this DSPy and BAML example with CocoIndex, there are some differences in the data flow and the schema enforcement.
| Aspect | This tutorial (DSPy) | Previous intake tutorials (OpenAI/BAML) |
|--------------------|--------------------------------------------------------------------|-------------------------------------------------------------------------------|
| Model interaction | DSPy Signatures + Modules, trainable and optimizable. | Direct prompts or BAML functions with generated clients. |
| Input format | PDF bytes → images → Gemini Vision (no Markdown step). | PDF/DOCX → Markdown via parsers (e.g., MarkItDown) → text LLM. |
| Schema enforcement | Pydantic models used directly as DSPy output types. | Dataclasses or BAML types mapped into CocoIndex collectors. |
| Pipeline engine | CocoIndex incremental flow with LocalFile source + Postgres sink. | Same engine
DSPy and BAML both help build LLM applications with structured outputs, but they emphasize different things: DSPy focuses on modular Python workflows and optimizer-driven prompt tuning, while BAML focuses on a typed DSL, schema guarantees, and multi-language client generation.
## Support CocoIndex ❤️
If this example was helpful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub.
Your stars help us grow the community, stay motivated, and keep shipping better tools for real-time data ingestion and transformation.
---
# A knowledge graph from meeting notes that auto-updates
URL: https://cocoindex.io/blogs/meeting-notes-graph/
Date: 2025-12-08
Summary: Build a self-updating Neo4j knowledge graph from meeting notes with CocoIndex v1: LLM-extracted decisions, tasks, and owners, with embedding-based dedupe.
Meeting notes capture decisions, action items, who was in the room, and how people and tasks connect. Most teams keep them as static documents, reachable only through text search. A knowledge graph lets you query them like a database instead: *"Who attended the budget planning meetings?"* or *"What tasks did Sarah get assigned across every meeting?"*
To answer questions like these you extract structured information from unstructured notes and store it as a graph, which makes relationship queries direct instead of impossible.
This post walks through the **Meeting Notes Graph** example: a CocoIndex v1 pipeline that reads Markdown meeting notes from Google Drive, extracts structure with an LLM, deduplicates person names with embedding-based entity resolution, and builds a queryable graph in Neo4j.
The project is open-sourced in the CocoIndex v1 repo.
Neo4j Property Graph
## Why a graph, and why incremental
Meeting notes are living documents. Attendee names get corrected, tasks get reassigned, and decisions get revised. A batch pipeline that rebuilds the whole graph on every edit is wasteful once you have more than a handful of files, and it makes near-real-time freshness expensive.
CocoIndex processes incrementally: it detects which files changed and reprocesses only those. Unchanged notes never hit the LLM, and the graph write is a no-op for anything that did not change. The heavy LLM extraction step is memoized, so as long as a section's text and the model are unchanged, the cached result is reused.
## The graph model
We build a property graph with three node types and three edge types:
Property graph model: a Person node ATTENDED a Meeting as organizer, is ASSIGNED_TO a Task, and the Meeting DECIDED that Task. Circles are nodes; arrows are relationships.
**Nodes**
- **Meeting**: one per meeting section, with its date and notes
- **Person**: a canonical individual (organizer, participant, or task assignee)
- **Task**: an action item decided in a meeting
**Edges**
- **ATTENDED**: Person → Meeting, carrying an is_organizer flag
- **DECIDED**: Meeting → Task
- **ASSIGNED_TO**: Person → Task
To learn more about property graphs, see CocoIndex's Property Graph Targets documentation.
## Define the pipeline
CocoIndex v1 is declarative and Python-native: you write plain async Python and the Rust engine handles incremental processing. The whole example lives in one main.py, wrapped in a coco.App:
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import google_drive, neo4j
from cocoindex.ops.entity_resolution import ResolvedEntities, resolve_entities
from cocoindex.ops.entity_resolution.llm_resolver import LlmPairResolver
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.id import IdGenerator
app = coco.App(coco.AppConfig(name="MeetingNotesGraphNeo4j"), app_main)
```
### Shared resources
The Neo4j connection, the two LLM models, and the embedder are provided once in the app lifespan and retrieved anywhere with coco.use_context. The neo4j.ConnectionFactory reads standard connection settings from the environment, defaulting to a local Neo4j:
```python title=main.py
KG_DB = coco.ContextKey[neo4j.ConnectionFactory]("kg_db")
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
RESOLUTION_LLM_MODEL = coco.ContextKey[str]("resolution_llm_model", detect_change=True)
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.provide(
KG_DB,
neo4j.ConnectionFactory(
uri=os.environ.get("NEO4J_URI", "bolt://localhost:7687"),
auth=(
os.environ.get("NEO4J_USER", "neo4j"),
os.environ.get("NEO4J_PASSWORD", "cocoindex"),
),
database=os.environ.get("NEO4J_DATABASE", "neo4j"),
),
)
builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-5-mini"))
builder.provide(
RESOLUTION_LLM_MODEL,
os.environ.get("RESOLUTION_LLM_MODEL", "openai/gpt-5-mini"),
)
builder.provide(
EMBEDDER,
SentenceTransformerEmbedder("Snowflake/snowflake-arctic-embed-xs"),
)
yield
```
detect_change=True means swapping a model or the embedder re-runs only the affected steps.
### Node and edge schemas
Node rows are plain dataclasses. The primary key of each becomes the node's identity in Neo4j, so re-runs reconcile existing nodes in place instead of duplicating them:
```python title=main.py
@dataclass
class Meeting:
id: int
note_file: str
time: datetime.date
note: str
@dataclass
class Person:
name: str # canonical
@dataclass
class Task:
description: str
@dataclass
class AttendedRel:
is_organizer: bool
```
AttendedRel is the edge payload for ATTENDED. DECIDED and ASSIGNED_TO carry no payload, so they are declared without a record. Relation primary keys are auto-derived by the Neo4j connector from (from_id, to_id), giving exactly one edge per endpoint pair.
Each dataclass maps directly to the graph: a record becomes a node keyed by its primary key, and a declared relation becomes an edge between two endpoint keys.
Dataclass record schemas map to Neo4j: the Meeting, Person, and Task node dataclasses plus the AttendedRel edge payload become Person, Meeting, and Task nodes joined by ATTENDED, DECIDED, and ASSIGNED_TO relationships.
### The extraction schema
The LLM extraction target is a set of Pydantic models. instructor uses these as the response schema, so the model returns structured data that matches the Python types instead of free-form text:
```python title=main.py
class ExtractedPerson(pydantic.BaseModel):
name: str = pydantic.Field(
description="Full name of the person, as written in the note."
)
class ExtractedTask(pydantic.BaseModel):
description: str = pydantic.Field(
description="Concise, standalone description of the task or action item."
)
assigned_to: list[ExtractedPerson] = pydantic.Field(default_factory=list)
class ExtractedMeeting(pydantic.BaseModel):
time: datetime.date
note: str
organizer: ExtractedPerson
participants: list[ExtractedPerson] = pydantic.Field(default_factory=list)
tasks: list[ExtractedTask] = pydantic.Field(default_factory=list)
```
### Split, then extract
A single note file often holds several meetings. _split_meetings breaks the text at Markdown headings (# or ##) so each section becomes one meeting:
```python title=main.py
_HEADING_RE = re.compile(r"\n\n##?\s+")
def _split_meetings(text: str) -> list[str]:
parts = _HEADING_RE.split("\n\n" + text)
return [p.strip() for p in parts if p.strip()]
```
Each section goes to extract_meeting, which calls the model through LiteLLM and instructor. The model comes from the LLM_MODEL context (default openai/gpt-5-mini), and memo=True caches the result so an unchanged section is never re-extracted:
```python title=main.py
@coco.fn(memo=True)
async def extract_meeting(section_text: str) -> ExtractedMeeting:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=ExtractedMeeting,
messages=[
{"role": "system", "content": EXTRACT_PROMPT},
{"role": "user", "content": section_text},
],
)
return ExtractedMeeting.model_validate(result.model_dump())
```
## Three phases
The pipeline runs in three phases. Phase 1 extracts each file and declares the nodes and edges it can build on its own. Phase 2 resolves person names across the whole corpus. Phase 3 declares the canonical Person nodes and the edges that touch people.
Three phases: Phase 1 extracts each file with an LLM and declares Meeting and Task nodes plus DECIDED edges; Phase 2 resolves person names across the corpus so Sarah and Sarah Chen collapse to one; Phase 3 declares canonical Person nodes and the ATTENDED and ASSIGNED_TO edges, writing the graph to Neo4j.
### Phase 1: per-file extraction
process_file reads a Drive file, splits it, and for each meeting declares a Meeting node and its Task nodes plus DECIDED edges. Meeting IDs come from an IdGenerator keyed on the meeting date. Person names are not resolved yet, so the raw names are collected into a MeetingExtraction and returned for later phases:
```python title=main.py
@coco.fn(memo=True)
async def process_file(
file: google_drive.DriveFile,
meeting_table: neo4j.TableTarget[Meeting],
task_table: neo4j.TableTarget[Task],
decided_rel: neo4j.RelationTarget[Any],
) -> list[MeetingExtraction]:
text = await file.read_text()
note_file = file.file_path.path.as_posix()
id_generator = IdGenerator()
extractions = []
for section in _split_meetings(text):
extracted = await extract_meeting(section)
meeting_id = await id_generator.next_id(extracted.time)
meeting_table.declare_record(
row=Meeting(
id=meeting_id,
note_file=note_file,
time=extracted.time,
note=extracted.note,
)
)
for task in extracted.tasks:
task_table.declare_record(row=Task(description=task.description))
decided_rel.declare_relation(from_id=meeting_id, to_id=task.description)
extractions.append(
MeetingExtraction(
meeting_id=meeting_id,
organizer=extracted.organizer.name,
participants=[p.name for p in extracted.participants],
task_assignees=[
(t.description, [a.name for a in t.assigned_to])
for t in extracted.tasks
],
)
)
return extractions
```
### Phase 2: person entity resolution
The same person shows up under different names across files: "Sarah" in one note, "Sarah Chen" in another. If you create a Person node per raw string, the graph fragments and relationship queries miss edges.
_resolve_persons deduplicates raw names with embedding-based entity resolution. Candidate matches are found by embedding similarity, then confirmed pairwise by an LLM. The result maps every raw name to a canonical one:
```python title=main.py
@coco.fn(memo=True)
async def _resolve_persons(raw_persons: set[str]) -> ResolvedEntities:
return await resolve_entities(
entities=raw_persons,
embedder=coco.use_context(EMBEDDER),
resolve_pair=LlmPairResolver(model=coco.use_context(RESOLUTION_LLM_MODEL)),
)
```
ResolvedEntities exposes .canonicals() for the deduplicated set and .canonical_of(raw_name) to look up which canonical a raw name folds into. So "Sarah" and "Sarah Chen" collapse to a single Person node.
### Phase 3: canonical people and their edges
With names resolved, create_person_relations declares one Person node per canonical name, then the person-touching edges. ATTENDED aggregates organizer and participants so a person listed as both gets a single edge with is_organizer=True. ASSIGNED_TO dedupes per canonical person and task:
```python title=main.py
@coco.fn
async def create_person_relations(
meetings: list[MeetingExtraction],
persons: ResolvedEntities,
person_table: neo4j.TableTarget[Person],
attended_rel: neo4j.RelationTarget[Any],
assigned_rel: neo4j.RelationTarget[Any],
) -> None:
for canonical_name in persons.canonicals():
person_table.declare_record(row=Person(name=canonical_name))
for m in meetings:
attendees: dict[str, bool] = {persons.canonical_of(m.organizer): True}
for p in m.participants:
attendees.setdefault(persons.canonical_of(p), False)
for canonical, is_organizer in attendees.items():
attended_rel.declare_relation(
from_id=canonical,
to_id=m.meeting_id,
record=AttendedRel(is_organizer=is_organizer),
)
for task_desc, assignees in m.task_assignees:
seen: set[str] = set()
for raw in assignees:
canonical = persons.canonical_of(raw)
if canonical in seen:
continue
seen.add(canonical)
assigned_rel.declare_relation(from_id=canonical, to_id=task_desc)
```
### Wiring it together
app_main mounts the three node tables and three relation targets, then runs the phases. Node tables take a TableSchema and a primary key; relations mount without a schema so the connector derives their PK from the endpoints. The Google Drive source is iterated with async for, and each file is processed under its own component path so change detection is per-file:
```python title=main.py
@coco.fn
async def app_main() -> None:
meeting_table = await neo4j.mount_table_target(
KG_DB, "Meeting",
await neo4j.TableSchema.from_class(Meeting, primary_key="id"),
primary_key="id",
)
person_table = await neo4j.mount_table_target(
KG_DB, "Person",
await neo4j.TableSchema.from_class(Person, primary_key="name"),
primary_key="name",
)
task_table = await neo4j.mount_table_target(
KG_DB, "Task",
await neo4j.TableSchema.from_class(Task, primary_key="description"),
primary_key="description",
)
attended_rel = await neo4j.mount_relation_target(
KG_DB, "ATTENDED", person_table, meeting_table
)
decided_rel = await neo4j.mount_relation_target(
KG_DB, "DECIDED", meeting_table, task_table
)
assigned_rel = await neo4j.mount_relation_target(
KG_DB, "ASSIGNED_TO", person_table, task_table
)
# Phase 1: per-file extraction
source = google_drive.GoogleDriveSource(
service_account_credential_path=os.environ["GOOGLE_SERVICE_ACCOUNT_CREDENTIAL"],
root_folder_ids=[
f.strip()
for f in os.environ["GOOGLE_DRIVE_ROOT_FOLDER_IDS"].split(",")
if f.strip()
],
)
file_coros = []
async for path_key, file in source.items():
file_coros.append(
coco.use_mount(
coco.component_subpath("file", path_key),
process_file, file, meeting_table, task_table, decided_rel,
)
)
per_file = list(await asyncio.gather(*file_coros))
all_meetings = [m for ms in per_file for m in ms]
# Phase 2: person entity resolution
raw_persons: set[str] = set()
for m in all_meetings:
raw_persons.add(m.organizer)
raw_persons.update(m.participants)
for _task_desc, assignees in m.task_assignees:
raw_persons.update(assignees)
persons = await coco.use_mount(
coco.component_subpath("resolve_persons"), _resolve_persons, raw_persons
)
# Phase 3: declare Person nodes + person-touching relations
await coco.mount(
coco.component_subpath("person_relations"),
create_person_relations,
all_meetings, persons, person_table, attended_rel, assigned_rel,
)
```
When a component finishes, CocoIndex compares its declared nodes and edges against the previous run and applies only the differences: new items are inserted, changed ones updated, and items whose source file disappeared are removed. Nothing that stayed the same is rewritten.
## Run
### Prerequisites
- Neo4j running locally. Default connection bolt://localhost:7687, browser at http://localhost:7474, username neo4j, password cocoindex.
- An OpenAI API key (OPENAI_API_KEY). The default models are openai/gpt-5-mini for both extraction and resolution.
- Google Drive service-account credentials. Create a Google Cloud service account, download its JSON key, share the source folders with the service-account email, and collect the folder IDs. See Setup for Google Drive.
Set the environment variables:
```sh
export OPENAI_API_KEY=sk-...
export GOOGLE_SERVICE_ACCOUNT_CREDENTIAL=/absolute/path/to/service_account.json
export GOOGLE_DRIVE_ROOT_FOLDER_IDS=folderId1,folderId2
```
GOOGLE_DRIVE_ROOT_FOLDER_IDS accepts a comma-separated list.
### Build the graph
```sh
pip install -e .
cocoindex update main
```
Use cocoindex update -L main for live mode, which keeps watching Drive and updates the graph as notes change.
### Browse the graph
Open Neo4j Browser at http://localhost:7474 and run Cypher:
```cypher
// All relationships
MATCH p=()-->() RETURN p
// Who attended which meetings (including organizer)
MATCH (p:Person)-[:ATTENDED]->(m:Meeting)
RETURN p, m
// Tasks decided in meetings
MATCH (m:Meeting)-[:DECIDED]->(t:Task)
RETURN m, t
// Task assignments
MATCH (p:Person)-[:ASSIGNED_TO]->(t:Task)
RETURN p, t
```
Neo4j Property Graph
## Where else this applies
The same pattern (ingest → split → extract → resolve entities → build graph) fits any text-heavy domain where the entities and their relationships matter more than the raw text:
- **Research papers**: build a graph of concepts, authors, and citations across a repository, and keep it current as papers are added or revised.
- **Support tickets**: link issues, solutions, customers, and recurring patterns across a large, frequently edited ticket base.
- **Compliance documents**: extract requirements from policies and trace how a policy change cascades through related obligations.
- **Competitive intelligence**: connect companies, products, and market moves extracted from public filings and news.
## Summary
This example turns unstructured meeting notes into a queryable Neo4j graph: an LLM extracts structure, embedding-based entity resolution keeps one node per real person, and CocoIndex's incremental processing keeps the graph in sync as notes change without reprocessing the whole corpus. The same source → split → extract → resolve → graph template carries over to any domain where relationships are the thing you want to query.
## Support CocoIndex ❤️
If this example was helpful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub.
Your stars help us grow the community, stay motivated, and keep shipping better tools for real-time data ingestion and transformation.
---
# What HackerNews is talking about: trending topics with an LLM
URL: https://cocoindex.io/blogs/hackernews-trending-topics/
Date: 2025-12-02
Summary: Rank trending HackerNews topics in Postgres with an incremental CocoIndex v1 pipeline: scrape threads and comments, extract topics with an LLM.
The tech community's attention is hiding in plain sight, scattered across story titles and thousands of comments. This post builds a pipeline that reads recent HackerNews threads and their comments, asks an LLM to pull the topics out of every message, and ranks what's trending in Postgres, rewritten for CocoIndex v1.
Two pieces do the work: the LLM extraction that turns free-form text into normalized topics, and a SQL score that turns those topics into a leaderboard. This post focuses on both. The source here is a custom one (two async functions over the Algolia HN API); the custom-source walkthrough covers that side in detail, so we won't repeat it here.
The full example is in the CocoIndex v1 repo.
## How it works
Each recent story becomes its own processing component. The component fetches the story and its comments, extracts topics from each message with an LLM, and declares rows into two Postgres tables:
- hn_messages: one row per message (the story and each comment), for full-text search
- hn_topics: one row per topic mention, keyed on (topic, message_id)
Because the same topic mentioned in many messages becomes many hn_topics rows, ranking becomes a simple SQL aggregate. Re-running catches up on new stories, and because per-message extraction is memoized, only content CocoIndex hasn't seen hits the LLM.
## The topic extraction is the product
The interesting part is not calling an LLM; it's getting topics back in a shape you can rank. Raw model output would give you "postgres" in one comment and "PostgreSQL" in another, or a whole phrase like "books for autistic kids" as a single blob. Those never aggregate.
The fix lives entirely in the prompt, expressed as the description of a Pydantic field. The model normalizes three ways: it splits combined phrases into parts, keeps proper nouns canonical, and emits aliases for the same thing:
Topic extraction: an LLM reads a HackerNews message and returns a normalized list of topics. It splits combined phrases into parts (books for autistic kids becomes book, autistic, autistic kids), keeps proper nouns canonical (postgres becomes PostgreSQL), and emits aliases for the same thing (JFK and John Kennedy).
```python title=models.py
class TopicsResponse(BaseModel):
"""Response containing a list of extracted topics."""
topics: list[str] = Field(
description="""List of extracted topics.
Each topic can be a product name, technology, model, people, company name, business domain, etc.
Capitalize for proper nouns and acronyms only.
Use the form that is clear alone.
For topics that are a phrase combining multiple things, normalize into multiple topics. Examples:
- "books for autistic kids" -> "book", "autistic", "autistic kids"
When there're multiple common ways to refer to the same thing, use multiple topics. Examples:
- "John Kennedy", "JFK"
"""
)
```
That normalization is what makes the trending ranking meaningful later. Get it wrong and "PostgreSQL", "postgres", and "psql" split their votes three ways.
Extraction itself is one CocoIndex function: text in, a list of topics out, through litellm so any provider works. memo=True caches the result, so an unchanged message is never sent to the model twice:
```python title=main.py
@coco.fn(memo=True)
async def extract_topics(text: str | None) -> list[str]:
if not text or not text.strip():
return []
response = await acompletion(
model=LLM_MODEL,
messages=[
{"role": "user", "content": f"Extract topics from the following text:\n\n{text[:4000]}"}
],
response_format=TopicsResponse,
)
content = response.choices[0].message.content
return TopicsResponse.model_validate_json(content).topics
```
## Declare a row per message and per topic
The two target tables are plain dataclasses. A thread and each of its comments both become an HnMessage (distinguished by content_type), and every extracted topic becomes an HnTopic keyed on (topic, message_id):
```python title=main.py
@dataclass
class HnMessage:
id: str
thread_id: str
content_type: str # "thread" or "comment"
author: str | None
text: str | None
url: str | None
created_at: datetime | None
@dataclass
class HnTopic:
topic: str
message_id: str
thread_id: str
content_type: str
created_at: datetime | None
```
process_thread fetches one story, extracts topics for the story and every comment, and declares the rows. You declare what should exist; CocoIndex works out the inserts, updates, and deletes:
```python title=main.py
@coco.fn
async def process_thread(thread_id: str, targets: TableTargets) -> None:
async with aiohttp.ClientSession() as session:
thread = await fetch_thread(session, thread_id)
thread_topics = await extract_topics(thread.text)
targets.messages.declare_row(row=HnMessage(
id=thread.id, thread_id=thread.id, content_type="thread",
author=thread.author, text=thread.text, url=thread.url, created_at=thread.created_at,
))
for topic in thread_topics:
targets.topics.declare_row(row=HnTopic(
topic=topic, message_id=thread.id, thread_id=thread.id,
content_type="thread", created_at=thread.created_at,
))
for comment in thread.comments:
comment_topics = await extract_topics(comment.text)
targets.messages.declare_row(row=HnMessage(..., content_type="comment", ...))
for topic in comment_topics:
targets.topics.declare_row(row=HnTopic(..., content_type="comment", ...))
```
app_main mounts the two Postgres tables and fans out one process_thread per recent story with coco.mount_each. The two async fetch functions that make up the custom source are covered in the custom-source post.
## Ranking what's trending
Trending is a SQL query over hn_topics. Each row contributes a weight to its topic's score: a thread-level mention counts for more than a comment-level one, because a topic in the story title is the subject, while a topic in a comment is supporting discussion. Group by topic, order by score:
Trending ranking: the hn_topics table holds one row per topic mention. A SQL score sums a weight per row, 5 for a thread-level mention and 1 for a comment-level mention, grouped by topic and ordered by score, producing a leaderboard where Rust scores 7, PostgreSQL 5, LLM 3, and Claude 1.
```sql
SELECT
topic,
SUM(CASE WHEN content_type = 'thread' THEN 5 ELSE 1 END) AS score,
MAX(created_at) AS latest_mention,
COUNT(DISTINCT thread_id) AS thread_count
FROM coco_examples.hn_topics
GROUP BY topic
ORDER BY score DESC, latest_mention DESC
LIMIT 20
```
Weighting thread mentions 5 to 1 over comment mentions is what makes the ranking read as "what HackerNews is about" rather than "what people mentioned in passing." The same score, filtered to one topic and grouped by thread_id, ranks which threads are driving a given topic.
## Run it
Install the example and build the index:
```sh
pip install -e .
cocoindex update main
```
This fetches the most recent stories, runs topic extraction on every message, and writes coco_examples.hn_messages and coco_examples.hn_topics. This source is a one-shot catch-up per run; re-run cocoindex update main anytime to pick up new stories, and only new messages hit the LLM.
Then explore from the command line, top trending topics or a single topic:
```sh
python main.py # top 20 trending topics + interactive search
python main.py "rust" # search content for one topic
```
## What you can build on top
Once topics are structured and ranked, the pipeline is a foundation:
- **Trending dashboards.** Serve the leaderboard and per-topic threads to a live page that shows what's hot and which discussions are driving it.
- **Sentiment-aware trends.** Add a second extraction step for sentiment to track not just what's trending, but how people feel about it.
- **Contributor tracking.** Every message stores its author, so you can find who drives conversation around a given topic.
- **A knowledge graph.** Feed the topics into a graph linking companies, products, and people mentioned across HackerNews.
Each of these is a new transform or target on the same declared source, not a rewrite. The extraction and the ranking, the two pieces this post focused on, stay exactly as they are.
## Support us ❤️
If this was useful, a star on GitHub helps more developers find CocoIndex and keeps the project growing.
---
# CocoIndex Changelog 0.2.21 - 0.3.10
URL: https://cocoindex.io/blogs/changelog-0310/
Date: 2025-11-25
Summary: Featuring batching support for CocoIndex functions, execution robustness, schema & type system improvements, custom source support, and more.
CocoIndex is moving at insane speed! 🚀
This release is packed with some of our biggest upgrades yet, from automatic batching, to robust async execution, schema and type-system upgrades, and the long-awaited launch of Custom Sources.
CocoIndex is rapidly becoming the best framework for **persistent-state–driven data processing**, where it continuously transforms and updates the target data (e.g. AI context, feature stores, knowledge graphs and beyond) upon source change with zero manual orchestration.
We’re incredibly excited to keep building in the open with our community. Together, we’re redefining how AI systems maintain, evolve, and reason over long-lived state. Onward! 🎉
## Core capability
### Batching support for CocoIndex functions
CocoIndex now supports **automatic batching** for all CocoIndex functions. When embedding the CocoIndex codebase with *sentence-transformers/all-MiniLM-L6-v2*,
batching delivered **~5× higher throughput** (≈80% lower runtime) compared to processing items one-by-one.
#### Why it’s fast
fixed overhead vs data-dependent compute
Each call normally pays:
- **Fixed overhead** (GPU kernel launches, Python↔C transitions, memory allocator work)
- **Data-dependent compute** (FLOPs, bytes moved, tokens processed)
Batching amortizes the fixed cost and lets GPUs run larger, denser GEMMs with fewer data transfers, dramatically increasing utilization.
#### What’s new
CocoIndex introduces **adaptive, knob-free batching**:
framework level batching
- Requests queue while the GPU is busy
- As soon as a batch completes, all queued requests are flushed as the next batch
- No timers, target batch sizes, or tuning: batching automatically adapts to workload traffic
Batching significantly enhances processing speed by amortizing fixed overhead across multiple items, enabling more efficient GPU operations, and reducing data transfer.
CocoIndex simplifies this by offering automatic batching for several built-in functions and an easy batching=True parameter for custom functions.
You can read more about **batching and the benchmark** in the blog post.
*Changes:* #1229, #1230, #1232, #1233, #1236, #1238, #1261
## Custom source support
custom source
We're thrilled to introduce **Custom Sources** in CocoIndex, a feature that lets you pull data from **any system**: APIs, databases, file systems, cloud storage, or other services. CocoIndex now ingests data incrementally, tracks changes efficiently, and integrates seamlessly into your workflows.
With Custom Sources, you're no longer limited by prebuilt connectors or targets. Use CocoIndex for **anything**, leveraging its robust incremental computing to continuously build fresh knowledge for AI.
A **custom source** defines how CocoIndex reads data from an external system.
Custom sources are defined by two components:
- A **source spec** that configures the behavior and connection parameters for the source.
```python
class CustomSource(cocoindex.op.SourceSpec):
"""
Custom source for my external system.
"""
param1: str
param2: int | None = None
```
- A **source connector** that handles the actual data reading operations. It provides the following required methods:
- create(): Create a connector instance from the source spec.
- list(): List all available data items. Return keys.
- get_value(): Get the full content for a specific data item by a given key.
```python
@cocoindex.op.source_connector(
spec_cls=CustomSource,
key_type=DataKeyType,
value_type=DataValueType
)
class CustomSourceConnector:
@staticmethod
async def create(spec: CustomSource) -> "CustomSourceConnector":
"""Initialize connection, authenticate, and return connector instance."""
...
async def list(self, options: SourceReadOptions) -> AsyncIterator[PartialSourceRow[DataKeyType, DataValueType]]:
"""List available data items with optional metadata (ordinal, content)."""
...
async def get_value(self, key: DataKeyType, options: SourceReadOptions) -> PartialSourceRowData[DataValueType]:
"""Retrieve full content for a specific data item."""
...
def provides_ordinal(self) -> bool:
"""Optional: Return True if the source provides timestamps or version numbers."""
return False
```
Check out the article on Bring your own data: Index any data with Custom Sources for more details.
Get started and read the documentation now.
*Changes:* #1195, #1197, #1198, #1199, #1201
## Execution robustness / debugability enhancement
Robust async runtime, better error propagation and observability for HTTP calls, function‑level timeouts and contextual execution in CocoIndex.
### Runtime and async safety
- The runtime now detects misuse of the sync API from async code and guards it explicitly. Sync entrypoints now safely handle calls from different event loops, avoiding deadlocks.
- Async cancellation is now propagated correctly through CocoIndex's async contexts, so task cancellations (for example, from higher‑level orchestration or timeouts) reliably unwind work instead of silently hanging.
- Async custom functions are fully supported and wired into the runtime so user‑defined async operations behave like first‑class citizens, including proper scheduling and error handling.
*Changes:* #1224, #1228, #1255
### HTTP utility improvements
- A new **http::request()** utility centralizes HTTP calls, providing:
- Better, more actionable error messages for network and protocol failures.
- Built‑in retry behavior so transient errors are handled consistently across integrations.
- This utility is intended as the recommended path for HTTP interactions in sources, functions and sinks, improving both debuggability and resilience.
*Changes:* #1235
### Function timeout support
- User‑configured operations (sources, functions, etc.) now support a function timeout to prevent unbounded or long‑running executions.
- The timeout is fully wired through the batching executor so batched executions respect per‑function limits instead of letting a single slow item stall the entire batch.
- Timeouts integrate with the new cancellation propagation so timed‑out work is properly cancelled and cleaned up, rather than left in an indeterminate state.
*Changes:* #1241, #1284
### Clear context in error messages
CocoIndex now attaches clear context information (like source, function and target names) with error messages.
*Changes:* #1275
## Schema & type system
This set of changes improves how CocoIndex handles JSON Schemas, type metadata, and schema merging, with a focus on configurability, correctness, and compatibility across tools and providers.
### Collector schema alignment
- Collectors now automatically merge and align schemas when collect() is called multiple times with different shapes, producing a unified schema instead of requiring manual pre‑alignment.
- The new collector implementation has been simplified in a follow‑up change while preserving the automatic merge‑and‑align behavior for easier maintenance.
*Changes:* #1153, #1265
### Improved schema support
- additionalProperties is now configurable via supports_additional_properties on ToJsonSchemaOptions, so providers that support it (OpenAI, Anthropic, Bedrock, Ollama, etc.) still get strict schemas while Gemini (AI Studio, Vertex) receives schemas without this keyword, removing the previous Gemini‑specific strip workaround.
- Forward‑referenced field types are now resolved before export, improving compatibility with BAML and other tooling that expects fully concrete types, and type descriptions are correctly encoded so documentation survives serialization.
- A previously incorrect assertion condition in the type/schema logic has been fixed to avoid spurious failures in valid configurations.
*Changes:* #1254, #1272, #1294, #1312
## More accurate logic change detection
CocoIndex now detects logic changes with much finer granularity, so incremental runs only reprocess data that is truly affected.
- New per‑source SourceLogicFingerprint and field‑level FieldDefFingerprint track exactly which sources and operations each export/field depends on.
- Execution and indexing now use these fingerprints (including target schema IDs) to avoid unnecessary recomputation while still catching real logic changes.
*Changes:* #1292
## Building blocks
### Builtin sources
#### File size management across sources
CocoIndex’s built-in file-based sources now support consistent file size limits to protect pipelines from unexpectedly large inputs.
- max_file_size is now supported on Azure Blob, Amazon S3, LocalFile, and Google Drive sources.
- Files larger than the configured max_file_size are treated as non-existent in both list() and get_value() calls, letting you cap resource usage and skip oversized blobs uniformly across all these sources.
*Changes:* #1257, #1259, #1260, #1269
#### GoogleDrive source enhancements
GoogleDrive now supports included_patterns and excluded_patterns, using glob-style patterns to decide exactly which Drive files to index. This makes it easy to focus on specific file types or paths (for example, only *.md docs) while excluding noisy locations like temp folders or logs.
*Changes:* #1263
#### S3 configuration enhancements
Amazon S3 sources gain a force_path_style configuration flag, improving compatibility with S3-compatible object stores and strict networking setups that require path-style URLs instead of virtual-hosted–style.
*Changes:* #1290
#### S3 event notifications via Redis queue
CocoIndex can now consume S3 event notifications via a Redis queue, enabling push-based, near–real-time updates from S3 buckets into your indexing flows. This reduces the need for frequent full scans or tight polling loops while still keeping downstream indexes fresh.
*Changes:* #1189
#### Postgres source stability
- The Postgres source’s change‑listening connection is now more stable, reducing disconnects and flaky behavior when capturing updates via **LISTEN/NOTIFY**.
- This makes long‑running live update pipelines against Postgres more resilient, especially under network hiccups or database restarts.
*Changes:* #1264
#### UTF-16/UTF-32 file support
- File decoding now auto‑detects BOMs and supports UTF‑16 and UTF‑32 in addition to UTF‑8, so mixed‑encoding corpora can be indexed without pre‑normalizing everything.
- The internal bytes_to_string() helper was optimized to return a Cow instead of always allocating, avoiding unnecessary copies when data is already valid UTF‑8.
*Changes:* #1185, #1186
### Builtin functions
#### Performance optimization for SentenceTransformerEmbed
**SentenceTransformerEmbed** now sorts inputs by length before batching, which reduces padding, improves GPU utilization, and lowers per-batch compute cost for sentence-transformer models. This change is transparent to users but can significantly improve throughput for large embedding workloads.
*Changes:* #1245
#### Ollama embedding support
The Ollama embedding integration now correctly parses the nested embeddings array returned by the **/api/embed** endpoint. This fixes shape/format mismatches so multi-input embedding calls against Ollama work reliably in CocoIndex.
*Changes:* #1227
## Operations & monitoring
CocoIndex now exposes basic server health checks and clearer runtime progress reporting.
### HTTP server & health checks
- A new **/healthz** endpoint on the CocoIndex HTTP server returns a lightweight JSON payload with status and version, making it easy to plug the server into Kubernetes, load balancers, and external monitoring.
- Documentation has been added for the HTTP server, including how to start it via CLI or Python and how to wire it up with CocoInsight or custom frontends.
*Changes:* #1270, #1271
### Statistics & progress reporting
- Processing commands now report elapsed time, so you can quickly see how long an update or run took.
- Progress reporting has been reworked: there is a consolidated, clearer progress bar, continuous stats updates during batch operations, and better handling of errors so counts and statuses remain accurate throughout the run.
*Changes:* #1204, #1223, #1231, #1240, #1247
## CLI simplification
CocoIndex simplifies the CLI around flow setup by making setup the default and deprecating the extra flag.
- The --setup flag on cocoindex update and cocoindex server is now enabled by default and marked as deprecated, since these commands will automatically perform any required setup before running.
- When --setup is used explicitly, the CLI shows a deprecation warning, and docs have been updated to remove references to manually passing this flag in common workflows.
*Changes:* #1212, #1237
## Claude skills integration
CocoIndex now ships a dedicated Claude Code skill so developers can work with CocoIndex flows directly from their IDE-like Claude environment.
- A CocoIndex Claude skill has been introduced and moved into its own cocoindex-claude repository, giving Claude richer, structured knowledge of CocoIndex concepts, data types, and workflows.
- Documentation was added showing how to install and enable the CocoIndex skill in Claude Code, so Claude can help scaffold flows, edit pipelines, and navigate CocoIndex projects more effectively.
claude skills
## New tutorials
### Index PDF elements
PDFs are rich with both text and visual content, from descriptive paragraphs to illustrations and tables. This example builds an end-to-end flow that parses, embeds, and indexes both, with full traceability to the original page.
Check out the blog for more details.
pdf elements
### Extracting intake forms with BAML and CocoIndex
This tutorial shows how to use BAML together with CocoIndex to build a data pipeline that extracts structured patient information from PDF intake forms. The BAML definitions describe the desired output schema and prompt logic, while CocoIndex orchestrates file input, transformation, and incremental indexing.
Check out the blog for more details.
extraction baml
## Special edition
CocoIndex recently hit **3K GitHub stars** and became the **#1 trending Rust repo globally**!
trending
🎉 We shared an article reflecting on why we built CocoIndex, the journey so far, and some fresh thoughts:
Data for AI (CocoIndex Blog)
data for ai
For a look back at when we reached 1K stars, see:
CocoIndex at 1K (Reflection)
We're just getting started and can't wait to see what comes next on our way to **5K stars**!
## Thanks to the community
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @Haleshot
Thanks @Haleshot for making additionalProperties configurable in JSON schemas in #1312, improving compatibility with Gemini while keeping richer schema support for other providers, and for improving the README note for better GitHub rendering in #1288, making the docs clearer and more readable.
### @Gohlub
Thanks @Gohlub for adding with_context to user‑configured operations in #1275, enabling more powerful context‑aware flows, and for adding included_patterns and excluded_patterns to the GoogleDrive source in #1263, making it easier to control which files are indexed.
### @dcbark01
Thanks @dcbark01 for adding the force_path_style parameter to S3 config in #1290, improving support for S3‑compatible storage.
### @prabhath004
Thanks @prabhath004 for adding max_file_size support to the AzureBlob source in #1259, LocalFile source in #1260, and AmazonS3 source in #1257, making large‑file handling safer and more configurable.
### @xuzijan
Thanks @xuzijan for adding a README with CocoIndex project examples in #1273, giving users concrete starting points for building projects.
### @AdwitaSingh1711
Thanks @AdwitaSingh1711 for adding function timeout support in #1241, preventing long‑running functions from blocking pipelines.
### @skalwaghe-56
Thanks @skalwaghe-56 for teaching the collector to automatically merge and align multiple collect() calls with different schemas in #1153, simplifying complex aggregation flows.
### @ansu86d
Thanks @ansu86d for enhancing UpdateStats with a progress bar and better error handling in #1223, making batch updates more transparent.
### @samojavo
Thanks @samojavo for guarding sync APIs inside async contexts in #1224, improving runtime safety.
### @CAPsMANyo
Thanks @CAPsMANyo for correctly parsing nested embeddings arrays from the Ollama /api/embed endpoint in #1227, ensuring robust embedding ingestion.
### @belloibrahv
Thanks @belloibrahv for adding Redis queue support for S3 event notifications in #1189 and improving CLI --help/Markdown docstring formatting in #1210, making event‑driven updates and CLI usage smoother.
### @GooglyBlox
Thanks @GooglyBlox for deprecating the setup flag in #1212, simplifying the CLI surface and guiding users toward the preferred setup flow.
---
# Extract HackerNews into Postgres with a custom source
URL: https://cocoindex.io/blogs/custom-source-hackernews/
Date: 2025-11-25
Summary: Build a custom CocoIndex source for the HackerNews API: fetch threads and comments in async Python, extract topics with an LLM, keep Postgres in sync.
Custom sources are one of the most useful ideas in CocoIndex. They let you turn any API, internal or external, into a first-class incremental data stream that the framework can diff, track, and sync.
Think of it as React for data flows: you declare what your target should look like as a function of the source, and CocoIndex handles incremental updates, state persistence, and downstream sync. In CocoIndex v1 this needs no special connector API. A custom source is ordinary async Python that fetches data, plus processing components that declare the rows that should exist downstream.
This post walks through a custom source for HackerNews built that way. It fetches recent stories and their nested comments through the Algolia HN API, extracts topics from every message with an LLM, and keeps two Postgres tables in sync for search and analytics.
The project is open-sourced in the CocoIndex v1 repo.
## Why use a custom source?
In many scenarios, pipelines don't just read from clean tables. They depend on:
- Internal REST services
- Partner APIs
- Legacy systems
- Non-standard data models that don't fit traditional connectors
CocoIndex makes these integrations declarative and incremental without a plugin interface. You fetch data with plain async def functions, each item runs as its own processing component, and the component declares the target rows it owns. The engine takes it from there: change detection, syncing to the target, and cleanup when an item disappears from the source.
## Project walkthrough: building a HackerNews index
### Goals
1. Call the HackerNews search API
2. Fetch nested comments for each story
3. Extract topics from every message with an LLM
4. Store messages and topics in Postgres
5. Search the index by topic
CocoIndex handles change detection, idempotency, and state sync automatically.
### Overview
Pipeline overview: fetch_thread_list pulls recent story IDs from the HackerNews API, mount_each fans out one processing component per thread, each component fetches the thread, extracts topics with an LLM, and declares rows that CocoIndex syncs into the hn_messages and hn_topics Postgres tables.
The pipeline consists of three parts:
1. Two async functions over the Algolia HN API
- fetch_thread_list lists recent story IDs
- fetch_thread pulls one story with its full comment tree
2. One processing component per thread
- process_thread extracts topics from the story and every comment
- Declares rows into two Postgres tables (hn_messages, hn_topics)
3. Query utilities
- Rank trending topics in SQL
- Search messages by topic
Each cocoindex update catches up on the latest stories and only does the work it hasn't seen before.
## Prerequisites
- Postgres running locally. The example repo ships a compose file: docker compose -f dev/postgres.yaml up -d.
- An LLM API key. The default model is gemini/gemini-2.5-flash through litellm; set GEMINI_API_KEY, or set LLM_MODEL to any litellm model id and the matching provider key.
## Define the data models
Everything the pipeline handles is a plain dataclass. Thread and Comment model what we scrape from the API:
```python title=main.py
@dataclass
class Comment:
id: str
author: str | None
text: str | None
created_at: datetime | None
@dataclass
class Thread:
id: str
author: str | None
text: str
url: str | None
created_at: datetime | None
comments: list[Comment]
```
Each source item is one thread: a story plus all of its comments, flattened out of HackerNews' nested reply tree into a single list.
A HackerNews thread is a story with a nested comment tree; parsing flattens it into one Thread object holding the story text and a flat list of Comment objects.
HnMessage and HnTopic are the schemas of the two target tables:
```python title=main.py
@dataclass
class HnMessage:
"""Schema for hn_messages table."""
id: str
thread_id: str
content_type: str
author: str | None
text: str | None
url: str | None
created_at: datetime | None
@dataclass
class HnTopic:
"""Schema for hn_topics table."""
topic: str
message_id: str
thread_id: str
content_type: str
created_at: datetime | None
```
A thread and each of its comments both become an HnMessage row, distinguished by content_type. Every extracted topic becomes an HnTopic row keyed on (topic, message_id), so the same topic mentioned in many messages shows up many times, which is exactly what makes ranking meaningful.
Primary keys are the identity of a row, the same role the key type played for a v0 custom source: they must be stable over time so re-runs reconcile rows in place instead of duplicating them.
Dataclass schemas map to Postgres: HnMessage and HnTopic dataclasses become the hn_messages and hn_topics tables, one row per primary key, declared with declare_row and diffed per component.
## Fetch from the HackerNews API
The source itself is two small async functions over the Algolia HN API. No decorators, no connector class.
### Listing recent threads
fetch_thread_list returns the most recent story IDs. CocoIndex uses this list to know which threads exist right now:
```python title=main.py
async def fetch_thread_list(
session: aiohttp.ClientSession, max_results: int = MAX_THREADS
) -> list[str]:
"""Fetch list of recent thread IDs from HackerNews."""
search_url = "https://hn.algolia.com/api/v1/search_by_date"
params: dict[str, str | int] = {"tags": "story", "hitsPerPage": max_results}
async with session.get(search_url, params=params) as response:
response.raise_for_status()
data = await response.json()
thread_ids = [
hit["objectID"] for hit in data.get("hits", []) if hit.get("objectID")
]
return thread_ids
```
### Fetching a full thread
fetch_thread pulls one story with all its comments and parses the raw JSON into the structured Thread. The inner parse_comments walks the reply tree recursively and flattens it; the story title and body are combined into the thread text:
```python title=main.py
async def fetch_thread(session: aiohttp.ClientSession, thread_id: str) -> Thread:
"""Fetch a single thread with all its comments."""
item_url = f"https://hn.algolia.com/api/v1/items/{thread_id}"
async with session.get(item_url) as response:
response.raise_for_status()
data = await response.json()
comments: list[Comment] = []
# Parse comments recursively
def parse_comments(parent: dict[str, Any]) -> None:
for child in parent.get("children", []):
if comment_id := child.get("id"):
ctime = child.get("created_at")
comments.append(
Comment(
id=str(comment_id),
author=child.get("author"),
text=child.get("text"),
created_at=datetime.fromisoformat(ctime) if ctime else None,
)
)
parse_comments(child)
parse_comments(data)
ctime = data.get("created_at")
text = data.get("title", "")
if more_text := data.get("text"):
text += "\n\n" + more_text
return Thread(
id=thread_id,
author=data.get("author"),
text=text,
url=data.get("url"),
created_at=datetime.fromisoformat(ctime) if ctime else None,
comments=comments,
)
```
These are ordinary async def functions. Any HTTP API, message queue, or third-party SDK can be a source the same way: fetch the data in plain Python and hand it to the pipeline.
## Extract topics with an LLM
The core transformation is a single CocoIndex function: text in, a list of topics out. The response schema is a small Pydantic model, so the LLM returns structured data instead of free-form text:
```python title=models.py
class TopicsResponse(BaseModel):
"""Response containing a list of extracted topics."""
topics: list[str] = Field(
description="""List of extracted topics.
Each topic can be a product name, technology, model, people, company name, business domain, etc.
Capitalize for proper nouns and acronyms only.
Use the form that is clear alone.
..."""
)
```
```python title=main.py
@coco.fn
async def extract_topics(text: str | None) -> list[str]:
"""Extract topics from text using LLM."""
if not text or not text.strip():
return []
response = await acompletion(
model=LLM_MODEL,
messages=[
{
"role": "user",
"content": f"Extract topics from the following text:\n\n{text[:4000]}",
}
],
response_format=TopicsResponse,
)
content = response.choices[0].message.content
return TopicsResponse.model_validate_json(content).topics
```
The prompt in TopicsResponse does the heavy lifting: it normalizes phrases into separate topics ("books for autistic kids" → "book", "autistic", "autistic kids"), keeps proper nouns canonical ("PostgreSQL", "Claude"), and emits aliases for the same thing ("JFK", "John Kennedy").
## Process each thread
Each thread runs as its own processing component. process_thread fetches the thread, extracts topics for the story and every comment, and declares the resulting rows:
```python title=main.py
@coco.fn
async def process_thread(
thread_id: str,
targets: TableTargets,
) -> None:
"""Fetch and process a single thread and its comments."""
async with aiohttp.ClientSession() as session:
thread = await fetch_thread(session, thread_id)
thread_topics = await extract_topics(thread.text)
# Declare thread message row
targets.messages.declare_row(
row=HnMessage(
id=thread.id,
thread_id=thread.id,
content_type="thread",
author=thread.author,
text=thread.text,
url=thread.url,
created_at=thread.created_at,
),
)
# Declare thread topic rows
for topic in thread_topics:
targets.topics.declare_row(
row=HnTopic(
topic=topic,
message_id=thread.id,
thread_id=thread.id,
content_type="thread",
created_at=thread.created_at,
),
)
# Process comments
for comment in thread.comments:
comment_topics = await extract_topics(comment.text)
# Declare comment message row
targets.messages.declare_row(
row=HnMessage(
id=comment.id,
thread_id=thread.id,
content_type="comment",
author=comment.author,
text=comment.text,
url="",
created_at=comment.created_at,
),
)
# Declare comment topic rows
for topic in comment_topics:
targets.topics.declare_row(
row=HnTopic(
topic=topic,
message_id=comment.id,
thread_id=thread.id,
content_type="comment",
created_at=comment.created_at,
),
)
```
You declare what rows should exist; you don't write inserts or deletes. When the component finishes, CocoIndex diffs the declared rows against the previous run at the same component path and applies only the creates, updates, and deletes needed. If a thread drops out of the feed, the rows it owned are cleaned up automatically.
Why a component per thread? A processing component groups one thread's work together with the target rows it owns. Each component runs independently and in parallel, and its rows are committed to Postgres as soon as that thread is done, without waiting for the rest of the batch.
## Wire up the app
The Postgres pool is provided once in the app lifespan and referenced by a context key:
```python title=main.py
PG_DB = coco.ContextKey[asyncpg.Pool]("hn_db")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.settings.db_path = pathlib.Path("./cocoindex.db")
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
yield
```
app_main mounts the two table targets, fetches the recent thread IDs, and fans out one process_thread component per thread with coco.mount_each:
```python title=main.py
@dataclass
class TableTargets:
"""Container for table targets."""
messages: postgres.TableTarget[HnMessage]
topics: postgres.TableTarget[HnTopic]
@coco.fn
async def app_main() -> None:
"""Main pipeline function."""
# Set up table targets
messages_table = await postgres.mount_table_target(
PG_DB,
table_name="hn_messages",
table_schema=await postgres.TableSchema.from_class(
HnMessage, primary_key=["id"]
),
pg_schema_name="coco_examples",
)
topics_table = await postgres.mount_table_target(
PG_DB,
table_name="hn_topics",
table_schema=await postgres.TableSchema.from_class(
HnTopic, primary_key=["topic", "message_id"]
),
pg_schema_name="coco_examples",
)
targets = TableTargets(messages=messages_table, topics=topics_table)
# Fetch thread IDs from HackerNews
async with aiohttp.ClientSession() as session:
thread_ids = await fetch_thread_list(session)
# Process threads (each component fetches its own thread data)
await coco.mount_each(process_thread, ((tid, tid) for tid in thread_ids), targets)
app = coco.App(
coco.AppConfig(name="HNTrendingTopics"),
app_main,
)
```
mount_each takes one (component_key, *args) tuple per item, so each thread gets a stable component path keyed on its ID. The TableSchema.from_class calls derive the SQL columns straight from the dataclasses. That's the whole app: plain Python on top, the Rust engine handling incremental sync underneath.
## Query and search the index
The index is two plain Postgres tables, so you can query them with any library or framework. The example ships two query helpers over asyncpg. Trending topics are ranked in SQL, weighting a thread-level mention higher than a comment-level one:
```sql
SELECT
topic,
SUM(CASE WHEN content_type = 'thread' THEN 5 ELSE 1 END) AS score,
MAX(created_at) AS latest_mention,
COUNT(DISTINCT thread_id) AS thread_count
FROM coco_examples.hn_topics
GROUP BY topic
ORDER BY score DESC, latest_mention DESC
LIMIT 20
```
And topic search joins the two tables to return the underlying messages:
```sql
SELECT m.id, m.thread_id, m.author, m.content_type, m.text, m.created_at, t.topic
FROM coco_examples.hn_topics t
JOIN coco_examples.hn_messages m ON t.message_id = m.id
WHERE LOWER(t.topic) LIKE LOWER($1)
ORDER BY m.created_at DESC
```
## Run it
Install the project in editable mode from the example directory:
```sh
pip install -e .
```
Then build the index:
```sh
cocoindex update main
```
This fetches the most recent MAX_THREADS (default 10) stories, runs LLM topic extraction on every message, and writes coco_examples.hn_messages and coco_examples.hn_topics. This source is a one-shot catch-up per run: each cocoindex update syncs the tables to the current feed.
Re-running is where the incremental model pays off:
- New threads in the feed get processed; threads already in the database reuse their committed rows, so the expensive LLM calls only run for content CocoIndex hasn't seen.
- Change the extraction prompt or switch LLM_MODEL, and the next update re-extracts to keep hn_topics consistent with the new logic. No manual migration.
Explore the results from the command line:
```sh
python main.py # top 20 trending topics + interactive search
python main.py "rust" # search content for one topic
```
## What you can build next
This example opens the door to a lot more:
- Run LLM summarization on top of indexed threads
- Add embeddings and vector search
- Mirror HN into your internal data warehouse
- Build a real-time HN dashboard
- Extend to other news sources (Reddit, Lobsters, etc.)
Because the whole pipeline is declarative and incremental, extending it is a matter of declaring more rows.
Since a custom source is just Python code that fetches data, the best use cases are usually "hard-to-reach" data: systems that don't have standard database connectors, have complex nesting, or require heavy pre-processing.
### The knowledge aggregator for LLM context
Building a context engine for an AI bot often requires pulling from non-standard documentation sources: wikis, ticket systems, internal APIs. Each becomes a fetch function and a component, and the index stays current as the sources change.
### The composite entity (data stitching)
Most companies have user data fragmented across multiple microservices. A custom source can act as a virtual join before the data ever hits your index. For example, a component:
1. Fetches a user ID from an auth service (Okta, Auth0)
2. Uses that ID to fetch billing status from the Stripe API
3. Uses that ID to fetch usage logs from an internal Redis
Instead of managing ETL joins downstream, the component declares a single User360 row. CocoIndex tracks the state of that composite object; if the user upgrades in Stripe or changes their email in Auth0, the index updates automatically.
### The legacy wrapper (modernization layer)
Enterprises often have valuable data locked in systems that are painful to query (SOAP, XML, mainframes). Wrap the legacy read path in a fetch function and you get a modern, queryable SQL interface via the CocoIndex target, without rewriting the legacy system itself.
### Public data monitor (competitive intelligence)
Tracking changes on public websites or APIs that don't offer webhooks: competitor pricing pages, regulatory feeds, market data APIs. Because CocoIndex diffs declared rows against the previous run, downstream tables only change when the data actually changes, rather than filling up with identical polling results.
## Why this matters
Custom sources extend the same declarative model to any API: internal, external, legacy, or real-time.
This unlocks a simple pattern:
> If you can fetch it, CocoIndex can index it, diff it, and sync it.
Whether you're indexing HackerNews or stitching together dozens of enterprise services, the framework gives you a stable backbone: persistent state, deterministic incremental updates, automatic cleanup, and flexible target exports with minimal infrastructure overhead.
## Support CocoIndex ❤️
If this example was helpful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub.
Your stars help us grow the community, stay motivated, and keep shipping better tools for real-time data ingestion and transformation.
---
# Extracting Intake Forms with BAML and CocoIndex
URL: https://cocoindex.io/blogs/extraction-baml/
Date: 2025-11-21
Summary: How to use BAML and CocoIndex to extract structured data from patient intake forms in PDF/Word with LLMs continuously for production.
This tutorial shows how to use **BAML** together with **CocoIndex** to build a data pipeline that extracts structured information with an LLM from PDF intake forms. The BAML definitions describe the desired output schema and prompt logic, while CocoIndex orchestrates file input, transformation, and incremental indexing.
We’ll walk through setup, defining the BAML schema, generating the Python client, writing the CocoIndex flow, and running the pipeline. Throughout, we follow best practices (e.g. caching heavy steps) and cite documentation for key concepts.
The full project is open sourced here ⭐. To see more examples built with CocoIndex, you could refer to the examples page.
## BAML
**BAML**, created by BoundaryML, is a typed prompt engineering language that makes LLM workflows predictable, testable, and production-safe. Instead of treating prompts as fragile strings, BAML lets developers define clear input parameters, output schemas, and model configurations, transforming prompts into strongly typed functions.
## CocoIndex
**CocoIndex** is a unified data processing engine built for AI-native applications. It lets you define transformations in one declarative workflow, then keeps everything continuously up to date with real-time, incremental processing. Designed for reliability and scale, CocoIndex ensures that every derived artifact (embeddings, metadata, extractions, models) always reflects the latest source data, making it the foundation for fast, consistent RAG, analytics, and automation pipelines.
## Flow overview
Flow Overview
- Read PDF files from a directory.
- For each file, call the BAML function to get a structured Patient.
- Collect results and export to Postgres.
## Prerequisites
1. Install Postgres if you don't have one.
2. Install dependencies
```
pip install -U cocoindex baml-py
```
1. Create a .env file. You can copy it from .env.example first:
```
cp .env.example .env
```
Then edit the file to fill in your GEMINI_API_KEY.
## Structured extraction component with BAML
Create a baml_src/ directory for your BAML definitions. We’ll define a schema for patient intake data (nested classes) and a function that prompts Gemini to extract those fields from a PDF. Save this as baml_src/patient.baml
### Define patient schema
**Classes**: We defined Pydantic-style classes (Contact, Address, Insurance, etc.) to match the FHIR-inspired patient schema. These become typed output models. Required fields are non-nullable; optional fields use ?.
Schema
```python
class Contact {
name string
phone string
relationship string
}
class Address {
street string
city string
state string
zip_code string
}
class Pharmacy {
name string
phone string
address Address
}
class Insurance {
provider string
policy_number string
group_number string?
policyholder_name string
relationship_to_patient string
}
class Condition {
name string
diagnosed bool
}
class Medication {
name string
dosage string
}
class Allergy {
name string
}
class Surgery {
name string
date string
}
class Patient {
name string
dob string
gender string
address Address
phone string
email string
preferred_contact_method string
emergency_contact Contact
insurance Insurance?
reason_for_visit string
symptoms_duration string
past_conditions Condition[]
current_medications Medication[]
allergies Allergy[]
surgeries Surgery[]
occupation string?
pharmacy Pharmacy?
consent_given bool
consent_date string?
}
```
### Define the BAML function to extract patient info from a PDF
```python
function ExtractPatientInfo(intake_form: pdf) -> Patient {
client Gemini
prompt #"
Extract all patient information from the following intake form document.
Please be thorough and extract all available information accurately.
{{ _.role("user") }}
{{ intake_form }}
Fill in with "N/A" for required fields if the information is not available.
{{ ctx.output_format }}
"#
}
```
We specify client Gemini and a prompt template. The special variable {{ intake_form }} injects the PDF, and {{ ctx.output_format }} tells BAML to expect the structured format defined by the return type. The prompt explicitly asks Gemini to extract all fields, filling “N/A” if missing.
:::tip[BAML PDF Extraction: Crucial Prompt Role Gotcha]
When using BAML to extract structured data (like a Patient record) from PDFs, it is absolutely critical to ensure the PDF content is injected as part of the user message in the prompt.
Specifically, you need to include {{ _.role("user") }} *before* you insert your file data with {{ intake_form }}:
:::
**Why role("user") matters**
- For **OpenAI models (e.g., GPT-4, GPT-4o)**, if the file's content is *not* presented in the user message, the model won't "see" the PDF at all; your extraction will fail or be empty.
- For **Gemini and Anthropic**, it's more forgiving and can sometimes work anyway, which makes this confusing to debug across providers.
We only discovered this after a discussion on the BAML repo and our own investigations.
If you skip the explicit role("user"), you might waste hours debugging inconsistent extractions.
**Takeaway:**
When building extraction flows with BAML, **always set the role to "user" before adding file content to your prompt**. That makes your workflow robust and portable across LLM providers.
Thanks to Deepu and Prashanth from our Discord community for working with us on this issue. You can see a real-world debugging journey in our Discord thread.
## Configure the LLM client to use Google’s Gemini model
```python
client Gemini {
provider google-ai
options {
model gemini-2.5-flash
api_key env.GEMINI_API_KEY
}
}
```
### Configure BAML generator
In baml_src folder add generator.baml:
```python
generator python_client {
output_type python/pydantic
output_dir "../"
version "0.213.0"
}
```
The generator block tells baml-cli to create a Python client with Pydantic models in the parent directory.
When we run baml-cli generate, this will compile the .baml definitions into a baml_client/ Python package in your project root. It contains:
- baml_client/types.py with Pydantic classes (Patient, etc.).
- baml_client/sync_client.py and async_client.py with a callable b object. For example, b.ExtractPatientInfo(pdf) will return a Patient.
## Continuous data transformation flow with incremental processing
Next we will define a data transformation flow with CocoIndex. Once you declare the state and transformation logic, CocoIndex will take care of all the state change for you from source to target.
### CocoIndex flow
#### Declare flow
Declare a CocoIndex flow, connect to the LocalFile source, add a data collector to collect processed data.
```python
@cocoindex.flow_def(name="PatientIntakeExtractionBaml")
def patient_intake_extraction_flow(
flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.LocalFile(
path=os.path.join("data", "patient_forms"), binary=True
)
)
patients_index = data_scope.add_collector()
```
This iterates over each document. We transform doc["content"] (the bytes) by our extract_patient_info function. The result is stored in a new field patient_info. Then we collect a row with the filename and extracted patient info.
Ingesting Data
#### Define a custom function to use BAML extraction to transform a PDF
```python
@cocoindex.op.function(cache=True, behavior_version=1)
async def extract_patient_info(content: bytes) -> Patient:
pdf = baml_py.Pdf.from_base64(base64.b64encode(content).decode("utf-8"))
return await b.ExtractPatientInfo(pdf)
```
- The extract_patient_info function is a custom function decorated with @cocoindex.op.function(cache=True, behavior_version=1). Setting cache=True causes CocoIndex to cache outputs of this function for incremental runs (so unchanged inputs skip rerunning the LLM). We increase behavior_version (start at 1) so that any prompt or logic changes will force a refresh.
- Inside the function, we convert bytes to a BAML Pdf (via base64) and then call await b.ExtractPatientInfo(pdf). This returns a Patient dataclass instance (mapped from the BAML output).
#### Process each document
1. Transform each doc with BAML
2. Collect the structured output
```python
with data_scope["documents"].row() as doc:
doc["patient_info"] = doc["content"].transform(extract_patient_info)
patients_index.collect(
filename=doc["filename"],
patient_info=doc["patient_info"],
)
```
Transforming Data
It is common to have heavily nested data, and CocoIndex is natively designed to handle heavily nested data structures.
Nested Data
### Export to Postgres
```python
patients_index.export(
"patients",
cocoindex.storages.Postgres(),
primary_key_fields=["filename"],
)
```
We export the collected index to Postgres. This will create/maintain a table patients keyed by filename, automatically deleting or updating rows if inputs change. Because CocoIndex tracks data lineage, it will handle updates/deletions of source files incrementally.
## Running the pipeline
**Generate BAML client code** (required step, in case you didn’t do it earlier).
```sh
baml generate
```
This generates the baml_client/ directory with Python code to call your BAML functions.
Update the index:
```sh
cocoindex update main
```
**CocoInsight**
I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with zero pipeline data retention.
```sh
cocoindex server -ci main
```
## Composable by default: use the best components for your use case
While CocoIndex provides a rich set of building blocks for building LLM pipelines, it is fundamentally designed as an open system. Developers can bring in their preferred transformation components tailored to their domain, from document parsers to structured extractors like BAML.
This flexibility enables deep composability with other open ecosystems. The synergy between CocoIndex and BAML highlights this philosophy: BAML brings powerful prompt-driven schema extraction, while CocoIndex orchestrates and maintains the flow at scale. There’s no lock-in: developers and enterprises experimenting at the frontier can adapt, extend, and integrate freely.
## Summary
By combining BAML and CocoIndex, we get a robust, schema-driven workflow: BAML ensures the prompt-to-schema mapping is correct and type-safe, while CocoIndex handles data ingestion, transformation, and incremental storage. This example extracted patient intake information (names, insurance, medications, etc.) from PDFs, but the pattern applies to any structured data extraction task.
- For more details on CocoIndex flows, see its documentation.
- For BAML usage and prompt engineering, refer to the BoundaryML docs.
---
# Adaptive Batching - 5x throughput on your data pipelines
URL: https://cocoindex.io/blogs/batching/
Date: 2025-11-10
Summary: CocoIndex now batches GPU and ML workloads automatically: 5x throughput on text embeddings and AI ops, with zero configuration required.
CocoIndex just launched batching support for CocoIndex functions.
With batching, throughput increased to ~5X the non-batched baseline (≈80% lower runtime) when embedding the CocoIndex codebase using the *sentence-transformers/all-MiniLM-L6-v2* model.
## Why batching makes processing fast
When we call a function or remote API, every call has:
- **Fixed overhead** you pay once per call: GPU kernel launch setup, Python/C transition, scheduling, memory allocator work, framework bookkeeping, etc.
- **Data-size-dependent work** that scales with input: floating-point operations (FLOPs) for the model, bytes copied, tokens processed.
Doing 1,000 items one by one pays the fixed overhead 1,000 times.
Doing them in batches pays that overhead once per batch, and the hardware can run the size-dependent work more efficiently (fewer launches, better cache use, fuller pipelines).
fixed overhead vs data-size-dependent work
**Batching helps because it:**
- **Spreads one-time overhead across many items.** You do fewer GPU kernel launches and fewer Python-to-C boundary crossings, etc.
- **Lets the GPU run bigger, more efficient matrix math.** Larger batches map to dense matrix multiplies (*General Matrix–Matrix Multiplication*, a.k.a. GEMM), which use the hardware more effectively (higher utilization).
- **Cuts down on data copies.** You reduce transfers between CPU memory and GPU memory, H2D (*Host-to-Device*) and D2H (*Device-to-Host*), so more time is spent computing, not moving bytes.
## What batching looks like for normal Python code
### Non-batching code – simple but less efficient
The most natural way to organize a pipeline is to process data piece-by-piece.
For example, a two-layer loop like this:
```python
for file in os.listdir(directory):
content = file.read()
chunks = split_into_chunks(content)
for chunk in chunks:
vector = model.encode([chunk.text]) # one item at a time
index.upsert(file_id=file.name, chunk_offset=chunk.offset, vector=vector)
```
This is easy to read and reason about: each chunk flows straight through multiple steps.
### Batching manually – more efficient but complicated
gather data across loops to batch
You can speed it up by batching, but even the simplest “just batch everything once” version makes the code significantly more complicated:
```python
# 1) Collect payloads and remember where each came from
batch_texts = []
metadata = [] # (file_id, chunk_id)
for file in os.listdir(directory):
content = file.read()
chunks = split_into_chunks(content)
for chunk in chunks:
batch_texts.append(chunk.text)
metadata.append((file.name, chunk.offset))
# 2) One batched call (library will still mini-batch internally)
vectors = model.encode(batch_texts)
# 3) Zip results back to their sources
for (file_name, chunk_offset), vector in zip(metadata, vectors):
index.upsert(file_id=file.name, chunk_offset=chunk.offset, vector=vector)
```
Moreover, batching everything at once is usually not ideal, e.g., the next steps can only start after this step is done for all data.
## CocoIndex's batching support
CocoIndex bridges the gap and allows you to get the best out of the two – keep the simplicity of your code by following the natural flow, while getting the efficiency from batching provided by CocoIndex runtime.
We already enabled batching support for the following builtin functions:
- *EmbedText*
- *SentenceTransformerEmbed*
- *ColPaliEmbedImage*
- *ColPaliEmbedQuery*
It doesn’t change the API. **Your existing code will just work without any change – still following the natural flow, while enjoying the efficiency of batching.**
For custom functions, enabling batching is as simple as:
- Set batching=True in the custom function decorator.
- Change the arguments and return type to list.
For example, suppose you want to create a custom function that calls an API to build thumbnails for images.
```python
@cocoindex.op.function(batching=True)
def make_image_thumbnail(self, args: list[bytes]) -> list[bytes]:
...
```
See the batching documentation for more details.
## How CocoIndex batches
### Common approaches
To batch requests, you first accumulate them in a queue, then decide when to flush that queue as a batch. Two widely used policies are:
- **Time-based (W ms)**: Flush whatever arrived in the last *W* milliseconds.
- *Pros:* predictable wait bound; simple.
- *Cons:* adds idle latency when traffic is sparse; needs tuning per workload.
- **Size-based (K items)**: Flush when at least *K* items are waiting.
- *Pros:* predictable batch size; easy to reason about memory.
- *Cons:* under sparse traffic, the head request can wait too long; still needs tuning.
Many systems combine them (“flush when **W** or **K** triggers first”), which helps, but you’re still tuning knobs and making trade-offs that shift with traffic.
### CocoIndex’s approach
#### Framework level: adaptive, knob-free
CocoIndex keeps batching simple:
- While a batch is running on the device, new requests keep queuing.
- When that batch finishes, CocoIndex ships all currently queued requests as the next batch window and immediately starts processing again.
- No timers. No target batch size. The batch naturally reflects whatever arrived during the previous service time.
batching at framework level
Why is this good?
- **Low latency when sparse**: With few requests, batches are tiny (often size 1), so you’re effectively running at near single-call latency.
- **High throughput when busy**: When traffic spikes, more requests accumulate during the in-flight batch, so the next batch is larger. Utilization rises automatically.
- **No tuning**: You don’t need to tune *W* or *K*. The system adapts to your traffic pattern by design.
#### Function level: pack the batch intelligently
Each function receives the batch window (all queued requests at that moment) and decides how to process it efficiently and safely for its model/library.
We use the *SentenceTransformerEmbed* function as an example.
The underlying *sentence-transformer* library accepts batches with arbitrary length, but splits them into micro-batches (default size: 32) so that each can fit into the device memory and keep kernels in the model's sweet spot.
We use the default micro-batch size.
Besides, transformer runtimes pad every sequence in a batch to the length of the longest one so the GPU can run uniform, fast kernels. Short texts pay the cost of the longest text in the batch (e.g., mixing 64-token with 256-token items makes the 64-token ones ~4X more expensive).
We sort by token count and form micro-batches of similar lengths to keep padding minimal and throughput high.
Other functions can simply ship the entire batch to the backend, or apply their own packing (e.g., SIMD tiles, merge-writes).
The framework is agnostic.
It just delivers the batch window promptly.
## Performance evaluation
We ran all benchmarks on a MacBook Pro (Apple M1 Pro, 16 GB unified memory, 2022). For each configuration:
- We run on two different versions, batching **on** (cocoindex v0.3.1) and batching **off** (cocoindex v0.2.23).
- We report two timings: (1) end-to-end benchmark wall-clock time, and (2) the cumulative time inside the embedding function (where batching happens).
- We executed each test 5 times, discarded the fastest and slowest run to dampen outliers, and reported the mean of the remaining three runs (a 20% trimmed mean).
### *text_embedding* and *code_embedding*
We run on the 2 basic examples from cocoindex:
- *text_embedding*: with 3 input files, 106 chunks in total.
- *code_embedding*: with 273 input files, 3383 chunks in total.
Both are using the *SentenceTransformerEmbed* function, using the *all-MiniLM-L6-v2* embedding model (with 22.7M parameters).
This is the evaluation outcome:
example name
end-to-end execution time
runtime in function to batch
off (s) on (s) saving off (s) on (s) saving
text_embedding 1.959 0.630 67.84% 1.853 0.567 69.42%
code_embedding 58.931 12.516 78.76% 58.343 12.117 79.23%
*code_embedding* has significantly more chunks, so there are more opportunities for batching, thus the runtime saving is more significant.
### *code_embedding* on different micro-batch sizes
We experimented with different micro batch sizes on sentence-transformer:
micro batch size
end-to-end execution time
runtime in function to batch
runtime (s) saving runtime (s) saving
batch off 58.931 N/A 58.343 N/A
4 23.254 60.54% 23.083 60.44%
8 16.522 71.96% 16.210 72.22%
16 12.812 78.26% 12.640 78.34%
32 (default) 12.516 78.76% 12.117 79.23%
64 11.925 79.76% 11.577 80.16%
128 11.939 79.74% 11.597 80.12%
There are significant improvements when we increase the batch size to 4, 8, and 16.
As the batch size increases, the improvements become smaller and smaller, since the amortized fixed overhead is already low.
Although we're able to achieve slightly better performance with 128, we keep using the default micro batch size offered by the sentence-transformer library (currently 32), as we trust it will provide and maintain a reasonable default.
### *code_embedding* on *nomic-embed-text-v1.5* model
In this experiment, we switch *all-MiniLM-L6-v2* to *nomic-embed-text-v1.5*. *nomic-embed-text-v1.5* is a larger model, with 0.1B parameters. We tested it:
end-to-end execution time
runtime in function to batch
off (s) on (s) saving off (s) on (s) saving
137.170 131.442 4.18% 136.607 130.377 4.56%
The improvement is much smaller compared to the *all-MiniLM-L6-v2* model.
This is because once the model is large, the fixed overhead is usually much less than the data-size-dependent work.
### *code_embedding* on Ollama with *all-minilm* model
In this experiment, we switched from *SentenceTransformerEmbed* to *EmbedText* function, which offers the functionality to use a remote API to embed text.
We picked the *all-minilm* model, which is the same as the default model we used with *SentenceTransformerEmbed* above.
For ease of comparison, we copy the data from running *SentenceTransformerEmbed* on the same model from above:
function for embedding
end-to-end execution time
off (s) on (s) saving
SentenceTransformerEmbed 58.931 12.516 78.76%
EmbedText with Ollama 44.248 38.343 13.35%
Here we can see that with no batching when calling Ollama, the execution time is better than using *SentenceTransformerEmbed* without batching.
But when there's batching, the savings are quite small, so it's much slower than using *SentenceTransformerEmbed*.
After digging deeper, we noticed that Ollama computes embeddings for different inputs separately (Ollama code), even if the inputs come from the same request.
So batching doesn't save much for Ollama – it still saves a little bit, likely because of the reduced HTTP API calls.
## Conclusion
In conclusion, batching significantly enhances processing speed by amortizing fixed overhead across multiple items, enabling more efficient GPU operations, and reducing data transfer.
CocoIndex simplifies this by offering automatic batching for several built-in functions and an easy batching=True decorator for custom functions.
The greatest impact of batching is seen when fixed overhead constitutes a larger portion of the total work, such as with smaller models.
It's also most effective when the underlying API or library fully supports batched operations, as demonstrated by the limited gains observed with Ollama.
## Support us
⭐ Star CocoIndex on GitHub and share with your community if you find it useful!
---
# AI-Native Data Pipeline - Why We Made It
URL: https://cocoindex.io/blogs/data-for-ai/
Date: 2025-10-29
Summary: Why the next wave of AI needs open-source, scalable, AI-native data infrastructure, and how CocoIndex is building the foundation for intelligent data pipelines.
There’s more need for **open** data infrastructure for AI than ever.
## Data for humans → to data for AI
Traditionally, people build data frameworks heavily in this space to prepare data for humans.
Over the years, we’ve seen massive progress in analytics-focused data infrastructure.
Platforms like Spark and Flink fundamentally changed how the world processes and transforms data, at scale.
But with the rise of AI, entirely new needs, and new capabilities, have emerged.
A new generation of data transformations is now required to support AI-native workloads.
### So, what has changed?
- AI can process new types of data, e.g., by running models to process unstructured data.
- AI can handle high data throughput, and AI agents need rich, high-quality, fresh data to make effective decisions.
- Data preparation for AI must happen dynamically, with flexible schemas and constantly evolving sources.
- Developers need a faster iteration cycle than ever, quickly iterating data transformation heuristics, e.g., dynamically adding new computations and new columns without waiting hours or days.
### Why quality and freshness are non-negotiable for AI
For analytics, slightly stale or slightly noisy data degrades a dashboard at the margins. For AI, the data *is* the context the model reasons over, so the failure mode is sharper. A retrieval pipeline built on poorly chunked or out-of-date content doesn't return a marginally worse answer - it grounds the model in the wrong facts, and no amount of prompt engineering recovers from that. An agent acting on stale state takes the wrong action. Quality and freshness therefore stop being nice-to-haves and become correctness requirements: when a source document is edited, the embeddings and structured fields derived from it have to follow within seconds, not on the next nightly rebuild, or the system is confidently wrong in between.
### New capabilities require new infrastructure
The new patterns require capabilities to
- Process data beyond tabular formats - videos, images, PDFs, complex nested data structures - where fan-outs become the norm.
- Process data within data pipelines with AI workloads. Running on your own GPUs, massive calls to remote LLMs expose new challenges for data infrastructure, such as scheduling heterogeneous computing workloads across both CPU and GPU, and also respecting remote rate limits and backpressure.
- Be able to understand what AI is doing with the data - explainable AI.
- Work well with various changes – data changes, schema changes, logic changes.
On top of all this, we need to think about:
- How to do it at scale, with all the things and best practices we’ve learned from building scalable data pipelines in the past.
- How to democratize it, and make it accessible to anyone beyond data engineers.
### Why isn't patching existing data pipelines enough?
Challenges with existing data pipelines
It’s not something that can be fixed with small patches to existing data pipelines. Many traditional frameworks fall short in several key ways:
- Limited to tabular data models, which is a stretch for data with hierarchical/nested structures.
- Strict requirement for determinism. Many smart operations are not perfectly deterministic, e.g., running LLMs or operations with upgradable dependencies (e.g., an OCR service).
- Require more than a single general-purpose language (like Python), e.g., many use SQL. Users not only face a steeper learning curve, but also need to switch between two languages: e.g., for advanced transformations not supported by SQL, they still need to fall back to Python for UDFs. Ideally, a single language should handle all transformations. Many frameworks force users to switch between two languages, often with overlapping functionality, adding unnecessary complexity.
- Focus on CPU-intensive workloads, and don’t achieve the best throughput when there are GPU and remote API calls.
- Don’t deal with changes. Any schema change, data change, or logic change usually requires rebuilding the output from scratch.
With so many limitations, developers start to handle AI-native data “natively”, by hand-writing Python and wrapping it in orchestrations.
Begin from a demo, then start to worry about scale, tolerating backend failures, picking up from where it left off when pipelines break, rate limiting and backpressure,
building tons of manual integrations when data freshness is needed, making sure stale input data is purged from output, and all of the issues are hard to handle when it runs at scale, and things start to break.
There are so many things to “fix”, and patching existing systems doesn’t work anymore. A new way of thinking about it, from the ground up, is needed.
## So what are some of the design choices for CocoIndex?
### Declarative data pipeline
Users declare "what", we take care of "how". Once the data flow is declared, you have a production-ready pipeline - infrastructure setup (like creating target tables with the right schema, and schema upgrades), data processing and refresh, fault tolerance, rate limiting and backpressure, batching requests to backends for efficiency, etc.
### Persistent computing model
Most traditional data pipelines treat data processing as transient things – they terminate once all data have been processed.
If anything changes (data or code), you process it again from scratch.
We treat pipelines as live things with memory of existing states and only perform necessary reprocessing on data or code change, making the output data continuously reflect the latest input data and code.
This programming model is essential for AI-native data pipelines. It unlocks out-of-the-box incremental processing - the output data to be continuously updated as the input data and code change with minimal computation, and provides the ability to trace the lineage of the data for explainable AI.
### Clear data ownership with fine-granular lineage
Output of pipelines is data derived from source data via certain transformation logic, so each row of data created by the pipeline can be traced back to the specific rows or files from the data source, plus certain pieces of logic. This is essential for refreshing the output on data or logic updates, and also makes the output data explainable.
The key idea is that everything in the target is *derived* data, never authored data. Nothing in the index is hand-maintained; every chunk, embedding, and extracted field exists only as the output of applying declared logic to a specific source item. This framing is what makes correct incremental updates possible at all: because CocoIndex knows precisely which source rows and which pieces of logic produced each output row, a change to one document can be propagated by recomputing exactly the derived rows it owns - and a deleted source row's derived data can be safely purged rather than left to rot as stale context. It's also what makes the pipeline auditable: you can ask of any indexed value where it came from and what produced it. The mechanics of this state-driven, derive-everything model are covered in core concepts and the overview.
### Strong type safety
The schema of data created by each processing step is determined at pipeline declaration time with validation – before running on specific items of data. This catches issues earlier and enables automatic inference of the output data schema for automatic target infrastructure setup.
### Open ecosystem
An open system that allows developers to dock their choice of ecosystem as building blocks.
AI agents should be tailored to specific domains and there will be different technology choices from source to storage to domain-specific processing.
It has to be an open data stack that is easily customizable and brings in the user's own building blocks.
With the rapid growth of the ecosystem - new sources, targets, data formats, transformation building blocks, etc.,
the system shouldn’t be bound to any specific one. Instead of waiting for specific connectors to be built, anyone should be able to use it to create their own data pipelines, assembling
flexible building blocks that can work directly with internal APIs or external systems.
## It needs to stay open
We believe this AI-native data stack must be open.
The space is moving too fast; closed systems can’t keep up. Open infrastructure enables:
- Transparent design and rapid iteration
- Building a community, collaboratively solving it for the future
- Composability with other open ecosystems (agents, analytics, compute, real-time, orchestration frameworks)
- No lock-in for developers or companies experimenting at the frontier
It should be something everyone can contribute to, learn from, and build upon.
## Build with the ecosystem
CocoIndex fits seamlessly into the broader data ecosystem, working well with orchestration frameworks, agentic systems, and analytical pipelines.
As an open, AI-native data infrastructure, it aims to drive the rise of the next generation of applied AI.
## The road ahead
We are just getting started.
AI is notoriously bad at writing data infrastructure code.
The abstractions, data feedback loop, and programming model from CocoIndex are deliberately designed from the ground up, tailored for an AI co-pilot.
This unlocks the full potential of CocoIndex on the path of a self-driving data pipeline, with data auditable and controllable along the way.
:rocket: To the future of building!
## Support us and join the journey
Thank you, everyone, for your support and contributions to CocoIndex. Thank you so much for your suggestions, feedback, stars and for sharing the love for CocoIndex.
We are especially grateful for our beloved community and users.
Your passion, continuous feedback, and collaboration as we got started have been invaluable, helping us iterate and improve the project every step of the way.
Looking forward to building the future of data for AI together!
:star: Star CocoIndex on GitHub here to help us grow! The project has since passed 10,000 stars - thank you to everyone who has joined us along the way.
3k stars
---
# Index PDF elements with mixed embedding models
URL: https://cocoindex.io/blogs/pdf-elements/
Date: 2025-10-27
Summary: Extract, embed, and store multimodal PDF elements (text with SentenceTransformers, images with CLIP) for unified semantic search with traceable metadata.
PDFs are rich with both text and visual content, from descriptive paragraphs to illustrations and tables. This example builds an end-to-end flow that parses, embeds, and indexes both, with full traceability to the original page. For a simpler variant that converts whole PDFs to Markdown, see the PDF to Markdown example.
In this example, we split out both text and images, link them back to page metadata, and enable unified semantic search. We’ll use **CocoIndex** to define the flow, **SentenceTransformers** for text embeddings, and **CLIP** for image embeddings, all stored in **Qdrant** for retrieval.
## 🔍 What it does
This flow automatically:
- Extracts page texts and images from PDF files
- Skips tiny or redundant images
- Creates consistent thumbnails (up to 512×512)
- Chunks and embeds text with SentenceTransformers (all-MiniLM-L6-v2)
- Embeds images with CLIP (openai/clip-vit-large-patch14)
- Stores both embeddings and metadata
(e.g., where the text or image came from) in Qdrant, enabling unified semantic search across text and image modalities
## 📦 Prerequisite:
### Run Qdrant
If you don’t have Qdrant running locally, start it via Docker:
```sh
docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant
```
### 📁 Input Data
We’ll use a few sample PDFs (board game manuals). Download them into the source_files directory:
```sh
./fetch_manual_urls.sh
```
Or, feel free to drop in any of your own PDFs.
## ⚙️ Run the flow
Install dependencies:
```sh
pip install -e .
```
Then build your index (sets up tables automatically on first run):
```sh
cocoindex update --setup main
```
Or run in CocoInsight
```sh
cocoindex server -ci main
```
## Define the flow
### Flow definition
Flow Definition
Let’s break down what happens inside the **PdfElementsEmbedding** flow.
```python
@cocoindex.flow_def(name="PdfElementsEmbedding")
def multi_format_indexing_flow(
flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.LocalFile(
path="source_files", included_patterns=["*.pdf"], binary=True
)
)
text_output = data_scope.add_collector()
image_output = data_scope.add_collector()
```
We define the flow, add a source, and add data collectors.
For flow definition, the decorator:
```python
@cocoindex.flow_def(name="PdfElementsEmbedding")
```
marks the function as a CocoIndex flow definition, registering it as part of the data indexing system.
When executed via CocoIndex runtime, it orchestrates data ingestion, transformation, and collection.
When started, CocoIndex’s runtime executes the flow in either **one-time update** or **live update** mode, and incrementally embeds only what changed.
### Process each document
#### Extract PDF documents
We iterate through each document row and run a custom transformation that extracts PDF elements.
```python
with data_scope["documents"].row() as doc:
doc["pages"] = doc["content"].transform(extract_pdf_elements)
```
#### Extract PDF elements
Define a dataclass for structured extraction: we want to extract a list of PdfPage, each of which has a page number, text, and a list of images.
```python
@dataclass
class PdfImage:
name: str
data: bytes
@dataclass
class PdfPage:
page_number: int
text: str
images: list[PdfImage]
```
Next, define a **CocoIndex function** called **extract_pdf_elements** that **extracts both text and images from a PDF file**, returning them as structured, page-wise data objects.
```python
@cocoindex.op.function()
def extract_pdf_elements(content: bytes) -> list[PdfPage]:
"""
Extract texts and images from a PDF file.
"""
reader = PdfReader(io.BytesIO(content))
result = []
for i, page in enumerate(reader.pages):
text = page.extract_text()
images = []
for image in page.images:
img = image.image
if img is None:
continue
# Skip very small images.
if img.width < 16 or img.height < 16:
continue
thumbnail = io.BytesIO()
img.thumbnail(IMG_THUMBNAIL_SIZE)
img.save(thumbnail, img.format or "PNG")
images.append(PdfImage(name=image.name, data=thumbnail.getvalue()))
result.append(PdfPage(page_number=i + 1, text=text, images=images))
return result
```
The extract_pdf_elements function reads a PDF file from bytes and extracts both text and images from each page in a structured way.
Using pypdf, it parses every page to retrieve text content and any embedded images, skipping empty or very small images to avoid noise.
Each image is resized to a consistent thumbnail size (up to 512×512) and converted into bytes for downstream use.
The result is a clean, per-page data structure (PdfPage) that contains the page number, extracted text, and processed images, making it easy to embed and index PDFs for multimodal search.
pdf-elements
#### Process each page
Once we have the pages, we process each page.
1. Chunk the text
This takes each PDF page's text and splits it into smaller, overlapping chunks.
```python
with doc["pages"].row() as page:
page["chunks"] = page["text"].transform(
cocoindex.functions.SplitRecursively(
custom_languages=[
cocoindex.functions.CustomLanguageSpec(
language_name="text",
separators_regex=[
r"\n(\s*\n)+",
r"[\.!\?]\s+",
r"\n",
r"\s+",
],
)
]
),
language="text",
chunk_size=600,
chunk_overlap=100,
)
```
Split Recursively
2. Process each chunk
Embed and collect the metadata we need. Each chunk includes its embedding, original text, and references to the filename and page where it originated.
```python
with page["chunks"].row() as chunk:
chunk["embedding"] = chunk["text"].call(embed_text)
text_output.collect(
id=cocoindex.GeneratedField.UUID,
filename=doc["filename"],
page=page["page_number"],
text=chunk["text"],
embedding=chunk["embedding"],
)
```
embed-text
3. Process each image
We use CLIP to embed the image and collect the data, embedding, and metadata filename, page_number.
```python
with page["images"].row() as image:
image["embedding"] = image["data"].transform(clip_embed_image)
image_output.collect(
id=cocoindex.GeneratedField.UUID,
filename=doc["filename"],
page=page["page_number"],
image_data=image["data"],
embedding=image["embedding"],
)
```
embed-image
When we collect image outputs, we also want to preserve relevant metadata alongside the embeddings.
For each image, we not only collect the image embedding and binary image data, but also metadata such as the original file name and page number.
This association makes it possible to trace embedded images back to their source document and page during retrieval or exploration.
### Export to Qdrant
Finally, we export the collected data to the target store.
```python
text_output.export(
"text_embeddings",
cocoindex.targets.Qdrant(
connection=qdrant_connection,
collection_name=QDRANT_COLLECTION_TEXT,
),
primary_key_fields=["id"],
)
image_output.export(
"image_embeddings",
cocoindex.targets.Qdrant(
connection=qdrant_connection,
collection_name=QDRANT_COLLECTION_IMAGE,
),
primary_key_fields=["id"],
)
```
## 🧭 Explore with CocoInsight (free beta)
Use **CocoInsight** to visually trace your data lineage and debug the flow.
It connects locally with **zero data retention**.
Start your local server:
```sh
cocoindex server -ci main
```
Then open the UI 👉 https://cocoindex.io/cocoinsight
## 💡 Why this matters
Traditional document search only scratches the surface: it’s text-only and often brittle across document layouts.
This flow gives you **multimodal recall**, meaning you can:
- Search PDFs by text *or* image similarity
- Retrieve related figures, diagrams, or captions
- Build next-generation retrieval systems across rich content formats
## Compare with ColPali vision model (OCR free)
We also have an example for ColPali.
To compare the two approaches:
| **Aspect** | **ColPali Multi-Vector Image Grids** | **Separate Text and Image Embeddings** |
| --- | --- | --- |
| Input | Whole page as an image grid (multi-vector patches) | Text extracted via OCR + images processed separately |
| Embedding | Multi-vector patch-level embeddings preserving spatial context | Independent text and image vectors |
| Query Matching | Late interaction between text token embeddings and image patches | Separate embedding similarity / fusion |
| Document Structure Handling | Maintains layout and visual cues implicitly | Layout structure inferred by heuristics |
| OCR Dependence | Minimal to none; model reads text visually | Heavy dependence on OCR (for scanned PDFs) and text extraction |
| Use Case Strength | Document-heavy, visual-rich formats | General image and text data, simpler layouts |
| Complexity | Higher computational cost, more complex storage | Simpler architecture; fewer compute resources needed |
In essence, ColPali excels in deep, integrated vision-text understanding, ideal for visually complex documents, while splitting text and images for separate embeddings is more modular but prone to losing spatial and semantic cohesion. The choice depends on your document complexity, precision needs, and resource constraints.
## Support us
⭐ Star CocoIndex on GitHub and share with your community if you find it useful!
---
# Bring your own data: Index any data with Custom Sources
URL: https://cocoindex.io/blogs/custom-source/
Date: 2025-10-21
Summary: Read data from any system with CocoIndex: in v1 a custom source is plain async Python plus processing components. No connector API to implement.
CocoIndex can read data from any system: internal APIs, databases, file systems, cloud storage, legacy services. You are not limited to the built-in connector library. Whatever you can fetch from Python, CocoIndex can ingest incrementally, track for changes, and keep in sync with your targets.
:::note[Updated for CocoIndex v1]
The original version of this post introduced a dedicated source connector API (SourceSpec, source_connector, list() / get_value()). CocoIndex v1 removed the need for that interface: a custom source is now ordinary async Python plus processing components. This post describes the v1 approach, with a mapping from the v0 API for readers of the original.
:::
Custom sources are the complement to custom targets, giving you control over both ends of a pipeline: where data comes from and where it lands.
## Why not build another thousand connectors?
We could, and expanding the connector library is on the roadmap. But more connectors alone don't solve the problem. Enterprise data is siloed behind complex systems, inconsistent schemas, and fragile integrations. What pipelines actually need is infrastructure that handles ever-changing datasets, reconciles differences across systems, and keeps data flows durable and observable. That is where CocoIndex focuses: the incremental processing engine underneath, with sources as a thin layer you can supply yourself.
## Assemble pipelines from building blocks
CocoIndex apps are modular: sources, transformations, and targets compose, and each block swaps in a single line of code. Custom sources are the input-side block you supply yourself, alongside:
- Custom targets: write to any system, with the same diff-and-sync treatment as built-in targets.
- Custom transformations: domain-specific logic as ordinary decorated functions.
- Built-in connectors and operations for the common cases.
Mix and match to build end-to-end pipelines that stay incremental and traceable, whatever they read from or write to.
CocoIndex building blocks: a sources column (local files, cloud storage, postgres, your custom source), a transform group (parse, chunk, extract structure, embed, dedup, custom logic), and a targets column (relational, vector, and graph databases, your custom target), all running on the incremental engine.
## What a custom source looks like in v1
A custom source is two ordinary async functions and one processing component: no decorators on the fetch code, connector class, or plugin registration.
1. A listing function that returns what exists right now, as stable keys.
2. A fetch function that returns the full content for one key.
3. A processing component, one per item, that transforms the content and declares the rows that should exist downstream.
Here is the shape, using an internal ticketing API as the example:
```python
import cocoindex as coco
from cocoindex.connectors import postgres
@dataclass
class TicketRow:
id: str
title: str
body: str
updated_at: datetime
async def list_ticket_ids() -> list[str]:
"""Ask your system what exists right now."""
...
async def fetch_ticket(ticket_id: str) -> TicketRow:
"""Fetch the full content for one item."""
...
@coco.fn
async def process_ticket(
ticket_id: str, table: postgres.TableTarget[TicketRow]
) -> None:
ticket = await fetch_ticket(ticket_id)
table.declare_row(row=ticket)
@coco.fn
async def app_main() -> None:
table = await postgres.mount_table_target(
PG_DB,
table_name="tickets",
table_schema=await postgres.TableSchema.from_class(
TicketRow, primary_key=["id"]
),
)
ticket_ids = await list_ticket_ids()
await coco.mount_each(process_ticket, ((t, t) for t in ticket_ids), table)
```
A custom source in CocoIndex v1: your system is read by plain Python fetch functions, mount_each fans out one processing component per item, each component declares target rows, and the CocoIndex engine diffs, syncs, cleans up, and memoizes underneath.
Three details carry the weight here:
- coco.mount_each fans out one processing component per item, keyed by the ID you pass. That key gives each item a stable component path across runs, which is how the engine matches an item to its previous run.
- declare_row states what should exist in the target. You never write inserts, updates, or deletes. The engine compares the declared rows against the previous run and applies only the difference.
- The fetch functions are plain async def. Any HTTP API, message queue, SDK, or database driver works the same way. Authentication and connection pooling live where they normally would in Python: create the client in the app lifespan and share it through a context key.
The downstream half of the pipeline (chunking, embedding, LLM extraction, graph building) is written exactly the same way as for a built-in connector. A transformation doesn't know or care whether its input came from S3, Postgres, or your ticketing API.
## How it stays incremental
The engine never assumes it must reprocess everything. On each run:
- Items whose key is new get processed and their rows created.
- Items already seen are matched by component path. Unchanged declared rows produce no writes; changed rows produce exactly the update needed.
- Items that disappear from your listing have their target states cleaned up automatically. If a ticket is deleted upstream, its rows go away downstream without any delete logic on your side.
Every @coco.fn participates in change detection: the engine fingerprints inputs and logic, so it knows when a function's result may have changed, including when you edit the code. For expensive steps such as embedding or LLM extraction, add memo=True and the memoized result is reused whenever inputs and logic are unchanged. Re-running a pipeline over a mostly-unchanged source only pays for what actually changed.
In v0 this efficiency depended on your connector exposing an ordinal (a timestamp or version number) so the engine could skip unchanged items. In v1 the engine fingerprints and diffs on its own, so any source gets incremental behavior without cooperating. To keep a source fresh over time, run cocoindex update on a schedule; sources with native change watching plug into live mode.
## From the v0 connector API
If you read the original version of this post, or wrote a v0 custom source, here is where each piece went:
| v0 connector API | v1 equivalent |
| --- | --- |
| SourceSpec | Plain function arguments and app config |
| create(spec) | Your own client setup, typically in the app lifespan |
| list() | A listing function plus the keys passed to mount_each |
| get_value(key) | A fetch call inside the per-item component |
| Key and value types | The component key, plus dataclasses for rows |
| provides_ordinal() | Not needed: change detection and memoization |
The v0 interface asked you to fit your system into a connector contract so the engine could read through it. The v1 model inverts that: you read your system with whatever code is natural, and the engine tracks the results. Less to implement, and nothing to learn beyond the concepts the rest of the framework already uses.
## When to build a custom source
Reach for a custom source when there is genuinely no built-in connector for where your data lives: an internal or proprietary API, a legacy database with an unusual access path, or a SaaS product CocoIndex doesn't ship a connector for yet. If a built-in connector already covers your system, use it. The value of the custom-source model is the long tail of your systems, brought in without forking or patching the engine.
Common shapes we see:
- Knowledge aggregation for LLM context: wikis, ticket systems, and internal APIs each become a fetch function and a component, and the index stays current as they change.
- Data stitching: one component fetches from several services (auth, billing, usage) and declares a single composite row, so the join happens before the data hits your index.
- Legacy wrappers: wrap a painful read path (SOAP, XML, mainframe exports) in a fetch function and get a modern, queryable table on the other side.
## See it in action
The HackerNews example is a complete custom source built this way: it fetches recent stories and their nested comments from the Algolia HN API, extracts topics from every message with an LLM, and keeps two Postgres tables in sync for search and analytics.
The custom-source walkthrough builds it step by step, and the example gallery has more pipelines to start from.
If you build a source for a system others use too, we'd like to hear about it: CocoIndex on GitHub.
---
# CocoIndex Changelog 2025-10-19
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-10-19/
Date: 2025-10-19
Summary: Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.
:rocket: Over the 20+ releases, CocoIndex continued to advance its ultra-performant AI-native ETL infrastructure with a series of targeted features and enhancements focused on reliable incremental processing, error resilience, and extensibility.
We are constantly improving CocoIndex to make it easier to use, more powerful, and more reliable. :star: Star us on GitHub to stay updated!
## Core capability
### Durable execution
CocoIndex is designed for massive, ever-changing datasets. Reliability and efficiency are critical. With durable execution, CocoIndex now automatically captures failures and retries only the affected rows, ensuring progress is never lost and avoiding full reprocessing.
- **Automatic recovery on failure**: Failed rows are automatically retried in the next update #995, with optional fine-grained control through configurable retry lists #997. CocoIndex also maintains precise processing order tracking to guarantee consistent reprocessing sequences #1001.
- **Stable transient authentication**: Improved handling of short-lived auth keys ensures reliability and prevents drift across retries #1002.
These enhancements make incremental processing **more fault-tolerant, efficient, and self-healing**, allowing CocoIndex to continuously move forward - no wasted computation, no data left behind.
### Avoid unnecessary reprocessing
In CocoIndex, reprocessing can be triggered by changes in either **source data** or **flow definitions**. We improved our flow change detection mechanism, significantly reducing false positives and avoiding unnecessary reprocessing.
- **Improved change detection logic**: Refined how CocoIndex evaluates flow and dependency changes, cutting down false positives and ensuring reprocessing only happens when it’s truly needed. This leads to faster, more predictable runs and reduced compute costs. #1182
- **Stable transient auth keys**: Enhanced retry behavior now keeps authentication keys consistent across transient failures, preventing drift and improving reliability for long-running flows. #1002
Together, these updates make CocoIndex’s incremental engine more **deterministic, efficient, and resilient**. So your flows stay lean even as your data and logic evolve.
### Enhanced build & update target data
We’ve expanded how CocoIndex builds and updates target data, giving users more control over replication and recovery workflows.
#### Supported Modes
- **One-time update**: Builds or updates target data based on the current state of the source.
- **Live update**: Starts with a one-time update, then continuously captures source changes to keep targets up to date.
#### New Option: Reexport Targets with --reexport
We’ve added a new **--reexport** option that can be applied to both modes. When enabled, CocoIndex **ignores the previous state of the target** and rebuilds it entirely, even if neither the source data nor the flow definition has changed.
This is especially useful for **data recovery**, **corrections**, or **full target refreshes** after data loss. In live update mode, reexport applies only to the initial build phase.
Example:
```sh
cocoindex update --reexport main
```
This enhancement broadens pipeline flexibility and improves data reliability in recovery or replay scenarios.
Read more in Build and update data.
#### New option: --reset for setup, update, and server commands
We’ve added a new --reset flag that makes it easier to start fresh by automatically clearing existing setups before running key commands.
When you use --reset, CocoIndex will **drop the existing setup or state** before performing the new operation, equivalent to running cocoindex drop first.
Example:
```sh
cocoindex setup --reset main
```
This ensures a clean environment for rebuilds, testing, or reinitializing workflows, without needing to manually clear prior state.
### Incremental engine optimizations with **fast fingerprint collapsing**
CocoIndex has smart incremental processing out of the box that only processes what’s changed. In the latest version, we’ve included more **engine-level optimizations** to improve efficiency for large datasets:
- **Source content fingerprinting**: the engine tracks a lightweight “fingerprint” of each source row or file, allowing it to detect when content hasn’t changed. *#892*
- **Fast collapse**: unchanged data can be skipped without reading the full source, significantly reducing processing time and resource usage. *#895*
This optimization is particularly useful for sources that provide a **content hash, version tag, or fingerprint** (e.g., GitHub files, S3 objects, or custom sources). CocoIndex can save a **large number of external API calls in addition to compute costs**, making workflows faster and more cost-efficient.
While primarily designed for versioned sources like GitHub, this approach can benefit **any source that exposes change indicators**, improving incremental processing across diverse pipelines.
### None handling
CocoIndex supports *None* values. A *None* value represents the absence of data or an unknown value, distinct from empty strings, zero numbers, or false boolean values. None input values passed to required arguments of CocoIndex functions will short-circuit the function and result in a None output value. See the documentation for more details.
In this release, None value handling in transformations was improved to better manage required and nullable fields, increasing robustness.
## Robustness
### Isolate GPU workloads in subprocess
Added full subprocess support for GPU workloads, improving stability and isolation while removing the need for global locks. These enhancements mark a strong step forward in robustness and performance for GPU‐driven workloads within CocoIndex. They reduce the risk of GPU jobs hanging, and provide better tolerance to GPU failures. For users leveraging GPU processing, the update is highly recommended and should bring a smoother, more resilient operational experience.
#### What to expect for users / developers
If you’re using CocoIndex’s GPU-capable workloads, you may notice improved stability. Temporary failures caused by the GPU will be isolated in the subprocess, which will be restarted gracefully.
### Error tolerance improvements
Fixed live-mode behaviour so that source-level errors are now logged instead of aborting the process. Also improved coverage of retrying for several external APIs, e.g. OpenAI and Vertex AI. This reduces failures and improves stability.
## Query handler and CocoInsight support
### Query handler
Query handlers let you expose a simple function that takes a query string and returns structured results. They are discoverable by tools like CocoInsight so you can query your indexes without writing extra glue code.
```python
@my_flow.query_handler(name="run_query") # Name is optional, use the function name by default
def run_query(query: str) -> cocoindex.QueryOutput:
# 1) Perform your query against the input `query`
...
# 2) Return structured results
return cocoindex.QueryOutput(results=[{"filename": "...", "text": "..."}])
```
### CocoInsight
We are launching a major feature in both CocoIndex and CocoInsight to help users iterate quickly on the indexing strategy, and trace back all the way to the data.
:tada: Announcement: https://cocoindex.io/blogs/query-support
Query Support
Demo:
https://www.youtube.com/watch?v=crV7odEVYTE
## Python SDK
### Support pydantic with field-level description for structured extraction
ExtractByLlm is one of the native building blocks in CocoIndex to make it easier to extract structured information from unstructured documents with an LLM.
This change allows users to directly define the description at the field level when doing structured extraction.
For example, a user could define the following class:
```python
class ProductTaxonomyInfo(BaseModel):
"""
Taxonomy information for the product.
"""
taxonomies: list[ProductTaxonomy] = Field(
...,
description="Taxonomies for the current product."
)
complementary_taxonomies: list[ProductTaxonomy] = Field(
...,
description="Think about when customers buy this product, what else they might need as complementary products. Put labels for these complentary products."
)
```
And you can plug it into any ExtractByLlm transformation.
```python
taxonomy = data["detail"].transform(
cocoindex.functions.ExtractByLlm(
llm_spec=cocoindex.LlmSpec(
api_type=cocoindex.LlmApiType.OPENAI, model="gpt-4.1"
),
output_type=ProductTaxonomyInfo,
)
)
```
### support @cocoindex.settings and make init() optional
A more flexible approach is to provide a setting function that returns a cocoindex.Settings dataclass object. The setting function can have any name, and needs to be decorated with the @cocoindex.settings decorator, for example:
```python
@cocoindex.settings
def cocoindex_settings() -> cocoindex.Settings:
return cocoindex.Settings(
database=cocoindex.DatabaseConnectionSpec(
url="postgres://cocoindex:cocoindex@localhost/cocoindex"
)
)
```
This setting function will be called once when CocoIndex is initialized, whenever you're running the CLI or your own main script, hence you can provide settings in a single way within your code. Once the settings function is provided, environment variables will be ignored.
## Building blocks
### Postgres source enhancements
The Postgres source allows users to take data from any PostgreSQL table as input. The schema of the imported data is automatically inferred from the source table. It supports change-capture based on LISTEN/NOTIFY.
- **Composite-key Postgres source**: Auto-detects schema, supports single or multi-column keys, and maps Postgres → CocoIndex types. #910
- **Faster small-row performance**: Refactored Postgres source and list() API for high-volume, many-small-rows workloads. #948
- **Real-time Postgres updates**: New LISTEN/NOTIFY change-capture mode with auto channel naming and full source wiring. #952, #953, #954, #955
### Postgres target enhancements
- **Postgres filtering support**: added the ability to apply filters directly on Postgres sources, enabling more efficient data selection and ingestion. #1178
- **Half-precision vector (halfvec) support**: added halfvec support for Postgres targets, reducing storage overhead for embeddings while maintaining compatibility with vector operations. #1171
- **Schema support for Postgres targets**: users can now specify and write to custom schemas, allowing better organization in multi-schema database setups #1138
### Target enhancements - new & improved built-ins
#### Support for LanceDB target
We are officially supporting LanceDB as a target.
Get started with LanceDB target.
#### Postgres target enhancements
**Target Attachment / PostgreSQL SQL Command Attachment**: The core engine now supports attachments on targets, including a PostgresSqlCommand type that allows you to execute arbitrary SQL (setup/teardown) on the target side #1131.
Example (create a custom index):
```python
collector.export(
"doc_embeddings",
cocoindex.targets.Postgres(table_name="doc_embeddings"),
primary_key_fields=["id"],
attachments=[
cocoindex.targets.PostgresSqlCommand(
name="fts",
setup_sql=(
"CREATE INDEX IF NOT EXISTS doc_embeddings_text_fts "
"ON doc_embeddings USING GIN (to_tsvector('english', text));"
),
teardown_sql= "DROP INDEX IF EXISTS doc_embeddings_text_fts;",
)
],
)
```
Check the full documentation here.
#### **Neo4j improvements**
- Support vector index method
- Changed the KEY constraint to UNIQUE in Neo4j mappings for compatibility with the Community Edition. *#983*
#### New LLM integrations - add AWS Bedrock LLM support
CocoIndex provides builtin functions integrating with various LLM APIs, for various inference tasks. In the new release we’ve added AWS Bedrock LLM Support in addition to all major commercial and open source LLMs / APIs supported. See the full documentation here.
## Native functions
CocoIndex’s native building blocks for **codebase indexing** received major upgrades this month, driven by extensive community feedback. We’ve refined language detection, splitting strategies, and parsing robustness for diverse codebases.
- DetectProgrammingLanguage automatically identifies programming languages from file content or extensions, improving indexing precision across mixed-language repositories.
- Improved SplitRecursively function. Improved its tree-sitter based chunking method with better markdown handling, richer punctuation handling for plain text, configurable chunk overlap, and Solidity support.
- SplitBySeparators is a lightweight regex-only splitter for simpler, high-performance text segmentation workflows.
To see how to build customized codebase indexing, you can get started here.
## New blogs and examples:
https://cocoindex.io/blogs/etl-to-snowflake
- Incrementally Transform Structured + Unstructured Data from Postgres with AI
Incrementally Transform Structured + Unstructured Data from Postgres with AI
This blog introduces the new PostgreSQL source and shows how to take data from a PostgreSQL table as a source, transform with both AI models and non-AI calculations, and write them into a new PostgreSQL table for semantic + structured search.
- Automated invoice processing with AI, Snowflake and CocoIndex - with incremental processing
Snowflake ETL with CocoIndex
This blog shows how to automate invoice processing using CocoIndex, an open-source ETL framework built for AI-powered data transformation. It walks through how to extract supplier invoices from Azure Blob Storage, use LLMs to read and structure the data, and then load everything into Snowflake, all with incremental processing for scalability and data freshness.
- Build a Visual Document Index from multiple formats all at once - PDFs, Images, Slides - with ColPali
Multi-format indexing with ColPali
This blog shows how to build a visual document index from multiple formats all at once - PDFs, Images, Slides - with ColPali. It walks through how to use ColPali to extract text from multiple formats, and then use CocoIndex to index the text into a visual document index.
## **Thanks to the Community 🤗🎉**
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @MrAnayDongre
Thanks @MrAnayDongre for the work on SplitBySeparators regex-only splitter #1010, enabling simpler and high-performance text segmentation workflows.
### @lemorage
Thanks @lemorage for adding explicit type hints in engine value encoding for the Python SDK #807,
making value serialization more robust and type-safe.
### @banrovegrie
Thanks @banrovegrie for fixing the Gemini embedding configuration and exposing Postgres index tuning #1050,
improving integration reliability.
### @thisisharsh7
Thanks @thisisharsh7 for the work on:
adding the --reset flag for setup, update, and server commands #1106
improving Postgres schema support #1138
enhancing error messages when no flows are registered #1070
making example paths OS-friendly #1066
These updates improve usability, reliability, and cross-platform compatibility, making workflows smoother and more robust for all users.
### @Davda-James
Thanks @Davda-James for the work on:
binding Pydantic models to CocoIndex Struct #1072
reorganizing the Python SDK package structure #1081, #1082
generating full CLI docs and adding them to pre-commit hooks #1096
collecting additional row processing counters for stats #1105
and splitting convert.py into modular components #1120
These updates improve developer productivity, maintainability, and observability, making the Python SDK and CLI easier to use, extend, and monitor for large-scale workflows.
### @princyballabh
Thanks @princyballabh for auto-applying safe defaults for missing fields in load_engine_object #1104,
improving reliability and reducing errors when loading engine objects.
### @shresthashim
Thanks @shresthashim for implementing __str__ and __repr__ methods for Python schema classes #1095,
enhancing developer experience when working with schema objects.
### @belloibrahv
Thanks @belloibrahv for the work on:
Adding AWS Bedrock LLM #1173
Neo4j vector index methods #1111
Field-level descriptions to FieldSchema #1087
These updates expand integration capabilities, improve graph database support, and enhance schema clarity.
### @mensonones
Thanks @mensonones for updating documentation to use cocoindex update --setup main #1093,
making setup instructions clearer.
### @esther-anierobi
Thanks @esther-anierobi for improving documentation navigation #1136.
### @TheVijayVignesh
Thanks @TheVijayVignesh for moving built-in sources to top-level navigation in the docs #1119,
improving documentation structure.
### @aryasoni98
Thanks @aryasoni98 for adding tests for json_schema.rs #1133,
improving code reliability.
### @skalwaghe-56
Thanks @skalwaghe-56 for adding the legacy-states-v0 Cargo feature for backward compatibility #1135,
making it easier to maintain legacy state logic.
### @siddharthbaleja7
Thanks @siddharthbaleja7 for flattening recursion in SplitRecursively to prevent stack overflow #1127,
improving stability.
## Support us
We are constantly improving CocoIndex, and more features are coming soon! Stay tuned and follow us by starring our GitHub repo.
---
# Automated invoice processing with AI and Snowflake
URL: https://cocoindex.io/blogs/etl-to-snowflake/
Date: 2025-10-11
Summary: Incremental ETL from Azure Blob Storage to Snowflake with CocoIndex v1: LLM invoice extraction from PDFs, processing only new or changed files.
I recently worked with a clothing manufacturer who wanted to simplify their invoice process. Every day they receive around 20 to 22 supplier invoices as PDFs, all stored in Azure Blob Storage.
The finance team used to open each PDF by hand and copy the details into their system. It took a lot of time, and they already had a backlog of 8,000 old invoices waiting to be processed.
My first version used n8n: read the invoices from Azure Blob Storage, pull the fields out of each PDF with an AI model, then load the results into Snowflake. It worked for a while. As the number of invoices grew, the workflow started to break, and debugging errors inside a no-code tool got harder and harder. That is when I switched to code.
I came across CocoIndex, an open-source ETL framework for transforming data for AI, with incremental processing built in. It let me build a pipeline that was reliable and easy to reason about.
This post walks through that pipeline, rewritten for CocoIndex v1. The v1 Snowflake connector is now first-class, so the custom target code from the original version is gone.
## What is CocoIndex?
CocoIndex moves data from one place to another in a structured way. In technical terms that is ETL (Extract, Transform, Load), built around a declarative model: you describe the target state you want, and the engine keeps it in sync with the source.
A useful mental picture is a conveyor belt in a factory. You place raw material on one end (the PDF invoices). As the belt moves, items pass through stations that clean, format, and label the data, with an AI model acting as the inspector that reads each invoice. At the far end, the finished product is packed into boxes (your Snowflake table).
The part that matters most here: CocoIndex remembers what it has already processed. Add a new invoice tomorrow and it processes only that file, not the whole backlog again. That is incremental processing, and it is where most of the time and cost savings come from.
## The pipeline
The flow reads invoice PDFs from Azure Blob Storage, converts each to text, extracts the fields with an LLM, and loads the result into Snowflake:
Pipeline: Azure Blob Storage holds the PDF invoices; one processing component per file converts the PDF to Markdown, extracts fields with an LLM, and declares a row that CocoIndex merges into the Snowflake INVOICES table.
Each invoice is handled by its own processing component. A component groups one invoice's work with the row it produces, runs in parallel with the others, and commits to Snowflake as soon as it finishes. This walkthrough covers the Extract and Load steps; the heavier Transform work can happen inside Snowflake afterward.
CocoIndex keeps its own processing state in a small local database file, so it knows which invoices it has already seen. In v1 that state lives in a db_path file you set at startup. There is no separate tracking database to run, which is one fewer moving part than the original version needed.
## Set up credentials
Store the credentials for the LLM, Azure Blob Storage, and Snowflake in a .env file, out of the main code:
```bash
# LLM (any litellm model id + the matching provider key)
OPENAI_API_KEY=sk-*********************
LLM_MODEL=openai/gpt-4o
# Azure Blob
AZURE_ACCOUNT_NAME=your_account_name
AZURE_CONTAINER=invoice
AZURE_SAS_TOKEN=sv=**************************
AZURE_PREFIX=
# Snowflake
SNOWFLAKE_ACCOUNT=your_account_id
SNOWFLAKE_USER=your_username
SNOWFLAKE_PASSWORD=***************
SNOWFLAKE_WAREHOUSE=COMPUTE_WH
SNOWFLAKE_DATABASE=INVOICE
SNOWFLAKE_SCHEMA=DBO
SNOWFLAKE_TABLE=INVOICES
```
For local development against Azure you can also authenticate with the Azure CLI (az login) instead of a SAS token; see the Azure Blob connector docs.
The invoice container the pipeline reads from holds the supplier PDFs, one blob per invoice:
The invoice container in the Azure portal, listing supplier invoice PDFs as block blobs with their names, sizes, and last-modified times.
## Define the extraction schema
The fields to pull from each invoice are described once, as a Pydantic model. The field descriptions do double duty: they document the schema and they instruct the LLM on how to fill it in. This is the same idea as the original version's docstring prompt, now expressed as typed fields.
We extract the invoice header (number, date, customer, totals, and so on) plus a list of line items. Here is a real invoice from the sample set, with each captured field highlighted:
A sample supplier invoice with the extracted fields highlighted: invoice number, date, customer, bill-to and ship-to blocks, totals, order ID, and the line-item table.
```python title=main.py
import pydantic
class LineItem(pydantic.BaseModel):
description: str = pydantic.Field("", description="Item or product name, e.g. \"Newell 330\".")
quantity: str = pydantic.Field("", description="Numeric quantity.")
rate: str = pydantic.Field("", description="Unit price, numeric, no currency symbol.")
amount: str = pydantic.Field("", description="Line total, numeric.")
sku: str = pydantic.Field("", description="Product ID if listed, e.g. \"OFF-AR-5309\".")
category: str = pydantic.Field("", description="Category or sub-category if listed.")
class Invoice(pydantic.BaseModel):
invoice_number: str = pydantic.Field("", description="Invoice number, e.g. \"36259\".")
date: str = pydantic.Field("", description="Invoice date, e.g. \"Mar 06 2012\".")
customer_name: str = pydantic.Field("", description="Name under \"Bill To\", without the address.")
bill_to: str = pydantic.Field("", description="Full \"Bill To\" block, including address.")
ship_to: str = pydantic.Field("", description="Full \"Ship To\" block, including address.")
subtotal: str = pydantic.Field("", description="Subtotal, numeric, no currency symbol.")
discount: str = pydantic.Field("", description="Discount, numeric, no percent sign.")
shipping: str = pydantic.Field("", description="Shipping or handling amount, numeric.")
total: str = pydantic.Field("", description="Total amount, numeric.")
balance_due: str = pydantic.Field("", description="Balance due, numeric.")
order_id: str = pydantic.Field("", description="Purchase order or Order ID.")
ship_mode: str = pydantic.Field("", description="Shipping method, e.g. \"First Class\".")
notes: str = pydantic.Field("", description="Free-text notes.")
terms: str = pydantic.Field("", description="Payment terms, if present.")
line_items: list[LineItem] = pydantic.Field(default_factory=list)
```
The rules that used to live in the docstring (return numbers without currency symbols, use an empty string for missing fields, never swap values between fields) go into a short system prompt:
```python title=main.py
EXTRACT_PROMPT = (
"Extract the invoice into the Invoice schema. Return numbers as plain "
"numeric text (\"58.11\", not \"$58.11\"). Use an empty string for any "
"field not present. Never swap values between fields. Keep line_items as "
"a structured list."
)
```
## Convert the PDF and extract
An LLM cannot read a raw PDF well, so each file is first converted to Markdown with MarkItDown, which keeps headings and tables in order:
```python title=main.py
import os
import tempfile
from markitdown import MarkItDown
_md = MarkItDown()
def to_markdown(content: bytes, filename: str) -> str:
suffix = os.path.splitext(filename)[1] or ".pdf"
with tempfile.NamedTemporaryFile(delete=True, suffix=suffix) as tmp:
tmp.write(content)
tmp.flush()
return _md.convert(tmp.name).text_content or ""
```
Extraction is a single CocoIndex function. It calls the model through litellm and instructor, which forces the response to match the Invoice schema. memo=True caches the result, so an unchanged invoice is never sent to the model twice:
```python title=main.py
import instructor
import litellm
import cocoindex as coco
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
@coco.fn(memo=True)
async def extract_invoice(markdown: str) -> Invoice:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=Invoice,
messages=[
{"role": "system", "content": EXTRACT_PROMPT},
{"role": "user", "content": markdown},
],
)
return Invoice.model_validate(result.model_dump())
```
## Load into Snowflake
This is where v1 changes the most. The original version needed a hand-written target connector, roughly 120 lines of raw MERGE SQL, to load rows into Snowflake. In v1, Snowflake is a built-in connector: you declare the table as a Python dataclass and CocoIndex creates it, maps the types, and upserts by primary key.
Define the row you want in Snowflake as a dataclass. The line items are a nested list, which CocoIndex stores in a VARIANT column as JSON:
```python title=main.py
from dataclasses import dataclass
@dataclass
class InvoiceRow:
invoice_number: str # primary key
date: str
customer_name: str
bill_to: str
ship_to: str
subtotal: str
discount: str
shipping: str
total: str
balance_due: str
order_id: str
ship_mode: str
notes: str
terms: str
line_items: list[dict[str, object]]
filename: str
```
CocoIndex maps each Python field to a Snowflake column automatically, so string fields become VARCHAR and the nested line_items list becomes a VARIANT written with PARSE_JSON:
The Invoice dataclass maps to the Snowflake INVOICES table: string fields become VARCHAR, and the nested line_items list becomes a VARIANT column written with PARSE_JSON. CocoIndex creates the table and upserts rows by primary key.
## Process each invoice
One component processes one invoice: read the PDF, convert it, extract the fields, and declare the row. declare_row states the row that should exist in Snowflake; CocoIndex works out whether that means an insert or an update:
```python title=main.py
from cocoindex.resources.file import FileLike
@coco.fn(memo=True)
async def process_invoice(
file: FileLike[str],
table: snowflake.TableTarget[InvoiceRow],
) -> None:
markdown = to_markdown(await file.read(), file.file_path.path.name)
inv = await extract_invoice(markdown)
table.declare_row(
row=InvoiceRow(
invoice_number=inv.invoice_number or file.file_path.path.name,
date=inv.date,
customer_name=inv.customer_name,
bill_to=inv.bill_to,
ship_to=inv.ship_to,
subtotal=inv.subtotal,
discount=inv.discount,
shipping=inv.shipping,
total=inv.total,
balance_due=inv.balance_due,
order_id=inv.order_id,
ship_mode=inv.ship_mode,
notes=inv.notes,
terms=inv.terms,
line_items=[li.model_dump() for li in inv.line_items],
filename=file.file_path.path.name,
)
)
```
The invoice number is the primary key, so re-processing the same invoice updates its row in place instead of creating a duplicate. If the number is missing, the filename stands in as a fallback key.
## Wire up the app
Connections are provided once in the app lifespan and read anywhere with coco.use_context. This is also where CocoIndex's local processing-state file is set, via builder.settings.db_path:
```python title=main.py
import os
from collections.abc import AsyncIterator
import pathlib
from azure.storage.blob.aio import ContainerClient
from cocoindex.connectors import azure_blob, snowflake
SNOWFLAKE = coco.ContextKey[snowflake.ConnectionConfig]("snowflake")
AZURE = coco.ContextKey[ContainerClient]("azure_container")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.settings.db_path = pathlib.Path("./cocoindex.db")
builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-4o"))
builder.provide(
SNOWFLAKE,
snowflake.ConnectionConfig(
account=os.environ["SNOWFLAKE_ACCOUNT"],
user=os.environ["SNOWFLAKE_USER"],
password=os.environ["SNOWFLAKE_PASSWORD"],
warehouse=os.environ.get("SNOWFLAKE_WAREHOUSE"),
),
)
async with ContainerClient(
account_url=f"https://{os.environ['AZURE_ACCOUNT_NAME']}.blob.core.windows.net",
container_name=os.environ["AZURE_CONTAINER"],
credential=os.environ["AZURE_SAS_TOKEN"],
) as client:
builder.provide(AZURE, client)
yield
```
app_main mounts the Snowflake table target, lists the PDFs in the container, and fans out one process_invoice component per file with mount_each:
```python title=main.py
from cocoindex.resources.file import PatternFilePathMatcher
@coco.fn
async def app_main() -> None:
table = await snowflake.mount_table_target(
SNOWFLAKE,
table_name=os.environ.get("SNOWFLAKE_TABLE", "INVOICES"),
table_schema=await snowflake.TableSchema.from_class(
InvoiceRow, primary_key=["invoice_number"]
),
database=os.environ.get("SNOWFLAKE_DATABASE"),
schema=os.environ.get("SNOWFLAKE_SCHEMA"),
)
client = coco.use_context(AZURE)
files = azure_blob.list_blobs(
client,
prefix=os.environ.get("AZURE_PREFIX", ""),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
)
await coco.mount_each(process_invoice, files.items(), table)
app = coco.App(
coco.AppConfig(name="InvoiceExtraction"),
app_main,
)
```
## Run the pipeline
Install the dependencies and run the pipeline:
```sh
pip install "cocoindex[snowflake,azure_blob]" instructor litellm markitdown python-dotenv
cocoindex update main
```
The first run is a full load: every invoice in the container is converted, extracted, and merged into Snowflake. After that, re-running does only the new work.
This is the payoff from incremental processing. On the client project we first loaded about 8,000 old invoices, then ran the pipeline daily on the 20-odd new ones. The second run does not touch the 8,000 unchanged invoices, so it skips those LLM calls entirely:
Incremental processing across runs: the first run extracts all 8,000 backlog invoices with the LLM; the next day's run extracts only the 22 new invoices while the 8,000 unchanged ones are reused, so cost stays flat as the backlog grows.
Because per-invoice extraction is memoized, the expensive model calls run only for files CocoIndex has not seen. Change the prompt or switch LLM_MODEL and the next run re-extracts to stay consistent, with no manual migration.
Here is the result in Snowflake after a run, one row per invoice with the line items in the VARIANT column:
The INVOICES table in the Snowflake UI, populated with one row per processed invoice.
## Challenges
Because this uses an LLM to read invoices, it is not perfect, and evaluation is the real work. On the original client data, many fields were mismatched at first, and refining the prompt is what improved accuracy. Clean, structured PDFs are the easy case; messy or inconsistent ones can still trip the model up.
CocoIndex itself is not a parser. It is the framework and incremental engine, and you plug in whatever extraction logic fits your data. I used OpenAI here, but Google Document AI, a specialized invoice parser, or a fine-tuned model could all slot into the same extract_invoice step.
For more, see the CocoIndex documentation and the Snowflake connector guide. If it was useful, a star on GitHub helps a lot.
---
# Thinking in Rust: Ownership, Access, and Memory Safety
URL: https://cocoindex.io/blogs/rust-ownership-access/
Date: 2025-10-10
Summary: A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.
I'm an experienced C++ programmer, but still feel that I need a mindset shift when starting to work with Rust.
- References can be immutable (&) or mutable (&mut), which is straightforward for the simplest cases. There are more complex cases, e.g. there are data types with interior mutability (e.g. RefCell), there are multi-level references (& &mut).
- Rust also models memory safety in multi-thread scenarios, so there's Send and Sync. They have intricate rules related to various types of references, e.g. when is &T Send or Sync? How about &mut T?
- There are various data types, like Rc , Arc, Cell, RefCell, Mutex, RwLock, Cow . When to pick which?
Each single rule is well documented from the bottom up, but when putting things together to design a complex system, it's easy to get overwhelmed by all these intricacies. So it would be very helpful if we had a clear mental model to help us think in a more rationalized way.
This article is not a tutorial to introduce these concepts, but provides a mental framework, as a simple and natural interpretation of Rust's memory safety models. I hope with this, Rust's rules around memory safety and capability of related built-in data types will be more easily understandable, based on rationalized coherent ideas instead of memorizing many rules.
## Basic concepts: ownership and access
A variable represents some resource in the memory. When talking about the relationship between the variable and the relevant resource, there are two types of roles: **ownership** and **access**.
```rust
let s: String = "Hello".to_string(); // Variable `s` owns the data
println!("{}", s); // Variable `s` has access to the data
let p: &String = &s; // Create a reference `p`
println!("{}", p); // `p` also has access to the same data
```
### Access: shared vs exclusive
Rust classifies variables and their references as immutable (default) and mutable (mut). When it comes to some more intricate data types, the boundary is less clear. e.g. if my data type uses a cache to accelerate access, is it mutating things? What about some statistics collections? What about talking to a remote backend?
I find things become clearer suddenly after switching to a different viewpoint: ***thinking in terms of shared access and exclusive access***.
- **Shared Access** (default): You can access the resource, without excluding others from holding shared access simultaneously.
- **Exclusive Access** (mut): You have sole permission to use and modify the resource. No one else can access it during this period.
The level of access you have determines what you can do with the resource. Shared access is sufficient for read operations. Write operations require exclusive access. For methods of a resource, its receiver type (&self or &mut self) indicates what level of access is needed by this method. When you wrap a resource with something offering interior mutability (e.g. Mutex ), the wrapped resource can be accessed in a shared way, while its implementation provides run-time protection to only grant access exclusively at runtime.
*Exclusive access implies shared access*, i.e. what you can achieve with shared access, can always be achieved with exclusive access.
### Ownership implies lifetime keeper + exclusive access
When a variable owns a resource, there are two implications:
- **Lifetime Keeper**: You hold the lifetime of the resource. Before you're out of scope (e.g. end of the block that defines the variable), the ownership is either moved, or the resource needs to be dropped.
- **Access**: You have access to it through the entire lifetime of the ownership (unless during the period when it's lent exclusively).
Note that the access here can also be shared or exclusive, depending on whether you used mut when introducing it. However, even without mut, since you have the ownership, you can easily *upgrade* it to a mutable variable, e.g.
```rust
fn foo(s: String) {
// You cannot do this, as you don't have exclusive access yet
//! s.push_str(" world");
// However, since you have ownership, it's easy to claim exclusive access
let mut s = s;
// Now we have exclusive access, so can modify it.
s.push_str(" world");
...
}
```
So once you own a resource, whether or not you have mut isn't part of the contract. You have the same permission as exclusive access, i.e. ***ownership implies exclusive access***.
Lifetime keeping isn't another permission. For a resource, you can do almost everything with exclusive access: with std::mem::take() and std::mem::replace(), you can even drop the resource with exclusive access. Hence ***exclusive access is equivalent to ownership in terms of permission.*** Keep this in mind - it will be helpful to understand rules around Send and Sync later.
What each form grants: a shared reference has shared access (read); an exclusive reference has shared and exclusive access (read and write) but no lifetime role; the owner has identical permission to an exclusive reference, plus the lifetime-keeper role (move, drop, keep alive). In short: exclusive access + lifetime keeper = ownership.
## Moves vs. borrows: transferring ownership or access
*Move* and *borrow* are two very unique concepts in Rust. Both are about passing around resources: one for ownership, one for access.
Rust provides two ways to pass around resources:
- **Move**: Permanently transfers ownership to another variable. It's a one-way ticket, and after moving, the original owner loses ownership and access.
```rust
let s = "Hello".to_string();
let t = s; // ownership moves to t; s is no longer valid
```
- **Borrow (Exclusive or Shared Access)**: Temporarily grants access to a reference without giving up ownership. It will suspend your exclusive access (as you no longer exclusively hold it) until the end of the borrow.
```rust
let mut s = String::from("hello");
let r = &mut s; // exclusive access borrowed
r.push_str(" world");
// Borrow ends here; s is usable again
```
You can grant the access as long as you have the same level of access, even if you don't have the ownership. Once an exclusive access is lent out, your access is suspended until the borrow ends, by the definition of exclusivity. Once shared access is lent out, you only retain shared access, as you no longer exclusively hold the access.
:::note
Technically, when you grant access from &T or &mut T, slightly different things happen:
- It's called *reborrow* when you pass on access from &mut T.
- A *copy* happens when you grant access from &T, as shared access can be freely shared without suspending the current access.
:::
Move versus borrow. A move hands ownership from s to t one way; s is gone and t is the one owner. An exclusive borrow lends &mut s to a single reference; s stays the owner, pauses while lent, and resumes when the borrow ends. A shared borrow lends &s to many readers at once, and the owner can still read too.
## Send and Sync: access across threads
Rust ensures thread-safety explicitly using two traits: Send and Sync. There are 3 rules around trait transitivity on &T and &mut T; however, anchoring them to the concepts of ownership and access will make things simpler to understand.
To define Send and Sync:
- **Send**: Indicates that ownership can safely be transferred to another thread. Hence when T is Send, T can be moved across the thread boundary.
Corollary: As we discussed earlier, exclusive access is equivalent to ownership in terms of permission. So whenever we allow transferring ownership across threads, nothing can go wrong for shared access across the thread boundary, and vice versa. So Send also indicates the safety of transferring exclusive access across threads.
- **Sync**: Indicates shared access can be safely transferred across the thread boundary.
Send and Sync across a thread boundary: Send means ownership (equivalently exclusive access) may move to another thread; Sync means shared access (a reference) may be shared with another thread. A reference &T is Send if and only if T is Sync.
From here, we can reason about the 3 rules regarding trait transitivity:
1. By the definition of Sync above, &T is Send if and only if T is Sync.
2. By the definition and corollary of Send above, &mut T is Send if and only if T is Send.
3. For both &T and &mut T, their shared access (& &T and & &mut T) grants and only grants shared access of T . So &T and &mut T are Sync if and only if T is Sync.
You don't need to memorize any rules. Think clearly about which levels of access are allowed to happen across threads, and nothing can go wrong.
## Smart wrappers to defer compile-time safety to runtime
Rust tries hard to enforce safety at compile time, but it can be rigid sometimes. Some scenarios need more flexibility. Rust provides smart wrappers to defer compile-time safety guards to runtime:
- **Smart wrappers for interior mutability** (Cell, RefCell, Mutex, RwMutex). With these, only shared access on the wrapper is needed for exclusive access on the interior resource.
- Cell is the simplest one, exposing no API to return references with exclusive access. The main API to set the value is:
```rust
pub fn set(&self, val: T)
```
It doesn't expose any exclusive access outside the function. Also, it declares to be !Sync, so all methods are guaranteed not to be called simultaneously. Hence it guarantees exclusive access isn't held concurrently without any extra runtime check.
- RefCell also declares to be !Sync, while it exposes methods that return proxy to access:
```rust
// Expose shared access to T
pub fn borrow(&self) -> Ref
// Expose exclusive access to T
pub fn borrow_mut(&self) -> RefMut
```
While Ref and RefMut hold access to the interior resource T after the methods return, it needs to perform a runtime check to make sure whenever RefMut is returned no other Ref or RefMut is live.
- Mutex and RwMutex offer a similar interface to RefCell, but are both Sync - they're safely shared across threads. To achieve safety, it uses lock and read-write lock internally.
- **Smart wrappers for shared ownership** (Rc , Arc ). With these, ownership on the interior resource can be shared among multiple parties, and it's dropped when the last owner no longer holds the smart wrapper. Shared ownership on the interior resource can only safely grant shared access. When it comes to exclusive access, they provide the following methods:
```rust
pub fn get_mut(this: &mut Rc) -> Option<&mut T>
pub fn get_mut(this: &mut Arc) -> Option<&mut T>
```
which do a runtime check to make sure this is the sole owner, and return None otherwise. Internally, they do reference counting, while Rc is cheaper but thread unsafe (!Send + !Sync), and Arc is thread safe (Send + Sync) when T is thread safe (Send + Sync).
- **Smart wrappers for clone-on-write** (Cow) leave ownership versus shared access undecided at compile time. Shared access is always guaranteed without overhead. Whenever exclusive access is needed, it clones the interior resource.
When thinking from the angle of ownership and access, the purpose, contract, and behavior of these smart wrappers become more easily explainable. The choice comes down to two questions: are you sharing access or sharing ownership, and does it need to cross threads?
Choosing a smart wrapper by two questions: sharing access or sharing ownership, and does it cross threads. Sharing access: Cell and RefCell single-thread, Mutex and RwLock thread-safe. Sharing ownership: Rc single-thread, Arc thread-safe. Cow stays undecided until a write and clones only when exclusive access is needed.
## Why this matters for a data engine
This mental model isn't academic for us — it's the daily reality of building CocoIndex's engine. The core is async-first on Tokio, which means values routinely cross thread boundaries: a future started on one worker thread can be polled to completion on another. The moment you put a value into a task that the runtime may move between threads, the compiler demands that value be Send. Reading that requirement through the ownership lens above — "Send means ownership (and therefore exclusive access) can safely move across a thread boundary" — turns a cryptic compiler error into an obvious design question: *should this resource really be handed off to another thread, and is everything it transitively owns safe to hand off too?*
The smart-wrapper choices fall out of the same reasoning. State that's genuinely shared across concurrent tasks — connection pools, cached schema, context values — lives behind Arc, because we need shared *ownership* with a lifetime that ends only when the last task lets go, and Arc is the thread-safe counter (Rc would be !Send and wouldn't compile into a multi-threaded task). Where that shared state also needs mutation, we reach for the Sync interior-mutability wrappers (Mutex/RwLock) so that shared access to the wrapper can yield exclusive access to the interior, with the runtime check enforcing exclusivity. The single-writer discipline around our internal LMDB storage is exactly this principle applied at the I/O layer: many tasks hold shared access to the database handle, but writes are funneled so that mutation is serialized rather than racing.
The mental model at work in CocoIndex's async engine. Futures hop between Tokio worker threads, so their values must be Send. State shared across tasks lives behind Arc with Mutex or RwLock: shared ownership plus runtime-checked exclusive writes. LMDB storage takes many shared readers in parallel while writes are funneled to one exclusive writer at a time.
The same separation of *what a value is* from *how it may be accessed* underpins our type-guided serialization: a value the engine owns can be moved into the serializer and reconstructed on the other side, while a value merely borrowed can only be read. Thinking in ownership and access keeps these decisions principled instead of trial-and-error against the borrow checker.
## Conclusion
By clearly separating and defining **ownership** and **exclusive versus shared access**, Rust's complexity transforms into logical clarity. Moves, borrows, Send, Sync, and runtime checks become intuitive and predictable tools in your programming toolbox.
With this mental model, you can confidently explore even advanced Rust, while keeping your understanding grounded and practical.
## Support us
CocoIndex is an open source project built on Rust. Rust is the number one choice for any modern data engine, and these ownership and access ideas underpin how CocoIndex handles type-guided serialization and its internal storage.
If this article is helpful to you, please drop a star ⭐ at GitHub to support this project.
---
# Trace search results back to source data
URL: https://cocoindex.io/blogs/query-support/
Date: 2025-09-21
Summary: Define query handlers in CocoIndex and trace search results back to source data in CocoInsight to close the loop on indexing strategy.
We are launching a major feature in both CocoIndex and CocoInsight to help users iterate faster on the indexing strategy, and trace back all the way to the data, to make the transformation experience more seamlessly integrated with the end goal.
We deeply care about making the overall experience seamless. With the new launch, you can define query handlers, so that you can easily run queries in tools like CocoInsight.
## CocoInsight
### Does my data transformation create a meaningful index for retrieval?
In CocoInsight, we’ve added a Query mode. You can enable this by adding a CocoIndex Query Handler. You can quickly query the index, and view the collected information for any entity.
CocoInsight Query Mode
The result is directly linked and can be traced back step by step to how data is generated on the indexing path.
### Where are the results coming from?
For example, this snippet comes from the file docs/docs/core/flow_def.mdx. The file was split into 30 chunks after transformation.
trace back data
### Why is my chunk / snippet not showing in the search result?
When you perform a query, on the ranking path, you’d usually have a scoring mechanism. On CocoInsight, you can quickly find any files you have in your mind, and for any chunks, you can scan the scoring in the same context.
missing chunks
This gives you a powerful toolset with direct insight into end-to-end data transformation, to quickly iterate on the data indexing strategy without any headaches of building any additional UI or tools.
## Integrate query logic with CocoIndex
### Query handler
To run queries in CocoInsight, you need to define query handlers. You can use any libraries or frameworks of your choice to perform queries.
You can read more in the documentation about Query Handler.
Query handlers let you expose a simple function that takes a query string and returns structured results. They are discoverable by tools like CocoInsight so you can query your indexes without building your own UI.
For example:
```python
# Declaring it as a query handler, so that you can easily run queries in CocoInsight.
@code_embedding_flow.query_handler(
result_fields=cocoindex.QueryHandlerResultFields(
embedding=["embedding"], score="score"
)
)
def search(query: str) -> cocoindex.QueryOutput:
# Get the table name, for the export target in the code_embedding_flow above.
table_name = cocoindex.utils.get_target_default_name(
code_embedding_flow, "code_embeddings"
)
# Evaluate the transform flow defined below with the input query, to get the embedding.
query_vector = code_to_embedding.eval(query)
# Run the query and get the results.
with connection_pool().connection() as conn:
register_vector(conn)
with conn.cursor() as cur:
cur.execute(
f"""
SELECT filename, code, embedding, embedding <=> %s AS distance, start, "end"
FROM {table_name} ORDER BY distance LIMIT %s
""",
(query_vector, TOP_K),
)
return cocoindex.QueryOutput(
query_info=cocoindex.QueryInfo(
embedding=query_vector,
similarity_metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY,
),
results=[
{
"filename": row[0],
"code": row[1],
"embedding": row[2],
"score": 1.0 - row[3],
"start": row[4],
"end": row[5],
}
for row in cur.fetchall()
],
)
```
This code defines a query handler that:
1. Turns the input query into an embedding vector. code_to_embedding is a shared transformation flow between the Query and Index paths; see the detailed explanation below.
2. Searches a database of code embeddings using cosine similarity.
3. Returns the top matching code snippets with their filename, code, embedding, score, and positions.
### Sharing logic between indexing and query
Sometimes, transformation logic needs to be shared between indexing and querying, e.g. when we build a vector index and query against it, the embedding computation needs to be consistent between indexing and querying.
You can find the documentation about Transformation Flow.
You can use @cocoindex.transform_flow() to define shared logic. For example:
```python
@cocoindex.transform_flow()
def text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[NDArray[np.float32]]:
return text.transform(
cocoindex.functions.SentenceTransformerEmbed(
model="sentence-transformers/all-MiniLM-L6-v2"))
```
In your indexing flow, you can directly call it:
```python
with doc["chunks"].row() as chunk:
chunk["embedding"] = text_to_embedding(chunk["text"])
```
In your query logic, you can call the eval() method with a specific value:
```python
def search(query: str) -> cocoindex.QueryOutput:
# Evaluate the transform flow defined below with the input query, to get the embedding.
query_vector = code_to_embedding.eval(query)
```
## Examples
- Text Embedding (PostgreSQL)
- Text Embedding (Qdrant)
- Code Embedding
## Beyond vector index
We use a vector index in this blog, following the text embedding example. CocoIndex is a powerful data transformation framework that goes beyond vector indexes. You can use it to build vector indexes, knowledge graphs, structured extraction and transformation, and any custom logic towards your need for efficient retrieval from fresh data.
## Support us
We’re constantly adding more examples and improving our runtime. ⭐ Star CocoIndex on GitHub and share the love ❤️ !
And let us know what you are building with CocoIndex. We’d love to feature it.
---
# Turn a Postgres table into a semantic index
URL: https://cocoindex.io/blogs/postgres-source/
Date: 2025-09-01
Summary: Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.
Most of your data already lives in a database. This example takes an existing Postgres table of products, reads it row by row, derives a couple of fields, embeds each row, and writes the result, including the vector, back into Postgres with pgvector. Structured data in, searchable-by-meaning index out, no separate export step.
You declare the transformation in native Python and your own types, target_state = transformation(source_state), and a Rust engine handles the heavy lifting underneath: incremental processing, change tracking, and managed targets. Only the rows that changed get re-embedded and re-upserted.
The full code is in the CocoIndex v1 repo. A star on GitHub is appreciated if this helps.
## What we'll build
The pipeline reads product rows from source_products, computes a total_value and a combined full_description, embeds that description, and stores the enriched rows plus their vectors in a second Postgres table.
Indexing flow: a Postgres source table feeds a CocoIndex app; one processing component per row derives total_value and a full_description, embeds the description with a sentence-transformer model, and declares a row that CocoIndex stores back in a Postgres pgvector table.
Both ends are Postgres here, but they need not be: point the source at one database and the target at another by setting a different connection string. The source can be any table you already have.
## Define the row shapes
Each source row maps into a plain dataclass, and each output row into another. The embedding field is annotated with the embedder so CocoIndex infers the vector's size automatically:
```python title=main.py
from dataclasses import dataclass
from typing import Annotated
from numpy.typing import NDArray
import cocoindex as coco
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@dataclass
class SourceProduct:
product_category: str
product_name: str
description: str
price: float
amount: int
@dataclass
class OutputProduct:
product_category: str
product_name: str
description: str
price: float
amount: int
total_value: float
embedding: Annotated[NDArray, EMBEDDER]
```
The database pools and the embedder are created once, in the app lifespan. Set SOURCE_DATABASE_URL to read from a separate database, or leave it pointing at the same instance:
```python title=main.py
import asyncpg
from cocoindex.connectors import postgres
PG_DB = coco.ContextKey[asyncpg.Pool]("postgres_source_db")
SOURCE_POOL = coco.ContextKey[asyncpg.Pool]("source_pool")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with (
asyncpg.create_pool(DATABASE_URL) as target_pool,
asyncpg.create_pool(SOURCE_DATABASE_URL) as source_pool,
):
builder.provide(PG_DB, target_pool)
builder.provide(SOURCE_POOL, source_pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
```
## Transform and embed each row
process_product runs once per source row. It computes the derived fields, embeds the composed description, and declares the output row. memo=True caches this work, so a row whose content and code are unchanged is skipped on the next run:
```python title=main.py
@coco.fn(memo=True)
async def process_product(
product: SourceProduct,
table: postgres.TableTarget[OutputProduct],
) -> None:
full_description = (
f"Category: {product.product_category}\n"
f"Name: {product.product_name}\n\n{product.description}"
)
total_value = product.price * product.amount
embedding = await coco.use_context(EMBEDDER).embed(full_description)
table.declare_row(
row=OutputProduct(
product_category=product.product_category,
product_name=product.product_name,
description=product.description,
price=product.price,
amount=product.amount,
total_value=total_value,
embedding=embedding,
),
)
```
Embeddings carry more meaning when they see more context. We embed the *composed* description, category and name included, so a search for "wireless audio" matches even when the body text never says it.
The AI step and the plain arithmetic sit side by side in the same function. total_value = price * amount is an ordinary Python expression; embed(...) is a model call. To CocoIndex both are just transforms that derive an output field from the source row.
## Wire up the app
app_main mounts the target table, opens the source table, and runs one processing component per source row:
```python title=main.py
@coco.fn
async def app_main() -> None:
target_table = await postgres.mount_table_target(
PG_DB,
table_name="output",
table_schema=await postgres.TableSchema.from_class(
OutputProduct,
primary_key=["product_category", "product_name"],
),
pg_schema_name="coco_examples_v1",
)
source = postgres.PgTableSource(
coco.use_context(SOURCE_POOL),
table_name="source_products",
row_type=SourceProduct,
)
await coco.mount_each(
process_product,
source.fetch_rows().items(lambda p: (p.product_category, p.product_name)),
target_table,
)
app = coco.App(
coco.AppConfig(name="PostgresSourceV1"),
app_main,
)
```
PgTableSource reads the existing table directly, and row_type=SourceProduct maps each row straight into the dataclass. items(...) tags each row with its (product_category, product_name) composite key, which becomes the output row's primary key. mount_table_target creates and manages the target table for you: schema, idempotent upserts, and cleanup of rows whose source disappeared.
Because the output key derives from the source row, incremental updates are exact. Change one product and only that row is re-embedded and re-upserted; delete a source row and its output row is removed. That is what memo=True and the derived key buy you: the expensive embedding work scales with how much actually changed, not with the size of the table.
On a re-run, CocoIndex compares the source table to what it processed before. Only the edited row is re-embedded and re-upserted, a deleted source row has its output row removed, and unchanged rows are skipped on a memo hit, so the embedding cost scales with what changed rather than with table size.
## Run it
Start Postgres with pgvector (the repo ships a compose file), configure the connection strings, and seed the source table:
```sh
docker compose -f ../../dev/postgres.yaml up -d
cp .env.example .env # set POSTGRES_URL and SOURCE_DATABASE_URL
pip install -e .
psql "$SOURCE_DATABASE_URL" -f ./prepare_source_data.sql
```
Build the index. The Postgres source runs as a one-shot catch-up: it scans the source table, syncs the target, and exits.
```sh
cocoindex update main
```
Re-run it any time; only new or changed rows are processed. To keep the index fresh on a schedule, run cocoindex update from cron.
## Query the index
Searching is the mirror image of indexing: embed the query with the same model, then ask Postgres for the nearest vectors. The operator is pgvector's cosine distance, so smaller means more similar:
```python title=main.py
async def query_once(pool, embedder, query: str, *, top_k: int = 5) -> None:
query_vec = await embedder.embed(query)
async with pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT product_category, product_name, description, amount,
total_value, embedding <=> $1 AS distance
FROM "coco_examples_v1"."output"
ORDER BY distance ASC
LIMIT $2
""",
query_vec,
top_k,
)
for r in rows:
score = 1.0 - float(r["distance"])
print(f"[{score:.3f}] {r['product_category']} | {r['product_name']}")
print(f" {r['description']}")
```
Run it and ask in plain language:
```sh
python main.py "wireless headphones"
```
The closest products come back ranked, even when they share none of the words in your query. That is the point of a vector index over your structured data.
## Where to go next
You now have a semantic index over an existing Postgres table, kept in sync incrementally. The same shape extends a long way: swap the source for Amazon S3, Azure Blob, or Google Drive; swap the target for a dedicated vector database like Qdrant; or feed the retrieved rows to an LLM for RAG. Because the pipeline is declarative and incremental, none of that changes how you keep the index fresh.
## Support us
We are constantly adding examples and improving the runtime. If this guide helped, please star CocoIndex on GitHub and let us know what you are building.
---
# Index PDFs, images, and slides with ColPali, no OCR
URL: https://cocoindex.io/blogs/multi-format-indexing/
Date: 2025-08-20
Summary: Build a unified visual document index from multiple file formats (including PDFs, images, and slides) using CocoIndex and ColPali. No OCR needed.
Do you have a messy collection of scanned documents, PDFs, academic papers, presentation slides, and standalone images, all mixed together with charts, tables, and figures, that you want to process into the same vector space for semantic search or to power an AI agent?
In this example, we’ll walk through how to build a visual document indexing pipeline using ColPali for embedding both PDFs and images, and then query the index using natural language.
We’ll skip OCR entirely: ColPali can directly understand document layouts, tables, and figures from images, making it perfect for semantic search across visual-heavy content.
If this helps you, ⭐ Star CocoIndex GitHub!
## Use cases
- Semantic search across scanned documents and visual-heavy reports
- Searching tables, charts, or diagrams without OCR
- Cross-modal search (e.g., “find the page showing a bar chart about Q2 revenue”)
- Research paper figure indexing
- Historical archive search without text extraction
## Flow overview
Multi-Format Indexing Flow
We’ll build a pipeline that:
- **Ingests PDFs and images** from a local directory
- **Converts PDF pages** into high-resolution images (300 DPI)
- **Generates visual embeddings** for each page/image using ColPali
- **Stores embeddings + metadata** in a Qdrant vector database
- **Supports natural language queries** directly against the visual index
Example queries:
- *"handwritten lab notes about physics"*
- *"architectural floor plan with annotations"*
- *"pie chart of Q3 revenue"*
Full code is open source and available here.
:rocket: Only ~70 lines of Python on the indexing path (super simple!)
## Core components
### Image ingestion
We use CocoIndex’s LocalFile source to read PDFs and images:
```python
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="source_files", binary=True)
)
```
LocalFile Source
### Convert files to pages
We classify files by MIME type and process accordingly.
Define a dataclass:
- page_number: The page number (if applicable, only for PDFs)
- image: The binary content of that page as a PNG image
```python
@dataclass
class Page:
page_number: int | None
image: bytes
```
This normalizes different file formats into a **list of page images** so the rest of the pipeline can process them uniformly. This file_to_pages custom function takes a **filename** and its **raw binary content** (bytes) and returns a list of Page objects, where each Page contains:
```python
@cocoindex.op.function()
def file_to_pages(filename: str, content: bytes) -> list[Page]:
mime_type, _ = mimetypes.guess_type(filename)
if mime_type == "application/pdf":
images = convert_from_bytes(content, dpi=300)
pages = []
for i, image in enumerate(images):
with BytesIO() as buffer:
image.save(buffer, format="PNG")
pages.append(Page(page_number=i + 1, image=buffer.getvalue()))
return pages
elif mime_type and mime_type.startswith("image/"):
return [Page(page_number=None, image=content)]
else:
return []
```
For each document:
- If the file is an image → file_to_pages returns a single Page where page["image"] is just the original image binary.
- If the file is a PDF → file_to_pages converts each page to a PNG, so page["image"] contains that page’s PNG binary.
File to Pages
In the flow we convert all the files to pages. This makes each page and all images in the output data: pages.
```jsx
output_embeddings = data_scope.add_collector()
with data_scope["documents"].row() as doc:
doc["pages"] = flow_builder.transform(
file_to_pages, filename=doc["filename"], content=doc["content"]
)
```
File to Pages
### Generate visual embeddings
We use ColPali to generate embeddings for images on each page.
```python
with doc["pages"].row() as page:
page["embedding"] = page["image"].transform(
cocoindex.functions.ColPaliEmbedImage(model=COLPALI_MODEL_NAME)
)
output_embeddings.collect(
id=cocoindex.GeneratedField.UUID,
filename=doc["filename"],
page=page["page_number"],
embedding=page["embedding"],
)
```
ColPali Embed Image
ColPali Architecture fundamentally rethinks how documents, especially visually complex or image-rich ones, are represented and searched.
Instead of reducing each image or page to a single dense vector (as in traditional bi-encoders), ColPali breaks an image into many smaller patches, preserving local spatial and semantic structure.
Each patch receives its own embedding, which together form a multi-vector representation of the complete document.
Multi-Modal Architecture
For a detailed explanation of ColPali Architecture, please refer to our previous blog with image search examples.
## Collect & export to Qdrant
Note the ways to embed an image and a query are different, as they’re two different types of data.
Create a transform flow to embed the query:
```python
@cocoindex.transform_flow()
def query_to_colpali_embedding(
text: cocoindex.DataSlice[str],
) -> cocoindex.DataSlice[list[list[float]]]:
return text.transform(
cocoindex.functions.ColPaliEmbedQuery(model=COLPALI_MODEL_NAME)
)
```
We store metadata and embeddings in Qdrant:
```jsx
output_embeddings.export(
"multi_format_indexings",
cocoindex.targets.Qdrant(
connection=qdrant_connection,
collection_name=QDRANT_COLLECTION,
),
primary_key_fields=["id"],
)
```
## Query the index
ColPali supports **text-to-visual embeddings**, so we can search using natural language:
```python
query_embedding = query_to_colpali_embedding.eval(query)
search_results = client.query_points(
collection_name=QDRANT_COLLECTION,
query=query_embedding,
using="embedding",
limit=5,
with_payload=True,
)
```
Check out the full code here.
## Debugging with CocoInsight
Run CocoInsight locally:
```sh
cocoindex server -ci main
```
Open https://cocoindex.io/cocoinsight to:
- View extracted pages
- See embedding vectors and metadata
## Support us
We’re constantly adding more examples and improving our runtime.
⭐ Star CocoIndex on GitHub and share the love :heart:!
And let us know what you are building with CocoIndex. We’d love to feature them.
---
# CocoIndex Changelog 2025-08-18
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-08-18/
Date: 2025-08-18
Summary: CocoIndex updates: production readiness, scalability, and reliability, plus more customization, native integrations, and multi-modal pipeline features.
We’ve shipped 20+ releases, packed with production-ready features, scalability upgrades, and runtime improvements.
🚀 Huge thanks to our amazing users for the feedback and for running CocoIndex at scale!
Full changelog: v0.1.58...v0.1.79.
We've made it to GitHub Rust Trending this week along with some other cool projects like OpenAI Codex, Tree-sitter, Polars, LanceDB, and more!
rust-trending
## Performance, scalability, and reliability
CocoIndex is built to be production-ready from day one, empowering teams to process data at scale without compromising performance or reliability.
Our continuous improvements to the runtime ensure that CocoIndex can handle large-scale, real-time workloads with ease.
### Control processing concurrency in CocoIndex
CocoIndex now has concurrency control to balance maximum throughput with system stability.
The default settings work well for most cases, but we also expose layered controls for advanced scenarios.
flow-control
- Global limits: keep total concurrency within safe thresholds
- Per-source limits: tune for different workloads
- Nested iteration limits: control fan-out during chunk/sub-row processing
Read more about the concurrency control in the blog post.
### Auto-retry for remote LLM requests
CocoIndex now automatically backs off and retries rate-limited requests (HTTP 429) to remote LLMs, ensuring smoother and more resilient data processing.
More optimizations are coming soon.
### Automatically combined update operations
CocoIndex now automatically combines multiple update calls into a single operation when it's safe to do so.
This optimization allows users to blindly call the update API whenever an update is potentially needed,
without worrying about too frequent or conflicting updates.
### Enhanced PostgreSQL connection pooling
The latest update introduces configurable minimum and maximum connection limits for PostgreSQL connections.
Check out DatabaseConnectionSpec in the docs.
This enhancement allows fine-tuning connection pool settings to better match your workload requirements,
improving resource utilization and performance.
## API updates
### Custom targets
CocoIndex envisions building pipelines like assembling building blocks by standardizing the interfaces.
It should be composable, interchangeable, and fun to build.
We provide a rich set of default components to help you build vector indexes, knowledge graphs,
and your own custom building block creations, with a standard interface.
Now, we’re introducing custom targets, further pushing the boundaries of what you can build with CocoIndex. You define two things:
1. **Target Spec** – how to configure the target (such as setting a file path or API key).
2. **Target Connector** – how to write data to that target (the logic).
You can think of this as plugging in your own target with a few lines of Python.
custom-targets
Read more about custom targets with an example in the blog post.
Read the documentation for more details.
### New transform method in FlowBuilder
The new FlowBuilder.transform() method provides an alternative to the chained-style transformation, and makes the flow more clear when the CocoIndex function takes more than one argument.
e.g. in the multi_format_indexing example, instead of writing this:
```py
doc["pages"] = doc["filename"].transform(file_to_pages, content=doc["content"])
```
Now you can write:
```py
doc["pages"] = flow_builder.transform(
file_to_pages, filename=doc["filename"], content=doc["content"]
)
```
### Status update polling API
We introduced capabilities for tracking and polling the status of long-running data flows.
A new next_status_updates() method has been added to the FlowLiveUpdater class.
This API enables users to programmatically poll for status changes and receive updates during live flow execution.
This allows you to react to updates in your application, for example, by notifying users or triggering downstream processes, like an agentic workflow.
```python
while True:
updates = my_updater.next_status_updates()
for source in updates.updated_sources:
# Perform downstream operations on the target of the source.
run_your_downstream_operations_for(source)
# Break the loop if there are no more active sources.
if not updates.active_sources:
break
```
Read the documentation here.
### Flow identifier naming validation
This validation standardizes name patterns for CocoIndex flow and field names, ensuring relevant backend resources (e.g. tables) can be named directly based on these names.
Flow identifiers are now enforced to match the pattern [a-zA-Z_][a-zA-Z0-9_]* and no more than 64 characters.
You’ll experience improved reliability and more predictable behavior when defining and referencing flows, reducing risks of unexpected bugs or collisions.
## Typing
### Multi-dimensional vector
CocoIndex now provides robust and flexible support for typed vector data, from simple numeric arrays to deeply nested multi-dimensional vectors.
This support is designed for seamless integration with high-performance vector databases such as Qdrant,
and enables advanced indexing, embedding, and retrieval workflows across **diverse data modalities**.
:::tip
CocoIndex automatically infers types, so when defining a flow, you don’t need to explicitly specify any types.
You only need to specify types explicitly for return types of custom functions, etc.
:::
In CocoIndex, we represent a multi-dimensional vector using Vector[Vector[T, N], M], meaning M vectors, each of dimension N. M and N are optional - CocoIndex doesn't require them to be fixed, while some targets have requirements, e.g. a multi-vector exported to Qdrant needs to have a fixed inner dimension, i.e. Vector[Vector[T, N]].
If you are building a multimodal search engine with Qdrant, this is the mapping you need to know:
| CocoIndex Python Type | Qdrant Type |
|--------------------------------------|------------------------------------------|
| Vector[Float32, Literal[N]] | Dense Vector |
| Vector[Vector[Float32, Literal[N]]] | MultiVector |
| _Other types_ | Stored in Qdrant’s JSON payload |
Read more about multi-dimensional vector support in the blog post.
### None propagation behavior
A function may specify whether each input argument is optional or not.
A non-optional argument means the function needs a known value for the argument to work.
However, it doesn't forbid the argument from being *None* at runtime. When a non-optional argument receives a *None* value,
the function execution is skipped and the result is *None*.
For your custom function, this none-propagation is also automatic whenever you annotate your argument with a non-optional type (e.g. T instead of T | None / Optional[T]).
For example, for SplitRecursively function, the text and chunk_size arguments are not optional. If the input value of either of them is None, the function will return None.
Read more about None values and None propagation here.
### Support unannotated type bindings on Rust->Python path
Now you don't have to provide explicit type annotations for Python variables binding to values coming from the Rust engine
(e.g. arguments of custom functions). When it's omitted, we create Python values in the default type according to specific
CocoIndex data type. For example, for the custom function in
multi_file_indexing example, you can write:
```python
@cocoindex.op.function()
def file_to_pages(filename, content) -> list[Page]:
...
```
which omits explicit type hints for filename and content arguments.
When it comes to Struct types, we use dict[str, Any] where the key is the name of each field.
See our data types documentation for more details.
Besides, we also enhanced robustness of value bindings for Union types.
### ISO8601 parsing enhancement
The ISO8601 date/time parser now supports fractional values, allowing more precise parsing of timestamps with fractional seconds.
This improvement aligns parsing behavior with wider ISO8601 variations that include decimal fractions in time units, enhancing temporal data accuracy and compatibility with external data sources.
## Expanded LLM, cloud AI & data source integrations
We integrated with the following LLM APIs:
- vLLM (for text generation)
- Ollama (for text embedding)
- Google Vertex AI (for text generation and embedding)
We support the following types of LLM APIs. Read the latest docs.
| API Name | LlmApiType enum | Text Generation | Text Embedding |
|-----------------|--------------------------|-----------------|---------------|
| OpenAI | LlmApiType.OPENAI | ✅ | ✅ |
| Ollama | LlmApiType.OLLAMA | ✅ | ✅ **New** |
| Google Gemini | LlmApiType.GEMINI | ✅ | ✅ |
| Vertex AI | LlmApiType.VERTEX_AI | ✅ **New** | ✅ **New** |
| Anthropic | LlmApiType.ANTHROPIC | ✅ | ❌ |
| Voyage | LlmApiType.VOYAGE | ❌ | ✅ |
| LiteLLM | LlmApiType.LITE_LLM | ✅ | ❌ |
| OpenRouter | LlmApiType.OPEN_ROUTER | ✅ | ❌ |
| vLLM | LlmApiType.VLLM | ✅ **New** | ❌ |
Example Usage:
```python
cocoindex.LlmSpec(
api_type=cocoindex.LlmApiType.VERTEX_AI,
model="gemini-2.0-flash",
api_config=cocoindex.llm.VertexAiConfig(project="your-project-id"),
)
```
### ColPali embedding
CocoIndex now supports native integration with ColPali, enabling multi-vector, patch-level image indexing using cutting-edge multimodal models.
With just a few lines of code, you can now embed and index images with ColPali’s late-interaction architecture, fully integrated into CocoIndex’s composable flow system.
colpali
Read more about ColPali integration in the blog post.
We are constantly adding new integrations to support your use case. Please contact us if you need any support. You can also use custom ops to bring any of your custom logic for LLM inference.
### Azure Blob Storage as data source
CocoIndex now supports Azure Blob Storage as a data source. Read more about the integration here.
```python
cocoindex.StorageSpec(
api_type=cocoindex.StorageApiType.AZURE_BLOB_STORAGE,
account_name="your-account-name",
container_name="your-container-name",
)
```
## CLI clarity improvements & vector storage notes
- CocoIndex now shows clear notes when vectors are stored as JSON in either Postgres or Qdrant. In some deployments, vectors may be stored as JSON instead of native vector column types, affecting performance, indexing, and retrieval methods. The new note alerts users so they can locate potential bugs earlier (e.g. forgetting to specify vector length in type annotation).
- Improved the clarity of CLI output messages and added more detailed error messages during interactive and automated runs.
- Added context-sensitive hints with more actionable guidance when running in live mode.
## Tutorials
We're expanding the documentation with more in-depth tutorials!
### Live updates
CocoIndex is designed to keep your indexes synchronized with your data sources. This is achieved through a feature called live updates, which automatically detects changes in your sources and updates your indexes accordingly.
This tutorial walks you through deep tech details of how live updates work, and how to use it in your own projects.
Check out the tutorial.
### Manage flow dynamically
You write a function, a.k.a. flow definition, to define indexing logic.
Sometimes you want to reuse the same flow definition for multiple flow instances (a.k.a. flow), e.g. each takes input from different sources, exports to different targets, and even with slightly different parameters for transformation logic.
Check out the tutorial.
### ColPali - multi-modal search engine
In this tutorial, we’ll walk through how to build a multi-modal search engine with CocoIndex and ColPali.
We’ll show how to embed and index images with ColPali’s late-interaction architecture, fully integrated into CocoIndex’s composable flow system.
Check out the blog and source code.
### Custom targets
In this tutorial, we'll explain how custom targets work and show a simple example to continuously convert JSON from local files to HTML
and export them as custom targets.
Check out the blog and source code.
### Face Detection - build your own Google Photos search
In this blog, we’ll walk through a comprehensive example of building a scalable face recognition pipeline using CocoIndex.
We’ll show how to extract and embed faces from images, structure the data relationally, and export everything into a vector database for real-time querying.
Check out the blog and source code.
### Rust ownership access
Open source is about knowledge sharing! In this blog, we are proposing a mental model to help you understand Rust ownership and memory safety models.
By clearly separating and defining ownership and exclusive versus shared access, Rust's complexity transforms into logical clarity. Moves, borrows, Send, Sync, and runtime checks become intuitive and predictable tools in your programming toolbox.
Check out the blog.
### Academic papers indexing
In this tutorial, we will walk through a comprehensive example of indexing research papers with extracting different metadata, beyond full text chunking and embedding.
Check out the blog and source code.
## Thanks to the community 🤗🎉
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @lemorage
Thanks to @lemorage for the contributions!
CocoIndex has received a list of high quality PRs from him, and we really appreciate his work.
- New Features
- Support image input to LLM clients #653
- Add transform method to FlowBuilder #675
- Add EmbedText for Ollama #770
- Codebase Robustness
- Add utilities to streamline tests for flow functions #723
- Add unit tests to execute transform flow e2e #737
- Cover roundtrip validation for all data types #703
- Tutorial for performing live updates #781
### @vumichien
Thanks to @vumichien for contributions around data type conversion and for making the data type environment more robust between Rust and Python.
- Add LTable and KTable decoders for list and dict bindings #767
- Enhance dict and struct encoding with type awareness #727
### @kingkushal16
Thanks to @kingkushal16 for the contributions! We appreciate the work on adding default values for field decoding #788.
### @theparthgupta
Thanks to @theparthgupta for the contributions! We appreciate the work on adding naming validation for flow identifiers #779 to make the system more robust.
### @par4m
Thanks to @par4m for the contributions! We appreciate the work on adding vLLM support #683.
### @wykrrr
Thanks to @wykrrr for the contributions! We appreciate the work on adding Azure Blob Storage as a data source #746.
## Support us
We are constantly improving CocoIndex, and more features are coming soon!
Stay tuned and follow us by starring our GitHub repo.
---
# Control Processing Concurrency in CocoIndex
URL: https://cocoindex.io/blogs/flow-control/
Date: 2025-08-13
Summary: How CocoIndex's layered concurrency controls optimize data-processing performance, prevent system overload, and keep pipelines stable and efficient at scale.
CocoIndex is designed to be **production-ready from day one**, built to process data in parallel, maximizing throughput while keeping your systems safe. Today, we’ll look at how to **optimize performance without overloading your environment**. With CocoIndex, it’s just **one configuration away**.
CocoIndex’s parallelism model boosts speed by processing multiple data items at once, but **more parallelism isn’t always better**. Left unconstrained, excessive concurrency can strain, or even destabilize, your systems. That’s why CocoIndex includes **built-in concurrency control mechanisms** that strike the right balance between **raw performance** and **system stability**, even at massive scale.
Processing too many items simultaneously can cause:
- **Memory exhaustion** – large datasets loaded at once consume massive amounts of RAM.
- **Resource contention** – CPU, disk I/O, and network bandwidth get overwhelmed by competing operations.
- **System instability** – timeouts, degraded performance, or outright crashes.
Unlike generic concurrency features, CocoIndex lets you:
- Constrain both data volume (rows) and memory usage (bytes).
- Set limits at multiple layers: global, per source, and per-row iteration.
- Combine controls: *all* specified constraints must be satisfied before processing proceeds.
This layered approach ensures that resource-heavy sources don’t overwhelm the system, and nested tasks (such as splitting documents into chunks) remain predictable and safe.
flow-control
You can review the full concurrency control documentation. CocoIndex is powering users to process at the scale of millions in production, :star: star us if you like it!
## Concurrency options
CocoIndex provides two primary settings:
| Option | Purpose | Unit |
| --- | --- | --- |
| **max_inflight_rows** | Maximum number of rows processed concurrently. | rows |
| **max_inflight_bytes** | Maximum memory footprint of concurrently processed data (before transformations). | bytes |
When a limit is reached, CocoIndex **pauses new processing** until some existing work completes. This keeps throughput high without pushing your system past its limits.
:::note
For simplicity, max_inflight_bytes only measures the size of data already in memory before any transformations; it does **not** include temporary memory used during processing steps.
:::
## Where to apply concurrency controls
### 1. Source level
Controls how many rows from a data source are processed simultaneously. This prevents overwhelming your system when ingesting large datasets.
Source level control happens at two different granularities:
- Global, in which all sources across all indexing flows share the same budget.
- Per-source, in which each source has its own budget.
Both **global** and **per-source** limits must pass before a new row is processed, providing two layers of safety.
#### Global concurrency: one setting to shield all flows
global-level concurrency
Global limits ensure your system never overshoots safe operating thresholds, even if individual flows attempt higher concurrency.
Apply system-wide protections either via environment variables or programmatic control:
The easiest way is to control it via environment variables:
```sh
export COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS=256
export COCOINDEX_SOURCE_MAX_INFLIGHT_BYTES=1048576
```
Programmatically, configure it when calling cocoindex.init(), which will take precedence over the environment variable:
```python
from cocoindex import GlobalExecutionOptions
cocoindex.init(
cocoindex.Settings(
...,
global_execution_options = GlobalExecutionOptions(
source_max_inflight_rows=256,
source_max_inflight_bytes=1_048_576
)
)
)
```
Currently, CocoIndex uses 1024 as the default value of global max inflight rows, if you don't explicitly set it.
#### Per-source concurrency: granular customization
per-source concurrency
Set different limits for each source according to workload and data characteristics:
```python
@cocoindex.flow_def(name="DemoFlow")
def demo_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
data_scope["documents"] = flow_builder.add_source(
DemoSourceSpec(...),
max_inflight_rows=10,
max_inflight_bytes=100*1024*1024 # 100 MB
)
```
### 2. Nested iteration level concurrency: deep structural control
nested-iteration concurrency
When processing nested rows, such as processing each chunk of each document, you can configure the maximum concurrent rows and/or bytes:
```python
with data_scope["documents"].row() as doc:
doc["chunks"] = doc["content"].transform(SplitRecursively(...))
with doc["chunks"].row(max_inflight_rows=100, max_inflight_bytes=100*1000*1000):
# Process up to 100 chunks in parallel per document
...
```
### Summary table: concurrency configuration in CocoIndex
| Level | Configuration Path | Applies To |
| --- | --- | --- |
| Global | Environment variables, or pass GlobalExecutionOptions to cocoindex.init() | All sources, all flows, added together |
| Per-Source | Arguments to FlowBuilder.add_source() | Specific source/flow |
| Row Iteration | Arguments to DataSlice.row(max_inflight_rows=...) | Nested iterations |
## Best practices
In actual incremental pipelines, the processing bottleneck is usually at a few heavy operations, such as running inference using an AI model locally or via a remote API. It's common to keep more data in memory even if it cannot be processed immediately. In this way, once the busy backend becomes available, new workloads can be taken on right away to keep the backends busy. However, we need a reasonable bound on this to prevent memory exhaustion and similar issues. That's where concurrency control comes in.
tuning-guide
### Most situations - default is good enough
For most cases, the default global source max rows limit (COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS: 1024) already fits the situation described above: loading more than what heavy operations can consume, but still within a reasonable bound. You don't need to do anything.
### When and what to tune
Decrease the limit if it's not stable enough (e.g. memory overuse, timeout observed on certain operations), or increase the limit if it's very stable but you want it to go faster. What to tune?
- In most cases, tuning the global source row limit (COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS) will just work.
- Only start to touch more specific knobs when deemed necessary - usually only for some complex and unbalanced situations.
- On highly unbalanced row (file) size, set the max bytes limit to prevent a small number of abnormally large inputs from overloading the system, e.g., when the distribution of your input data size follows a long-tail distribution rather than a normal distribution.
- On highly unbalanced complexity across sources, set per-source concurrency. This happens when you want to run multiple flows within the same process, or have multiple sources within the same flow, and they vary in processing complexity (e.g., one source goes through a very heavy and slow model, and another only does simple data movement).
- On high fanout in nested iterations and unbalanced nested rows, set concurrency control options on nested iterations. For example, you have a high number of nested rows to process, and the specific number varies significantly.
This concurrency control framework gives you **safe, scalable, and customizable flow performance**. You gain flexibility (configure per-flow), control (set global limits), and the confidence to scale CocoIndex flows smoothly across diverse workloads.
## Support us
We’re constantly improving our runtime. Please ⭐ star CocoIndex on GitHub and share it with others.
Need help crafting a more detailed code snippet, or insight into using byte-based or default concurrency settings? Just let me know!
---
# Index Images with ColPali: multi-vector visual search
URL: https://cocoindex.io/blogs/colpali/
Date: 2025-08-12
Summary: Index images with ColPali multi-vector patch embeddings and a Qdrant MaxSim collection using CocoIndex v1: incremental, live, in plain async Python.
Most image search squeezes each image into a single vector. That works until the image is dense: a page with a chart and a table, a form with many fields, a photo with several objects. One vector blurs all of that together.
ColPali takes a different approach. It splits each image into a grid of patches and embeds every patch into its own vector, so an image becomes a *bag* of vectors instead of one. A text query is matched patch by patch, late-interaction style, which holds up on exactly the dense, text-heavy images where a single embedding falls apart.
This post builds an image index with ColPali and a Qdrant multivector collection, rewritten for CocoIndex v1. CocoIndex handles the incremental multi-vector indexing, the managed Qdrant collection, and live updates; ColPali does the embedding in plain Python. The full example is in the CocoIndex v1 repo.
Star CocoIndex on GitHub if you like it.
## Why ColPali
ColPali (Contextual Late-interaction over Patches) rethinks how image-rich documents are represented and searched. Instead of one dense vector per image, it keeps one vector per patch and scores queries against those patches directly:
Single dense vector versus ColPali multi-vector with MaxSim: a traditional bi-encoder squeezes a whole image into one vector and loses the layout; ColPali splits the image into a grid of patches, embeds each into its own vector, and at query time matches each query token to its best-scoring patch (MaxSim), so a query like "long neck" localizes the giraffe's neck region.
- **Fine-grained visual search.** Each image is split into a grid (commonly 32×32, about 1,024 patches per page), and every patch is embedded with awareness of both visual and textual cues. Queries are broken into token embeddings and matched against the most relevant patches, far finer than a single-vector model.
- **Spatial structure is preserved.** Global vectors lose layout. Patch embeddings keep spatial relationships, so a match can localize to a region: a diagram in a manual, a table on a form.
- **Late interaction with MaxSim.** Following ColBERT, each query token is compared against all patch embeddings; MaxSim keeps only the best-matching patch per query token, then sums those maxima into the final score. It is precise, interpretable, and cheap at query time because there is no up-front cross-attention.
- **No OCR pipeline.** Images are processed natively, so there is no error-prone text extraction step, and visual elements OCR skips (charts, drawings, logos) are still captured.
## The indexing flow
The flow reads images from a local folder, embeds each with ColPali, and stores the result as one multivector point per image in Qdrant:
ColPali indexing flow: a local img folder holds images; one processing component per image reads it, embeds it with ColPali into a bag of patch vectors, and declares a point that CocoIndex syncs into a Qdrant multivector collection configured for MaxSim, kept live.
Each image is its own processing component, so change detection is per image and the flow runs live.
## Embed with ColPali
The embedding is plain Python using the colpali_engine library. Load the model once, then turn image bytes into a list of patch vectors (list[list[float]]):
```python title=pipeline.py
import functools
import io
from PIL import Image
import torch
from colpali_engine import ColPali, ColPaliProcessor
from colpali_engine.utils.torch_utils import (
get_torch_device,
unbind_padded_multivector_embeddings,
)
@functools.cache
def get_colpali() -> tuple[ColPali, ColPaliProcessor, str]:
model = ColPali.from_pretrained(COLPALI_MODEL_NAME)
processor = ColPaliProcessor.from_pretrained(COLPALI_MODEL_NAME)
device = get_torch_device("auto")
return model.to(device).eval(), processor, device
def embed_image_bytes(img_bytes: bytes) -> list[list[float]]:
model, processor, device = get_colpali()
image = Image.open(io.BytesIO(img_bytes)).convert("RGB")
batch = processor.process_images([image]).to(device)
with torch.no_grad():
embeddings = model(**batch)
padding_side = getattr(processor.tokenizer, "padding_side", "right")
unpadded = unbind_padded_multivector_embeddings(embeddings, padding_side=padding_side)
return unpadded[0].cpu().tolist()
```
The same colpali_engine model embeds a text query into its own per-token vectors at search time, which is what makes the query and the image comparable patch by patch.
## Index each image
One component processes one image: read the bytes, embed them, and declare a Qdrant point keyed by a stable uuid5 of the path. memo=True skips images that have not changed:
```python title=pipeline.py
import cocoindex as coco
from cocoindex.connectors import qdrant
from cocoindex.resources.file import FileLike
@coco.fn(memo=True)
async def process_file(file: FileLike[str], target: qdrant.CollectionTarget) -> None:
content = await file.read()
embedding = embed_image_bytes(content) # list[list[float]]: multi-vector
point = qdrant.PointStruct(
id=_image_id(file.file_path.path),
vector=embedding,
payload={"filename": str(file.file_path.path)},
)
target.declare_point(point)
```
## Configure the Qdrant multivector collection
The collection is where late interaction actually happens. A MultiVectorSchema with multivector_comparator="max_sim" tells Qdrant to do the MaxSim scoring, so the query side just hands over the query's bag of vectors. app_main mounts the collection, walks the image folder in live mode, and fans out one process_file per image:
```python title=pipeline.py
import numpy as np
from cocoindex.connectors import localfs
from cocoindex.resources.file import PatternFilePathMatcher
from cocoindex.resources.schema import MultiVectorSchema, VectorSchema
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
model, _, _ = get_colpali()
dim = int(getattr(model, "dim", 128))
target = await qdrant.mount_collection_target(
QDRANT_DB,
collection_name=QDRANT_COLLECTION,
schema=await qdrant.CollectionSchema.create(
vectors=qdrant.QdrantVectorDef(
schema=MultiVectorSchema(
vector_schema=VectorSchema(dtype=np.dtype(np.float32), size=dim)
),
distance="cosine",
multivector_comparator="max_sim",
)
),
)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(
included_patterns=["**/*.jpg", "**/*.jpeg", "**/*.png"]
),
live=True,
)
await coco.mount_each(process_file, files.items(), target)
app = coco.App(
coco.AppConfig(name="ImageSearchColpali"),
app_main,
sourcedir=pathlib.Path("./img"),
)
```
## What is stored
Unlike a typical image pipeline that stores one vector per image, ColPali stores a multi-vector: the outer dimension is the number of patches, the inner dimension is the model's hidden size. That is what makes the collection compatible with late-interaction queries. See how CocoIndex models a vector index schema for details.
## Serve it live
The example wraps the flow in a FastAPI app. Its lifespan starts the flow in live mode, blocks startup until the first sweep is READY, then keeps watching the folder in the background while serving search:
```python title=api.py
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
async with coco.runtime():
_client = qdrant.create_client(pipeline.qdrant_url(), prefer_grpc=True)
update_handle = pipeline.app.update(live=True)
async for snap in update_handle.watch():
if snap.status is coco.UpdateStatus.READY:
break
update_task = asyncio.create_task(update_handle.result())
try:
yield
finally:
update_task.cancel()
```
Because the source runs live, a new photo in img/ is searchable within a second, with no rebuild step. Each image is its own component, so deleting one removes its Qdrant point automatically.
## Query the index
Search embeds the query text into ColPali's per-token space and lets Qdrant do the MaxSim scoring against every image's patches:
```python title=api.py
@app.get("/search")
async def search(q: str, limit: int = 5) -> dict[str, Any]:
query_embedding = pipeline.embed_query(q) # list[list[float]]: one vector per token
results = pipeline._qdrant_search(_client, pipeline.QDRANT_COLLECTION, query_embedding, limit)
return {"results": [
{"filename": (r.payload or {}).get("filename"), "score": r.score}
for r in results
]}
```
## Run it
Start Qdrant, install the example (cocoindex[colpali,qdrant] pulls in torch, transformers, and pillow), and run the server. It indexes img/ in live mode on startup, so there is no separate indexing command:
```sh
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
pip install -e .
python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000
```
Type a query like *"long neck"* and you get the giraffe back. On dense, text-heavy, or busy images the results hold up better than CLIP with a single dense vector, because the match is patch by patch instead of image-wide:
The ColPali image search frontend returning ranked image results for a text query.
## Connect to any source, kept in sync
Local files are just one source. Swapping the source leaves the rest of the flow unchanged, and CocoIndex keeps the index fresh as files are added, updated, or deleted. Beyond the local filesystem, CocoIndex supports source connectors including:
- Google Drive
- Amazon S3
- Azure Blob Storage
## Support us
We are constantly adding examples and improving the runtime. If this was helpful, please star CocoIndex on GitHub and share it.
---
# Multi-Dimensional Vector Support in CocoIndex
URL: https://cocoindex.io/blogs/multi-vector/
Date: 2025-08-10
Summary: CocoIndex natively handles typed multi-dimensional vectors, from simple arrays to multi-vector embeddings, unlocking multimodal AI pipelines at scale.
CocoIndex now provides robust and flexible support for typed vector data, from simple numeric arrays to deeply nested multi-dimensional vectors. This support is designed for seamless integration with high-performance vector databases such as Qdrant,
and enables advanced indexing, embedding, and retrieval workflows across diverse *data modalities*.
In this post, we’ll break down:
- What types of vectors CocoIndex supports
- How they are represented in Python
- How we handle dimensionality
- How they map to Qdrant vectors or payloads
- Practical examples of usage
:star: Star CocoIndex on GitHub if you like it!
## Multi-vector embedding search architecture
Multi-Dimensional Vector Support
This architecture enables fine-grained retrieval by leveraging multi-vector embeddings for both images and queries.
The system supports high-resolution semantic matching by comparing dense sets of embeddings from both modalities using a similarity operator like MaxSim.
1. Input Image Preprocessing: The image is divided into patches (typically non-overlapping or slightly overlapping regions).
Each patch is treated as a semantic unit and will be embedded individually.
2. Multi-Vector Image Embedding: A multi-vector embedding model processes each image patch independently.
Instead of encoding the entire image into a single vector, the model produces a set of embedding vectors per image (one or more vectors per patch).
This results in a multi-vector representation of the image, preserving local semantic information.
3. Query Embedding: A query (text or image) is passed through a similar multi-vector embedding model,
producing a set of embedding vectors that capture different semantic aspects of the query.
4. MaxSim Similarity Matching: The MaxSim operation compares all query vectors with all image vectors.
For each query vector, it finds the maximum similarity score with any image vector.
These max scores are aggregated (e.g., summed or averaged) to yield a final relevance score.
This method allows the query to flexibly match any relevant part of the image representation.
5. Retrieval
Based on the final similarity scores, the system retrieves the most relevant images for the given query.
## What Python vector types does CocoIndex support?
CocoIndex accepts a range of vector types with strong typing guarantees.
See CocoIndex data type documentation for more details.
:::tip
CocoIndex automatically infers types, so when defining a flow, you don’t need to explicitly specify any types. You only need to specify types explicitly when defining custom functions.
:::
### 1. One-dimensional vectors
```python
from cocoindex import Vector, Float32
from typing import Literal
Vector[Float32] # dynamic dimension
Vector[Float32, Literal[384]] # fixed dimension of 384
```
These types are interpreted as:
- **Underlying Python type**: numpy.typing.NDArray[np.float32] or list[float]
- **Dimension-aware**: When used with Literal[N], dimension size is explicitly enforced
- **Supported number types**:
- Python native types: float, int
- CocoIndex type aliases: Float32, Float64, Int64. They're aliases for Python native types but annotated with CocoIndex type information to indicate the exact bit size. Using them as custom function return types, you can control the exact type used downstream, within the engine and target, and more
- numpy number types: numpy.float32, numpy.float64, numpy.int64
### 2. Two-dimensional vectors (multi-vectors)
```python
Vector[Vector[Float32, Literal[3]], Literal[2]]
```
This declares a vector with two rows of 3-element vectors. These are treated as **multi-vectors** (e.g., multiple embeddings per point).
- **Underlying Python type**:
- numpy.typing.NDArray[np.float32, Literal[3, 2]] or
- nested list[list[float]]
- **Semantics**: Useful for scenarios like comparing sets of embeddings per item, multi-view representations, or batched encodings
## What is a multi-dimensional vector?
A **multi-dimensional vector** is simply a vector whose elements are themselves vectors, essentially a matrix or a nested list. In CocoIndex, we represent this using Vector[Vector[T, N], M], meaning *M* vectors, each of dimension *N*. *M* and *N* are optional: CocoIndex doesn't require them to be fixed, while some targets have requirements, e.g. a multi-vector exported to Qdrant needs to have a fixed inner dimension, i.e. Vector[Vector[T, N]].
This concept is crucial in deep learning and multi-modal applications, for example:
- An image might be represented as a collection of patch-level embeddings.
- A document could be broken down into paragraph-level vectors.
- A user session might be a sequence of behavioral vectors.
Instead of flattening these rich representations, CocoIndex supports them natively through nested Vector typing.
Imagine a 1024x1024 image processed by a Vision Transformer (ViT). Instead of one global image vector, ViT outputs **a vector per patch** (e.g. 8x8 patches → 64 vectors).
```python
Vector[Vector[Float32, Literal[768]]]
```
- 196 patches, each with a 768-dim embedding
- Enables **local feature matching** and **region-aware retrieval**
## Why not just use a flat vector?
You could flatten a 2x3 multi-vector like [[1,2,3],[4,5,6]] into [1,2,3,4,5,6], but you’d lose **semantic boundaries**.
With true multi-vectors:
- Each sub-vector retains its meaning
- You can match/query at the **sub-vector level**
- Vector DBs like Qdrant can **score across multiple embeddings**,
choosing the best match (e.g., closest patch, paragraph, or step)
- The inner vector can have a fixed dimension even if the outer dimension is variable, which is quite common,
e.g. the number of patches of images and the number of paragraphs of articles depend on specific data.
A vector having a fixed dimension is required by most vector databases, as it's essential for building a vector index.
## How do CocoIndex vector types map to Qdrant?
Qdrant is a popular vector database that supports both **dense vectors** and **multi-vectors**. CocoIndex maps vector types to Qdrant-compatible formats as follows:
| CocoIndex Type | Qdrant Type |
| --- | --- |
| Vector[Float32, Literal[N]] | Dense Vector |
| Vector[Vector[Float32, Literal[N]]] | MultiVector |
| Anything else | Stored as part of Qdrant’s JSON **payload** |
:::warning[Vector type mapping to Qdrant]
Qdrant only accepts vectors with **fixed dimension**. CocoIndex automatically detects vector shapes and maps unsupported or dynamic-dimension vectors into payloads instead.
:::
In CocoIndex, data is structured as rows and fields:
- A **row** corresponds to a **Qdrant point**
- A **field** can either:
- Be a **named vector**, if it fits Qdrant’s vector constraints
- Be part of the **payload**, for non-conforming types
This hybrid approach gives you the flexibility of structured metadata with the power of vector search.
### How the type becomes a column
The mapping above isn't something you configure by hand — it falls out of the type. A vector column is fully described by two facts: the dtype of each element and the number of dimensions. CocoIndex captures exactly that in a vector schema, and a multi-vector is simply that single-vector schema wrapped to say "a variable number of these." When a connector sets up a target, it reads the schema off the field's type annotation and translates it into whatever the destination requires — vector(N) in Postgres, fixed_size_list(N) in LanceDB, or a multi-vector collection in Qdrant.
This is also where the fixed-inner-dimension rule becomes concrete. The outer count can stay dynamic — one image yields as many patch vectors as it has patches — but each inner vector has to declare a fixed dtype and size, because that's the unit the vector index is actually built over. The schema is what carries that guarantee from your Python types down to the storage layer, so the column is provisioned correctly the first time rather than inferred from whatever the first row happened to contain.
## Finally
With the ability to support deeply typed vectors and multi-dimensional embeddings, CocoIndex brings structure and semantic clarity to vector data pipelines.
Whether you're indexing images, text, audio, or abstract graph representations, our typing system ensures compatibility, debuggability, and correctness at scale.
We’re constantly adding more examples and improving our runtime. If you found this helpful, please ⭐ star
CocoIndex on GitHub and share it with others.
---
# Custom Targets: export your data anywhere
URL: https://cocoindex.io/blogs/custom-targets/
Date: 2025-08-03
Summary: Export CocoIndex data anywhere: implement a TargetHandler with a tracking record and an action sink; the engine handles diffing, syncing, and cleanup.
CocoIndex supports custom targets: you can export data to any destination, whether it's a local file, cloud storage, a REST API, or your own bespoke system, and still get incremental updates, change tracking, and automatic cleanup from the engine.
:::note[Updated for CocoIndex v1]
The original version of this post introduced the v0 connector API (TargetSpec, target_connector, get_persistent_key() / apply_setup_change() / mutate()). CocoIndex v1 replaced it with the target-state system: a TargetHandler that reconciles desired state against tracking records. This post describes the v1 approach, with a mapping from the v0 API for readers of the original.
:::
Thanks to the community for the suggestions and early feedback that shaped this capability.
## Swap targets without rewriting your pipeline
CocoIndex targets share one declarative pattern: you state what should exist, and the connector keeps the external system in sync. Writing HTML files to a folder and rows to Postgres look the same at the call site:
```python
# Local files
localfs.declare_file(outdir / name, html, create_parent_dirs=True)
# Postgres: same declare pattern, different connector
table.declare_row(row=DocEmbedding(id=id, embedding=vec))
```
Because the transformation logic never touches write paths, swapping a target doesn't ripple through the pipeline. Custom targets extend the same pattern to systems CocoIndex doesn't ship a connector for: you implement the reconciliation once, and your pipeline code keeps declaring state the same way. They are the write-side counterpart of custom sources.
## What a custom target looks like in v1
A custom target connector is the piece that connects CocoIndex's declarative target states to an external system. You implement three small things:
1. A **tracking record**: the minimal information persisted per item to detect changes on future runs. For a file, a content fingerprint; for a row, a hash of the row data.
2. An **action** and a **sink**: the action describes one operation (write this file, delete this row); the TargetActionSink executes a batch of them against the external system.
3. A **TargetHandler** with one required method, reconcile(): given the desired state for a key and the tracking records from previous runs, return an action if something must change, or None to skip.
Registration ties it together: coco.register_root_target_states_provider() takes your handler and returns a provider your user-facing API uses to declare target states.
At runtime the engine drives the loop: it collects everything your pipeline declared, calls reconcile() per key (passing NON_EXISTENCE for keys that disappeared, which is how cleanup happens), batches the returned actions by sink, executes them, and persists the new tracking records for next time. reconcile() itself must be non-blocking; all I/O lives in the sink.
## How change detection works
The reason a custom target gets incremental updates "for free" is that you never write an imperative sync loop. You describe the desired state and the handler reconciles it against what existed before. To detect changes without re-reading the whole target, the tracking record stores just enough to compare, typically a content fingerprint from the connectorkits.fingerprint utilities, not the full content. When the fingerprint of the desired state matches the stored record, reconcile() returns None and the item is skipped cheaply.
One subtlety: because an update can be interrupted, the engine may carry multiple possible previous states for an item until a run confirms which one is real. That's why reconcile() receives a *collection* of previous records, plus a prev_may_be_missing flag, and why the conservative skip condition checks that all of them match. It's also why actions must be idempotent: re-applying a write that already landed has no ill effect, so a resumed run converges to the correct target state regardless of where the prior one stopped.
## Example: export markdown files to local HTML
Let's walk through the same example the original post used: converting .md files to .html on disk, incrementally. The pipeline side is the files_transform example; it uses the built-in localfs target, and afterwards we'll build a simplified version of that file target ourselves.
### The pipeline
One processing component per markdown file: read, render, declare the HTML file that should exist.
```python
@coco.fn(memo=True)
async def process_file(file: FileLike, outdir: pathlib.Path) -> None:
html = _markdown_it.render(await file.read_text())
outname = "__".join(file.file_path.path.parts) + ".html"
localfs.declare_file(outdir / outname, html, create_parent_dirs=True)
@coco.fn
async def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:
files = localfs.walk_dir(
sourcedir,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
live=True,
)
await coco.mount_each(process_file, files.items(), outdir)
```
declare_file states what should exist; the connector writes it, overwrites it on change, and deletes the .html when the source .md disappears. No file I/O glue in the pipeline.
Markdown to HTML flow: walk_dir lists markdown files, mount_each fans out one processing component per file, each renders HTML and declares the output file, and the file target's handler reconciles fingerprints and writes only what changed.
### Build the file target yourself
Here is a simplified version of that localfs connector, following the custom target connector guide. First the types: an action describing one file operation, and a tracking record holding a content fingerprint.
```python
class _FileAction(NamedTuple):
path: pathlib.Path
content: bytes | None # None = delete
@dataclass(frozen=True, slots=True)
class _FileTrackingRecord:
fingerprint: bytes
```
The sink applies a batch of actions. Note every operation is idempotent (missing_ok, exist_ok):
```python
def _apply_actions(
context_provider: coco.ContextProvider, actions: Sequence[_FileAction]
) -> None:
for action in actions:
if action.content is None:
action.path.unlink(missing_ok=True)
else:
action.path.parent.mkdir(parents=True, exist_ok=True)
action.path.write_bytes(action.content)
_file_sink = coco.TargetActionSink[_FileAction, None].from_fn(_apply_actions)
```
The handler holds the reconciliation logic: delete when the state is gone, skip when every possible previous record matches the new fingerprint, write otherwise.
```python
class _FileHandler(coco.TargetHandler[_FileContent, _FileTrackingRecord]):
def __init__(self, base_path: pathlib.Path):
self._base_path = base_path
def reconcile(
self,
key: coco.StableKey,
desired_target_state: _FileContent | coco.NonExistenceType,
prev_possible_records: Collection[_FileTrackingRecord],
prev_may_be_missing: bool,
/,
) -> coco.TargetReconcileOutput[_FileAction, _FileTrackingRecord] | None:
path = self._base_path / key
if coco.is_non_existence(desired_target_state):
if not prev_possible_records and not prev_may_be_missing:
return None
return coco.TargetReconcileOutput(
action=_FileAction(path=path, content=None),
sink=_file_sink,
tracking_record=coco.NON_EXISTENCE,
)
target_fp = fingerprint_bytes(desired_target_state)
if not prev_may_be_missing and all(
prev.fingerprint == target_fp for prev in prev_possible_records
):
return None
return coco.TargetReconcileOutput(
action=_FileAction(path=path, content=desired_target_state),
sink=_file_sink,
tracking_record=_FileTrackingRecord(fingerprint=target_fp),
)
```
Registering the handler returns a provider, and a thin user-facing wrapper turns it into the declare_file call the pipeline uses:
```python
_provider = coco.register_root_target_states_provider(
"myproject.io/file", _FileHandler(base_path)
)
def declare_file(filename: str, content: bytes) -> None:
coco.declare_target_state(_provider.target_state(filename, content))
```
Containers (a directory whose files are children, a table whose rows are children) follow the same pattern one level deeper: the parent's sink returns a ChildTargetDef with a handler for the children. The connector guide covers that, along with attachments for auxiliary state like vector indexes.
### Run it
```sh
pip install -e .
cocoindex update main # catch-up: scan, sync, exit
cocoindex update -L main # live: keep watching for file changes
```
Add, modify, or remove files under data/ and only the changed files are reprocessed; a removed source's .html is deleted automatically. Live mode keeps the output continuously synchronized, useful for fast-changing content like internal wikis.
## From the v0 connector API
If you wrote a v0 custom target, here is where each piece went:
| v0 connector API | v1 equivalent |
| --- | --- |
| TargetSpec | Plain value types declared per target state |
| get_persistent_key() | The StableKey per state, with ContextKey for external resources |
| apply_setup_change() | The container target's reconcile() and sink |
| mutate() | Per-item reconcile() plus batched actions in the sink |
| Automatic tracking | A tracking record you define (usually a fingerprint) |
| prepare() | Handler construction, or context_provider at action time |
The v0 API split targets into setup (DDL-like) and data (DML-like) methods. The v1 model unifies both as target states reconciled the same way, which is what makes containers, children, and attachments compose: a table and its rows are the same concept at two levels.
## Best practices
- **Idempotency matters**: actions should be safe to run more than once (ON CONFLICT DO UPDATE, missing_ok=True, exist_ok=True), because interrupted runs re-apply.
- **Keep tracking records minimal**: a fingerprint beats storing content; the engine persists them per item across runs.
- **Handle multiple previous states**: skip only when all prev_possible_records match and prev_may_be_missing is false.
- **Use ContextKey for external resource identity**: put the stable logical name in the target state key, never hosts or credentials, so rotating a password doesn't orphan tracked state.
## When to build a custom target
Most pipelines use built-in connectors and never need one. Reach for a custom target connector when you integrate with a system CocoIndex doesn't cover, need custom change detection (such as content-based fingerprinting), or manage hierarchical state (containers with children). For a simple "write this somewhere" step with no cleanup requirement, a regular memoized function is often enough; the connector machinery earns its keep when you want the engine to track and clean up what it wrote.
If you build a connector for a system others use too, tell us about it: CocoIndex on GitHub.
---
# Index faces for visual search: your own Google Photos
URL: https://cocoindex.io/blogs/face-detection/
Date: 2025-07-24
Summary: Build a scalable face detection and recognition pipeline with CocoIndex: embed faces, structure for search, and export to a vector DB.
**CocoIndex** supports multi-modal processing natively:
it can process both text and images with the same programming model and observe in the same user flow (in **CocoInsight**).
In this blog, we’ll walk through a comprehensive example of building a scalable face recognition pipeline using CocoIndex.
We’ll show how to extract and embed faces from images, structure the data relationally,
and export everything into a vector database for real-time querying.
**CocoInsight** can now visualize identified sections of an image based on the bounding boxes and make it easier to understand and
evaluate AI extractions - seamlessly attaching computed features in the context of unstructured visual data.
If you find this tutorial helpful, we’d greatly appreciate it if you could ⭐ star
CocoIndex on GitHub.
## Use cases
- Photo search
- Face-based access control and surveillance
- Visual deduplication and identity detection
- Multimodal search involving people or facial identity
- Social graph analysis from photos
## What we will achieve
The photo taken of this conference's participants is sometimes entitled
*"The Most Intelligent Picture Ever Taken"*, for its depiction of the world's leading physicists gathered together in one shot
(Wikipedia).
Solvay Conference
Here’s what we want to accomplish:
- Detect all faces in the image and extract their bounding boxes
- Crop and encode each face image into a 128-dimensional face embedding
- Store metadata and vectors in a structured index to support queries like:
“Find all similar faces to this one” or “Search images that include this person”
Extracted faces
⭐ You can find the full code here.
Indexing Flow
Indexing Flow
1. We ingest a list of images.
2. For each image, we:
- Extract faces from the image.
- Compute embeddings for each face.
3. We export the following fields to a table in Postgres with PGVector:
- Filename, rect, embedding for each face.
## Core components
### Image ingestion
We monitor an images/ directory using the built-in LocalFile source. All newly added files are automatically processed and indexed.
```python
python
CopyEdit
@cocoindex.flow_def(name="FaceRecognition")
def face_recognition_flow(flow_builder, data_scope):
data_scope["images"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="images", binary=True),
refresh_interval=datetime.timedelta(seconds=10),
)
```
This creates a table with filename and content fields. 📂
Image Ingestion
You can connect it to your S3 Buckets (with SQS integration, example)
or Azure Blob store.
### Detect and extract faces
We use the face_recognition library under the hood, powered by dlib’s CNN-based face detector. Since the model is slow on large images, we downscale wide images before detection.
Extract faces
```python
@cocoindex.op.function(
cache=True,
behavior_version=1,
gpu=True,
arg_relationship=(cocoindex.op.ArgRelationship.RECTS_BASE_IMAGE, "content"),
)
def extract_faces(content: bytes) -> list[FaceBase]:
orig_img = Image.open(io.BytesIO(content)).convert("RGB")
# The model is too slow on large images, so we resize them if too large.
if orig_img.width > MAX_IMAGE_WIDTH:
ratio = orig_img.width * 1.0 / MAX_IMAGE_WIDTH
img = orig_img.resize(
(MAX_IMAGE_WIDTH, int(orig_img.height / ratio)),
resample=Image.Resampling.BICUBIC,
)
else:
ratio = 1.0
img = orig_img
# Extract face locations.
locs = face_recognition.face_locations(np.array(img), model="cnn")
faces: list[FaceBase] = []
for min_y, max_x, max_y, min_x in locs:
rect = ImageRect(
min_x=int(min_x * ratio),
min_y=int(min_y * ratio),
max_x=int(max_x * ratio),
max_y=int(max_y * ratio),
)
# Crop the face and save it as a PNG.
buf = io.BytesIO()
orig_img.crop((rect.min_x, rect.min_y, rect.max_x, rect.max_y)).save(
buf, format="PNG"
)
face = buf.getvalue()
faces.append(FaceBase(rect, face))
return faces
```
We transform the image content:
```python
with data_scope["images"].row() as image:
image["faces"] = image["content"].transform(extract_faces)
```
After this step, each image has a list of detected faces and bounding boxes.
Each detected face is cropped from the original image and stored as a PNG.
Extract faces
Sample Extraction:
Solvay Extraction
Sample Extraction:
Gates Jobs Extraction
### Compute face embeddings
We encode each cropped face using the same library. This generates a 128-dimensional vector representation per face.
```python
@cocoindex.op.function(cache=True, behavior_version=1, gpu=True)
def extract_face_embedding(
face: bytes,
) -> cocoindex.Vector[cocoindex.Float32, typing.Literal[128]]:
"""Extract the embedding of a face."""
img = Image.open(io.BytesIO(face)).convert("RGB")
embedding = face_recognition.face_encodings(
np.array(img),
known_face_locations=[(0, img.width - 1, img.height - 1, 0)],
)[0]
return embedding
```
We plug the embedding function into the flow:
```python
with image["faces"].row() as face:
face["embedding"] = face["image"].transform(extract_face_embedding)
```
After this step, we have embeddings ready to be indexed!
Embedding
### Collect and export embeddings
We now collect structured data for each face: filename, bounding box, and embedding.
```python
face_embeddings = data_scope.add_collector()
face_embeddings.collect(
id=cocoindex.GeneratedField.UUID,
filename=image["filename"],
rect=face["rect"],
embedding=face["embedding"],
)
```
We export to a Qdrant collection:
```python
face_embeddings.export(
QDRANT_COLLECTION,
cocoindex.targets.Qdrant(
collection_name=QDRANT_COLLECTION
),
primary_key_fields=["id"],
)
```
Now you can run cosine similarity queries over facial vectors.
CocoIndex supports a 1-line switch to other vector databases like Postgres.
## Query the index
You can now build facial search apps or dashboards. For example:
- Given a new face embedding, find the most similar faces
- Find all face images that appear in a set of photos
- Cluster embeddings to group visually similar people
Similarity Search
For querying embeddings, check out the Image Search project.
If you’d like to see a full example on the query path with image match, give it a shout at
our group.
## Support us
We’re constantly adding more examples and improving our runtime. If you found this helpful, please ⭐ star CocoIndex on GitHub and share it with others.
Thanks for reading!
Let us know what pipelines you’re building. We’d love to feature them.
---
# Index academic papers and extract metadata for AI agents
URL: https://cocoindex.io/blogs/academic-papers-indexing/
Date: 2025-07-09
Summary: Index academic papers with CocoIndex: read the first page of each PDF, extract title, authors, and abstract with an LLM, and embed them for semantic search.
The first page of a paper holds almost everything you'd want to query: title, authors, abstract. But it's locked in PDF prose. This walkthrough reads just that page, hands the text to an LLM with a strict schema, gets back clean typed JSON, and embeds the metadata so you can search papers by meaning. One PDF fans out into three Postgres tables, and CocoIndex keeps all three in sync as the folder changes.
The full code is in the CocoIndex v1 repo. A star on GitHub is appreciated if this helps.
## What we'll build
Each PDF flows through one processing component: slice out the first page, pull its text, extract metadata with the LLM, and embed the title and abstract. The component declares rows into three tables.
Indexing flow: a folder of PDF papers feeds a CocoIndex app; one processing component per file slices the first page, extracts title, authors, and abstract with an LLM, embeds the title and abstract chunks, and declares rows into three Postgres tables: paper_metadata, author_papers, and metadata_embeddings with pgvector.
The three tables serve three kinds of queries. paper_metadata holds one row per paper for lookups and joins. author_papers maps each author name to a filename, so "all the papers by Jeff Dean" is a plain SQL WHERE. metadata_embeddings holds one vector per title and per abstract chunk, for semantic search with pgvector.
## Define the models and rows
Two pydantic models describe what the LLM must return. Validation is the contract: a malformed response fails loudly instead of writing junk.
```python title=models.py
from pydantic import BaseModel, Field
class AuthorModel(BaseModel):
name: str
email: str | None = None
affiliation: str | None = None
class PaperMetadataModel(BaseModel):
title: str
authors: list[AuthorModel] = Field(default_factory=list)
abstract: str
```
The pydantic models are the LLM contract only. What lands in Postgres is described separately, by three dataclasses, one per table. Keeping the two layers apart lets the table shape differ from the LLM's: rows add filename and num_pages, and authors are stored as plain dicts. The embedding field is annotated with the embedder so CocoIndex infers the vector size:
```python title=main.py
import uuid
from dataclasses import dataclass
from typing import Annotated
from numpy.typing import NDArray
import cocoindex as coco
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@dataclass
class PaperMetadataRow:
filename: str
title: str
authors: list[dict[str, str | None]]
abstract: str
num_pages: int
@dataclass
class AuthorPaperRow:
author_name: str
filename: str
@dataclass
class MetadataEmbeddingRow:
id: uuid.UUID
filename: str
location: str
text: str
embedding: Annotated[NDArray, EMBEDDER]
```
The database pool and the embedder are created once, in the app lifespan:
```python title=main.py
import asyncpg
from cocoindex.connectors import postgres
PG_DB = coco.ContextKey[asyncpg.Pool]("paper_metadata_db")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(os.getenv("POSTGRES_URL")) as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
```
## Read the first page
A paper can be large. The Attention Is All You Need PDF in the sample folder is 2,215,244 bytes, and other arXiv PDFs run to dozens of pages. The metadata we want lives on page one, so we slice it out with pypdf and never send the rest anywhere:
```python title=main.py
import io
from pypdf import PdfReader, PdfWriter
@dataclass
class PaperBasicInfo:
num_pages: int
first_page: bytes
@coco.fn
def extract_basic_info(content: bytes) -> PaperBasicInfo:
"""Extract first page bytes and page count from a PDF."""
reader = PdfReader(io.BytesIO(content))
output = io.BytesIO()
writer = PdfWriter()
writer.add_page(reader.pages[0])
writer.write(output)
return PaperBasicInfo(num_pages=len(reader.pages), first_page=output.getvalue())
```
extract_basic_info slices page one out of the PDF and counts the pages: a 53-page, 557,854-byte arXiv PDF becomes a PaperBasicInfo with num_pages 53 and a 50,521-byte single-page first_page. Only that first page moves on to text extraction and the LLM.
A second small function pulls the text off that single page:
```python title=main.py
@coco.fn
def pdf_to_markdown(content: bytes) -> str:
"""Convert PDF bytes to text using pypdf."""
reader = PdfReader(io.BytesIO(content))
page_text = reader.pages[0].extract_text() if reader.pages else ""
return page_text or ""
```
pypdf keeps the example dependency-light. If your PDFs are heavier on layout, swap this function's body for Marker or Docling; the rest of the pipeline does not change.
## Extract metadata with an LLM
One call turns the first-page prose into typed fields. The system prompt pins the schema, response_format forces JSON, temperature=0 keeps it deterministic, and model_validate_json parses the response into the pydantic model:
```python title=main.py
import functools
from openai import OpenAI
LLM_MODEL = "gpt-4o"
@functools.cache
def openai_client() -> OpenAI:
return OpenAI()
@coco.fn
def extract_metadata(markdown: str) -> PaperMetadataModel:
"""Extract paper metadata from first-page text using an LLM."""
client = openai_client()
response = client.chat.completions.create(
model=LLM_MODEL,
messages=[
{
"role": "system",
"content": (
"You extract metadata from academic paper first pages. "
"Return only JSON with keys: title, authors, abstract. "
"authors is a list of {name, email, affiliation}. "
"Use null for missing fields."
),
},
{
"role": "user",
"content": markdown[:4000],
},
],
response_format={"type": "json_object"},
temperature=0,
)
content = response.choices[0].message.content
if not content:
raise RuntimeError("LLM returned empty content.")
return PaperMetadataModel.model_validate_json(content)
```
One LLM call turns first-page prose into typed fields: the Attention Is All You Need first page goes in, and a validated PaperMetadataModel comes out with the title string, eight authors with name and affiliation, and the abstract. Malformed JSON fails validation instead of writing junk to the tables.
## Process one paper into three tables
process_file is the whole per-paper pipeline: read, slice, extract, embed, declare. memo=True caches it, so an unchanged PDF is skipped on the next run:
```python title=main.py
from cocoindex.ops.text import CustomLanguageConfig, RecursiveSplitter
from cocoindex.resources.file import FileLike
_abstract_splitter = RecursiveSplitter(
custom_languages=[
CustomLanguageConfig(
language_name="abstract",
separators_regex=[r"[.?!]+\s+", r"[:;]\s+", r",\s+", r"\s+"],
)
]
)
@coco.fn(memo=True)
async def process_file(
file: FileLike,
metadata_table: postgres.TableTarget[PaperMetadataRow],
author_table: postgres.TableTarget[AuthorPaperRow],
embedding_table: postgres.TableTarget[MetadataEmbeddingRow],
) -> None:
content = await file.read()
basic_info = extract_basic_info(content)
first_page_md = pdf_to_markdown(basic_info.first_page)
metadata = extract_metadata(first_page_md)
metadata_table.declare_row(
row=PaperMetadataRow(
filename=str(file.file_path.path),
title=metadata.title,
authors=[a.model_dump() for a in metadata.authors],
abstract=metadata.abstract,
num_pages=basic_info.num_pages,
),
)
for author in metadata.authors:
if author.name:
author_table.declare_row(
row=AuthorPaperRow(
author_name=author.name,
filename=str(file.file_path.path),
),
)
title_embedding = await coco.use_context(EMBEDDER).embed(metadata.title)
embedding_table.declare_row(
row=MetadataEmbeddingRow(
id=uuid.uuid4(),
filename=str(file.file_path.path),
location="title",
text=metadata.title,
embedding=title_embedding,
),
)
abstract_chunks = _abstract_splitter.split(
metadata.abstract,
chunk_size=500,
min_chunk_size=200,
chunk_overlap=150,
language="abstract",
)
for chunk in abstract_chunks:
embedding_table.declare_row(
row=MetadataEmbeddingRow(
id=uuid.uuid4(),
filename=str(file.file_path.path),
location="abstract",
text=chunk.text,
embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
),
)
```
Run the numbers for one concrete paper. Attention Is All You Need has 8 authors, and its 1,139-character abstract splits into 3 chunks with these settings (sentence-aware separators, chunk size 500, overlap 150). So the function declares 13 rows: 1 metadata row, 8 author rows, and 4 embedding rows (the title plus the 3 chunks):
One paper unrolls into 13 rows: Attention Is All You Need, with 8 authors and a 1,139-character abstract that splits into 3 chunks, becomes 1 paper_metadata row, 8 author_papers rows, and 4 metadata_embeddings rows (title vector plus one per abstract chunk). Delete the PDF and all 13 rows are cleaned up.
The abstract is chunked because a long abstract blurs into a single vector; per-chunk embeddings keep search sharp. And everything is declarative: remove the PDF from the folder and all 13 rows across the three tables are cleaned up on the next run.
## Wire up the app
app_main mounts the three target tables, walks the folder of PDFs, and runs one process_file per file:
```python title=main.py
import pathlib
from cocoindex.connectors import localfs
from cocoindex.resources.file import PatternFilePathMatcher
PG_SCHEMA_NAME = "coco_examples_v1"
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
metadata_table = await postgres.mount_table_target(
PG_DB,
table_name="paper_metadata",
table_schema=await postgres.TableSchema.from_class(
PaperMetadataRow,
primary_key=["filename"],
),
pg_schema_name=PG_SCHEMA_NAME,
)
author_table = await postgres.mount_table_target(
PG_DB,
table_name="author_papers",
table_schema=await postgres.TableSchema.from_class(
AuthorPaperRow,
primary_key=["author_name", "filename"],
),
pg_schema_name=PG_SCHEMA_NAME,
)
embedding_table = await postgres.mount_table_target(
PG_DB,
table_name="metadata_embeddings",
table_schema=await postgres.TableSchema.from_class(
MetadataEmbeddingRow,
primary_key=["id"],
),
pg_schema_name=PG_SCHEMA_NAME,
)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
live=True,
)
await coco.mount_each(
process_file, files.items(), metadata_table, author_table, embedding_table
)
app = coco.App(
coco.AppConfig(name="PaperMetadataV1"),
app_main,
sourcedir=pathlib.Path("./papers"),
)
```
mount_table_target creates and manages each table: schema, idempotent upserts, and removal of rows whose source vanished. walk_dir(..., live=True) means the source can watch the folder; drop a new PDF in and it gets indexed without re-running anything.
## Run it
Start Postgres with pgvector (the repo ships a compose file), set POSTGRES_URL and OPENAI_API_KEY, and build the index. The example folder ships sample papers, including Attention Is All You Need, so it works out of the box:
```sh
docker compose -f ../../dev/postgres.yaml up -d
pip install -e .
cocoindex update main
```
The first run extracts and embeds everything; re-runs only process changed files. Add -L (cocoindex update -L main) to watch the folder and update the index live.
## Query the index
Semantic search embeds the query with the same model and asks Postgres for the nearest vectors; is pgvector's cosine distance:
```python title=main.py
async def query_once(pool, embedder, query: str, *, top_k: int = 5) -> None:
query_vec = await embedder.embed(query)
async with pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT filename, location, text, embedding <=> $1 AS distance
FROM "coco_examples_v1"."metadata_embeddings"
ORDER BY distance ASC
LIMIT $2
""",
query_vec,
top_k,
)
for r in rows:
score = 1.0 - float(r["distance"])
print(f"[{score:.3f}] {r['filename']} ({r['location']})")
print(f" {r['text']}")
```
```sh
python main.py "sequence models without recurrence"
```
Matches come back from both titles and abstract chunks, with the location column telling you which. The author index needs no embeddings at all:
```sql
SELECT filename FROM "coco_examples_v1"."author_papers"
WHERE author_name = 'Ashish Vaswani';
```
## Where to go next
The shape here generalizes past papers: any document family with a metadata-rich region (contracts, invoices, intake forms) can use the same slice-extract-validate-embed pipeline. Swap the source for Amazon S3 or Google Drive, swap the target for Qdrant, or feed the metadata tables to an agent. For full-text embedding of the whole PDF body, see text embeddings 101.
## Support us
We are constantly adding examples and improving the runtime. If this guide helped, please star CocoIndex on GitHub and let us know what you are building.
---
# CocoIndex Changelog 2025-07-07
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-07-07/
Date: 2025-07-07
Summary: CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.
In the past weeks, we've added support for an in-process API and convenient CLI options for setup / drop,
native support for EmbedText as a building block, major improvement to support codebase indexing and many core improvements over 10+ releases.
We are also very excited about the great suggestions from our users and community, and many of our changes come directly from user feedback.
Thank you ❤️!
Full changelog: v0.1.45...v0.1.57.
## We officially crossed 2k stars on GitHub! 🎉
CocoIndex crossed 2k stars last week with one day on GitHub Trending in Rust.
Thank you to everyone who contributed, starred the repo and shared the love!
Let’s make the best ETL framework for AI 🚀 !
CocoIndex GitHub Stars
## CocoInsight
We announced a major milestone on the CocoIndex companion - CocoInsight.
It has zero pipeline data retention and connects to your on-premise CocoIndex server for pipeline insights.
This makes data directly visible and makes it easy to develop ETL pipelines.
## Flexible API for flow backend setup / drop
CocoIndex automatically handles the flow setup - keeping everything in sync for the target stores (e.g., Qdrant, Postgres)
automatically without explicit schema setup, all inferred from the flow itself.
Automatic flow setup
As a follow-up, we've provided an in-process API and convenient CLI options for setup / drop.
Previously, automatic flow backend setup / drop was only supported by cocoindex setup and cocoindex drop CLI commands.
This was less flexible and convenient:
- This must happen as a separate process.
- setup works for all loaded flows, but doesn't work for a single flow.
These limitations came from simplification of our early implementation. We rebuilt the related logic and supported a lightweight in-process API for flow backend setup and drop.
With the new change, you can run
```python
flow1.setup() # Apply setup updates for flow1. It's noop if already up-to-date.
flow1.drop() # Drop the setup for flow1
# We also support doing it for all
cocoindex.setup_all()
cocoindex.drop_all()
```
With this, you can programmatically set up your flow in your code before using the update() API to build the index:
```python
flow1.setup()
flow1.update()
```
In addition, cocoindex update CLI starts to support a --setup option, which automatically pushes setup changes
before running the flow in the same CLI invocation.
```sh
cocoindex update --setup main
```
See the document for more details.
This is a major framework update. We've made significant improvements to state management,
including smarter handling of when to load global metadata, when to trigger setup, and how each component updates.
We're committed to taking care of the underlying infrastructure so developers can focus on what matters: the data and the logic.
## New building block: EmbedText
CocoIndex provides native builtins for different sources, targets and transformations, based on a standardized interface.
Components can be switched by one-line code change.
We are committed to making it convenient to plug in the best from the ecosystem seamlessly.
Building Blocks
EmbedText embeds text into a vector space using various LLM APIs that support text embedding.
It has native OpenAI, Gemini, and Voyage support.
You can view more about EmbedText in this documentation.
Example:
```python
text.transform(
cocoindex.functions.EmbedText(
api_type=cocoindex.LlmApiType.VOYAGE,
model="voyage-code-3",
)
)
```
## Optimization: skip source row reprocessing on unchanged content hash
CocoIndex supports incremental processing out-of-the-box that only reprocesses what's changed at minimum.
It can be at a source level, or during a transformation, e.g., a few chunks in a document have changed.
This improvement saves reprocessing cost when the content of a source row doesn't change.
Previously we used an *ordinal* (e.g. based on file modification time for file-based sources) to determine if there's content change.
Sometimes modification time changes without content change (e.g. Git always uses the current time after checkout),
and we want to skip processing in this case too.
## Major improvements on SplitRecursively function
We've made major improvements on native support for SplitRecursively (for chunking) building blocks.
- Holistically planning the way to chunk to minimize "cost", considering the following factors:
- AST structural level
- Literal styles (new lines, double new lines)
- Efficiency of overlap leverage
- Also for non-dividable elements (e.g. large comments, large strings), fall back to regex-based text chunking
- Add line/column to output of SplitRecursively
- Support customizing separators for the splitter
Check out this example of how to use SplitRecursively to index a codebase:
codebase-indexing.
## Union types
We’ve added support for union types for basic types.
Union types are supported in Python (e.g. str | int).
You can read more about union types here.
## NumPy type support
CocoIndex started to support NumPy numeric types and array types in CocoIndex functions. Specifically:
- numpy.int64: binds to *Int64*
- numpy.float32: binds to *Float32*
- numpy.float64: binds to *Float64*
- numpy.typing.NDArray[T], where T is any NumPy numeric type above: binds to *Vector*
See the document for more details.
## Supporting more LLM APIs
We’ve added LiteLLM (Proxy) and OpenRouter support. Read more here.
## New target store - Kuzu
CocoIndex now provides native support for Kuzu as a target graph data store.
This integration features a high performance knowledge graph stack with real-time updates.
You can read more here.
## Thanks to the community
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @lemorage
Thanks to @lemorage for the contributions! CocoIndex has received a series of high-quality PRs from him, and we truly appreciate his excellent work and passion for the project. @lemorage has also been helping us keep the repo healthy as a first responder to breaking changes. Thank you so much!
- use theme aligning with user system preferences #572
- feat: add NumPy array support for vector representations #586
- fix: remove unsupported type casting in integer vector handling #619
- feat: support scalar NumPy value encodings #620
- ops: add pre-commit and hooks for code checking #641
### @vumichien
Thanks to @vumichien for the contributions, especially PR #629, which tackles some complex and core aspects of CocoIndex. We really appreciate the thoughtful work and sustained effort over the past few weeks.
- chore: update Rust version to 1.86 and remove unused as_any methods #597
- feat(cli): add force option to setup and drop commands for bypassing confirmation prompts #602
- feat: allow optional database configuration #608
- feat: add content hash support for change detection in source processing #629
### @chardoncs
Thanks to @chardoncs for the contributions! We appreciate the work on supporting union types for basic types.
- feat: support union type for basic types #510
### @par4m
Thanks to @par4m for the contributions! We appreciate the work on adding LiteLLM and OpenRouter support, which gives users more flexibility to pick their favorite models.
- feat(llm): Add LiteLLM (Proxy) Support #630
- feat(llm): Add OpenRouter Support #635
### @cijiugechu
Thanks to @cijiugechu for the contributions! We appreciate the work on performance optimizations.
- chore(example): replace deprecated startup event with lifespan #590
- chore(server): upgrade to axum 0.8 #591
- perf(google_drive): generate EXPORT_MIME_TYPES at compile time #593
### @theparthgupta
Thanks to @theparthgupta for the contributions! We appreciate the work on making sentence_transformers an optional dependency, which helps reduce the base package size and gives users more flexibility.
- feat: make sentence_transformers an optional dependency #674
### @dubin555
Thanks to @dubin555 for the contributions! We appreciate the work on improving logging information.
- chore: add more info in log #614
### @TwistingTwists
Thanks to @TwistingTwists for the contributions! We appreciate the work on improving unit tests for Python values.
- fix: unit tests for python values #452
### @SaiSakthidar
Thanks to @SaiSakthidar for the contributions! We appreciate the work on migrating custom UUID conversion logic to the pyo3.uuid package, which improves code maintainability.
- feat: Migrated custom UUID conversion logic to pyo3.uuid package #663
## Support our work
We are constantly improving CocoIndex, and more features are coming soon!
Stay tuned and follow us by starring our GitHub repo.
---
# Introducing CocoInsight
URL: https://cocoindex.io/blogs/cocoinsight/
Date: 2025-06-24
Summary: Introducing CocoInsight, a data lineage and observability tool that lets you inspect, trace, and debug every step of a CocoIndex pipeline in real time.
From day zero, we envisioned **CocoInsight** as a fundamental companion to CocoIndex,
not just a tool, but a philosophy:
making data explainable, auditable, and actionable at every stage of the data pipeline with AI workloads.
CocoInsight has been in private beta for a while, and it is one of the most loved features for our users building ETL with coco,
with a significant boost to developer velocity, and lowering the barrier to entry for data engineering.
We are officially **launching CocoInsight** today - it has zero pipeline data retention and connects to your on-premise
CocoIndex server for pipeline insights. This makes data directly visible and makes it easy to develop ETL pipelines.
The "zero data retention" point is worth stressing: CocoInsight renders the UI, but the pipeline data it shows never leaves your machine. The tool talks to the CocoIndex server you run locally, reads the dataflow and the per-step data from there, and keeps it there. Nothing about your documents, embeddings, or extracted entities is uploaded — it's an inspector pointed at a server you own, not a hosted copy of your data. That property is what makes it usable on real, sensitive workloads rather than only on toy examples, and it's part of why it became one of the most loved features among the developers building on top of CocoIndex (10,000+ GitHub stars).
## Getting started
Start using it by running:
```sh
cocoindex server -ci main
```
for your cocoindex projects.
## Overview
This is an example view for CocoInsight:
- right panel is dataflow, and
- left panel is step-by-step data preview. Each field is tied to an input or output of a step in the dataflow transformation.
CocoInsight Panels
## How do I inspect data lineage?
You could click on any field (either in data flow or data preview),
or any transformation step in the dataflow to inspect lineage - to understand where the data comes from.
Inspect Lineage
The clicked element will be set to a purple color, as the element being inspected.
- Visibility:
- Direct data/ops with transitive dependency (upstream or downstream) will stay in view.
- Data/ops unrelated to the current selected element will be dimmed.
- Color:
- Direct upstream data dependency (exact fields) will be colored blue.
- Direct downstream data output (exact fields) will be colored green.
Let's walk through some simple examples of how these AI pipelines work.
You don't need to know how to write code; you just need to make sense of the spreadsheet 😊.
## Codebase indexing example
1. Ingest files, which outputs file names and contents.
Ingest files for codebase
2. Take the filename and extract extension.
Extract extension
3. Take the content (source code) and extension (language, e.g., .py) to do split based on code boundaries
with Tree-sitter.
Split code
You could further click on each chunk of a document to expand the details of the chunks.
Code chunk details
## Knowledge graph example
In this example, we process a list of files and generate a knowledge graph with documents and entities as nodes,
and relationships between document/entity and entity/entity.
Some key steps:
1. Use LLM to summarize a document.
Knowledge Graph LLM Summary
2. Use LLM to extract entities and relationships between entities.
Knowledge Graph LLM Relation Extraction
Click on any relationship "row" to drill into the child table.
relationship
## How it works
At the core of **CocoIndex**, both **data** and **data operations** are first-class citizens.
Because of this pure dataflow foundation, **CocoIndex offers full observability by default**:
- Before/after of the data are available at every transformation node.
- Every output field can be traced back to the exact set of input fields and operations that created it.
- **Lineage is first-class**, not as metadata bolted on afterward, but as a structural property of how data is defined and transformed in the system.
This is why lineage in CocoInsight is exact rather than approximate. Because the engine already knows, for every output field, the precise set of input fields and operations that produced it, the tool doesn't have to reconstruct or guess those edges — it reads them straight from the dataflow definition. That's what lets you click a single chunk's embedding and have the view highlight its exact upstream (the chunk text, the split step, the source file) in blue and its downstream consumers in green, while dimming everything unrelated. The relationships you're inspecting are the same relationships the engine uses to run the pipeline.
The same structure makes the *state* you see live rather than a static trace. Each field in the left panel is bound to the actual input or output of a step, so as the pipeline runs against your local server you're looking at the real before-and-after values at each node — the source content, the extracted extension, the chunk boundaries, the LLM summary — not a rendering of what the code is supposed to do. Drilling into a relationship row to expand its child table, or into a chunk to see its details, is reading the engine's own data, one step at a time.
This lineage model is not just useful for debugging.
It enables features like incremental processing, intelligent caching, and transformation-level explainability,
all out of the box.
While CocoIndex is architecturally a dataflow engine,
its user experience is deeply inspired by spreadsheets. Just like in a spreadsheet:
- Values of cells are derived from others through clearly visible formulas or expressions.
- You can visually inspect how data looks before and after each transformation, cell by cell.
- There’s no implicit global state, and every value can be explained in terms of its formula and input values.
- Once the value of a source cell changes, we automatically update derived cell values based on formulas with minimum reprocessing.
This spreadsheet-inspired paradigm is more than a UI choice: it’s a cognitive model.
It bridges the gap between low-code users and developers,
allowing anyone familiar with spreadsheets to reason about data transformations intuitively.
We have lots of features planned for CocoInsight 😎, including query debugging, stats, and more.
Stay tuned and join our Discord for any questions.
---
# Flow-based schema inference for Qdrant
URL: https://cocoindex.io/blogs/schema-inference-for-qdrant/
Date: 2025-06-08
Summary: CocoIndex sets up Qdrant collections automatically by inferring the target schema from your indexing flow: no manual config, vector sizes kept in sync.
CocoIndex supports Qdrant natively. The integration features a high-performance Rust stack with incremental processing end to end for scale and data freshness.
🎉 We just rolled out our latest change that handles **automatic target schema setup** with Qdrant from the CocoIndex indexing flow.
That means developers don’t need to do any schema setup, including setting up tables, field types, keys, and indexes for target stores. The setup is the result of schema inference from the CocoIndex flow definition.
It is already supported with native integration with Postgres, Neo4j, and LanceDB.
This allows for more seamless operation between the indexing and target stores.
## No more manual setup
Previously, users had to manually create the collection before indexing:
```sh
curl -X PUT 'http://localhost:6333/collections/image_search' \
-H 'Content-Type: application/json' \
-d '{
"vectors": {
"embedding": {
"size": 768,
"distance": "Cosine"
}
}
}'
```
With the new change, users don’t need to do any manual collection management.
## How it works
### Flow definition
Following the dataflow programming model, the user defines a flow,
where every step has output data type information, and the next step takes in data type information.
See an example
(~100 lines of Python end to end).
In short, it can be presented as the following lineage graph.
flow
In the declarative dataflow as above:
*Target = Formula (Source)*
It implies both data and the expected target schema.
A single flow definition drives both data processing (including change handling) and target schema setup, providing a single source of truth for both data and schema.
A similar way to think about it is like type systems inferring data type from operators and inputs: type inference (for example, Rust).
In the indexing flow, exporting embeddings and metadata directly to Qdrant is all you need.
```python
doc_embeddings.export(
"doc_embeddings",
cocoindex.storages.Qdrant(collection_name=QDRANT_COLLECTION),
primary_key_fields=["id"],
)
```
In this example,
qdrant flow
As part of the Qdrant schema setup, it is necessary to specify the vector size for embedding fields.
Qdrant only needs a schema for vector fields, including the vector size and distance.
Other fields are not part of the schema.
The vector name and size need to be consistent with the flow, and users need to maintain them manually otherwise.
When using CocoIndex, the vector size is decided by the embedding model.
For example:
1. At the SentenceTransformerEmbed transformation step,
the data field embedding has Vector[float, 384] type.
It is automatically generated because we used SentenceTransformerEmbed
with the all-MiniLM-L6-v2 model, which has 384 dimensions.
2. When we added it to the doc_embeddings collector, the datatype of field embeddings was carried over.
3. doc_embeddings collector exports to Qdrant, and the schema setup is derived consistently in a robust way.
If you have multiple fields with different embedding models,
the vector size will be different for each embedding field.
It'll end up with multiple different Named Vectors
in Qdrant, with different sizes.
CocoIndex always automatically handles the schema, no matter how many fields/vectors are involved in your flow,
and you can just focus on the transformation logic.
### Setup and update
To start a CocoIndex process, users need to first run the setup, which covers all the necessary setup for any backends needed.
```sh
cocoindex setup main
```
cocoindex setup
- Creates new backends for the schema setup, like tables/collections/etc.
- Alters existing backends with schema changes: it’ll try to do a non-destructive update if possible, e.g. primary keys don’t change and target storage supports in-place schema update (e.g. ALTER TABLE in Postgres); otherwise, drop and recreate.
- Drops stale backends.
Developers then run
```sh
cocoindex update main [-L]
```
to start an indexing pipeline (-L for long-running).
If you’ve made logic updates that require the schema on the target store to be updated, don’t worry.
When you run cocoindex update again after the logic update, CocoIndex will infer the schema for the target store.
It requires a cocoindex setup to push the schema to the target store, which will notify you in the CLI.
As a choice of design, CocoIndex won’t update any schema without your notice, as some schema updates may involve destructive changes.
For example, in the example above, if users change the embedding model, the vector size may change.
cocoindex setup will drop the previous collection and create a new one, and in the next cocoindex update run,
values will be populated.
Here the cached intermediate computation data will be reused, so it'll be a lot faster than building the index from scratch.
Note that Qdrant doesn't support ALTER TABLE like most relational databases, so it's a drop-and-recreate.
### Drop a flow
To drop a flow, you'd run:
```sh
cocoindex drop main
```
cocoindex drop drops the backends when dropping the flow.
All backend entities for the target stores, such as a PostgreSQL table or a Qdrant collection, are owned by the flow as derived data, so they will be dropped too.
## Why automatic target schema inference?
The question should really be, why not?
The traditional way is users fully figure out *when* and *how* to set up/update the target schema themselves, including the specific schema. Indexing flows often span multiple systems. For example:
On the target store:
- Vector databases (PGVector, Qdrant, etc.)
- Relational databases (PostgreSQL)
- Graph databases (Neo4j, Kuzu, etc.)
The data types you're outputting and your target schema must match up.
If there’s any internal state tracking, e.g., in the case of incremental processing:
- Internal tables (state tracking)
It’s tedious and painful to do this manually, as all of these systems must agree on schema and structure. This typically requires:
- Manual setup and syncing of schemas.
- Tight coordination between developers, DevOps, and data engineers: people writing the code may not be the same people deploying / running it in an organization.
- Debugging misalignments between flow logic and storage layers.
- Production rollout is typically stressful.
Any additional moving parts to the indexing pipeline system add friction: any mismatch between the logic and the storage schema could result in silent failures or subtle bugs.
- In some cases, it’s not silent failures. The failure should be obvious, e.g. if users forget to create a table or collection, it’ll just error out when writing to the target. In this case, the way to figure out the exact schema/configuration for the target is still subtle though.
- Some other scenarios can lead to non-obvious issues, i.e. out of sync between storage for internal states and the target. e.g. users may drop the flow and recreate, but not do so for the target; or drop and recreate the target, but not do so for the internal storage. Then they’re out of sync and there will be hard-to-debug issues. The gist is, a pipeline usually needs multiple backends, and it can be error-prone to keep them in sync manually.
Continuous changes to a system introduce persistent pains in production. Every time a data flow is updated, the target schema must evolve alongside, making it *not* a one-off tedious process, but an ongoing source of friction.
In real-world data systems, new fields often need indexing, old ones get deprecated, and transformations evolve. If a type changes, the schema must adapt. These shifts magnify the complexity and underscore the need for more resilient, adaptable infrastructure.
Following the dataflow programming model, every step is derived data all the way to the end. Indexing infrastructure requires data consistency between the indexing pipeline and target stores, and the fewer loose ends, the easier and more robust it will be.
## Our vision: declarative, flow-based indexing
When we started CocoIndex, our vision was to allow developers to define data transformation and indexing logic declaratively, and CocoIndex does the rest. One big step toward this is **automatic schema setup**.
Coco does the rest
We’re committed to taking care of the underlying infrastructure, so developers can focus on what matters: the data and the logic. We are serious when we say, you can have a production-ready data pipeline for AI with ~100 lines of Python code.
If you’ve ever struggled with keeping your indexing logic and storage setup in sync, we’ve been there. Let us know what you’d love to see next.
---
# CocoIndex + Kuzu: Real-time knowledge graph with Kuzu
URL: https://cocoindex.io/blogs/kuzu-integration/
Date: 2025-06-03
Summary: Build a real-time knowledge graph with Kuzu as a native CocoIndex target: incremental updates, high-performance graph queries.
CocoIndex now provides native support for Kuzu as a target graph data store.
This integration features a high-performance knowledge graph stack with real-time updates.
## What is Kuzu?
Kuzu is a graph database that is designed to be fast, scalable, and easy to use.
We love Kuzu because it is highly performant, lightweight, and open source.
Kuzu has been officially archived as per their announcement.
CocoIndex is an ultra performant real-time data transformation framework.
With a dataflow programming model, CocoIndex simplifies building and maintaining knowledge graphs with continuous source updates.
You can read the official CocoIndex Documentation for Property Graph Targets here.
We understand preparing data is highly use-case based, and there is no one-size-fits-all solution.
We take the composition approach. Instead of you building everything, we provide tons of native building blocks.
By standardizing the interface, we make it easier to plug in and play with a 1-line code switch, as simple as assembling the building blocks.
If you are using CocoIndex to build your knowledge graph, you can use Kuzu as a target graph data store.
## How to map to Kuzu in CocoIndex
The GraphDB interface in CocoIndex is standardized. If you are already using Neo4j, you just need to switch the configuration to export to Kuzu as below.
CocoIndex supports exporting to Kuzu through its API server. You can bring up a Kuzu API server locally by running:
```sh
KUZU_DB_DIR=$HOME/.kuzudb
KUZU_PORT=8123
docker run -d --name kuzu -p ${KUZU_PORT}:8000 -v ${KUZU_DB_DIR}:/database kuzudb/api-server:latest
```
In your CocoIndex flow, you need to add the Kuzu connection spec to your flow.
```python
kuzu_conn_spec = cocoindex.add_auth_entry(
"KuzuConnection",
cocoindex.storages.KuzuConnection(
api_server_url="http://localhost:8123",
),
)
```
## What does it look like to build an indexing flow with CocoIndex + Kuzu
A CocoIndex knowledge graph example that got the most love is to build a knowledge graph with an LLM; here is a detailed step-by-step blog.
In the project, we process a list of documents, and use an LLM to extract relationships between the concepts in each document.
Knowledge Graph Extraction
We will generate two kinds of relationships from the documents:
1. Relationships between subjects and objects. E.g., "CocoIndex supports Incremental Processing"
2. Mentions of entities in a document. E.g., "core/basics.mdx" mentions CocoIndex and Incremental Processing.
The indexing flow looks like this for Kuzu:
Knowledge Graph Indexing Flow
The code is available here.
1. Ingest the documents into CocoIndex
2. Process the documents, for each document:
1. Map document nodes: Use an LLM to generate a summary, and map the documents to Graph nodes in Kuzu.
2. Map relationship nodes: Use an LLM to extract relationships, and export the relationships to Kuzu.
Notably, it only takes ~200 lines of Python to have a production-ready knowledge graph, including class definitions, prompts, and configs.
To highlight how the relationship extraction works, you will define a Python class for structured extraction.
```python
@dataclasses.dataclass
class Relationship:
"""
Describe a relationship between two entities.
Subject and object should be Core CocoIndex concepts only, should be nouns. For example, `CocoIndex`, `Incremental Processing`, `ETL`, `Data` etc.
"""
subject: str
predicate: str
object: str
```
If you have a predefined set of ontologies, you can skip the entity extraction and use existing entities.
Call a transformation in the flow to extract the relationships from the document.
```python
with data_scope["documents"].row() as doc:
# extract relationships from document
doc["relationships"] = doc["content"].transform(
cocoindex.functions.ExtractByLlm(
llm_spec=cocoindex.LlmSpec(
# Supported LLM: https://cocoindex.io/docs/ops/litellm/
api_type=cocoindex.LlmApiType.OPENAI,
model="gpt-4o",
),
output_type=list[Relationship],
instruction=(
"Please extract relationships from CocoIndex documents. "
"Focus on concepts and ignore examples and code. "
),
)
)
```
You could use CocoInsight to verify each pair of the relationships.
CocoInsight
and then collect the relationship using the entity_relationship collector.
```python
with doc["relationships"].row() as relationship:
# relationship between two entities
entity_relationship.collect(
id=cocoindex.GeneratedField.UUID,
subject=relationship["subject"],
object=relationship["object"],
predicate=relationship["predicate"],
)
```
CocoIndex follows a dataflow programming model.
Rather than defining data operations like creations, updates or deletions, developers only need to focus on transformations or formulas based on source data.
The framework takes care of the data operations such as when to create, update, or delete.
Once you have collected the relationships, you can directly map them to Kuzu as below.
```python
entity_relationship.export(
"entity_relationship",
cocoindex.storages.Kuzu(
connection=conn_spec,
mapping=cocoindex.storages.Relationships(
rel_type="RELATIONSHIP",
source=cocoindex.storages.NodeFromFields(
label="Entity",
fields=[
cocoindex.storages.TargetFieldMapping(
source="subject", target="value"
),
],
),
target=cocoindex.storages.NodeFromFields(
label="Entity",
fields=[
cocoindex.storages.TargetFieldMapping(
source="object", target="value"
),
],
),
),
),
primary_key_fields=["id"],
)
```
Amazingly, while working on this Kuzu example, I had a previous flow that I ran locally with Neo4j.
It was instant to export to Kuzu. CocoIndex is based on incremental processing, and if you have already run this flow before and just switched targets, the intermediate transformation results can be reused.
To run the Kuzu Explorer, an open source UI for Kuzu, you need to first bring down the Kuzu API server.
And then you can run the following command to start the Kuzu Explorer:
```sh
KUZU_EXPLORER_PORT=8124
docker run -d --name kuzu-explorer -p ${KUZU_EXPLORER_PORT}:8000 -v ${KUZU_DB_DIR}:/database -e MODE=READ_ONLY kuzudb/explorer:latest
```
We could then access the explorer at http://localhost:8124. We could run a Cypher query to explore the graph.
```cypher
MATCH p=()-->() RETURN p
```
Kuzu Explorer
## Support us
We are constantly improving, and more features and examples are coming soon. If this article is helpful, please drop us a star ⭐ at GitHub to help us grow.
Thanks for reading!
---
# CocoIndex Changelog 2025-05-31
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-05-31/
Date: 2025-05-31
Summary: CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.
In the past weeks, we've added support for Amazon S3 as a native data source, updated query handling,
and made many core improvements in performance and stability over 15 releases.
We're all in on building the best real-time incremental data framework, and we couldn’t be more excited.
Full changelog: v0.1.30...v0.1.44.
## Amazon S3 and Amazon Simple Queue Service (SQS) as native data source
Amazon S3 Integration
CocoIndex now natively supports Amazon S3 as a data source, allowing seamless integration with your storage.
Combined with support for AWS Simple Queue Service (SQS), CocoIndex enables true real-time,
incremental processing of data as it's updated in S3.
CocoIndex processes only S3 files that have been newly added or updated,
reducing unnecessary compute cycles and improving overall system efficiency.
Incremental processing focuses on handling only new or modified data since the previous run, rather than reprocessing the entire dataset every time.
It becomes essential when working with large-scale data or when up-to-date information is critical.
- Read more about this integration in the blog post.
- Get started now with the example.
- Documentation: Amazon S3 as a data source
## Query stack
On the query side, CocoIndex has removed support for SimpleQueryHandler.
We made this decision because:
- Flexibility - users directly talk to the database to have maximum control. Users often have specific requirements for query handling, such as custom filtering and search logic.
- Many databases already have optimized query implementations with their own best practices
- The query space has excellent solutions for querying, reranking, and other search-related functionality.
And we’re doubling down on what we do best: real-time indexing + transformation.
Here are some examples with query server:
- Postgres with psycopg
- Qdrant
## Transform flow
When building vector indexes and querying against them, you often need to share transformation logic between indexing and querying operations.
For instance, the embedding computation must remain consistent across both processes.
In this case, you can
1. Extract a sub-flow with the shared transformation logic into a standalone function. E.g.,
```python
def text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:
return text.transform(
cocoindex.functions.SentenceTransformerEmbed(
model="sentence-transformers/all-MiniLM-L6-v2"))
```
2. When you're defining your indexing flow, you can directly call the function. E.g.,
```python
with doc["chunks"].row() as chunk:
chunk["embedding"] = text_to_embedding(chunk["text"])
```
3. At query time, you usually want to directly run the function with specific input data, instead of letting it be called as part of a long-lived indexing flow.
To do this, declare the function as a transform flow, by decorating it with @cocoindex.transform_flow().
```python
@cocoindex.transform_flow()
def text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:
return text.transform(
cocoindex.functions.SentenceTransformerEmbed(
model="sentence-transformers/all-MiniLM-L6-v2"))
```
So that you can directly call it with specific input data.
```python
print(text_to_embedding.eval("Hello, world!"))
```
Learn more about transform flow in the documentation.
## New standalone CLI
CocoIndex now supports a standalone CLI, with cleaner CLI commands and tons of improvements.
You have two ways to launch CocoIndex, see the Documentation to launch CocoIndex for more details.
- Use CocoIndex CLI. It's handy for most routine indexing building and management tasks.
It will load settings from environment variables, either already set in your environment, or specified in a .env file.
Check out the CLI documentation for more details.
For example:
```sh
cocoindex setup main
```
```sh
cocoindex update main
```
- Call CocoIndex functionality from your own Python application or library.
For example:
```python
from dotenv import load_dotenv
import cocoindex
load_dotenv()
cocoindex.init()
```
It's needed when you want to leverage CocoIndex support for query, or have your custom logic to trigger indexing, etc.
Check out the full example here.
## Support app namespace
CocoIndex now supports application namespaces, making it easier to manage flows across different environments.
This feature allows you to prefix flow names with a namespace, helping to organize and separate flows between environments like staging and production.
You can set the namespace using the COCOINDEX_APP_NAMESPACE environment variable.
For example, if you set COCOINDEX_APP_NAMESPACE=Staging and define a flow with @cocoindex.flow_def(name="TextEmbedding"), the final flow name will be Staging.TextEmbedding.
This is particularly useful when:
- Managing flows across different environments
- Sharing flows within a team
- Organizing flows by project or application
You can also programmatically access the current namespace using cocoindex.get_app_namespace() to use it in your own code.
Learn more about this feature in the documentation.
## Pretty print flow spec / schema
CocoIndex now supports showing the flow spec and schema in a more readable format with cocoindex show.
For example, to print out the flow spec in this document to knowledge graph project,
you can run:
```sh
cocoindex show main
```
CocoIndex show with flow spec
It prints out
- the lineage of the flow from data source to the final output.
- the global data schema of the flow
## Type system updates
### TimeDelta type
CocoIndex now supports the TimeDelta type, which is a duration of time.
Learn more about the type system in the documentation.
## Examples and tutorials
### Real-time semantic search from files in S3
- Tutorial: Real-time semantic search from files in S3
- Code example: Amazon S3 Embedding
### Image search with vision model
- Tutorial: Image search with Vision model
- Code example: Image Search
### Text search with FastAPI server on Docker Compose
- Code example: Docker Compose for fast api server
## Thanks to the community!
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @lemorage
Thanks to @lemorage for the contributions!
CocoIndex has received a list of high quality PRs from him, and we really appreciate his work.
#### CLI:
- feat: create standalone CLI for CocoIndex #485
- feat(pyo3): implement flow spec pretty print and add verbose mode #442
- feat(cli): enhance flow display with schema rendering support#427
- feat(cli): make flow name retrieval efficient #414
#### DataTypes
- feat: add support for TimeDelta type in Python and Rust #497
- feat: support namedtuple for struct types #462
- feat(yaml): store variant name in tuple and struct variants#444
### @AbdelRahmanYaghi
Created an example of docker compose which runs pg17 along with a simple fastapi endpoint for querying text embeddings. 432
### @par4m
Created an example of image search with Vision model 447
## Full changelog
Review the detailed changelog here.
We are constantly improving CocoIndex, and more features are coming soon!
Stay tuned and follow us by starring our GitHub repo.
---
# Incremental ETL on Amazon S3 with CocoIndex
URL: https://cocoindex.io/blogs/s3-incremental-etl/
Date: 2025-05-29
Summary: Index Markdown from Amazon S3 into Postgres pgvector with CocoIndex v1: incremental processing re-embeds only the files and chunks that changed.
CocoIndex has native support for Amazon S3 as a data source. This post builds a small pipeline that reads Markdown files from an S3 bucket, chunks and embeds them, and writes the vectors to a Postgres pgvector table for semantic search.
The point of the example is incremental processing: adding one object to the bucket embeds one object, not the whole bucket. It is Semantic Search 101 with the source swapped from a local folder to S3; everything downstream is the same.
The full project is in the CocoIndex v1 repo.
## Why incremental processing
Incremental processing processes only the data that changed since the last run, instead of reprocessing everything. It matters most in three situations.
**High freshness requirements.** For user-facing search, stale results are a real problem: when someone updates a document, they expect it reflected in search right away. Feed a stale result into an AI agent and the model answers from out-of-date information, often without anyone noticing.
**Expensive transforms.** When each item goes through an embedding model or an LLM, the transform costs far more than the retrieval. Recomputing unchanged items burns money for no benefit.
**Large datasets.** With terabytes in S3, reprocessing everything on each new file is impractical on time, cost, and compute. Incremental processing keeps the work proportional to the change, not the corpus.
A useful way to think about it: if T is the staleness you can tolerate, and you do not want to recompute the whole corpus every T, you need incremental processing.
## How CocoIndex processes only what changed
CocoIndex tracks each unit of work by a fingerprint of its input and the transform code. On the next run, anything whose fingerprint is unchanged is skipped, and its downstream rows are left untouched.
The granularity is finer than the file. If a Markdown file has many chunks and you edit one paragraph, only the chunks that actually changed are re-embedded; the rest reuse their cached vectors:
Chunk-level incremental processing: when one Markdown file is edited, CocoIndex diffs its chunks and re-embeds only the two that changed, reusing cached embeddings for the four unchanged chunks, then upserts just the changed rows into pgvector.
That is where the savings come from. The expensive embedding calls run only for content CocoIndex has not seen, and the vector store ends up consistent either way.
## The indexing flow
The flow reads Markdown files from S3, and for each file splits it into chunks, embeds each chunk, and declares a row in the vector table:
Indexing flow: an Amazon S3 bucket holds Markdown files; one processing component per file splits it into chunks, embeds each chunk, and declares a row that CocoIndex syncs into a Postgres pgvector table.
Each file is handled by its own processing component, so change detection and parallelism are per file.
## Define the pipeline
Shared resources (the Postgres pool, the S3 client, and the embedder) are provided once in the app lifespan and read anywhere with coco.use_context:
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import amazon_s3, postgres
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
PG_DB = coco.ContextKey[asyncpg.Pool]("s3_embedding_db")
S3_CLIENT = coco.ContextKey[AioBaseClient]("s3_client")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
session = aiobotocore.session.get_session()
async with session.create_client("s3") as s3_client:
builder.provide(S3_CLIENT, s3_client)
yield
```
detect_change=True on the embedder means swapping the model re-runs only the embedding step, not the whole pipeline.
The target row is a plain dataclass. The embedding column is annotated with the embedder so CocoIndex knows the vector dimension:
```python title=main.py
@dataclass
class DocEmbedding:
id: int
filename: str
chunk_start: int
chunk_end: int
text: str
embedding: Annotated[NDArray, EMBEDDER]
```
## Chunk, embed, declare
Each chunk becomes one row. process_chunk embeds the chunk text and declares the row; CocoIndex works out whether that is an insert or an update:
```python title=main.py
@coco.fn
async def process_chunk(
chunk: Chunk,
filename: str,
id_gen: IdGenerator,
table: postgres.TableTarget[DocEmbedding],
) -> None:
table.declare_row(
row=DocEmbedding(
id=await id_gen.next_id(chunk.text),
filename=filename,
chunk_start=chunk.start.char_offset,
chunk_end=chunk.end.char_offset,
text=chunk.text,
embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
),
)
```
process_file reads one S3 file, splits it into chunks, and maps process_chunk over them. memo=True caches per-file work, so an unchanged file is skipped entirely:
```python title=main.py
_splitter = RecursiveSplitter()
@coco.fn(memo=True)
async def process_file(
file: amazon_s3.S3File,
table: postgres.TableTarget[DocEmbedding],
) -> None:
text = await file.read_text()
chunks = _splitter.split(
text, chunk_size=2000, chunk_overlap=500, language="markdown"
)
id_gen = IdGenerator()
await coco.map(process_chunk, chunks, file.file_path.path.as_posix(), id_gen, table)
```
## Wire up the app
app_main mounts the pgvector table target, lists the Markdown objects in the bucket, and fans out one process_file component per file with mount_each:
```python title=main.py
@coco.fn
async def app_main() -> None:
target_table = await postgres.mount_table_target(
PG_DB,
table_name=TABLE_NAME,
table_schema=await postgres.TableSchema.from_class(
DocEmbedding, primary_key=["id"]
),
pg_schema_name=PG_SCHEMA_NAME,
)
client = coco.use_context(S3_CLIENT)
files = amazon_s3.list_objects(
client,
S3_BUCKET,
prefix=S3_PREFIX,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
)
await coco.mount_each(process_file, files.items(), target_table)
app = coco.App(
coco.AppConfig(name="AmazonS3Embedding"),
app_main,
)
```
amazon_s3.list_objects returns the objects under the prefix; files.items() yields (key, file) pairs, and mount_each gives each file a stable component path keyed on its S3 path.
## Set up the S3 bucket
Follow the Amazon S3 connector docs to create a bucket and grant read access. The pipeline needs read-only S3 permissions (AmazonS3ReadOnlyAccess is enough), and standard AWS credentials in the environment. Point the pipeline at the bucket with a .env file:
```sh
POSTGRES_URL=postgresql://cocoindex:cocoindex@localhost/cocoindex
S3_BUCKET=your-bucket-name
S3_PREFIX=
```
Then upload your Markdown files to the bucket.
## Run the pipeline
Build the index. This is a one-shot catch-up over the objects in the bucket:
```sh
cocoindex update main
```
The first run embeds everything. Every run after that does only the new work: new and changed files are processed, unchanged files are skipped, and within a changed file only the changed chunks are re-embedded.
For fresh data, schedule the update on whatever cadence you need (cron, Airflow, GitHub Actions, an Azure or Lambda function). Because re-runs only touch what changed, running the pipeline often is cheap. If you need event-driven, second-level freshness instead of scheduled runs, use one of CocoIndex's streaming sources (Kafka, Iggy, or Valkey) as the source of your pipeline.
Once the index is built, query it:
```sh
python main.py "your query"
```
## Support us
We are constantly adding features and examples. If this was useful, a star on GitHub helps others find CocoIndex and keeps us going.
---
# Image search in natural language with CLIP
URL: https://cocoindex.io/blogs/live-image-search/
Date: 2025-05-20
Summary: Search a folder of photos by meaning with CocoIndex v1: CLIP embeds images and text into one vector space, the index runs live inside a FastAPI app, and vectors live in Qdrant.
A folder of photos becomes searchable by meaning the moment you stop relying on filenames and tags. You type "a cute animal" or "long neck" and the matching pictures come back, with no captions and no manual labeling.
The trick is CLIP: it embeds an image and a piece of text into the *same* vector space, so a text query and a matching picture land near each other. This post builds that search with CocoIndex v1. It walks a local image folder, embeds each picture with CLIP, and stores the vectors in Qdrant. The index runs in live mode inside a FastAPI server, so dropping a new photo into the folder makes it searchable within a second, with no rebuild step.
The full code is in the CocoIndex v1 repo.
## The flow
Each image is its own processing component: read the bytes, embed them with the CLIP image encoder, and declare a Qdrant point. There is no text to chunk, so the indexing path is one embedding per image.
Live image-search flow: a local img folder feeds a CocoIndex app in live mode; one processing component per image reads the bytes, embeds them with the CLIP image encoder into a 768-dimensional vector, and declares a Qdrant point that CocoIndex upserts into the ImageSearch collection under cosine distance.
## The shared image-text space
CLIP is trained to place a picture and its description near each other in one embedding space. At index time we only ever embed pixels (get_image_features); at query time we embed text (get_text_features). Because both land in the same space, "long neck" ends up close to the giraffe photo even though that image was never described in words.
The vector database never learns that one vector came from an image and the other from text. It just returns nearest neighbors by cosine distance. So making a new image searchable never requires writing anything about it: dropping the file in the folder is enough.
## Technologies
### CLIP ViT-L/14
CLIP ViT-L/14 is a vision-language model that embeds images and text into a shared 768-dimensional space. We use it two ways: to embed each image at index time, and to convert a natural-language query into that same space at search time.
The embedding size is not hardcoded. It comes from model.config.projection_dim, so swapping in a smaller variant like CLIP ViT-B/32 (faster, slightly lower accuracy) only changes one model name; the collection is created to match.
### CocoIndex
CocoIndex is an open-source data transformation framework for AI, with incremental processing built in. You declare the transformation in plain Python and your own types; change tracking, the live watch, and the managed Qdrant collection run in a Rust engine underneath.
### Qdrant
Qdrant is a vector database. We store the image vectors as points and query them by cosine distance through the CocoIndex Qdrant connector.
### FastAPI
FastAPI serves the search API and hosts the live index in its lifespan, so one process both keeps the collection current and answers queries.
## Prerequisites
- Install Qdrant: docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant.
- Python 3.11+ with torch, transformers, and pillow for CLIP.
CocoIndex v1 keeps its own processing state in a local file (set by COCOINDEX_DB, defaulting to ./cocoindex.db), so there is no separate tracking database to run.
## Define the pipeline
The pipeline lives in pipeline.py. The Qdrant client is created once in the app lifespan and read anywhere with a ContextKey:
```python title=pipeline.py
import cocoindex as coco
from cocoindex.connectors import qdrant
QDRANT_DB = coco.ContextKey[QdrantClient]("image_search_qdrant")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
client = qdrant.create_client(qdrant_url(), prefer_grpc=True)
builder.provide(QDRANT_DB, client)
yield
```
CLIP is loaded once and reused. The image and query encoders are ordinary helpers, so nothing about them is CocoIndex-specific:
```python title=pipeline.py
@functools.cache
def get_clip_model() -> tuple[CLIPModel, CLIPProcessor]:
model = CLIPModel.from_pretrained(CLIP_MODEL_NAME)
processor = CLIPProcessor.from_pretrained(CLIP_MODEL_NAME)
return model, processor
def embed_image_bytes(img_bytes: bytes) -> list[float]:
model, processor = get_clip_model()
image = Image.open(io.BytesIO(img_bytes)).convert("RGB")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
out = model.get_image_features(**inputs)
return _projected_features(out)[0].tolist()
```
### One component per image
process_file handles a single image: read its bytes, embed them, and declare a Qdrant point keyed by a stable uuid5 of the path. memo=True means an unchanged image is never re-embedded, and because each image is its own component, deleting a file removes its point automatically:
```python title=pipeline.py
@coco.fn(memo=True)
async def process_file(file: FileLike, target: qdrant.CollectionTarget) -> None:
content = await file.read()
embedding = embed_image_bytes(content)
point = qdrant.PointStruct(
id=_image_id(file.file_path.path), # stable uuid5 of the path
vector=embedding,
payload={"filename": str(file.file_path.path)},
)
target.declare_point(point)
```
### Wire up the app
app_main creates the Qdrant collection (its vector size comes straight from CLIP's projection_dim), walks the image folder in live mode, and fans out one process_file per image with mount_each:
```python title=pipeline.py
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
model, _ = get_clip_model()
dim: int = model.config.projection_dim
target = await qdrant.mount_collection_target(
QDRANT_DB,
collection_name=QDRANT_COLLECTION,
schema=await qdrant.CollectionSchema.create(
vectors=qdrant.QdrantVectorDef(
schema=VectorSchema(dtype=np.dtype(np.float32), size=dim),
distance="cosine",
)
),
)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(
included_patterns=["**/*.jpg", "**/*.jpeg", "**/*.png"]
),
live=True,
)
await coco.mount_each(process_file, files.items(), target)
app = coco.App(
coco.AppConfig(name="ImageSearchQdrantV1"),
app_main,
sourcedir=pathlib.Path("./img"),
)
```
## Query the index
Search embeds the text with the same CLIP model, this time through the text encoder, and asks Qdrant for the nearest image vectors:
```python title=pipeline.py
def embed_query(text: str) -> list[float]:
model, processor = get_clip_model()
inputs = processor(text=[text], return_tensors="pt", padding=True)
with torch.no_grad():
out = model.get_text_features(**inputs)
return _projected_features(out)[0].tolist()
```
The FastAPI /search endpoint embeds the query and returns the top matches with their scores:
```python title=api.py
@app.get("/search")
async def search(q: str = Query(...), limit: int = Query(5)) -> dict[str, Any]:
query_embedding = pipeline.embed_query(q)
results = pipeline._qdrant_search(
_client, pipeline.QDRANT_COLLECTION, query_embedding, limit
)
return {
"results": [
{"filename": (r.payload or {}).get("filename"), "score": r.score}
for r in results
]
}
```
## Run it as a live service
The index does not need a separate build step. api.py starts the flow in live mode inside the FastAPI lifespan: it blocks startup until the first sweep reports READY so the collection is queryable, then keeps watching img/ in the background while it serves requests:
```python title=api.py
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
global _client
async with coco.runtime():
_client = qdrant.create_client(pipeline.qdrant_url(), prefer_grpc=True)
update_handle = pipeline.app.update(live=True)
async for snap in update_handle.watch():
if snap.status is coco.UpdateStatus.READY:
break
update_task = asyncio.create_task(update_handle.result())
try:
yield
finally:
update_task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await update_task
_client = None
```
## Time to have fun
Start Qdrant, install the example, and run the server. The example ships an img/ folder with a few animals:
```sh
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
pip install "cocoindex[qdrant]" fastapi torch transformers pillow qdrant-client uvicorn python-dotenv
python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000
```
Then run the frontend and open it in the browser:
```sh
cd frontend && npm install && npm run dev # http://localhost:5173
```
Query "elephant" and the elephant ranks first; query "long neck" and the giraffe wins, then the other animals fall in line by CLIP similarity. None of these images was ever tagged with a word:
Two image searches side by side. Querying "elephant" ranks the elephant photo first; querying "long neck" ranks the giraffe first, followed by the elephant, cat, and puppies by CLIP similarity score.
Now drop another image into the img/ folder, for example this cute squirrel. Because the folder is watched in live mode, it is embedded and searchable within a second, no command to re-run.
## Acknowledgement
This image search project was largely contributed by @par4m, Param Arora, an open source developer at Google Summer of Code.
## Support us
We are constantly adding examples and improving the runtime. If this was helpful, please star CocoIndex on GitHub.
---
# How to build an index with text embeddings
URL: https://cocoindex.io/blogs/text-embeddings-101/
Date: 2025-05-19
Summary: Text embeddings 101: what they are, why you chunk and embed, and how to build a semantic search index with CocoIndex v1 and Postgres pgvector.
Keyword search only finds the words you typed. Ask "how do I log back in?" and a page titled "recover your account access" never comes up, because it shares no words with your query.
Semantic search fixes that. It matches on *meaning*, so a question finds the right answer even when the wording is completely different. This guide builds a semantic search index from scratch with CocoIndex v1 and Postgres, and explains each idea as it comes up. No prior experience with embeddings needed.
The full code is in the CocoIndex v1 repo. A star on GitHub is appreciated if this helps.
## What is a text embedding?
An embedding is a list of numbers (a *vector*) that a model assigns to a piece of text so that its position captures the text's meaning. Texts that mean similar things get vectors that sit close together; unrelated texts land far apart.
An embedding model turns each piece of text into a vector. Texts with similar meaning land close together in the vector space, so a query finds its nearest neighbors by meaning rather than by matching exact keywords: "reset my password" and "recover account access" cluster near the query "how do I log back in?", while "today's lunch menu" sits far away.
Once every piece of text is a point in this space, search becomes geometry. You embed the query the same way, then look for the nearest points. "How do I log back in?" lands next to "reset my password" and "recover account access," and nowhere near "today's lunch menu," with no shared keywords required.
The model here is all-MiniLM-L6-v2, which turns any text into a 384-number vector. It is small, fast, and good enough to feel magic on real documents.
## What we'll build
The index is a short pipeline: read files, split each into chunks, embed each chunk, and store the vectors in Postgres so you can search them.
Indexing flow: a folder of Markdown files feeds a CocoIndex app; one processing component per file splits it into chunks, embeds each chunk with a sentence-transformer model, and declares a row that CocoIndex stores in a Postgres pgvector table with a vector index for fast nearest-neighbor search.
Why split into *chunks* instead of embedding a whole file? A single vector can only capture so much. A long document is about many things, and squeezing all of them into one vector blurs them together, the same way one photo-wide vector blurs a busy image. Splitting the document into smaller passages and embedding each one keeps the meaning sharp and lets search point at the exact passage that answers a query.
## Why CocoIndex?
You could write this pipeline by hand. The hard part is not embedding once; it is keeping the index correct as your documents change. By hand you would track which files changed, figure out which chunks are new, embed only those, and insert, update, or delete rows so the database matches, forever.
By hand you write glue code to track which files changed, diff chunks, embed only the new ones, and insert/update/delete rows to keep the index in sync. With CocoIndex you declare target_state = transformation(source_state): you write only the chunk-embed-declare transform, and the engine does change detection, incremental re-embedding, and insert/update/delete sync automatically.
CocoIndex is declarative. You describe the target as a function of the source, target_state = transformation(source_state), and a Rust engine keeps the two in sync. You write only the transform (chunk, embed, declare a row). The engine handles change detection, re-embeds only what changed, and applies the right inserts, updates, and deletes, including cleaning up when a file is removed. Edit one paragraph and only that chunk is re-embedded.
## Build it
### Prerequisites
Install Postgres with the pgvector extension; that is where the embeddings live. Set the connection string:
```sh
export POSTGRES_URL="postgres://cocoindex:cocoindex@localhost/cocoindex"
```
### Define the row
Each chunk becomes one row. A plain dataclass describes the columns; the embedding field is annotated with the embedder so CocoIndex knows the vector's size:
```python title=main.py
from dataclasses import dataclass
from typing import Annotated
from numpy.typing import NDArray
import cocoindex as coco
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@dataclass
class DocEmbedding:
id: int
filename: str
chunk_start: int
chunk_end: int
text: str
embedding: Annotated[NDArray, EMBEDDER]
```
The database connection and the embedder are created once, in the app lifespan:
```python title=main.py
import asyncpg
from cocoindex.connectors import postgres
PG_DB = coco.ContextKey[asyncpg.Pool]("text_embedding_db")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
```
### Chunk and embed
process_chunk embeds one chunk and declares its row. process_file reads a file, splits it into chunks, and maps process_chunk over them. memo=True caches per-file work, so an unchanged file is skipped:
```python title=main.py
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.resources.chunk import Chunk
from cocoindex.resources.file import FileLike
from cocoindex.resources.id import IdGenerator
_splitter = RecursiveSplitter()
@coco.fn
async def process_chunk(
chunk: Chunk,
filename: pathlib.PurePath,
id_gen: IdGenerator,
table: postgres.TableTarget[DocEmbedding],
) -> None:
table.declare_row(
row=DocEmbedding(
id=await id_gen.next_id(chunk.text),
filename=str(filename),
chunk_start=chunk.start.char_offset,
chunk_end=chunk.end.char_offset,
text=chunk.text,
embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
),
)
@coco.fn(memo=True)
async def process_file(
file: FileLike[str],
table: postgres.TableTarget[DocEmbedding],
) -> None:
text = await file.read_text()
chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language="markdown")
id_gen = IdGenerator()
await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)
```
### Wire up the app
app_main mounts the Postgres table, adds a vector index for fast nearest-neighbor search, walks the folder of Markdown files, and runs one process_file per file:
```python title=main.py
from cocoindex.connectors import localfs
from cocoindex.resources.file import PatternFilePathMatcher
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
table = await postgres.mount_table_target(
PG_DB,
table_name="doc_embeddings",
table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=["id"]),
pg_schema_name="coco_examples",
)
table.declare_vector_index(column="embedding")
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
live=True,
)
await coco.mount_each(process_file, files.items(), table)
app = coco.App(
coco.AppConfig(name="TextEmbedding"),
app_main,
sourcedir=pathlib.Path("./markdown_files"),
)
```
### Build the index
```sh
pip install "cocoindex[postgres]" sentence-transformers asyncpg pgvector
cocoindex update main
```
The first run embeds everything. Re-run it any time and only new or changed files are processed. Add -L (cocoindex update -L main) to watch the folder and update the index live as files change.
## Query the index
Searching is the mirror image of indexing: embed the query with the same model, then ask Postgres for the nearest vectors. The operator is pgvector's cosine distance, so smaller means more similar:
```python title=main.py
async def query_once(pool, embedder, query: str, *, top_k: int = 5) -> None:
query_vec = await embedder.embed(query)
async with pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT filename, text, embedding <=> $1 AS distance
FROM "coco_examples"."doc_embeddings"
ORDER BY distance ASC
LIMIT $2
""",
query_vec,
top_k,
)
for r in rows:
print(f"[{1.0 - float(r['distance']):.3f}] {r['filename']}")
print(f" {r['text']}")
```
Run it and ask questions in plain language:
```sh
python main.py "how do I get started?"
```
You get back the passages whose meaning is closest to your question, ranked by similarity, even when they share none of your words.
## Where to go next
You now have the core of semantic search: chunk, embed, store, and query by nearest neighbor. The same shape extends a long way. Swap the source for Amazon S3, Azure Blob, or Google Drive; swap the target for a dedicated vector database like Qdrant; or feed the retrieved passages to an LLM to build RAG. Because the pipeline is declarative and incremental, none of that changes how you keep the index fresh.
## Support us
We are constantly adding examples and improving the runtime. If this guide helped, please star CocoIndex on GitHub.
---
# Story of CocoIndex, at 1k stars 🎉
URL: https://cocoindex.io/blogs/cocoindex-1k/
Date: 2025-05-08
Summary: The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.
We have been working on CocoIndex, a real-time data framework for AI, for a while, with lots of excitement from the community. We officially crossed 1k stars earlier this week. Huge thanks to everyone who starred, forked, contributed, or shared the love ❤️!
**CocoIndex** is an ultra performant data transformation framework, with its core engine written in Rust. The problem it tries to solve is to make it easy to prepare fresh data for AI (either creating embeddings, building knowledge graphs, or performing other data transformations) and take real-time data pipelines beyond traditional SQL.
CocoIndex Key Feature
The philosophy is to have the framework handle the source updates, and have developers only worry about defining a series of data transformations, inspired by spreadsheets.
Here's a bit of history about our open source journey:
Star history
## What is data flow programming?
Unlike a workflow orchestration framework where data is usually opaque, in CocoIndex, data and data operations are first-class citizens. CocoIndex follows the idea of the Dataflow programming model. Each transformation creates a new field solely based on input fields, without hidden states and value mutation. All data before/after each transformation is observable, with lineage out of the box.
For example, this is a kind of data flow: Parse files -> Data Mapping -> Data Extraction -> Knowledge Graph
Data Flow
**Particularly**, users don't define data operations like creation, update, or deletion. Rather, they define something like this:
for a set of source data, this is the transformation or formula. The framework takes care of the data operations such as when to create, update, or delete.
With operations like:
```python
# ingest
data['content'] = flow_builder.add_source(...)
# transform
data['out'] = data['content']
.transform(...)
.transform(...)
# collect data
collector.collect(...)
# export to db, vector db, graph db ...
collector.export(...)
```
## How does CocoIndex keep data fresh?
As a data framework, CocoIndex takes data freshness to the next level. Incremental processing is one of the core values provided by CocoIndex.
CocoIndex Engine
The framework takes care of:
1. Change data capture
2. Figuring out what exactly needs to be updated, and only updating that without having to recompute everything throughout, backed by memoization of computation results.
This makes it fast to reflect any source updates to the target store. If you have concerns with surfacing stale data to AI agents and are spending lots of effort working on infrastructure pieces to optimize latency, the framework actually handles it for you.
## Built-in modules and custom modules
We understand preparing data is highly use-case based and there is no one-size-fits-all solution. With all the latest advancements in models, there is no single best solution for all cases, and it's also a balance of multiple factors in terms of quality, performance, pricing, etc. There are a lot of efforts in the data infra ecosystem that optimize for different goals: parsers, embedding models, vector DBs, graph DBs, and many others.
We take the composition approach, and instead of building everything, we provide native plugins to embrace the ecosystem and make it easier to plug in and swap any module by standardizing the interface, exactly like assembling building blocks.
And we can focus on what we're best at: building the best incremental pipeline and compute engine in the AI era.
## Tooling
In addition to standard statistics and reports with CocoIndex, we are building a product called CocoInsight. It has zero pipeline data retention and connects to your on-premise CocoIndex server for pipeline insights. This makes data directly visible and makes it easy to develop ETL pipelines.
Especially, as we mentioned earlier, CocoIndex is heavily influenced by spreadsheets. This is directly reflected in CocoInsight, as you can see all the data in the spreadsheet and how it looks after each transformation.
This is an example of mapping:
Mapping
In addition to the data insights for data transformation, we also aim to provide tooling to make it directly visible to understand the LLM transformations, for example:
- Understand and troubleshoot the chunking, if there's any, and debug why a chunk is showing in search results
Embedding Understanding
- Understand the relation extraction for Graphs
Relation Understanding
- and more
Data should be transparent, especially for ETL frameworks.
## Support us
We are constantly improving, and more features and examples are coming soon. If you love this article, please drop us a star ⭐ at the GitHub repo to help us grow. You can also find us on Discord. I try to be there 24/7 😊 Looking forward to discussing CocoIndex or any topics in Data/AI infra with you!
---
# Build a product recommendation engine with LLM + Neo4j
URL: https://cocoindex.io/blogs/product-recommendation/
Date: 2025-05-07
Summary: An LLM extracts each product's taxonomy and complements; CocoIndex turns the labels into a Neo4j knowledge graph that answers 'bought this, also need…'.
A pile of product listings has recommendations hiding in plain sight: a *pen* pairs with *ink refills* and a *notebook*; a *monitor* pairs with a *stand* and an *HDMI cable*. That knowledge is locked in the prose of each listing. This post builds a recommendation engine from it: an LLM pulls the knowledge out, and a Neo4j graph connects it.
For each product, CocoIndex v1 asks an LLM two things: what the product *is* (its taxonomy), and what a buyer might *also need* (its complementary taxonomy). Those labels become nodes and edges in a Neo4j graph, and the recommendation falls out of the graph: a product whose "also need" matches another product's "is" is the thing to cross-sell.
The full code is in the CocoIndex v1 repo.
## The data model
Two node types and two edge types are enough:
- **Product** nodes, one per listing, keyed by id (the filename), carrying title and price.
- **Taxonomy** nodes, one per distinct label (gel pen, notebook, ink refill), keyed by value and **shared** across products.
- **PRODUCT_TAXONOMY** edges, Product → Taxonomy: what the product is.
- **PRODUCT_COMPLEMENTARY_TAXONOMY** edges, Product → Taxonomy: what pairs with it.
The nodes are plain dataclasses:
```python title=main.py
from dataclasses import dataclass
@dataclass
class Product:
id: str # primary key (the filename stem)
title: str
price: float
@dataclass
class Taxonomy:
value: str # primary key (the taxonomy label)
```
## The flow
Because taxonomy labels are shared across products, the pipeline runs in two phases: per-product extraction declares each Product node and carries its labels forward, then a single graph-building pass declares the deduplicated Taxonomy nodes and all the edges.
Two-phase product-recommendation flow: phase one reads each product JSON, declares a Product node, and extracts its taxonomy labels with an LLM; phase two collects all labels, declares the deduplicated Taxonomy nodes, and declares the PRODUCT_TAXONOMY and PRODUCT_COMPLEMENTARY_TAXONOMY edges into Neo4j.
## Connect Neo4j
The Neo4j connection is provided once in the app lifespan and read anywhere with a ContextKey. The LLM model name is a context value too, declared with detect_change=True so swapping models re-extracts everything against the new one:
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import neo4j
KG_DB = coco.ContextKey[neo4j.ConnectionFactory]("kg_db")
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.provide(
KG_DB,
neo4j.ConnectionFactory(
uri=os.environ.get("NEO4J_URI", "bolt://localhost:7687"),
auth=(os.environ["NEO4J_USER"], os.environ["NEO4J_PASSWORD"]),
),
)
builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-4.1"))
yield
```
## Extract the taxonomy
The extraction schema is the whole prompt. A Pydantic model asks for two lists, and the field descriptions tell the model exactly what a good label looks like: a concise noun, US English, specific rather than broad (gel pen, not office supplies):
```python title=main.py
import pydantic
class ProductTaxonomy(pydantic.BaseModel):
name: str = pydantic.Field(
description="A concise noun for the product's core function, no branding "
"or style. Common US English, lowercase. Prefer specific ('pen') over "
"broad ('office supplies')."
)
class ProductTaxonomyInfo(pydantic.BaseModel):
taxonomies: list[ProductTaxonomy] = pydantic.Field(
description="Taxonomies describing what this product is."
)
complementary_taxonomies: list[ProductTaxonomy] = pydantic.Field(
description="Taxonomies for complementary products a buyer might also need."
)
```
Extraction is one CocoIndex function that calls the model through instructor over LiteLLM, forcing the response to match the schema. memo=True caches the result by content, so an unchanged product is never sent to the model twice:
```python title=main.py
@coco.fn(memo=True)
async def extract_taxonomy(detail: str) -> ProductTaxonomyInfo:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=ProductTaxonomyInfo,
messages=[
{"role": "system", "content": TAXONOMY_PROMPT},
{"role": "user", "content": detail},
],
)
return ProductTaxonomyInfo.model_validate(result.model_dump())
```
For the gel pen, the model returns gel pen as the taxonomy, and notebook, pen refill, and the like as complementary taxonomies:
The extract_taxonomy function reads the gel pen's product detail and returns a typed ProductTaxonomyInfo: taxonomies (gel pen) and complementary_taxonomies (notebook, pen refill, highlighter, sticky notes).
## Phase 1: one Product node per file
process_file handles a single product: read the JSON, declare its Product node into the graph, extract the taxonomy, and return the labels for phase two. declare_record writes one node; CocoIndex reconciles it into Neo4j by the primary key:
```python title=main.py
@coco.fn(memo=True)
async def process_file(
file: FileLike,
product_table: neo4j.TableTarget[Product],
) -> ProductTaxonomies:
raw = json.loads(await file.read_text())
product_id = file.file_path.path.name.removesuffix(".json")
price = float(str(raw["price"]).lstrip("$").replace(",", ""))
product_table.declare_record(row=Product(id=product_id, title=raw["title"], price=price))
info = await extract_taxonomy(PRODUCT_TEMPLATE.render(**raw))
return ProductTaxonomies(
product_id=product_id,
taxonomies=[t.name for t in info.taxonomies],
complementary=[t.name for t in info.complementary_taxonomies],
)
```
Each declared Product becomes a node with id as its key and every field as a property:
Each product record becomes one Product node in Neo4j via declare_record: id is the primary key and every field (title, price) is carried across as a node property.
## Phase 2: shared Taxonomy nodes and the edges
Taxonomy labels repeat across products, so one pass owns them. It gathers every label, declares the Taxonomy nodes, and declares the edges. Because a node is keyed by value, declaring gel pen from ten different products still produces exactly one node: CocoIndex deduplicates by the primary key.
Node deduplication by primary key: taxonomy labels collected from many products, where gel pen and notebook each appear more than once, collapse on their key (value) into one Neo4j Taxonomy node each.
declare_relation connects a product to a taxonomy by their keys. Two relation targets, one per edge type, share the same Product and Taxonomy node tables:
```python title=main.py
@coco.fn
async def build_graph(
products: list[ProductTaxonomies],
taxonomy_table: neo4j.TableTarget[Taxonomy],
product_taxonomy_rel: neo4j.RelationTarget[Any],
complementary_rel: neo4j.RelationTarget[Any],
) -> None:
labels = {t for p in products for t in (*p.taxonomies, *p.complementary)}
for value in labels:
taxonomy_table.declare_record(row=Taxonomy(value=value))
for p in products:
for t in set(p.taxonomies):
product_taxonomy_rel.declare_relation(from_id=p.product_id, to_id=t)
for t in set(p.complementary):
complementary_rel.declare_relation(from_id=p.product_id, to_id=t)
```
The PRODUCT_TAXONOMY edges link each product to what it is:
Each Product links to its own Taxonomy by a PRODUCT_TAXONOMY edge: Pink Gel Pen to GelPen, Premium Spiral Notepad to Notepad.
The PRODUCT_COMPLEMENTARY_TAXONOMY edges add what pairs with it. These are the edges that cross between products and make recommendations possible:
On top of the solid taxonomy edges, each Product gains a dashed PRODUCT_COMPLEMENTARY_TAXONOMY edge crossing to a buy-together Taxonomy: Pink Gel Pen to Notepad, Premium Spiral Notepad to GelPen.
## Wire up the app
app_main mounts the two node tables and the two relation targets, walks the product folder, fans out one process_file per product, then runs the single build_graph pass over the collected labels:
```python title=main.py
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
product_table = await neo4j.mount_table_target(
KG_DB, "Product",
await neo4j.TableSchema.from_class(Product, primary_key="id"),
primary_key="id",
)
taxonomy_table = await neo4j.mount_table_target(
KG_DB, "Taxonomy",
await neo4j.TableSchema.from_class(Taxonomy, primary_key="value"),
primary_key="value",
)
product_taxonomy_rel = await neo4j.mount_relation_target(
KG_DB, "PRODUCT_TAXONOMY", product_table, taxonomy_table
)
complementary_rel = await neo4j.mount_relation_target(
KG_DB, "PRODUCT_COMPLEMENTARY_TAXONOMY", product_table, taxonomy_table
)
files = localfs.walk_dir(
sourcedir, recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.json"]),
)
products = list(await asyncio.gather(*[
coco.use_mount(coco.component_subpath("file", key), process_file, file, product_table)
async for key, file in files.items()
]))
await coco.mount(
coco.component_subpath("build_graph"),
build_graph, products, taxonomy_table, product_taxonomy_rel, complementary_rel,
)
app = coco.App(coco.AppConfig(name="ProductRecommendation"), app_main,
sourcedir=pathlib.Path("./products"))
```
## Run it
Start Neo4j, install the example, and build the graph. The example ships a products/ folder of sample listings:
```sh
docker run -d -p 7474:7474 -p 7687:7687 -e NEO4J_AUTH=neo4j/cocoindex neo4j:5.26-community
pip install "cocoindex[neo4j]" instructor litellm pydantic jinja2
cocoindex update main
```
On the nine sample products that is nine Product nodes, around forty Taxonomy nodes, and both edge types wired up. Editing one product re-extracts only that product, then the graph diffs: new nodes and edges are added, and ones no longer supported anywhere are removed.
## Query the recommendations
Open Neo4j Browser (neo4j / cocoindex) and let the graph do the recommending. To find products to pair with a gel pen, walk from the pen's complementary taxonomy to any product that *is* that taxonomy:
```cypher
MATCH (:Taxonomy {value: "gel pen"})<-[:PRODUCT_TAXONOMY]-(:Product)
-[:PRODUCT_COMPLEMENTARY_TAXONOMY]->(need:Taxonomy)
MATCH (rec:Product)-[:PRODUCT_TAXONOMY]->(need)
RETURN DISTINCT rec.title
```
On the sample data, recommending for a pen surfaces the notepad and the multipurpose paper, the cross-sell you would want. To see the whole graph, MATCH p=()-->() RETURN p:
The product recommendation knowledge graph in Neo4j Browser: green Product nodes connected to purple Taxonomy nodes by PRODUCT_TAXONOMY and PRODUCT_COMPLEMENTARY_TAXONOMY edges.
## Support us
We are constantly adding examples and improving the runtime. If this was helpful, please star CocoIndex on GitHub.
---
# CocoIndex Changelog 2025-04-30
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-04-30/
Date: 2025-04-30
Summary: CocoIndex updates: knowledge graph support, Qdrant and Supabase targets, KTable and LTable data types, additional LLM providers, and more.
In the past 2 weeks, we've added support for Knowledge Graphs, Qdrant, Supabase, KTable/LTable, and more LLM providers,
along with tons of assorted core/performance improvements - full changelog.
## Knowledge graphs
CocoIndex makes it easy to build and maintain knowledge graphs with continuous updates.
The knowledge graph is automatically updated when the source data changes with incremental processing, ensuring that your graph always reflects the latest information.
Users can follow the CocoIndex data flow to collect nodes and relationships. Data from collectors is mapped to graph elements in various types:
- Rows from collectors → Nodes in the graph
- Rows from collectors → Relationships in the graph (including source and target nodes of the relationship)
Knowledge Graph Support in CocoIndex
By declaring the mapping rules, CocoIndex will map data from collectors to graph nodes and relationships.
### Getting started
- Documentation: Property Graph Targets
- Step-by-step example: Knowledge Graphs for docs
## Supporting Qdrant
CocoIndex is officially supporting Qdrant 🎉!
Qdrant is the leading open-source vector database designed to handle high-dimensional vectors for performance and massive-scale AI applications.
This integration combines a high-performance Rust 🦀 stack with real-time ETL to a vector store.
- Documentation: Qdrant as storage target
- Example: Qdrant for docs
We are excited to see this integration!
## Supporting Supabase
CocoIndex now supports Supabase as a storage target with simple configuration.
Supabase is a modern database platform that combines the best of PostgreSQL with the power of AI.
If you don't want to self-host any database for the data pipeline, Supabase is a great option.
Follow the Database connection spec to get started.
## Type system updates
### KTable
KTable is a Table type whose first column serves as the key. The row order of a KTable is not preserved.
The type of the first column (key column) must be a key type.
In Python, a KTable type is represented by dict[K, V]. The V should be a dataclass, representing the value fields of each row.
For example,
```python
@dataclass
class Person:
first_name: str
last_name
dob: datetime.date
```
You can use dict[str, Person] to represent a KTable, with 4 columns: the key field (Str), first_name (Str), last_name (Str), dob (Date).
### LTable
LTable is a Table type whose row order is preserved. LTable has no key column.
In Python, an LTable type is represented by list[R], where R is a dataclass representing a row.
For example, you can use list[Person] to represent an LTable with 3 columns: first_name (Str), last_name (Str), dob (Date).
Learn more about the type system in the documentation.
## Supporting more LLM providers in the data flow
CocoIndex now supports Gemini and Anthropic in the data flow.
Thanks to @par4m for the contributions!
- Documentation for Gemini
- Documentation for Anthropic
You could easily switch between the providers by changing the configuration. Try it out for some of our existing examples -
Text Embedding from PDFs,
LLM structured extraction.
## Thanks to the community 🤗🎉!
Welcome new contributors to the CocoIndex community! We are so excited to have you!
### @Anush008 from Qdrant
- Submitted the PR to support Qdrant storage in #205.
### @par4m
- Add unit tests for Python to Engine format conversion in #315
- Colors to CLI Status for better readability - Super neat! #329, #372
Colorful CLI output for better readability
- Support Gemini, and Anthropic in Data Flow #360, #395
### @FranFigueroa
- Add unit tests for SplitRecursively function in #348
### @lemorage
- Beautify show cli command with colorful structured output - Super Sleek! #404
Beautify show cli command with colorful structured output
Give it a try by running cocoindex show!
## Full changelog
Review the detailed changelog here.
We are constantly improving CocoIndex, and more features are coming soon!
Stay tuned and follow us by starring our GitHub repo.
---
# Build Real-Time Knowledge Graph For Documents with LLM
URL: https://cocoindex.io/blogs/knowledge-graph-for-docs/
Date: 2025-04-29
Summary: Turn a folder of Markdown docs into a Neo4j knowledge graph: an LLM extracts subject-predicate-object triples, and CocoIndex keeps the graph in sync as the docs change.
Documentation is a web of concepts pretending to be a list of files. Every page asserts relationships ("CocoIndex supports incremental processing"), but they are locked in prose. This post turns a folder of Markdown documents (CocoIndex's own docs) into a Neo4j knowledge graph: an LLM reads each document and extracts the relationships between the concepts it talks about, and CocoIndex reconciles them into the graph incrementally as documents change.
We generate two kinds of relationships:
1. Relationships between subjects and objects. E.g., "CocoIndex supports Incremental Processing".
2. Mentions of entities in a document. E.g., basics.md mentions CocoIndex and Incremental Processing.
The end result is a small, well-defined graph schema: two node labels (Document and Entity) and two relationship types (RELATIONSHIP between entities, and MENTION connecting a document to the entities it references). Everything the LLM extracts lands in one of those four shapes, which is what keeps the graph queryable rather than a loose pile of triples.
Example knowledge graph: the Basics.md document node mentions the CocoIndex and Incremental Processing entity nodes, which are connected by a "supports" relationship.
The source code is available at CocoIndex Examples - docs_to_knowledge_graph.
## Prerequisites
* Install Neo4j, a graph database. A docker one-liner is in the Run it section below.
* An LLM API key. The example defaults to OpenAI, and the model is a LiteLLM id, so you can set LLM_MODEL=ollama/llama3.2 to run the extraction locally instead.
## The graph schema
All nodes for Neo4j need two things:
1. Label: the type of the node. E.g., Document, Entity.
2. Primary key: the field that uniquely identifies the node. E.g., filename for Document nodes.
CocoIndex uses the primary key to match and deduplicate nodes: declaring the same key twice produces one node. In CocoIndex v1 the schema is plain dataclasses:
```python title=main.py
from dataclasses import dataclass
@dataclass
class Document:
filename: str # primary key
title: str
summary: str
@dataclass
class Entity:
value: str # primary key — the concept name
@dataclass
class Relationship:
"""RELATIONSHIP edge payload. ``id`` is a stable hash of the triple so the
same (subject, predicate, object) always maps to a single edge; the
``predicate`` is stored as an edge property."""
id: int
predicate: str
```
MENTION carries no payload, so it needs no dataclass at all: the Neo4j connector derives its identity from the (document, entity) endpoints, one edge per pair.
## The flow
Entities are shared across documents: the same concept, say CocoIndex, shows up in triples from many files, and every one of those references must collapse onto a single Entity node. So the pipeline runs in two phases. Phase 1 processes each document independently: it declares the Document node and extracts the relationship triples. Phase 2 is a single pass over all the triples: it owns the deduplicated Entity nodes and declares the RELATIONSHIP and MENTION edges.
Two-phase flow: Markdown files feed per-document extraction components that declare Document nodes and produce triples; a single build_graph pass declares the deduplicated Entity nodes and the RELATIONSHIP and MENTION edges into Neo4j.
Because phase 1 runs one component per file, editing one document re-extracts only that document. The graph pass then diffs: nodes and edges that are no longer supported by any document are removed automatically.
## Connect Neo4j
The Neo4j connection is provided once in the app lifespan and read anywhere with a ContextKey. The LLM model name is a context value too, declared with detect_change=True so swapping models re-extracts everything against the new one:
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import localfs, neo4j
KG_DB = coco.ContextKey[neo4j.ConnectionFactory]("kg_db")
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.provide(
KG_DB,
neo4j.ConnectionFactory(
uri=os.environ.get("NEO4J_URI", "bolt://localhost:7687"),
auth=(
os.environ.get("NEO4J_USER", "neo4j"),
os.environ.get("NEO4J_PASSWORD", "cocoindex"),
),
database=os.environ.get("NEO4J_DATABASE", "neo4j"),
),
)
builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-5-mini"))
yield
```
## Summarize each document
The extraction schemas are Pydantic models, and the field descriptions are part of the prompt. For the summary we ask for a title and one paragraph:
```python title=main.py
import pydantic
class DocumentSummary(pydantic.BaseModel):
title: str = pydantic.Field(description="A concise title for the document.")
summary: str = pydantic.Field(
description="A one-paragraph summary of what the document covers."
)
```
Extraction is one CocoIndex function that calls the model through instructor over LiteLLM, forcing the response to match the schema. memo=True memoizes the result by content, so an unchanged document is never sent to the model twice:
```python title=main.py
@coco.fn(memo=True)
async def extract_summary(content: str) -> DocumentSummary:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=DocumentSummary,
messages=[
{"role": "system", "content": SUMMARY_PROMPT},
{"role": "user", "content": content},
],
)
return DocumentSummary.model_validate(result.model_dump())
```
For each document the model returns a typed summary:
The extract_summary function reads each Markdown file and returns a typed DocumentSummary: for basics.md, the title "CocoIndex Basics" and a one-paragraph summary.
## Extract relationship triples
A relationship is a (subject, predicate, object) triple. The subject and object become Entity nodes, and the predicate is the label on the edge between them:
A triple in the graph: the subject (CocoIndex) and object (Incremental Processing) are entity nodes, and the predicate (supports) is stored on the relationship edge connecting them.
The Pydantic model mirrors that shape, and the system prompt steers the model toward concepts rather than code:
```python title=main.py
class ExtractedRelationship(pydantic.BaseModel):
subject: str = pydantic.Field(
description="The concept the statement is about, e.g. 'CocoIndex'."
)
predicate: str = pydantic.Field(
description="How subject relates to object, e.g. 'supports'."
)
object: str = pydantic.Field(
description="The related concept, e.g. 'Incremental Processing'."
)
class RelationshipList(pydantic.BaseModel):
relationships: list[ExtractedRelationship] = pydantic.Field(default_factory=list)
RELATIONSHIP_PROMPT = """\
You extract a concept knowledge graph from technical documentation.
List the salient (subject, predicate, object) relationships between concepts.
Focus on concepts and ignore code examples and implementation details.
Use concise noun phrases for subjects and objects and a short verb phrase for
the predicate. Return only relationships supported by the text.
"""
```
The extraction function has the same shape as the summary one, memoized the same way:
```python title=main.py
@coco.fn(memo=True)
async def extract_relationships(content: str) -> list[Triple]:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=RelationshipList,
messages=[
{"role": "system", "content": RELATIONSHIP_PROMPT},
{"role": "user", "content": content},
],
)
validated = RelationshipList.model_validate(result.model_dump())
return [Triple(r.subject, r.predicate, r.object) for r in validated.relationships]
```
On the CocoIndex docs, one file yields a few dozen triples:
The extract_relationships function reads basics.md and returns a list of typed triples, such as (Index, is collected in, Data) and (Data Sources, are built from, Indexes).
## Phase 1: one Document node per file
process_file handles a single document: read the content, extract the summary, declare the Document node into the graph, and return the triples for phase 2. declare_record writes one node; CocoIndex reconciles it into Neo4j by the primary key:
```python title=main.py
@coco.fn(memo=True)
async def process_file(
file: localfs.File,
document_table: neo4j.TableTarget[Document],
) -> DocTriples:
content = await file.read_text()
filename = file.file_path.path.as_posix()
summary = await extract_summary(content)
document_table.declare_record(
row=Document(filename=filename, title=summary.title, summary=summary.summary)
)
triples = await extract_relationships(content)
return DocTriples(filename=filename, triples=triples)
```
Each declared record becomes a Document node keyed by filename, with the title and summary carried across as node properties:
Each Document record declared by process_file becomes one Document node in Neo4j: filename is the primary key, and title and summary become node properties.
## Phase 2: build the concept graph
Phase 2 receives the triples from every document. Because Entity nodes are keyed by value, declaring CocoIndex from ten different documents still produces exactly one node: CocoIndex deduplicates by the primary key. The edges then fan out from that one node, which is exactly what you want when you later query "what mentions this concept?".
Deduplication by primary key: two triples from different documents both name CocoIndex as their subject, and the graph pass collapses them into a single CocoIndex entity node with two outgoing relationship edges.
One pass gathers the distinct entities and mentions, then declares everything. declare_relation connects two nodes by their primary keys, and generate_id hashes each triple so the same fact always maps to the same edge, whether it is re-extracted or asserted again by another document:
```python title=main.py
@coco.fn
async def build_graph(
docs: list[DocTriples],
entity_table: neo4j.TableTarget[Entity],
relationship_rel: neo4j.RelationTarget[Relationship],
mention_rel: neo4j.RelationTarget[Any],
) -> None:
entities: set[str] = set()
mentions: set[tuple[str, str]] = set() # (filename, entity value)
for doc in docs:
for t in doc.triples:
entities.add(t.subject)
entities.add(t.object)
mentions.add((doc.filename, t.subject))
mentions.add((doc.filename, t.object))
rel_id = await generate_id((t.subject, t.predicate, t.object))
relationship_rel.declare_relation(
from_id=t.subject,
to_id=t.object,
record=Relationship(id=rel_id, predicate=t.predicate),
)
for value in entities:
entity_table.declare_record(row=Entity(value=value))
for filename, entity in mentions:
mention_rel.declare_relation(from_id=filename, to_id=entity)
```
The RELATIONSHIP edges connect Entity nodes, with the predicate stored on the edge:
RELATIONSHIP edges between entity nodes: CocoIndex connects to Incremental Processing with the predicate "supports" stored as an edge property.
The MENTION edges record which document named which concept. For example, basics.md has a sentence "CocoIndex supports Incremental Processing", so it mentions both CocoIndex and Incremental Processing:
MENTION edges: the Basics.md document node connects with dashed mention edges to both entity nodes it references, alongside the solid relationship edge between them.
## Wire up the app
app_main mounts the two node tables and the two relation targets, walks the docs folder, fans out one process_file per document, then runs the single build_graph pass over the collected triples:
```python title=main.py
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
document_table = await neo4j.mount_table_target(
KG_DB,
"Document",
await neo4j.TableSchema.from_class(Document, primary_key="filename"),
primary_key="filename",
)
entity_table = await neo4j.mount_table_target(
KG_DB,
"Entity",
await neo4j.TableSchema.from_class(Entity, primary_key="value"),
primary_key="value",
)
relationship_rel = await neo4j.mount_relation_target(
KG_DB,
"RELATIONSHIP",
entity_table,
entity_table,
await neo4j.TableSchema.from_class(Relationship, primary_key="id"),
primary_key="id",
)
mention_rel = await neo4j.mount_relation_target(
KG_DB, "MENTION", document_table, entity_table
)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md", "**/*.mdx"]),
)
file_coros = []
async for path_key, file in files.items():
file_coros.append(
coco.use_mount(
coco.component_subpath("file", path_key),
process_file,
file,
document_table,
)
)
docs: list[DocTriples] = list(await asyncio.gather(*file_coros))
await coco.mount(
coco.component_subpath("build_graph"),
build_graph,
docs,
entity_table,
relationship_rel,
mention_rel,
)
app = coco.App(
coco.AppConfig(name="DocsToKnowledgeGraph"),
app_main,
sourcedir=pathlib.Path("./markdown_files"),
)
```
RELATIONSHIP is mounted with the Relationship schema so each distinct triple, keyed by the hashed id, becomes its own edge. MENTION is mounted without a schema, so the connector keys it by the endpoints.
## Run it
Start Neo4j, install the example, and build the graph. The example ships a markdown_files/ folder of sample docs so it runs out of the box:
```sh
docker run -d -p 7474:7474 -p 7687:7687 -e NEO4J_AUTH=neo4j/cocoindex --name cocoindex-neo4j neo4j:5.26-community
cp .env.example .env # set OPENAI_API_KEY (or LLM_MODEL=ollama/llama3.2 to run locally)
pip install -e .
cocoindex update main
```
To graph your own docs, drop .md / .mdx files into markdown_files/ (or point sourcedir at your real docs folder) and re-run. Only changed documents are re-extracted; a no-change re-run makes zero LLM calls.
## Browse the knowledge graph
Open Neo4j Browser (username neo4j, password cocoindex) and run a Cypher query. To see how concepts relate:
```cypher
MATCH (a:Entity)-[r:RELATIONSHIP]->(b:Entity)
RETURN a.value, r.predicate, b.value
```
To find the concepts mentioned in the most documents:
```cypher
MATCH (d:Document)-[:MENTION]->(e:Entity)
RETURN e.value, count(DISTINCT d) AS docs
ORDER BY docs DESC LIMIT 10
```
Or MATCH p=()-->() RETURN p to see everything:
Neo4j Browser showing the knowledge graph built from the CocoIndex docs: document and entity nodes connected by MENTION and RELATIONSHIP edges.
:::tip[Near-duplicate entities]
The LLM will sometimes name the same concept two ways ("CocoIndex" vs "Cocoindex"). The meeting notes graph example adds an embedding plus LLM entity-resolution pass that collapses near-duplicates; it drops into this pipeline between the two phases.
:::
## Support us
We are constantly adding examples and improving the runtime. If this was helpful, please star CocoIndex on GitHub.
---
# CocoIndex Changelog 2025-04-07
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-04-07/
Date: 2025-04-07
Summary: CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.
In the past 2 weeks — six releases, v0.1.13 through v0.1.18 — we added incremental processing with live update mode, evaluation utilities, support for date/time types, Google Drive, and assorted core/performance improvements.
## Incremental processing
Incremental processing is one of the core values provided by CocoIndex.
CocoIndex creates and maintains indexes while keeping them up to date with source changes through minimal reprocessing.
Users don't really need to do anything special. Just focus on defining the transformation needed.
CocoIndex automatically tracks the lineage of the data and maintains a cache of computation results.
This approach ensures low latency between source and index updates while minimizing computational costs.
For more details, please refer to Incremental Processing.
## Continuous synchronization between source and index
With the new live update mode, CocoIndex continuously captures changes from the source data and updates the target data accordingly.
It's long-running and only stops when being aborted explicitly.
This ensures that your applications always have access to the most current information without the performance overhead of full reindexing.
CocoIndex supports two main categories of change detection mechanisms:
1. **General Mechanism**: with refresh interval, CocoIndex periodically checks for changes.
2. **Source-Specific Mechanisms**: For example, push change notification and recent changes poll that lists recently modified entries.
Under the hood, after the change is detected, CocoIndex will use its incremental processing mechanism to update the target data. For more details, please refer to Continuous Sync between Source and Index.
It's super simple to get started. You just need to configure change data capture for your source:
```python
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.GoogleDrive(
...
recent_changes_poll_interval=datetime.timedelta(seconds=10)),
refresh_interval=datetime.timedelta(minutes=1))
```
recent_changes_poll_interval is the interval for polling recent changes from Google Drive.
refresh_interval is the interval for general change detection.
and run:
```sh
cocoindex update -L
```
-L is the flag for live update mode.
Or use cocoindex.FlowLiveUpdater in Python SDK.
For a complete example, please refer to Live Index from Google Drive Example.
For more detailed documentation, please refer to Live Update.
## Evaluation utilities
CocoIndex exposed a simple way to evaluate the quality of extraction via dumping output to files and comparing with goldens.
To dump the ETL output to YAML files, run:
```sh
python3 main.py cocoindex evaluate
```
It dumps what should be indexed to files under a directory.
And users could compare the output with golden files using tools like DirEqual or Meld.
A full example for evaluating and troubleshooting ETL (LLM data extraction) can be found in this blog.
## Support for date/time types
| Type | Description |Type in Python | Original Type in Python |
|------|-------------|---------------|-------------------------|
| Date | | datetime.date | datetime.date |
| Time | | datetime.time | datetime.time |
| LocalDatetime | Date and time without timezone | cocoindex.typing.LocalDateTime | datetime.datetime |
| OffsetDatetime | Date and time with a timezone offset | cocoindex.typing.OffsetDateTime | datetime.datetime |
For some types, CocoIndex Python SDK provides annotated types with finer granularity than Python's original type, e.g.
* *LocalDateTime* and *OffsetDateTime* for datetime.datetime, with different timezone awareness.
Review the full list of core data types.
An example of using date/time types in ETL data extraction can be found in this blog.
## New examples and tutorials
- Build text embeddings from Google Drive for RAG
- Build a patient intake form extraction pipeline
- New video - we listen to your suggestions! 🤗
## Thanks to the Community 🤗🎉!
- @Anush008 from qdrant made their first contribution in #182.
We are super excited for the upcoming official integration from our friends at qdrant!
We’re always improving CocoIndex and would love to hear your feedback.
## Full Changelog v0.1.13...v0.1.18
### v0.1.18 Incremental/Live update mode
- Support live update mode in CocoIndex engine, exposed by FlowLiveUpdater (Python SDK) and update -L / server -L (CLI).
- Offer a refresh_interval option for add_source() API in Python SDK for periodical metadata-traverse based change detection
- Update GoogleDrive data source to support detecting recent changes based on last modified time.
- Continuously show stats in live update mode.
- Make @main_fn decorator support async functions.
- Carry over more metadata for function and class decorators in Python SDK.
### v0.1.17 Core, Performance/Incremental updates
- Skip reprocessing a source row when source data and logic have no change.
- Keep source indexing states in memory to achieve lightweight incremental reprocessing when update called multiple times.
- Minor optimization for auto generated uuid as storage target key.
### v0.1.16 Core Data Types
- Support date/time types
### v0.1.15 Core, LLM structured extraction
- Add UUID type, and support automatically generating stable UUID.
- Support non-required field for LLM extraction.
- Storage target setup logic robustness improvement.
- Also allow collect() taking a constant.
### v0.1.14 Support Evaluation
- Support evaluate flow and dump output to files for offline evaluation purposes
### v0.1.13 Error handling, performance
- Add a field mime_type for GoogleDrive source.
- Count source rows with ERROR during indexing.
- Make error location caused by functions more clear.
- StructSchema takes an optional description and puts it in JSON schema
- Use class docstring as description for struct types in Python SDK.
- Correctly output unsupported type name.
- Correctly handle None values for composite types.
- Improve error message for encoding of field annotations.
- Bug fix: correctly deal with deleted/trashed files in GoogleDrive source
- Make indexing update stats more consistent with what really happened
- Bug fix: make sure dangling precommit states are properly handled.
---
# Keep derived data in sync with changing sources
URL: https://cocoindex.io/blogs/continuous-updates/
Date: 2025-04-07
Summary: CocoIndex continuously watches source changes and applies incremental updates to keep derived data in sync, with low latency and no full reindexing.
Continuous Updates
Today, we are excited to announce support for continuous updates for long-running pipelines in CocoIndex. This powerful feature automatically applies incremental source changes to keep your index up-to-date with minimal latency.
With continuous updates, your indexes remain synchronized with your source data in real-time, ensuring that your applications always have access to the most current information without the performance overhead of full reindexing.
## When to use it?
It fits into situations where you need to access the fresh target data continuously most of the time.
## How does it work?
It continuously captures changes from the source data and updates the target data accordingly. It's long-running and only stops when aborted explicitly.
A data source may enable one or multiple change capture mechanisms:
CocoIndex supports two main categories of change detection mechanisms:
1. **General Mechanism**:
- **Refresh Interval**: A universal approach applicable to all data sources, allowing periodic checks for changes by scanning and
comparing the current state with the previous state.
2. **Source-Specific Mechanisms**:
- **Push Change Notification**: Many data sources offer built-in mechanisms for real-time change notification which CocoIndex can subscribe to (coming soon!)
- **Recent Changes Poll**: Some data sources provide APIs for tracking recently modified entries, allowing efficient detection of changes; for example, Google Drive.
These mechanisms work together to ensure CocoIndex can detect and process changes as they happen, maintaining your index in perfect sync with source data with minimal latency and resource usage.
Under the hood, after the change is detected, CocoIndex will use its incremental processing mechanism to update the target data.
## How to enable continuous updates?
Here is an example of how to enable continuous updates for Google Drive. It is pretty simple:
### 1. Configure the change capture mechanisms for the source
```python
@cocoindex.flow_def(name="GoogleDriveIndex")
def my_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.GoogleDrive(
service_account_credential_path=credential_path,
root_folder_ids=root_folder_ids,
recent_changes_poll_interval=datetime.timedelta(seconds=10)),
refresh_interval=datetime.timedelta(minutes=1))
```
In this example, we've configured two change detection mechanisms:
1. recent_changes_poll_interval=datetime.timedelta(seconds=10): This is a Google Drive-specific mechanism that uses the Drive API's changes endpoint to efficiently detect modifications every 10 seconds.
This is an efficient fast scan to capture all latest modified files, and we could set it to a short interval to get fresher data.
It doesn't capture file deletions, so we need the fallback mechanism to ensure all changes are eventually captured.
2. refresh_interval=datetime.timedelta(minutes=1): This is the universal fallback mechanism that performs a complete scan of the data source every minute.
This is to scan all the files, to ensure all the changes - including the deleted files - are captured.
The refresh_interval parameter is particularly important as it serves as a safety net to ensure all changes are eventually captured,
even if source-specific mechanisms miss something. It works by:
- Periodically scanning the list of entries (e.g. file list with metadata like last modified time) at the specified interval
- Comparing the current list with the previously known list
- Identifying any differences (additions, modifications, deletions)
- Triggering updates only for the changed items
While source-specific mechanisms like recent_changes_poll_interval are more efficient for near real-time updates,
the refresh_interval provides comprehensive coverage.
We recommend setting it to a reasonable value based on your freshness requirements and resource constraints - shorter intervals provide fresher data but consume more resources.
You can read the full documentation:
- for source specific change detection mechanisms here. Different sources have different mechanisms supported.
- for the universal fallback mechanism, see the same documentation.
### 2. Run the flow in live update mode
#### Option 1: CLI
To run the CLI with live update mode, you can use the following command:
```sh
cocoindex update -L main
```
This will start the flow in live update mode, which will continuously capture changes from the source and update the target data accordingly.
#### Option 2: Python API
You can create cocoindex.FlowLiveUpdater. For example,
```python title="main.py"
async def main():
my_updater = cocoindex.FlowLiveUpdater(demo_flow)
await my_updater.wait()
...
if __name__ == "__main__":
asyncio.run(main())
```
And you run the flow with
```sh
python main.py
```
You can also use the updater as a context manager.
It will abort and wait for the updater to finish automatically when the context is exited.
See full documentation here.
Now you are all set! It is super simple to get started and have continuous updates for your data.
Get started now with the quickstart guide 🚀
It would mean a lot to us if you could support Cocoindex on Github with a star if you like our work. Thank you so much, with a warm coconut hug 🥥🤗.
---
# Incremental Processing with CocoIndex
URL: https://cocoindex.io/blogs/incremental-processing/
Date: 2025-04-06
Summary: What incremental processing is, who needs it, and how CocoIndex keeps an index in sync with source changes through caching, lineage tracking, and change data capture.
Incremental processing is one of the core values provided by CocoIndex.
In CocoIndex, users declare the transformation, and don't need to worry about the work to keep the index and source in sync.
CocoIndex creates & maintains an index, and keeps the derived index up to date based on source updates, with minimal computation and changes. That makes it suitable for ETL/RAG or any transformation tasks that stay low latency between source and index updates, and also minimizes the computation cost.
## What is incremental processing?
Figuring out what exactly needs to be updated, and only updating that without having to recompute everything throughout.
## How does it work?
You don't really need to do anything special, just focus on defining the transformation needed.
CocoIndex compute engine keeping the target store in sync: sources are added, updated, and deleted, and only the affected rows in the target store change
CocoIndex automatically tracks the lineage of your data and maintains a cache of computation results. When you update your source data, CocoIndex will:
1. Identify which parts of the data have changed
2. Only recompute transformations for the changed data
3. Reuse cached results for unchanged data
4. Update the index with minimal changes
And CocoIndex will handle the incremental processing for you.
CocoIndex provides two modes to run the pipeline:
- One-time update: Once triggered, CocoIndex updates the target data to reflect the version of source data up to the current moment.
- Live update: CocoIndex continuously reacts to changes of source data and updates the target data accordingly, based on various change capture mechanisms for the source.
Both modes run with incremental processing. You can view more details in Core Concepts.
## Who needs incremental processing?
Many people may think incremental processing is only beneficial for large scale data. Thinking carefully,
it really depends on the cost and requirement for data freshness.
Google processes huge scale data, backed by a huge amount of resources.
Your data scale is much less, but your resource provision is also much more limited.
Incremental processing is needed under the following conditions:
* **High freshness requirement**
For most user-facing applications this is needed. e.g. users update their documents,
and it's unexpected if they see stale information in search results.
If the search result is fed into an AI agent, it may mean an unexpected response to users (i.e. an LLM generates output based on inaccurate information).
It's more dangerous and users may even take the unexpected response without noticing.
* **Transformation cost is significantly higher than retrieval itself**
Overall, say T is your most acceptable staleness. If you don't want to recompute the whole thing repeatedly every cycle of T,
you will need incremental processing more or less.
Over a timeline with a tick every T, recomputing everything costs a full run each cycle, while incremental processing does a small delta of work each cycle. Same freshness, a fraction of the compute.
## How CocoIndex does incremental processing, by examples
We could take a look at a few examples to understand what CocoIndex handles behind the scenes for incremental processing.
### Example 1: Update a document
Consider this scenario:
* I have a document. Initially, it's split into 5 chunks, resulting in 5 rows with their embeddings in the index.
* After it's updated, 3 of them are exactly the same as previous. Others are changed.
Incremental Update Example
So we need to keep 3 rows, remove 2 previously existing rows, and add 2 new rows. These need to happen behind the scenes:
* Ideally, we only recompute embeddings for the 4 new rows, and reuse them for 3 unchanged chunks.
This can save computation cost especially when the embedding API charges by usage.
CocoIndex achieves this by maintaining a cache for heavy steps like embedding, and when the input for a transformation step isn't changed, the output will be reused.
* Besides, we also maintain a lineage tracking in the internal storage.
It keeps track of which rows in the index are derived from the previous version of this document, to make sure stale versions are properly removed.
CocoIndex takes care of this.
### Example 2: Delete a document
Continuing with the same example. If we delete the document later, we need to delete all 7 rows derived from the document.
Again, this needs to be based on the lineage tracking maintained by CocoIndex.
Deleting a document: lineage tracking links the document to all 7 rows derived from it, so all 7 rows are removed from the index without replaying the transformation.
### Example 3: Change of the transformation flow
The transformation flow may also be changed, for example, the chunking logic is upgraded, or a parameter passed to the chunker is adjusted.
This may result in the following scenario:
* Before the change, the document is split into 5 chunks, resulting in 5 rows with their embeddings in the index.
* After the change, they become 6 chunks: 4 of the previous chunks remain unchanged, the remaining 1 is split into 2 smaller chunks.
Logic Update Recomputation
This falls into a similar situation as document update (example 1), and CocoIndex will take care of it. The approach is similar, while this involves some additional considerations:
* We can still safely reuse embeddings for 4 unchanged chunks by the caching mechanism: this needs a prerequisite that the logic and parameters for embedding are unchanged.
If the changed part is the embedding logic itself, we will recompute the embeddings for everything.
CocoIndex is able to see if the logic or parameters for a step changed from the cached version, by putting this additional information in the cache key.
* To remove stale rows in the target index, the lineage tracking works well again.
Note that some other systems handle stale output deletions on source update/deletion by replaying the transformation logic on the previous version of input:
this only works well when transformation is fully deterministic and never upgraded.
CocoIndex's lineage tracking based approach doesn't have this limitation: it's robust to transformation logic non-determinism and changes.
### Example 4: Multiple inputs involved: merge / lookup / clustering
All examples above are simple cases: each single input row (e.g. a document) is involved independently during each specific transformation.
CocoIndex is a highly customizable framework, not only limited to simple chunking and embedding.
It allows users to perform more complex advanced transformations, such as:
* **Merge**.
For example, you're building an index for "all AI products", and you want to combine information from multiple sources, some products exist in one source and some in multiple.
For each product, you want to combine information from different sources.
* **Lookup**.
For example, you also have a data source about company information.
During your transformation for each product, you want to enrich it with information of the company building the product, so a lookup for the company information is needed.
* **Clustering**.
For example, you want to cluster different products into scenarios, and create cluster-level summaries based on information of products in the cluster.
The common theme is that during transformation, multiple input rows (coming from single or multiple sources) need to be involved at the same time.
Once a single input row is updated or deleted, CocoIndex will need to fetch other related rows from the same or other sources.
Here which other rows are needed is based on which are involved in the transformations.
CocoIndex keeps track of such relationships, and will fetch related rows and trigger necessary reprocessings incrementally.
Three multi-input patterns: merge combines the same item from two sources, lookup enriches a product row with its company row, clustering groups rows and summarizes each group. In every case a changed row only recomputes the outputs it was involved in; other outputs stay untouched.
## Change data capture (CDC)
Three change data capture mechanisms: push, where the source sends change notifications and CocoIndex computes on the diff; full scan, where every entry's metadata is compared against the previous state, which works for any source; and recent-changes polling, which is efficient but cannot detect deletions.
### 1. When source supports push change
Some source connectors support push change.
For example, Google Drive supports drive-level changelog and sends change notifications to your public URL,
which is applicable for team drive and personal drive (only by OAuth, service account not supported).
When a file is created, updated, or deleted, CocoIndex could compute based on the diff.
:::tip[Google Drive Change Log]
- Google Drive change log is only supported on the drive level, not the file/folder level.
- With service accounts, you can only enable changelog for team drives, by sharing the team drive with it.
:::
### 2. Metadata-based, fullscan
All source connectors in CocoIndex provide a basic list API capability, which enables a generic change detection mechanism applicable to any data source.
For example, with local file systems, we can traverse all directories and subdirectories recursively to get the full list of entries and their metadata (like modification time). By comparing the current state with the previous state, CocoIndex can detect changes even without source-specific change notifications.
This approach works universally across all data sources, though when the number of entries is large, performing a complete traversal can be resource-intensive.
This is a generic mechanism applicable to all data sources, so all data sources can leverage this even if there's no source-specific change capture.
### 3. Optimizations for specific sources
When the source connector provides advanced features to list entries, for example, listing most recently changed entries,
CocoIndex could take advantage of that to provide more efficient change detection.
For example, when changelog is not available for Google Drive (see the conditions in the tip above),
CocoIndex could monitor the change based on last modified vs last poll time, with a periodic trigger to check modified entries.
However, this cannot capture full change, for example when an entry has been deleted.
## Cache
In CocoIndex, every piece of the building block in the pipeline can be cached, keyed by memoization keys.
Currently, whether or not cache is enabled is decided by the implementation of the function.
For builtin functions, if it performs heavy transformation, cache is enabled.
The decision of whether a cached result can be reused comes down to a single question: did anything the function depends on change? CocoIndex answers it by reducing every input (the function arguments, declared dependencies, and context values) to a deterministic *fingerprint*. Each value is mapped to a canonical form (primitives and containers directly, dataclasses and Pydantic models field by field, and a pickle fallback for anything else), and those forms are combined into one key. If that key matches a previously stored entry, the cached output is a candidate for reuse. This is the mechanism behind every reuse decision described in the examples above: when the embedding step sees the same chunk text and the same logic, its fingerprint is unchanged, and the prior embedding is returned without recomputation.
Arguments, dependencies, and context values are each reduced to a canonical form, combined into one memo key, then looked up: a key seen before makes the cached output a candidate for reuse; a new key means compute and record.
The subtle part is separating *identity* from *freshness*. A file's path tells you *which* input you're looking at; its modification time and content tell you whether that input has *changed*. Folding everything into one key works, but it forces an expensive check on every run: for a file, that means reading and hashing the whole thing each time. CocoIndex lets you split the two: keep the stable identifier (path, URL, primary key) in the memoization key, and attach a *memo state* that validates freshness only after the fingerprint matches. A state function can do a cheap check first (compare the modification time) and fall back to the expensive one (hash the content) only when the cheap check fails. Even then, it can confirm the content is identical and reuse the cached result while quietly recording the new modification time. State validation can also run asynchronously, which is what makes freshness checks like an S3 HEAD request or an If-Modified-Since header practical. See memoization keys for the full model.
Identity, the path or URL or primary key, lives in the memo key and says which input this is. Freshness, the modification time and content hash, lives in the memo state and says whether it changed. After a fingerprint match: mtime unchanged means reuse on the fast path; otherwise hash the content, and an equal hash means reuse and record the new mtime; only truly changed content is recomputed.
Custom functions can take a parameter memo.
When True, CocoIndex memoizes the result of the function for reuse during reprocessing.
We recommend setting this to True for any function that is computationally intensive.
Output will be reused if both of these are unchanged: the input data and the behavior of the function.
When you change the function's logic, bump version so the previously memoized results are invalidated.
For example, this enables memoization for a function:
```python
import cocoindex as coco
@coco.fn(memo=True, version=1)
async def compute_something(arg1: str, arg2: int | None = None) -> str:
...
```
For a synchronous function you want to batch, force the async interface:
```python
@coco.fn.as_async(memo=True, batching=True)
def compute_batch(args: list[str]) -> list[str]:
...
```
Thanks for reading and we love to hear from you!
If you like our work, it would mean a lot to us if you could support Cocoindex on Github with a star.
---
# Structured Extraction from Patient Intake Form with LLM
URL: https://cocoindex.io/blogs/patient-intake-form-extraction-with-llm/
Date: 2025-03-26
Summary: Extract typed Patient records from PDF and DOCX intake forms with an LLM and CocoIndex v1: the nested schema is the whole prompt; results land in Postgres.
Patient intake forms arrive as PDFs and Word documents, in whatever layout each clinic uses. This post builds a CocoIndex v1 pipeline that reads those forms from Google Drive, extracts a fully typed Patient record with an LLM, and writes it to Postgres. The interesting part is that a single Python schema does all the instructing: there is no prompt template describing the fields, because the dataclass is the contract the model fills in.
This is the plain-LLM version. If you want the same extraction with a dedicated framework, see the BAML and DSPy variants. The full code is in the CocoIndex v1 repo.
## The flow
Each form is its own processing component: convert the document to Markdown, extract the Patient with the LLM, and declare a row in Postgres:
Flow: Google Drive holds intake forms as PDF and DOCX; one processing component per file converts the document to Markdown, extracts a typed Patient record with an LLM, and declares a row into the Postgres patients_info table.
## The schema is the prompt
The schema follows the FHIR Patient resource, simplified to a top-level Patient plus eight nested types. It is plain Python: nested objects (address, emergency_contact, pharmacy), optional fields marked | None, and lists for repeated items (past_conditions, current_medications, allergies, surgeries).
The Patient schema is the extraction contract: a top-level Patient with scalar fields, nested objects (Address, Contact, Insurance, Pharmacy), optional fields marked None, and lists of items (Condition, Medication, Allergy, Surgery). Address is reused by both Patient and Pharmacy rather than re-described.
The whole schema is a set of Pydantic models. instructor uses these as the response schema, so the model returns data that matches the Python types instead of free-form text:
```python title=models.py
import datetime
from pydantic import BaseModel, Field
class Address(BaseModel):
street: str
city: str
state: str
zip_code: str
class Contact(BaseModel):
name: str
phone: str
relationship: str
class Insurance(BaseModel):
provider: str
policy_number: str
group_number: str | None = None
policyholder_name: str
relationship_to_patient: str
class Pharmacy(BaseModel):
name: str
phone: str
address: Address
class Condition(BaseModel):
name: str
diagnosed: bool
class Medication(BaseModel):
name: str
dosage: str
class Allergy(BaseModel):
name: str
class Surgery(BaseModel):
name: str
date: str
class Patient(BaseModel):
name: str
dob: datetime.date
gender: str
address: Address
phone: str
email: str
preferred_contact_method: str
emergency_contact: Contact
insurance: Insurance | None = None
reason_for_visit: str
symptoms_duration: str
past_conditions: list[Condition] = Field(default_factory=list)
current_medications: list[Medication] = Field(default_factory=list)
allergies: list[Allergy] = Field(default_factory=list)
surgeries: list[Surgery] = Field(default_factory=list)
occupation: str | None = None
pharmacy: Pharmacy | None = None
consent_given: bool
consent_date: str | None = None
```
One declaration covers the whole intake form. The nested Address inside both Patient and Pharmacy is reused rather than re-described, and the | None markers tell the model which fields are allowed to be absent.
## Convert any format to Markdown
An LLM cannot read a raw PDF or DOCX well, so each file is first converted to Markdown with MarkItDown, which handles PDF, Word, Excel, images, and more:
```python title=main.py
import os
import tempfile
from markitdown import MarkItDown
_md = MarkItDown()
def to_markdown(content: bytes, filename: str) -> str:
suffix = os.path.splitext(filename)[1] or ".pdf"
with tempfile.NamedTemporaryFile(delete=True, suffix=suffix) as tmp:
tmp.write(content)
tmp.flush()
return _md.convert(tmp.name).text_content or ""
```
## Extract the Patient
Extraction is a single CocoIndex function. It calls the model through litellm and instructor, which forces the response to match the Patient schema. memo=True caches the result, so an unchanged form is never sent to the model twice. Swapping gpt-4o for a local Ollama model or another provider is a one-line change of LLM_MODEL:
```python title=main.py
import instructor
import litellm
import cocoindex as coco
from models import Patient
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
@coco.fn(memo=True)
async def extract_patient(markdown: str) -> Patient:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=Patient,
messages=[
{"role": "system", "content": "Extract patient information from the intake form."},
{"role": "user", "content": markdown},
],
)
return Patient.model_validate(result.model_dump())
```
## Declare a row per form
The Postgres row keeps a few flat columns for easy querying and stores the full nested record in a jsonb column (CocoIndex maps the dict to jsonb automatically). process_form reads one Drive file, converts it, extracts the patient, and declares the row:
```python title=main.py
from dataclasses import dataclass
import datetime
from cocoindex.connectors import google_drive, postgres
@dataclass
class PatientRecord:
filename: str # primary key
name: str
dob: datetime.date
reason_for_visit: str
patient: dict[str, object] # the full Patient, as jsonb
@coco.fn(memo=True)
async def process_form(
file: google_drive.DriveFile,
table: postgres.TableTarget[PatientRecord],
) -> None:
markdown = to_markdown(await file.read(), file.file_path.path.name)
patient = await extract_patient(markdown)
table.declare_row(
row=PatientRecord(
filename=file.file_path.path.name,
name=patient.name,
dob=patient.dob,
reason_for_visit=patient.reason_for_visit,
patient=patient.model_dump(mode="json"),
)
)
```
## Wire up the app
Connections are provided once in the lifespan; this is also where CocoIndex's local processing-state file is set. app_main mounts the Postgres table, points the Google Drive source at your folders, and fans out one process_form per file with mount_each:
```python title=main.py
import pathlib
from collections.abc import AsyncIterator
import asyncpg
DATABASE_URL = os.environ["POSTGRES_URL"]
PG_DB = coco.ContextKey[asyncpg.Pool]("patient_db")
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.settings.db_path = pathlib.Path("./cocoindex.db")
builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-4o"))
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
yield
@coco.fn
async def app_main() -> None:
table = await postgres.mount_table_target(
PG_DB,
table_name="patients_info",
table_schema=await postgres.TableSchema.from_class(
PatientRecord, primary_key=["filename"]
),
pg_schema_name="coco_examples",
)
source = google_drive.GoogleDriveSource(
service_account_credential_path=os.environ["GOOGLE_SERVICE_ACCOUNT_CREDENTIAL"],
root_folder_ids=[
f.strip()
for f in os.environ["GOOGLE_DRIVE_ROOT_FOLDER_IDS"].split(",")
if f.strip()
],
)
await coco.mount_each(process_form, source.items(), table)
app = coco.App(coco.AppConfig(name="PatientIntakeExtraction"), app_main)
```
To load from a local folder instead of Google Drive, swap the source for localfs.walk_dir; the rest of the flow is unchanged.
## Run and query
Set the environment (a Postgres URL, an OpenAI key, and Google Drive service-account credentials; see Setup for Google Drive), then build the index:
```sh
pip install "cocoindex[postgres,google_drive]" instructor litellm markitdown asyncpg
cocoindex update main
```
Re-running processes only new or changed forms, since per-form extraction is memoized. Query the result in Postgres, where the flat columns and the jsonb record are both available:
```sql
SELECT filename, name, reason_for_visit,
patient -> 'allergies' AS allergies
FROM coco_examples.patients_info;
```
## Extraction quality
LLM extraction is not perfect, and evaluation is the real work. Structured forms with clear labels are the easy case; a scan with an unusual layout can drop a field or misread a value. When accuracy matters, keep a set of hand-checked golden records and diff each run against them, and try a different model or converter on the forms that miss. Because the schema is a single declaration, tightening a field description or swapping the model is a small change with the same typed output.
## Support us
We are constantly adding examples and improving the runtime. If this was helpful, please star CocoIndex on GitHub.
---
# Search your Google Drive by meaning
URL: https://cocoindex.io/blogs/text-embedding-from-google-drive/
Date: 2025-03-23
Summary: Index the documents in a shared Google Drive folder as text embeddings in Postgres with CocoIndex v1, then search them by meaning instead of by filename.
The documents you actually search are rarely on your laptop; they are in a shared Google Drive folder. This post indexes that folder with CocoIndex v1: it reads each document, splits it into chunks, embeds each chunk, and stores the vectors in Postgres, so you can find a passage by what it means rather than by remembering its filename.
If text embeddings are new to you, Text embeddings 101 explains the idea first. This post is the same recipe with the source swapped to Google Drive; the interesting part here is connecting to Drive.
The full code is in the CocoIndex v1 repo.
## The flow
Each file in the folder is its own processing component: read it, split it into chunks, embed each chunk, and declare a row in Postgres pgvector:
Indexing flow: a shared Google Drive folder feeds a CocoIndex app; one processing component per file splits it into chunks, embeds each chunk with a sentence-transformer model, and declares a row that CocoIndex stores in a Postgres pgvector table for semantic search.
## Connect Google Drive
CocoIndex reads Drive through a Google Cloud service account: a robot identity with a JSON key. You grant that identity read access by sharing your folder with its email address, and CocoIndex authenticates with the key. That sharing step is the whole access model:
Google Drive access model: a service account has a JSON key and an email; you grant that email Viewer access on your Drive folder; CocoIndex authenticates with the JSON key and reads the shared folder's files.
The setup is one-time. Here is the full walkthrough; the same steps are in the Google Drive connector docs.
### Enable Google Drive access by service account
CocoIndex provides a native builtin to support Google Drive as a source. You could find the full documentation here.
### 1. Register / log in to Google Cloud.
First, you need to create a Google Cloud account if you don't have one already. Go to the Google Cloud Console and sign up or sign in.
Google Cloud Console
### 2. Select or create a GCP project
Once you've logged into Google Cloud Console, you need to select an existing project or create a new one. Click on the project selector dropdown at the top of the page:
- If you already have a project, you can select from here:
Select or Create a GCP Project
- If you are new to Google Cloud, it looks like this:
Select or Create a GCP Project New User
### 3. Create a service account
1. In Google Cloud Console, search for Service Accounts, to enter the IAM & Admin / Service Accounts page.
Service Account Search
2. Click on "CREATE SERVICE ACCOUNT" at the top of the page:
Create Service Account
3. Fill in the service account name, e.g. cocoindex-test.
Create Service Account Form
And make a note of that email address; you will need it in a later step.
4. Click on "CREATE" to create the service account.
You will see the service account created successfully.
Service Account Listing
### 4. Create and download the key for the service account
1. Click on "Actions" and select "Manage Keys".
Manage Keys
2. Select "Add Key" and select "Create new key".
Create New Key
Choose "JSON" as the key type and click "Create".
Create JSON Key
3. The key file will be downloaded to your computer. Depending on the browser setting, it starts downloading automatically or may pop up a dialog for the location to download. Keep this file secure as it provides access to your Google Drive resources.
Key Downloaded
It looks like this:
```json
{
"type": "service_account",
"project_id": "cocoindexdriveexample",
"private_key_id": "key_id",
"private_key": "PRIVATE_KEY",
"client_email": "cocoindex-test@cocoindexdriveexample.iam.gserviceaccount.com",
"client_id": "id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cocoindex-test%40cocoindexdriveexample.iam.gserviceaccount.com",
"universe_domain": "googleapis.com"
}
```
### 5. Enable Google Drive API
Search for "Google Drive API" and select it.
Enable Google Drive API
Make sure it is enabled. You can check it from the page.
Google Drive API Enabled
### 6. Prepare and share a folder
1. Create a new folder or use an existing folder in your Google Drive.
- For this project, we will create a folder in my own Google Drive, and share it with the service account email address we created in Step 3. For example, cocoindex-test@cocoindexdriveexample.iam.gserviceaccount.com.
- The files are also available in the example repo.
2. Share the folder with the service account. Enter the service account email address (e.g., cocoindex-test@cocoindexdriveexample.iam.gserviceaccount.com) and give it "Viewer" access.
Create a new folder in Google Drive
3. Note the folder ID from the URL when you open the folder. The URL will look like:
```
https://drive.google.com/drive/folders/1AbCdEfGhIjKlMnOpQrStUvWxYz
```
The folder ID is the part after folders/ (in this example: 1AbCdEfGhIjKlMnOpQrStUvWxYz).
You'll need this folder ID when connecting to the Google Drive API.
Now you are all set! 🎉 You can start to build your text embeddings from Google Drive. 📁✨
Then point the pipeline at the credential and folder with a .env file:
```sh
POSTGRES_URL=postgres://cocoindex:cocoindex@localhost/cocoindex
GOOGLE_SERVICE_ACCOUNT_CREDENTIAL=/path/to/service_account_credential.json
GOOGLE_DRIVE_ROOT_FOLDER_IDS=1AbCdEfGhIjKlMnOpQrStUvWxYz
```
## Define the pipeline
The Postgres pool and the embedder are provided once in the app lifespan and read anywhere with coco.use_context:
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import google_drive, postgres
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.ops.text import RecursiveSplitter
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
PG_DB = coco.ContextKey[asyncpg.Pool]("gdrive_text_embedding_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
_splitter = RecursiveSplitter()
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
```
Each chunk becomes one row. The embedding column is annotated with the embedder so CocoIndex knows the vector's size:
```python title=main.py
@dataclass
class DocEmbedding:
id: int
filename: str
text: str
embedding: Annotated[NDArray, EMBEDDER]
```
## Chunk, embed, declare
process_file reads one Drive file, splits it into chunks, and maps _emit_chunk over them; _emit_chunk embeds the chunk and declares its row. memo=True caches per-file work, so an unchanged file is skipped on the next run:
```python title=main.py
@coco.fn(memo=True)
async def process_file(
file: google_drive.DriveFile,
table: postgres.TableTarget[DocEmbedding],
) -> None:
text = await file.read_text()
chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language="markdown")
id_gen = IdGenerator()
await coco.map(_emit_chunk, chunks, file.file_path.path.as_posix(), id_gen, table)
@coco.fn
async def _emit_chunk(
chunk: Chunk,
filename: str,
id_gen: IdGenerator,
table: postgres.TableTarget[DocEmbedding],
) -> None:
table.declare_row(
row=DocEmbedding(
id=await id_gen.next_id(chunk.text),
filename=filename,
text=chunk.text,
embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
),
)
```
## Wire up the app
app_main mounts the pgvector table, points the Google Drive source at your folders, and fans out one process_file per file with mount_each:
```python title=main.py
@coco.fn
async def app_main() -> None:
table = await postgres.mount_table_target(
PG_DB,
table_name="doc_embeddings",
table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=["id"]),
pg_schema_name="coco_examples",
)
source = google_drive.GoogleDriveSource(
service_account_credential_path=os.environ["GOOGLE_SERVICE_ACCOUNT_CREDENTIAL"],
root_folder_ids=[
f.strip()
for f in os.environ["GOOGLE_DRIVE_ROOT_FOLDER_IDS"].split(",")
if f.strip()
],
)
await coco.mount_each(process_file, source.items(), table)
app = coco.App(coco.AppConfig(name="GoogleDriveTextEmbedding"), app_main)
```
source.items() yields one (path, DriveFile) per document, and mount_each gives each file a stable component path so change detection is per file.
## Run and search
Install and build the index:
```sh
pip install "cocoindex[postgres,google_drive]" sentence-transformers asyncpg pgvector
cocoindex update main
```
The first run embeds every document in the folder. Re-run cocoindex update main anytime and only new or edited files are re-embedded, so keeping the index current is cheap; schedule it (cron, a cloud function) for whatever freshness you need.
Then search from the command line. The query is embedded with the same model, and Postgres returns the nearest chunks by cosine distance, even when they share none of your words:
```sh
python main.py "how do I request time off?"
```
## Where it goes next
The source is the only thing specific to Google Drive here. Swap it for Amazon S3, Azure Blob, or the local filesystem and the chunk-embed-store pipeline is unchanged. Point the retrieved chunks at an LLM and you have a question-answering assistant over your team's Drive; keep them as-is and you have semantic search that finds the right document by meaning.
## Support us
We are constantly adding examples and improving the runtime. If this was helpful, please star CocoIndex on GitHub.
---
# CocoIndex Changelog 2025-03-20
URL: https://cocoindex.io/blogs/cocoindex-changelog-2025-03-20/
Date: 2025-03-20
Summary: First release of CocoIndex Changelog: LLM support, codebase indexing, custom functions, and assorted core/performance improvements
We're excited to share our progress with you! We'll be publishing these updates weekly, but since this is our first one, we're covering highlights from the last two weeks.
We had 9 releases in the last 2 weeks over 100+ PRs merged (Yes, we shipped a lot!). Here are the highlights.
Thank you for your support and warm coconut hugs to all our contributors! 🤗
Let's dive into what we've been building...
## 📖 Documentation
We've added a lot of new material to back you up. See our shining documentation site for cocoindex.
- Getting Started
- CocoIndex Core Concepts
- CocoIndex Built-in Ops
- AI Support
We've received a lot of feedback from our community, and we're excited to see you're using cocoindex in your projects! And we've added onboarding videos to help you get started.
## ✨ LLM Support
We've added support for LLM to extract structured information from text. See the documentation for LLM Support.
CocoIndex provides powerful built-in functions like ExtractByLlm that leverage Large Language Models to process your data. To use these functions, you'll need to configure an LLM Spec that defines which LLM integration you want to use (such as OpenAI, Ollama), specify the model, and set any additional parameters needed for your specific use case.
LLM Support
For Ollama, you can pull models from the Ollama Model Library, which offers a variety of open-source models including the latest published models like Gemma 3.0. Simply use the ollama pull command followed by the model name (e.g., ollama pull llama3.2).
We published a blog post with a step-by-step guide to use Ollama for structured extraction from a PDF. See the full code example here.
## 📦 Native support on chunking for codebases
We've added native support for chunking codebases with Tree-sitter integration. This allows for more intelligent code splitting based on syntax structure rather than arbitrary line breaks.
Code Indexing for RAG
CocoIndex leverages Tree-sitter to parse code and extract syntax trees for various programming languages. This enables syntactically coherent chunking that preserves the context and structure of your code, making your RAG system more effective for code retrieval.
We've added support for indexing codebases, with a step-by-step guide on how to index a codebase. Read the full blog post here and see the full code example here.
## 🛠️ Support composite types in Python custom functions
In CocoIndex, all data processed by the flow has a type determined when the flow is defined, before any actual data is processed at runtime. Read the full description and documentation for data types here. This makes the schema of data processed by CocoIndex clear, and makes it easy to determine the schema of your index.
These types were already supported in core, and data sources and functions like SplitRecursively already produced these data types.
Now we start to support representing struct and collection types in Python, so that they can be used in the following ways:
### 1. Used as input/output for your custom functions
For example, we can define a custom function:
```python
@cocoindex.op.function()
def summarize_module(module_info: ModuleInfo) -> ModuleSummary:
"""Summarize a Python module."""
return ModuleSummary(
num_classes=len(module_info.classes),
num_methods=len(module_info.methods),
)
```
Here ModuleInfo and ModuleSummary are custom dataclass types.
For a full example, see here.
### 2. For builtin functions taking custom types
For an extraction example, see here.
```python
doc["module_info"] = doc["markdown"].transform(
cocoindex.functions.ExtractByLlm(
llm_spec=cocoindex.LlmSpec(
api_type=cocoindex.LlmApiType.OLLAMA,
# See the full list of models: https://ollama.com/library
model="llama3.2"
),
# Replace by this spec below, to use OpenAI API model instead of ollama
# llm_spec=cocoindex.LlmSpec(
# api_type=cocoindex.LlmApiType.OPENAI, model="gpt-4o"),
output_type=ModuleInfo,
instruction="Please extract Python module information from the manual."))
```
## 🧩 Make custom functions easier to define
You can directly define your custom function with a function decorator. It fits into simple cases where the function doesn't need to take additional configurations and extra setup logic.
See documentation here.
Example usage below. See full code example here:
```python
@cocoindex.op.function()
def summarize_module(module_info: ModuleInfo) -> ModuleSummary:
"""Summarize a Python module."""
return ModuleSummary(
num_classes=len(module_info.classes),
num_methods=len(module_info.methods),
)
```
## ⚡ Support optimizations by caching
Custom functions can take a parameter cache. When True, the executor will cache the result of the function for reuse during reprocessing. We recommend setting this to True for any function that is computationally intensive.
Output will be reused if all these are unchanged: spec (if exists), input data, behavior of the function. For this purpose, a behavior_version needs to be provided, and should increase on behavior changes.
For example, this enables cache for a standalone function, see full code example here:
```python
@cocoindex.op.executor_class(gpu=True, cache=True, behavior_version=1)
class PdfToMarkdownExecutor:
"""Executor for PdfToMarkdown."""
...
```
See the documentation here.
## Thanks to the community 🤗🎉!
- @eltociear made their first contribution in #68
- @wykrrr made their first contribution in #112
## More upgrades
### v0.1.12 Basic Google Drive support, chunking improvement
- Support for more languages in SplitRecursively.
- Skip chunks without alphanumeric characters.
- Add GoogleDrive source.
### v0.1.11 Native support on chunking for codebase, @function decorator
- Add support for more languages in SplitRecursively.
- Add @function decorator for simple functions.
### v0.1.9 Improve native support for chunking, Core robustness
- Use OpArgsResolver to make resolving multiple input args easier.
- Extract Rust - Python binding logic into a module for reuse
- Robust support for passing constants to functions.
- Update SplitRecursively to take language and chunk sizes dynamically.
- Update the documents about SplitRecursively and transform arguments.
### v0.1.8 LLM Integration
- Added support for extracting structured information from text by LLM
- ExtractByLlm function
- LLM integration
- Manual Extraction example.
### v0.1.7 Support more data types, Core robustness
- Support Python->Rust Struct/Table type bindings.
- Skip update table if type changed without storage type change.
- Use List instead of Table for Python list type
- Serializing/deserializing struct to JSON object for Postgres.
- Support type annotations for both List and Table in Python SDK.
- Support any data type in Rust->Python.
- Refactor logic to parse Python type, to support dataclass type binding.
- Support ... | None (or Optional[...]) when analyzing type.
- Support creating dataclasses in Rust->Python value bindings.
- Use custom code to convert Rust values to Python so bytes are bytes
### v0.1.6 Serialization fix
- Bunch of serialization fixes in Python SDK.
### v0.1.5 Core, Performance, Python SDK, Query Metrics
- Correct similarity score for cosine and inner product.
- Leverage pythonize to bypass JSON serialization in Python to Rust.
- Support struct and table types in Python to Rust type encoding.
### v0.1.4 Core, Performance, Custom Function, Improve SentenceTransformerEmbed
- Avoid clone when reading data from cache.
- Custom function documentation: enable cache, links to examples.
- SentenceTransformerEmbed supports additional args passed to the library.
- Polish document for quickstart.
- Bug fix: for evaluate API, use position in source instead of schema.
- Enable evaluation cache in the readonly evaluation API.
### v0.1.3 Core & performance
- Add cache utilities and plumbing through the evaluator.
- Tolerate JSON null in memoization_info field in DB.
- Implement a generic fingerprinter for computing cache key.
- Evaluator starts to read/write cache if enabled
- A bunch of bug fixes / cleanups for cache feature.
- Output detailed error when indexing failed for a row.
- Backward compatible to old-format fingerprint already in database.
- Enable cache in SentenceTransformerEmbed and PdfToMarkdown.
---
# Build Real-Time Codebase Indexing for AI Code Generation
URL: https://cocoindex.io/blogs/index-code-base-for-rag/
Date: 2025-03-18
Summary: Indexing codebase for RAG with CocoIndex and Tree-sitter in real-time: chunking, embedding, semantic search, and build vector index for efficient retrieval.
In this blog, we will show you how to index a codebase for RAG with CocoIndex.
CocoIndex provides built-in support for codebase chunking, with native Tree-sitter support and real-time update.
Because we use incremental processing (only reprocessing what has changed) under the hood, the index is updated in near real-time.
It can be real-time if we integrate with any source that directly pushes change notifications, e.g., a code editor.
This can be built for code search applications, or for providing relevant context to AI code generation systems where low latency is critical for a seamless developer experience.
It'd mean a lot to us if you could **⭐ star CocoIndex on Github**, if this tutorial helps you. GitHub
## Tree-sitter
CocoIndex leverages Tree-sitter's capabilities to intelligently chunk code based on the actual syntax structure rather than arbitrary line breaks.
These syntactically coherent chunks are then used to build a more effective index for RAG systems, enabling more precise code retrieval and better context preservation.
Tree-sitter is a parser generator tool and an incremental parsing library;
it is available in Rust 🦀.
You can find the full code of this project here, and a walkthrough in the index-codebase example.
Only ~ 50 lines of Python on the indexing path, check it out :rocket:!
## Use case
- Build semantic code context for AI coding agents like Claude, Codex, Gemini CLI.
- Build MCP for code editors such as Cursor, Windsurf, and VSCode.
- Build context-aware code search applications - semantic code search, natural language code retrieval.
- Build context for code generation applications - AI code generation, code autocompletion, code assistant bots.
- Build context for code review applications - AI-powered code review, automated code analysis, code quality checks, pull request summarization, code diff understanding.
- Build context for code refactoring applications - automated code refactoring, code transformation, large-scale code migration.
CocoIndex Use Case
## Why build with CocoIndex?
- Super scalable, ultra-performant, core engine written in Rust. Works with very large codebases.
- Incremental processing, only re-process what has changed, no need to re-index the whole codebase on what's changed, real-time performance with minimal computation.
- Logic customizable, pick your favorite embedding model, strategy, vector database, and more.
## Prerequisites
If you don't have Postgres installed, please follow the installation guide.
CocoIndex uses Postgres to keep track of data lineage for incremental processing.
## Define indexing flow
### Flow design
CocoIndex Flow for Code Embedding
The flow diagram illustrates how we'll process our codebase:
1. Read code files from the local filesystem
2. Extract file extensions, to get the language of the code for Tree-sitter to parse
3. Split code into semantic chunks using Tree-sitter
4. Generate embeddings for each chunk
5. Store in a vector database for retrieval
### 1. Add the codebase as a source.
```python
@cocoindex.flow_def(name="CodeEmbedding")
def code_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
"""
Define an example flow that embeds files into a vector database.
"""
data_scope["files"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="../..",
included_patterns=["*.py", "*.rs", "*.toml", "*.md", "*.mdx"],
excluded_patterns=[".*", "target", "**/node_modules"]))
code_embeddings = data_scope.add_collector()
```
We ingest the CocoIndex codebase from the root directory.
We can change the path to the codebase you want to index.
We index all the files with the extensions of .py, .rs, .toml, .md, .mdx, and skip directories starting with ., target (in the root) and node_modules (under any directory).
flow_builder.add_source will create a table with sub fields (filename, content). We can refer to the documentation for more details.
Ingest Codebase
### 2. Process each file and collect the information.
#### 2.1 Extract the extension of a filename
We need to pass the language (or extension) to Tree-sitter to parse the code.
Let's define a function to extract the extension of a filename while processing each file.
You can find the documentation for custom functions here.
```python
@cocoindex.op.function()
def extract_extension(filename: str) -> str:
"""Extract the extension of a filename."""
return os.path.splitext(filename)[1]
```
Then we are going to process each file and collect the information.
```python
with data_scope["files"].row() as file:
file["extension"] = file["filename"].transform(extract_extension)
```
Here we extract the extension of the filename and store it in the extension field.
Extract Extension
#### 2.2 Split the file into chunks
We will chunk the code with Tree-sitter.
We use the SplitRecursively function to split the file into chunks.
It is integrated with Tree-sitter, so you can pass in the language to the language parameter.
To see all supported language names and extensions, see the documentation here. All the major languages are supported, e.g., Python, Rust, JavaScript, TypeScript, Java, C++, etc. If it's unspecified or the specified language is not supported, it will be treated as plain text.
```python
with data_scope["files"].row() as file:
file["chunks"] = file["content"].transform(
cocoindex.functions.SplitRecursively(),
language=file["extension"], chunk_size=1000, chunk_overlap=300)
```
Split Code
#### 2.3 Embed the chunks
We use SentenceTransformerEmbed to embed the chunks.
You can refer to the documentation here.
There are 12k models supported by 🤗 Hugging Face.
You can just pick your favorite model.
```python
@cocoindex.transform_flow()
def code_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:
"""
Embed the text using a SentenceTransformer model.
"""
return text.transform(
cocoindex.functions.SentenceTransformerEmbed(
model="sentence-transformers/all-MiniLM-L6-v2"))
```
Then for each chunk, we will embed it using the code_to_embedding function and collect the embeddings to the code_embeddings collector.
@cocoindex.transform_flow() is needed to share the transformation across indexing and query. We build a vector index and query against it;
the embedding computation needs to be consistent between indexing and querying. See documentation for more details.
```python
with data_scope["files"].row() as file:
with file["chunks"].row() as chunk:
chunk["embedding"] = chunk["text"].call(code_to_embedding)
code_embeddings.collect(filename=file["filename"], location=chunk["location"],
code=chunk["text"], embedding=chunk["embedding"])
```
Embed Chunks
#### 2.4 Collect the embeddings
Export the embeddings to a table.
```python
code_embeddings.export(
"code_embeddings",
cocoindex.storages.Postgres(),
primary_key_fields=["filename", "location"],
vector_indexes=[cocoindex.VectorIndex("embedding", cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])
```
We use Cosine Similarity to measure the similarity between the query and the indexed data.
To learn more about Cosine Similarity, see Wiki.
### 3. Query the index
We match against user-provided text by a SQL query, reusing the embedding operation in the indexing flow.
```python
def search(pool: ConnectionPool, query: str, top_k: int = 5):
# Get the table name, for the export target in the code_embedding_flow above.
table_name = cocoindex.utils.get_target_storage_default_name(code_embedding_flow, "code_embeddings")
# Evaluate the transform flow defined above with the input query, to get the embedding.
query_vector = code_to_embedding.eval(query)
# Run the query and get the results.
with pool.connection() as conn:
with conn.cursor() as cur:
cur.execute(f"""
SELECT filename, code, embedding <=> %s::vector AS distance
FROM {table_name} ORDER BY distance LIMIT %s
""", (query_vector, top_k))
return [
{"filename": row[0], "code": row[1], "score": 1.0 - row[2]}
for row in cur.fetchall()
]
```
Define a main function to run the query in the terminal.
```python
def main():
# Initialize the database connection pool.
pool = ConnectionPool(os.getenv("COCOINDEX_DATABASE_URL"))
# Run queries in a loop to demonstrate the query capabilities.
while True:
try:
query = input("Enter search query (or Enter to quit): ")
if query == '':
break
# Run the query function with the database connection pool and the query.
results = search(pool, query)
print("\nSearch results:")
for result in results:
print(f"[{result['score']:.3f}] {result['filename']}")
print(f" {result['code']}")
print("---")
print()
except KeyboardInterrupt:
break
if __name__ == "__main__":
main()
```
## Run the index setup & update
🎉 Now you are all set!
Run the following command to set up and update the index.
```sh
cocoindex update --setup main
```
You'll see the index updates state in the terminal.
Terminal showing index update process
## Test the query
At this point, you can start the CocoIndex server and develop your RAG runtime against the data. To test your index, you could
``` bash
python main.py
```
When you see the prompt, you can enter your search query. For example: spec.
You can find the search results in the terminal.
Search results in terminal
The returned results: each entry contains the score (Cosine Similarity), filename, and the code snippet that gets matched.
## Support us
We are constantly improving, and more features and examples are coming soon. If you love this article, please give us a star ⭐ at GitHub to help us grow.
Thanks for reading!
---
# On-premise structured extraction from PDFs with Ollama
URL: https://cocoindex.io/blogs/cocoindex-ollama-structured-extraction-from-pdf/
Date: 2025-03-17
Summary: Extract structured data from PDF manuals locally with Ollama and CocoIndex: docling converts PDFs to Markdown, a local LLM fills typed Postgres rows.
Manuals, datasheets, and reference docs are full of structure: titles, classes, functions, parameters, defaults. It's all laid out for humans, not machines. This post builds a CocoIndex pipeline that pulls that structure back out into typed rows, and does it entirely on your own hardware. Each PDF is converted to Markdown with docling, a local Ollama model extracts a nested schema, and the result lands in Postgres. The documents never leave your machine.
Every stage runs on one machine: PDF manuals are parsed to Markdown by docling, a local Ollama model extracts structure, and rows land in Postgres. No document is ever sent to a hosted API.
On-premise matters when the documents are sensitive: internal manuals, contracts, medical or financial filings. Running the LLM locally with Ollama means the data stays inside your network, and you can still deploy the same pipeline to your own cloud or server later without changing a line.
The full example is in the CocoIndex repo under examples/manuals_llm_extraction, and there's a step-by-step docs walkthrough too.
## Introducing CocoIndex
CocoIndex is an incremental data transformation framework. You declare target_state = transformation(source_state) in plain async Python and your own types, and a Rust engine underneath handles change tracking, incremental processing, and managed targets. Edit one manual and CocoIndex re-parses and re-extracts only that one, updating its row in place. The whole example is about 100 lines.
## Introducing Ollama
Ollama runs open LLMs on your local machine with a single command. Download and install it, then pull a model:
```sh
ollama pull llama3.2
```
CocoIndex talks to Ollama through LiteLLM, so the model is just a string: ollama/llama3.2. Swap that string for ollama/mistral, ollama/qwen2.5, or a hosted provider like openai/gpt-4o, and nothing else in the pipeline changes. For a fully on-premise setup, keep it pointed at Ollama.
## Overview of the pipeline
The pipeline: each PDF in the manuals folder is converted to Markdown, extracted into a typed ModuleInfo by the local LLM, and declared as one Postgres row.
At a high level, the flow watches a manuals/ folder and turns each PDF into one typed Postgres row:
1. **Ingest PDFs from a local folder.** CocoIndex watches manuals/ and reads new or updated PDF files as they appear.
2. **Convert each PDF to Markdown.** docling parses the document into clean Markdown, run on a coco.GPU runner so the heavy work stays off the event loop.
3. **Extract a nested schema with the local LLM.** The Markdown goes to Ollama through instructor, which constrains the output to a typed ModuleInfo (title, description, classes, methods, arguments).
4. **Declare one Postgres row per manual.** Summary counts plus the full structure as JSON.
5. **Stay incremental.** Both the parse and the extraction are cached by content, so re-running only touches what changed.
The output is a Postgres table that mirrors each document's real shape, kept in sync with the folder as manuals come and go.
## Define the pipeline
CocoIndex v1 is declarative and Python-native. The entry point is an app_main function wrapped in a coco.App: it mounts one Postgres table target, walks the manuals/ folder for PDFs in live mode, and runs one processor per file. The whole example lives in one main.py.
A manuals folder of PDFs flows through a CocoIndex app (source, transform f(x), target state) into a Postgres table.
```python title=main.py
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
table = await postgres.mount_table_target(
PG_DB,
table_name=TABLE_NAME,
table_schema=await postgres.TableSchema.from_class(
ModuleRecord, primary_key=["filename"]
),
pg_schema_name=PG_SCHEMA_NAME,
)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
live=True,
)
await coco.mount_each(process_file, files.items(), table)
app = coco.App(
coco.AppConfig(name="ManualsLlmExtraction"),
app_main,
sourcedir=pathlib.Path("./manuals"),
)
```
The live=True source keeps watching the folder, so a manual dropped in later is picked up without a restart. coco.mount_each runs process_file once per matched PDF.
### The extraction schema
The output type is nested Pydantic, and the structure itself is the instruction to the model. A ModuleInfo has classes (each with methods) and module-level methods (each with args), so the model knows exactly what to pull out at every level, no per-level prompt tuning.
The nested schema: ModuleInfo holds lists of ClassInfo and MethodInfo, each ClassInfo holds MethodInfo, and each MethodInfo holds ArgInfo, repeated for every class, method, and argument the LLM extracts.
```python
class ArgInfo(pydantic.BaseModel):
name: str
description: str = ""
class MethodInfo(pydantic.BaseModel):
name: str
args: list[ArgInfo] = pydantic.Field(default_factory=list)
description: str = ""
class ClassInfo(pydantic.BaseModel):
name: str
description: str = ""
methods: list[MethodInfo] = pydantic.Field(default_factory=list)
class ModuleInfo(pydantic.BaseModel):
title: str
description: str
classes: list[ClassInfo] = pydantic.Field(default_factory=list)
methods: list[MethodInfo] = pydantic.Field(default_factory=list)
```
### Convert PDF to Markdown
Convert step: docling parses each PDF into clean Markdown on the GPU runner.
Ollama does not take PDFs directly, so each document is converted to Markdown first. docling handles the parse; the function is decorated @coco.fn.as_async(runner=coco.GPU) so the heavy work runs on the GPU runner instead of blocking the event loop.
```python
@functools.cache
def pdf_converter() -> DocumentConverter:
options = PdfPipelineOptions(
accelerator_options=AcceleratorOptions(device=AcceleratorDevice.CPU)
)
return DocumentConverter(
format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=options)}
)
@coco.fn.as_async(runner=coco.GPU)
def pdf_to_markdown(content: bytes) -> str:
source = DocumentStream(name="manual.pdf", stream=io.BytesIO(content))
return pdf_converter().convert(source).document.export_to_markdown()
```
### Extract structured data with Ollama
Extract step: the local Ollama model turns the Markdown into a typed ModuleInfo.
The Markdown goes to the local model. instructor wraps LiteLLM and forces the response to validate against ModuleInfo, so you get a typed object back rather than free text to parse. The model comes from context (LLM_MODEL), which is ollama/llama3.2 for an on-premise run.
```python
@coco.fn(memo=True)
async def extract_module(markdown: str) -> ModuleInfo:
client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
result = await client.chat.completions.create(
model=coco.use_context(LLM_MODEL),
response_model=ModuleInfo,
messages=[
{"role": "system", "content": EXTRACT_PROMPT},
{"role": "user", "content": markdown},
],
)
return ModuleInfo.model_validate(result.model_dump())
```
@coco.fn(memo=True) caches the extraction by its input, so an unchanged manual is never re-sent to the model on later runs.
### Declare the Postgres row
Declare step: each manual becomes one Postgres row, kept in sync as files change.
process_file ties it together: read the file, convert, extract, then *declare* one row. You describe the row you want; CocoIndex inserts, updates, or deletes it to match. The row carries the summary counts and the full ModuleInfo as JSON.
```python
@dataclass
class ModuleRecord:
filename: str # primary key
title: str
description: str
num_classes: int
num_methods: int
module_info: str # the full ModuleInfo as JSON
@coco.fn(memo=True)
async def process_file(
file: FileLike,
table: postgres.TableTarget[ModuleRecord],
) -> None:
markdown = await pdf_to_markdown(await file.read())
info = await extract_module(markdown)
table.declare_row(
row=ModuleRecord(
filename=file.file_path.path.name,
title=info.title,
description=info.description,
num_classes=len(info.classes),
num_methods=len(info.methods),
module_info=json.dumps(info.model_dump()),
)
)
```
### Providing shared resources
The Postgres pool and the model name are provided once at startup with @coco.lifespan and read back inside the functions with coco.use_context. LLM_MODEL is declared with detect_change=True, so if you swap models later, CocoIndex re-extracts everything against the new one with no cache to clear by hand.
```python
PG_DB = coco.ContextKey[asyncpg.Pool]("manuals_db")
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(os.environ["POSTGRES_URL"]) as pool:
builder.provide(PG_DB, pool)
builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "ollama/llama3.2"))
yield
```
## Running the example
### Prerequisites
- Python 3.11+
- Ollama installed, with a model pulled: ollama pull llama3.2
- Postgres (the pipeline stores the index there)
Everything runs locally. With Ollama serving the model and Postgres on your own machine, no document content leaves the box.
### Quick start
The example ships a manuals/ folder of Python module reference PDFs (array, base64, copy).
```sh
cd examples/manuals_llm_extraction
# Start Postgres
docker compose -f ../../dev/postgres.yaml up -d
# Configure and install
cp .env.example .env # set POSTGRES_URL and LLM_MODEL=ollama/llama3.2
pip install -e .
# Build the index
cocoindex update main # catch-up run
cocoindex update -L main # or: live run, keep watching the manuals/ folder
```
The extraction is faithful to each module's shape: base64 comes out function-based (many module-level functions, no classes), while array is a single class.
### Query the results
The resulting modules_info table: one row per manual with filename, title, and class/method counts, plus the full nested structure as JSON, shown expanded for copy.pdf.
Each manual becomes one row in coco_examples.modules_info:
```sql
SELECT filename, title, num_classes, num_methods FROM coco_examples.modules_info;
-- pull the full nested structure for one module
SELECT module_info::jsonb -> 'classes' -> 0 -> 'methods'
FROM coco_examples.modules_info WHERE filename = 'copy.pdf';
```
## Incremental updates in action
The payoff shows up *after* the first build. Edit a manual, drop in a new one, or delete one, and CocoIndex reconciles the table:
- A changed PDF is re-parsed and re-extracted, and its row is updated in place.
- A new PDF gets its own row.
- A removed PDF's row is dropped.
Because both the docling parse and the LLM extraction are memoized by content, nothing else in the folder is touched. Re-run cocoindex update main anytime, or leave -L running and let it watch.
## Swap the model, keep the pipeline
Because the model is a single context value, going from local to hosted (or between local models) is one env var:
Swapping the model is one environment variable: llama3.2, qwen2.5, or a hosted gpt-4o all feed the same LLM_MODEL slot, extract_module, and Postgres target.
```sh
LLM_MODEL=ollama/llama3.2 # fully on-premise
LLM_MODEL=ollama/qwen2.5 # a different local model
LLM_MODEL=openai/gpt-4o # hosted, when privacy isn't a constraint
```
Since LLM_MODEL is declared with detect_change=True, switching re-extracts everything against the new model automatically.
## Community
We would love to hear from you. Find us on GitHub and Discord.
If this turned your manuals into structured rows, please star CocoIndex on GitHub to support us. GitHub
---
# We are officially open sourced! 🎉
URL: https://cocoindex.io/blogs/cocoindex-open-source/
Date: 2025-03-03
Summary: CocoIndex is now open source: the first engine to combine custom transformation logic with incremental processing built specifically for data indexing.
We are thrilled to announce the open-source release of CocoIndex, the world's first engine that
supports both custom transformation logic and incremental processing specialized for data indexing.
CocoIndex combines custom transformation logic and incremental processing
CocoIndex is an ETL framework to prepare data for AI applications such as semantic search and retrieval-augmented generation (RAG). It offers a data-driven programming model that simplifies the creation and maintenance of data indexing pipelines, ensuring data freshness and consistency.
### Why this is hard, and why we built an engine for it
Indexing pipelines have a property most ETL tooling doesn't account for: the output is *derived* data that has to stay correct as the source keeps changing. The clean way to express that is **TargetState = Transform(SourceState)** — you declare what the index should look like given the current source, and something keeps it in sync. The trouble is that doing the "keeping in sync" by hand is brutal. You have to figure out what changed, compute the exact delta of inserts, updates, and deletes for the target, preserve intermediate results so you aren't recomputing embeddings or LLM calls you've already paid for, and migrate the target schema when your own transformation code evolves. Get any of it wrong and you ship a stale or inconsistent index — the kind of bug that's painful to even detect.
CocoIndex exists to own that bookkeeping. You write the transformation; the engine diffs each run against the last, skips work whose inputs and code are unchanged (memoization), and applies only the difference to the target. That's the difference between a generic data framework and one built specifically for indexing.
CocoIndex is now open source under the Apache License 2.0. This means the core functionality of CocoIndex is freely available for anyone to use, modify, and distribute. We believe that open sourcing CocoIndex will foster innovation, enable broader adoption, and create a vibrant community of contributors who can help shape its future. By choosing the Apache License 2.0, we're ensuring that both individual developers and enterprises can confidently build upon and integrate CocoIndex into their projects while maintaining the flexibility to create proprietary extensions.
Open source also matters specifically for infrastructure like this. An indexing pipeline sits between your data and your AI applications — it's load-bearing. Teams should be able to read the engine that decides what gets reprocessed and what gets cached, extend it with the connectors and transforms their data actually needs, and run it without a vendor in the critical path. A permissive license is the honest way to make those commitments durable.
## 🔥 Key features
- **Data Flow Programming:** Build indexing pipelines by composing transformations like building blocks, with built-in state management and observability.
- **Support Custom Logic:** Plug in your choice of chunking, embedding, and vector stores. Extend with custom transformations like deduplication and reconciliation.
- **Incremental Processing:** Smart state management minimizes re-computation by tracking changes at the file level, with future support for chunk-level granularity.
- **Python SDK:** Built with a RUST core 🦀 for performance, exposed through an intuitive Python binding 🐍 for ease of use.
The Rust-core / Python-binding split is deliberate. The engine's job — change detection, state tracking, computing deltas, and reconciling target states — is concurrent, long-running, and performance-sensitive, which is exactly where Rust earns its keep. But the people writing indexing logic want to reach for the rest of the AI and data ecosystem: chunkers, embedding models, LLM clients, vector stores. So the heavy lifting stays in Rust while everything you actually author — sources, transforms, targets — is plain Python. You get the engine's guarantees without giving up the libraries you already use.
We are moving fast and a lot of features and improvements are coming soon.
## 🚀 Getting started
1. **Installation:** Install the CocoIndex Python library:
```sh
pip install cocoindex
```
2. **Set Up Postgres with pgvector Extension:** Ensure Docker Compose is installed, then start a Postgres database:
```sh
docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/postgres.yaml) up -d
```
3. **Define Your Indexing Flow:** Create a flow to index your data. For example:
```python
@cocoindex.flow_def(name="TextEmbedding")
def text_embedding(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
data_scope["documents"] = flow_builder.add_source(cocoindex.sources.LocalFile(path="markdown_files"))
doc_embeddings = data_scope.add_collector()
with data_scope["documents"].row() as doc:
doc["chunks"] = doc["content"].transform(
cocoindex.functions.SplitRecursively(language="markdown", chunk_size=300, chunk_overlap=100))
with doc["chunks"].row() as chunk:
chunk["embedding"] = chunk["text"].transform(
cocoindex.functions.SentenceTransformerEmbed(model="sentence-transformers/all-MiniLM-L6-v2"))
doc_embeddings.collect(filename=doc["filename"], location=chunk["location"],
text=chunk["text"], embedding=chunk["embedding"])
doc_embeddings.export(
"doc_embeddings",
cocoindex.storages.Postgres(),
primary_key_fields=["filename", "location"],
vector_indexes=[cocoindex.VectorIndex("embedding", cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])
```
For a detailed walkthrough, refer to our Quickstart Guide.
## 🤗 Community
We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests on GitHub, or discussions in our Discord.
Since this release the project has grown well past where it started: CocoIndex has crossed **10,000+ GitHub stars** and shipped **200+ releases**, a steady cadence of fixes, connectors, and capabilities driven in no small part by the community. If you find the project useful, the best way to help is to use it, tell us where it falls short, and contribute back.
- **GitHub:** Please give us a star on the repository 🤗.
- **Documentation:** Check out our documentation for detailed guides and API reference.
- **Discord:** Join discussions, seek support, and share your experiences on our Discord server.
- **Social Media:** Follow us on Twitter and LinkedIn for updates.
We would love to foster an inclusive, welcoming, and supportive environment. Contributing to CocoIndex should feel collaborative, friendly and enjoyable for everyone. Together, we can build better AI applications through robust data infrastructure.
Looking forward to seeing what you build with CocoIndex!
---
# Customizable Data Indexing Pipelines
URL: https://cocoindex.io/blogs/data-indexing-custom-logic/
Date: 2025-02-20
Summary: What customizable data indexing pipelines are and why custom transformation logic matters, with practical CocoIndex examples.
CocoIndex is an open-source engine built for data indexing that supports both custom transformation logic and incremental processing. So, what is custom transformation logic? In CocoIndex it is ordinary Python: any function decorated with @coco.fn can become a pipeline step, and marking it with memo=True makes it incremental, so an unchanged document never pays for the same parse, LLM call, or embedding twice.
Index-as-a-service (or RAG-as-a-service) tends to package a predesigned service and expose two endpoints to users: one to configure the source, and an API to read from the index. Many predefined pipelines for unstructured documents do this. The requirements are fairly simple: parse PDFs, perform some chunking and embedding, and dump into vector stores. This works well if your requirements are simple and primarily focused on document parsing.
We've talked to many developers across various verticals that require data indexing, and being able to customize logic is essential for high-quality data retrieval. For example:
- Basic choices for pipeline components
- which parser for different files?
- how to chunk files (documents with structure normally have different optimal chunking strategies)?
- which embedding model? which vector database?
- What should the pipeline do?
- Is it a simple text embedding?
- Is it building a knowledge graph?
- Should it perform simple summarization for each source for retrieval without chunking?
- What additional work is needed to improve pipeline quality?
- Do we need deduplication?
- Do we need to look up different sources to enrich our data?
- Do we need to reconcile and align multiple documents?
Here we'll walk through some examples of the topology of index pipelines, and we can explore more in the future. If you're new to the framework, the core concepts and the text embedding example are a good starting point.
## Basic embedding
Basic embedding pipeline: source documents flow into a per-document loop that converts each file to Markdown and splits it to chunks, a nested per-chunk loop embeds each chunk, and the embeddings land in a vector index
In this example, we do the following:
1. Read from sources, for example, a list of PDFs
2. For each source file, parse it to markdown with a PDF parser. There are lots of choices out there: Llama Parse, Unstructured, Docling, or a vision model such as Gemini.
3. Chunk all the markdown files. This is a way to break text into smaller chunks, or units, to help organize and process information.
Options range from flat to hierarchical chunking, with many publications in this area. There are also special chunking strategies for different verticals: for code, tools like Tree-sitter can help parse and chunk based on syntax. Normally the best choice is tied to your document structure and requirements.
4. Perform embedding for each chunk, with the model of your choice: Voyage, OpenAI, or a local model.
5. Collect the embeddings in vector stores. The embedding is normally attached with metadata, for example, which file this embedding belongs to. Popular choices include Chroma, Milvus, and Pinecone, and many databases now support vector indexing, for example PostgreSQL (pgvector) and MongoDB. CocoIndex ships target connectors for PostgreSQL, SQLite, LanceDB, Qdrant, SurrealDB, and Apache Doris.
In CocoIndex, this whole topology is a short piece of Python, and every step is a point where you plug in your own choice:
```python
@coco.fn(memo=True)
async def process_file(file: FileLike, table: postgres.TableTarget[DocChunk]) -> None:
text = await parse_to_markdown(file) # your parser choice
id_gen = IdGenerator()
for chunk in splitter.split(text, chunk_size=2000, chunk_overlap=500):
embedding = await coco.use_context(EMBEDDER).embed(chunk.text) # your model choice
table.declare_row(row=DocChunk(
id=await id_gen.next_id(chunk.text),
filename=str(file.file_path.path),
text=chunk.text,
embedding=embedding,
))
await coco.mount_each(process_file, files.items(), table)
```
The PDF embedding example is a runnable version of this pipeline.
## A combination of keyword search and vector search
Anthropic has published a great article about Contextual Retrieval that suggests combining embedding-based retrieval with BM25, a lexical ranking function built on TF-IDF.
The way to think about a data flow for the pipeline is:
Hybrid retrieval pipeline: each document is converted to Markdown, split to chunks, and embedded into a vector index; a second branch extracts keywords per document and declares keyword-plus-count rows into a keyword index; at query time the database groups by keyword to compute global keyword stats, scores matches BM25-style, and combines them with vector results
In addition to preparing the vector embedding as in the basic embedding example above, after the source data parsing, we can do the following:
1. For each document, extract keywords from it with their frequencies.
2. For each keyword, store the keyword with its per-document frequency into a keyword index.
The corpus-wide part of the computation belongs at query time. Ranking functions like TF-IDF and BM25 need to know, for each keyword, how many documents contain it, and that number changes whenever any document changes. Instead of recomputing it in the pipeline on every update, keep the pipeline per-document and let the database aggregate: a GROUP BY over the keyword index yields the document frequencies, and the score combines them with the per-document frequencies already stored.
At query time, we query both the vector index and the keyword index, score the keyword matches, and combine results from both. The HackerNews trending topics example uses the same split: per-item extraction in the pipeline, cross-item aggregation in SQL.
## Simple data lookup/enrichment example
Sometimes, you want to enrich your data with metadata looked up from other sources. For example, if we want to create an index on diagnostic reports, which use ICD-10 (International Classification of Diseases Version 10) codes to describe diseases, we can have a pipeline like this:
Lookup and enrichment pipeline with two paths: the first converts each ICD-10 description document to Markdown, splits it into items, and extracts the ICD-10 code and description of each item into a dictionary; the second converts each diagnostic report to Markdown, splits it into items, enriches each item with the ICD-10 descriptions looked up from the dictionary, embeds it, and stores it in a vector index
In this example, we do the following:
- On the first path, build an ICD-10 dictionary by
1. For each ICD-10 description document, convert it to markdown with a PDF parser.
2. Split into items.
3. For each item, extract the ICD-10 code and description, and collect them into a dictionary.
- On the second path, for each report
1. Parse it to markdown with a PDF parser.
2. Split into items.
3. For each item, look up the ICD-10 dictionary prepared above and enrich the item with the descriptions for ICD-10 codes.
The ordering between the two paths is exactly what use_mount expresses: mount the dictionary-building component, wait for its result, then fan out over the reports with it.
```python
@coco.fn
async def app_main(dict_dir: pathlib.Path, report_dir: pathlib.Path) -> None:
icd10 = await coco.use_mount(build_icd10_dictionary, dict_dir) # first path
reports = localfs.walk_dir(report_dir, recursive=True)
await coco.mount_each(process_report, reports.items(), icd10) # second path
```
Now we have a vector index, built based on diagnostic reports enriched with ICD-10 descriptions.
---
# How indexing pipelines differ from other data pipelines
URL: https://cocoindex.io/blogs/what-makes-indexing-pipelines-different/
Date: 2025-01-30
Summary: What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.
When building data processing systems, it's easy to think all pipelines are similar: they take data in, transform it, and produce outputs. However, indexing pipelines have unique characteristics that set them apart from traditional ETL, analytics, or transactional systems. Let's explore what makes indexing special.
The short version: an index is a *derived* view of source content that has to stay correct as that content changes underneath it. That one property — derived data with a long, unpredictable lifecycle — drives almost every design decision below.
## The nature of data: new vs derived
First, let's understand a fundamental difference in how data is created:
### Transactional systems: creating new data
In a typical application:
- A user creates a post
- The post is stored in a database
- This is new, original data being created
### Indexing systems: building derived data
In contrast, indexing:
- Takes existing content
- Processes and transforms it
- Creates derived data structures (like vector embeddings or knowledge graphs)
- Maintains these structures over time
Comparison of a transactional app and an indexing pipeline: an application writes new, original data into a database, which becomes the source of truth; an indexing pipeline takes existing source content that can change at any time, transforms it as index = F(source) reprocessing only what changed, and maintains a derived index where every row is traceable to its source
The distinction matters because it tells you where the *source of truth* lives. In a transactional system, the database row a user wrote is the truth. In an indexing system, the truth lives upstream — in the documents, files, or rows being indexed — and the index is just a projection of it. CocoIndex makes this explicit with a simple relationship: the target is a pure function of the source, **TargetState = Transform(SourceState)**. There are no other side effects hiding in the pipeline, which is exactly what makes it safe to recompute a slice of the index without recomputing the whole thing. The core concepts guide builds the rest of the system on top of this one equation.
## How do indexing pipelines compare with other data pipelines?
### Analytics ETL
Analytics pipelines often:
- Process data in time-bounded windows
- Generate aggregated metrics
- May be run as one-off or scheduled jobs
- Focus on historical analysis
### Time series / streaming
Streaming systems:
- Handle continuous flow of events
- Process data in real-time windows
- Today's events are distinct from tomorrow's
- Data naturally flows in and out of the system
### Indexing pipelines
Indexing is different because:
- Content is persistent and long-lived
- Same content may need reprocessing
- Updates can happen at any time
- Must maintain consistency over long periods
## The time dimension
The relationship with time is a key differentiator:
### Streaming/time series
- Data is inherently time-bound
- Events belong to specific time windows
- Processing is forward-moving
- Historical data rarely changes
### Indexing
- Data lifecycle isn't tied to time
- Content can remain unchanged for long periods
- Updates are unpredictable
- Must handle both fresh and historical content
## Why incremental processing matters
This persistence and longevity make incremental processing crucial for indexing:
1. **Efficiency**
- Reprocessing everything is costly
- Need to identify and process only what changed
- Must maintain consistency with unchanged content
2. **Consistency**
- Updates should preserve existing relationships
- Need to handle partial updates gracefully
- Must maintain referential integrity
3. **Resource Usage**
- Processing cost should scale with change size
- Avoid redundant computation
- Optimize storage and compute resources
This last point is worth dwelling on, because indexing usually wraps expensive work: LLM extraction, embedding model inference, OCR, large parses. Reprocessing a whole corpus to absorb a one-paragraph edit is not just slow, it can be the dominant cost of running the system. The goal is for processing cost to track the size of the *change*, not the size of the *corpus*.
## How the mechanism actually works
The hard part of incremental processing is not the idea — it's the bookkeeping. Doing it by hand means you have to figure out what changed, compute the delta of inserts, updates, and deletes for the target, preserve intermediate results so you don't recompute everything, and evolve the target schema when your own code changes. With that many moving parts, debugging a stale or missing index row is miserable.
A declarative engine like CocoIndex takes over that bookkeeping. You describe the target as a function of the source; the engine diffs the new declared state against the previous run and applies only the difference. Two ideas make this concrete:
- **Per-item processing components.** The source is broken into independently processable items — a file, a row, an entity. Each item's work plus its outputs form a unit that applies to the target as a batch when it completes (atomically, within a transaction where the backend supports it). Add a file and a new unit appears and inserts its rows; shrink a file from two chunks to one and the engine deletes the orphaned row and inserts the new one in the same transaction; delete the file and its rows are removed. Partial updates stay consistent because the unit is the boundary.
- **Function memoization.** If an item's input and the code that processes it are both unchanged since last run, the whole unit is skipped. Memoization can also apply at the transform level: when you edit one chunk of a document, the unchanged chunks reuse their cached embeddings and only the edited chunk is re-embedded. This is what lets cost scale with change size instead of corpus size — even when your logic changes, cached results are reused wherever the intermediate values come out the same.
## Catch-up vs. live: two ways to stay fresh
Because the index is derived, "staying fresh" is its own design axis. There are two natural modes:
- **Catch-up mode** scans the sources, processes whatever changed since the last run, syncs the targets, and returns. It is already incremental — unchanged work is skipped — but it only picks up changes when you trigger a run.
- **Live mode** keeps the app running and lets components stream changes continuously from their sources — a filesystem watcher, a database change feed, a Kafka topic — applying them to the index with low latency. Reach for it when you want near-real-time reactions, or when the source can push changes more cheaply than a full rescan can discover them.
The useful property is that the same pipeline definition serves both: you choose catch-up or live at run time, and the change-detection, memoization, and reconciliation logic is identical underneath.
## Practical implications
These characteristics influence how we build indexing systems:
1. **Change Detection**
- Must track content versions
- Need efficient diff mechanisms
- Handle various update patterns
2. **State Management**
- Maintain persistent state
- Track processing history
- Handle interrupted operations
3. **Update Strategies**
- Balance freshness vs efficiency
- Handle out-of-order updates
- Manage concurrent modifications
4. **Clear Ownership**
- Every piece of data needs clear provenance
- Schema-level ownership through pipeline definitions
- Row-level ownership traced to source data
Understanding these unique aspects of indexing pipelines is crucial for building effective systems. While other data processing patterns might seem similar, indexing's combination of persistence, long-lived data, and need for incremental processing creates distinct challenges and requirements.
Understanding these differences helps build more effective and efficient indexing systems that can maintain high-quality derived data structures over time.
---
# System updates and automatic schema inference
URL: https://cocoindex.io/blogs/handle-system-update-for-indexing-flow/
Date: 2025-01-20
Summary: How CocoIndex handles system updates in indexing flows: automatic schema inference and managing data + logic evolution without downtime.
When building data processing and indexing systems, one of the key challenges is handling system updates gracefully. These systems maintain state across multiple components (like Pinecone, PostgreSQL, etc.) and need to evolve over time. Let's explore the challenges and potential solutions.
## The two dimensions of change
### 1. Data evolution
Source data is constantly changing - new records are added, existing ones are updated or deleted.
### 2. Logic evolution
The business logic and processing rules also evolve, for example,
- New fields need to be indexed
- Transformation logic changes
- New analysis requirements emerge
This is similar to how spreadsheets work - changes in either source data or formulas trigger updates to the target data.
Two dimensions of change feeding one engine: data evolution (rows added, updated, deleted) and logic evolution (transformation code edited) both flow into CocoIndex, which infers storage and schema from the flow and reprocesses only the affected target rows, leaving the rest untouched
## Infrastructure and schema management challenges
When setting up a new indexing flow, there are multiple moving parts to configure:
1. Internal data storage
2. Target storage systems (PostgreSQL, Pinecone, Milvus, etc.)
3. Pipeline logic needs to match with the component setup. For example, the fields that need to be carried into the index need to be carefully managed.
Currently, this often requires manual setup and careful coordination. Small mismatches in schema or field definitions can cause subtle bugs that are hard to debug.
## CocoIndex Approach: reduce manual setup and infer from indexing flow
CocoIndex aims to simplify this by making infrastructure setup and schema management automatic and inference-based:
### Flow-driven setup
- Users define their indexing flow logic
- CocoIndex automatically infers required storage and schema configurations
- Internal and target storage is provisioned with correct schemas automatically
### Benefits of inference
Like modern programming languages that use type inference, for example, when using Java/TS to write code, developers don't need to define data types at every single step. CocoIndex can derive the necessary infrastructure setup from the flow definition. This:
- Reduces manual configuration
- Prevents schema mismatches
- Makes updates more reliable
- Allows the system to evolve more easily
## How updates are actually applied
The inference above is what makes updates safe to run. Because CocoIndex knows the schema up front and persists what it created, an update is a reconciliation rather than a rebuild.
### Declared state vs. previous state
A target state is what you declare should exist in an external system — a table, a row, a file, an embedding. CocoIndex treats your declarations as the source of truth and records them in its internal storage (an LMDB database that tracks target states and memoization results from previous runs). On the next run it compares what you now declare against what it stored last time and applies only the minimal changes needed:
| Target state | On first declaration | When declared differently | When no longer declared |
| --- | --- | --- | --- |
| A database table | Create the table | Alter the table | Drop the table |
| A row in a table | Insert the row | Update the row | Delete the row |
| A file in a directory | Create the file | Update the file | Delete the file |
This is the same mechanism for both dimensions of change. A new or deleted *source* record shows up as a row that is now (or no longer) declared. An edit to your *logic* — adding a field, changing a transformation — shows up as a row or table that is declared differently. Either way, CocoIndex computes the delta instead of reprocessing everything. Memoized components are skipped when their inputs haven't changed, so unchanged work isn't redone.
### What setup and drop do
When a *container* target state changes — for instance you add a column or change a primary key — CocoIndex detects it and does its best to alter the target in place. If the change is too large to alter (changing primary keys is the canonical example), the target is **dropped and recreated**. Crucially, when that happens CocoIndex automatically reprocesses the affected components to backfill the data; you don't have to manually trigger a full reprocess. This is driven by the target connector's child-invalidation mechanism, which tells the engine whether a change is destructive (all children lost) or merely lossy (some data may be lost).
You can also reach for these transitions explicitly through the CLI. cocoindex update runs the app in catch-up mode and applies the reconciliation above. Passing --reset drops the existing setup before updating (equivalent to running cocoindex drop first), while --full-reprocess reprocesses everything and invalidates existing caches. The standalone cocoindex drop command reverts all target states an app created — dropping tables, deleting rows — and clears the app's internal state database.
### Keeping the index fresh
Catch-up mode is already incremental, but each update() call still has to scan sources to discover what changed, and changes are only picked up when you trigger a run. For near-real-time indexes, live mode keeps the app running after the initial catch-up and lets change-aware sources (a filesystem watcher, a database change feed, a Kafka consumer) stream updates continuously into the same target-state reconciliation — new or modified items re-mount the affected component, deletions remove it and its target states.
## Looking forward
The future of data processing systems lies in smart automation that can:
- Infer infrastructure needs from processing logic
- Handle schema evolution gracefully
- Maintain consistency across distributed storage
- Make updates and changes reliable and predictable
By building these capabilities into CocoIndex, we can significantly reduce the operational burden on users while making systems more reliable and maintainable.
---
# Processing Large Files in Data Indexing Systems
URL: https://cocoindex.io/blogs/indexing-for-single-large-file/
Date: 2025-01-10
Summary: Handle large files in data indexing: processing granularity, fan-in/fan-out, and memory pressure, walked through a patent XML example in CocoIndex.
When building data indexing pipelines, handling large files efficiently presents unique challenges. For example, patent XML files from the USPTO can contain hundreds of patents in a single file, with each file being over 1GB in size. Processing such large files requires careful consideration of processing granularity and resource management.
## Understanding processing granularity
Processing granularity determines when and how frequently we commit processed data to storage. This seemingly simple decision has significant implications for system reliability, resource utilization, and recovery capabilities.
### The trade-offs of commit frequency
While committing after every small operation provides maximum recoverability, it comes with substantial costs:
- Frequent database writes are expensive
- Complex logic is needed to track partial progress
- Performance overhead from constant state synchronization
On the other hand, processing entire large files before committing can lead to:
- High memory pressure
- Long periods without checkpoints
- Risk of losing significant work on failure
### Finding the right balance
A reasonable processing granularity typically lies between these extremes. The default approach is to:
1. Process each source entry independently
2. Batch commit related entries together
3. Maintain trackable progress without excessive overhead
## Challenging scenarios
### 1. Non-independent sources (fan-in)
The default granularity breaks down when source entries are interdependent:
- Join operations between multiple sources
- Grouping related entries
- Clustering that spans multiple entries
- Intersection calculations across sources
After fan-in operations like grouping or joining, we need to establish new processing units at the appropriate granularity - for example, at the group level or post-join entity level.
### 2. Fan-out with heavy processing
When a single source entry fans out into many derived entries, we face additional challenges:
**Light Fan-out**
- Breaking an article into chunks
- Many small derived entries
- Manageable memory and processing requirements
**Heavy Fan-out**
- Large source files (e.g., 1GB USPTO XML)
- Thousands of derived entries
- Computationally intensive processing
- High memory multiplication factor
The risks of processing at full file granularity include:
1. **Memory Pressure**: Processing memory requirements can be N times the input size
2. **Long Checkpoint Intervals**: Extended periods without commit points
3. **Recovery Challenges**: Failed jobs require full recomputation
4. **Completion Risk**: In cloud environments with worker restarts:
- If processing takes 24 hours but workers restart every 8 hours
- Job may never complete due to frequent interruptions
- Resource priority changes can affect stability
## Best practices for large file processing
### 1. Adaptive granularity
After fan-out operations, establish new smaller granularity units for downstream processing:
- Break large files into manageable chunks
- Process and commit at chunk level
- Maintain progress tracking per chunk
A single 1 GB USPTO XML file fans out through CocoIndex into per-patent units of work, each processed and committed independently into the target index — memory stays flat and a restart resumes instead of recomputing the whole file
### 2. Resource-aware processing
Consider available resources when determining processing units:
- Memory constraints
- Processing time limits
- Worker stability characteristics
- Recovery requirements
### 3. Balanced checkpointing
Implement a checkpointing strategy that balances:
- Recovery capability
- Processing efficiency
- Resource utilization
- System reliability
## How CocoIndex helps
CocoIndex provides built-in support for handling large file processing:
1. **Smart Chunking**
- Automatic chunk size optimization
- Memory-aware processing
- Efficient progress tracking
2. **Flexible Granularity**
- Configurable processing units
- Adaptive commit strategies
- Resource-based optimization
3. **Reliable Processing**
- Robust checkpoint management
- Efficient recovery mechanisms
- Progress persistence
By handling these complexities automatically, CocoIndex allows developers to focus on their transformation logic while ensuring reliable and efficient processing of large files.
## Conclusion
Processing large files in indexing pipelines requires careful consideration of granularity, resource management, and reliability. Understanding these challenges and implementing appropriate strategies is crucial for building robust indexing systems. CocoIndex provides the tools and framework to handle these complexities effectively, enabling developers to build reliable and efficient large-scale indexing pipelines.
## Join our community!
Interested in learning more about CocoIndex? Join our community!
- Follow our GitHub repository to stay up to date with the latest developments.
- Check out our documentation to learn more about how CocoIndex can help build robust AI applications with properly indexed data.
- Join our Discord community to connect with other developers and get support.
- Follow us on Twitter for the latest updates.
---
# Data Consistency in Indexing Pipelines
URL: https://cocoindex.io/blogs/indexing-data-consistency/
Date: 2025-01-06
Summary: Data consistency in indexing pipelines: concurrent updates, exposure risks, and how CocoIndex's data-driven approach keeps indexes converging.
An indexing pipeline builds indexes derived from source data. The index should always be converging to the current version of source data. In other words, once a new version of source data is processed by the pipeline, all data derived from previous versions should no longer exist in the target index storage. This is called the *data consistency* requirement for an indexing pipeline.
Source versions flow through CocoIndex's three-phase commit — track new keys in the internal store, write atomic upserts and soft-deletes to the target index, then prune stale keys — keeping tracked keys a superset of stored keys so the index converges to the current source even across crashes and restarts
## The challenge of concurrent updates
Compared to building single-process in-memory transformation logic, an indexing pipeline has additional challenges in two aspects, to maintain data consistency:
1. Temporally, it's long-running, i.e. the execution may span across multiple processes. e.g. an early execution may terminate in the middle, due to a SIGTERM, or a failure to commit to the target storage or a reboot. When the process restarts later, it should pick up existing states, and keep moving forward to update the target storage to the desired status. This is both tedious and error-prone: for example, if source data is only partially processed, and it gets updated before the restarted pipeline starts to process it again, how can we make sure states derived from the old version are properly cleared?
2. Spatially, execution happens concurrently, sometimes distributed across multiple machines for large workloads. This brings up the problem caused by out-of-order processing. Consider a scenario where your source data gets updated rapidly:
1. Version 1 (V1) exists in the system
2. Version 2 (V2) and Version 3 (V3) arrive within a second of each other
3. V2 takes longer to process than V3
Without proper handling, you might end up with V2 data in your index even though V3 is the latest version - effectively rolling back to stale data.
## Common issues and gaps
### Data exposure risks
Data inconsistency in indexing systems can lead to several critical issues:
#### System integrity
- Stale data persists in indexes despite source updates
- Inconsistent states across different storage systems
- Mixed versions of the same source entry coexisting in the index
#### Security and compliance
- Deleted sensitive information remains accessible through search interfaces
- Outdated personal data violates privacy requirements (e.g., GDPR)
- Regulatory non-compliance due to improperly retained records
#### AI system reliability
- RAG systems incorporating outdated information into responses
- Semantic search surfacing deprecated or retracted content
- AI agents making decisions based on stale data
#### Business operations
- Customer-facing applications showing incorrect information
- Loss of trust due to inconsistent data presentation
- Financial impact from outdated pricing or product details
## Approaches for maintaining data consistency
### 1. Tracking source-to-target key mappings
If all your state management and storage systems are within a single transactional system (e.g., PostgreSQL), you can leverage database transactions for consistency. However, this is often not the case in real-world scenarios. For example, you may use an internal storage (e.g. PostgreSQL) to track per-source metadata, and use an external vector store (e.g. Pinecone or Milvus) to store vector embeddings.
There are extra challenges around this:
- We have to keep track of all the keys in target vector stores derived from each source key, in the internal storage. This is necessary when the old version of source data is updated or deleted: we need to locate previously derived keys and delete them if they no longer exist in the new version after processing.
- However, the internal storage is not transactional with the external vector store, as they're two independent storage systems. So if the process terminates abnormally in the middle, states in the two systems will be out of sync.
So we need to carefully design the order of write operations during commit, to avoid data leakage even when the process is only partially executed:
1. **Add new target keys to the internal store**
2. **Write to target storage**
- Create new records
- Delete old records
Note that for any target storage that supports atomic writing, this allows all updates derived from the same source key to be written atomically. This avoids mixed-version data at query time.
3. **Remove no-longer-exist target keys from the internal store**
This ensures a key invariant: keys tracked in the internal storage are always a superset of those that really exist in the target store. This makes sure all the data in the target store is always tracked so never leaked.
### 2. Ordinal bookkeeping
Ordinals are unique identifiers that establish a strict ordering of data updates in your pipeline. They help bookkeep the sequence and timing of changes to ensure consistency. Common examples include:
- Timestamps (e.g., 2024-01-06T12:34:56Z) - Precise time-based ordering
- Version numbers (e.g., v1.2.3, 20240106.1) - Sequential numbering
Bookkeep ordinals carefully in two key stages:
- During ingestion: Record and preserve the source ordinal (e.g., timestamp or version) from the original data
- During commit: Verify the source ordinal to prevent newer versions from being overwritten by older ones
Ordinal bookkeeping happens in Phase 1, which rejects out-of-order commits - they won't cause any updates in Phase 1, and Phase 2 and 3 won't be executed at all.
### 3. Versioned unique key assignment with soft deletion
When there are multiple ongoing processings on the same source key, updates writing to the target storage (Phase 2 above) may be out of order, which causes results derived from old versions to overwrite newer ones.
To avoid this, we need to:
- For data in the target storage, make sure the keys are unique even across different versions.
- Leverage soft deletion in the target storage, i.e. deleted versions are represented by a row with a deleted field set.
Specifically:
- In the transaction in Phase 1, we establish a full view of all derived keys for the current source key: collect existing keys, and generate new keys.
- In Phase 2, we do UPSERT for both new and deleted versions - the only difference is the deleted field is set for deleted versions, and not touched for new versions.
- During query time, we filter out all records with the deleted field set.
Besides, we need an offline GC process to garbage collect rows in the target storage with the deleted field set. Because of this, the deleted field could be a timestamp of the deletion time, hence we can decide when to GC a specific version based on it.
## The CocoIndex paradigm
The discussions above only cover a part of the complexity of building and maintaining long-running or distributed indexing pipelines. There are other additional complexities, compared to building single-process in-memory transformation logic, such as:
- CRUD across different APIs
- Incremental historical state management
- Reuse existing computations
- Data migration on processing logic changes
The CocoIndex framework aims at handling these complexities so users can focus on pure transformations:
- Users write transformations as pure functions
- Output depends only on input
- No side effects or hidden states
- Easy to understand and maintain
- Framework handles:
- Previous computation result management
- Incremental processing
- State management
- Data consistency
- Data upgrade on processing logic changes
By exposing a data-driven programming paradigm, CocoIndex allows developers to focus on their core business logic while maintaining data consistency and freshness.
## Join our community!
Interested in learning more about CocoIndex? Join our community!
- Follow our GitHub repository to stay up to date with the latest developments.
- Check out our documentation to learn more about how CocoIndex can help build robust AI applications with properly indexed data.
- Join our Discord community to connect with other developers and get support.
- Follow us on Twitter for the latest updates.
---
# Data Indexing and Common Challenges
URL: https://cocoindex.io/blogs/data-indexing-and-common-challenges/
Date: 2025-01-05
Summary: Fundamentals of data indexing pipelines for RAG: what makes a good one, common production pitfalls, and how CocoIndex addresses them.
At its core, data indexing is the process of transforming raw data into a format that's optimized for retrieval. Unlike an arbitrary application that may generate new source-of-truth data, indexing pipelines process existing data in various ways while maintaining trackability back to the original source. This intrinsic nature (being a derivative rather than a source of truth) creates unique challenges and requirements.
## Characteristics of a good indexing pipeline
A well-designed indexing pipeline should possess several key traits:
### 1. Ease of building
People should be able to build a new indexing pipeline without mastering techniques such as database manipulation / access, streaming processing, parallelization, fault recovery, etc. In addition, transformation components (a.k.a. operations) should be easily composable and reusable across different pipelines.
### 2. Maintainability
The pipeline should be easy to understand, modify, and debug. Complex transformation logic should be manageable without becoming a maintenance burden.
On the other hand, an indexing pipeline is a stateful system, so besides the transformation logic, it's also important to expose clear status of the pipeline states, e.g. statistics of the number of data entries, their freshness, and how a specific piece of derived data tracks back to the original source.
### 3. Cost-Effectiveness
Data transformation (with necessary tracking of relationships between data) should be done efficiently without excessive computational or storage costs. Moreover, existing computations should be reused whenever possible. For example, 1% of document change, or a chunking strategy change that only affects 1% of chunks, shouldn't need to entail rerunning the expensive embedding model over the entire dataset. CocoIndex reuses prior computation through memoization.
### 4. Indexing Freshness
For many applications, the source of truth for indexing is consistently updated, so it's important to make sure the indexing pipeline is also updated accordingly in a timely manner with live updates.
## Common challenges in indexing pipelines
Three ways indexing pipelines break: source-of-truth documents flow through a transformation pipeline F(source) into a derived index, with three challenge callouts pointing at the pipeline — incremental processing (1% of docs change, yet everything gets re-embedded), upgradability (modifying one step forces a manual full rebuild), and the deterministic logic trap (keys from old runs don't match new runs, so stale rows leak into the index)
### Incremental processing is challenging
The ability to process only new or changed data rather than reprocessing everything is crucial for both cost efficiency and indexing freshness. This becomes especially important as your data grows.
To make incremental processing work, we need to carefully track the state of the pipeline, to decide which portion of the data needs to be reprocessed, and make sure states derived from old versions are fully deleted or replaced. It's challenging to make things right while considering various complexities, like fan-in / fan-out in transformations, out-of-order processing, recovery after early termination, etc.
### Upgradability is often overlooked
Many implementations focus on the initial setup but neglect how the pipeline will evolve. When requirements change or new processing steps need to be added, the system should adapt without requiring a complete rebuild.
Traditional pipeline implementations often struggle with changes to the processing steps. Adding or modifying steps typically requires reprocessing all data, which can be extremely expensive and involves a manual process.
### The deterministic logic trap
Many systems require deterministic processing logic - meaning the same input should always produce the same output. This becomes problematic when:
- Entry deletion needs to be handled
- Processing logic naturally evolves
- Keys generated in previous runs don't match current runs, leading to data leaks
## How does CocoIndex solve these challenges?
CocoIndex approaches indexing pipelines with a fundamentally different mental model - similar to how React revolutionized UI development compared to vanilla JavaScript. Instead of focusing on the mechanics of data processing, users can concentrate on their business logic and desired state. See the core concepts for how flows, sources, transforms, and collectors fit together.
### The CocoIndex approach:
1. **Stateless Logic**: Users write pure transformation logic without worrying about state management
2. **Automatic Delta Processing**: CocoIndex handles incremental processing efficiently
3. **Built-in Trackability**: Every transformed piece of data maintains its lineage to source
4. **Flexible Evolution**: On pipeline changes, past intermediate states can still be reused whenever possible
5. **Non-Deterministic Friendly**: With data lineage clearly tracked, even without determinism of processing logic, CocoIndex can still make sure stale states are properly purged
## Subtle complexities we handle
CocoIndex takes care of many subtle but critical aspects:
- Managing processing state across pipeline updates
- Ensuring data consistency during partial updates
- Smooth recovery from early termination of the pipeline
- Optimizing resource usage automatically
- Maintaining data lineage and relationships
## The mental model shift
Just as React changed how developers think about UI updates by introducing the concept of declarative rendering, CocoIndex changes how we think about data indexing. Instead of writing imperative processing logic, users declare their desired transformations and let CocoIndex handle the complexities of efficient execution.
This shift allows developers to focus on what their data should look like rather than the mechanics of how to get it there. The result is more maintainable, efficient, and reliable indexing pipelines that can evolve with your application's needs.
## Conclusion
A well-designed indexing pipeline is crucial for production RAG applications, but building one that's maintainable, efficient, and evolvable is challenging. CocoIndex provides a framework that handles these complexities while allowing developers to focus on their core business logic. By learning from the challenges faced by traditional approaches, we've created a system that makes robust data indexing accessible to everyone building RAG applications.
---
# CocoIndex - A Data Indexing Platform for AI Applications
URL: https://cocoindex.io/blogs/cocoindex-data-indexing-platform/
Date: 2025-01-04
Summary: CocoIndex is a data indexing platform for AI: ingestion, chunking, embedding, and pipeline management for RAG, semantic search, and knowledge graphs.
CocoIndex Cover Image
High-quality data tailored for specific use cases is essential for successful AI applications in production. The old adage "garbage in, garbage out" rings especially true for modern AI systems - when a RAG pipeline or agent workflow is built on poorly processed, inconsistent, or irrelevant data, no amount of prompt engineering or model sophistication can fully compensate. Even the most advanced AI models can't magically make sense of low-quality or improperly structured data.
Whether we're building semantic search, RAG (Retrieval Augmented Generation), or agentic workflows on top of embeddings and knowledge graphs, the foundation of AI applications lies in how well the data is processed and indexed.
This is where CocoIndex comes in - we aim to be the best-in-class scalable data indexing infrastructure with built-in observability and lineage tracking. CocoIndex provides a data-driven programming model where indexing is defined by data flow with clear lineage, making it straightforward to understand and maintain data pipelines.
### The job: turn raw sources into AI-ready data
An indexing platform sits between your sources and your AI application. Its job is narrow but unglamorous: take documents, files, rows, and media from wherever they live, transform them into the representations an AI system actually consumes - chunks, embeddings, structured fields, graph relationships - and keep those representations correct as the sources change. Everything in a RAG or agent stack downstream of that - retrieval quality, grounding, the answers a user finally sees - inherits whatever the indexing layer produced. If the chunk boundaries are wrong or the embeddings are stale, the model has no way to recover. Getting this layer right is the highest-leverage work in an AI application, which is exactly why it shouldn't be re-implemented by hand for every project.
### The flow model
CocoIndex describes indexing as a *data flow*: you declare what the target should contain as a function of the source, and the framework figures out how to get there. This is the same mental model behind spreadsheets, React, and materialized views - you specify formulas or target state, and the system recomputes downstream values when inputs change (see core concepts and the overview). You write ordinary Python transformations; there is no separate DAG, no operators to wire up, and no DSL to learn. Because the flow is declared rather than imperatively scripted, the schema of every step is known at declaration time, before any data runs through it - which lets CocoIndex set up target tables and vector columns automatically and catch type mismatches up front instead of at row 10,000.
## What makes data preparation for AI hard?
Building production-ready AI applications requires solving several complex data challenges:
- Connecting to and ingesting data from multiple diverse sources
- Determining optimal chunking strategies for different content types
- Selecting and implementing appropriate embedding models
- Managing vector stores and knowledge graphs efficiently
- Tracking data lineage and ensuring observability
- Detecting and handling content updates across sources
- Managing content staleness and implementing refresh strategies
- Ensuring data consistency during updates and refreshes
- Deduplicating and reconciling relevant data
- Reusing existing computations when possible on incremental changes of source data or transformation logic
These challenges often require significant engineering effort that takes focus away from core business logic development.
## How CocoIndex helps
CocoIndex provides a comprehensive platform that handles all aspects of data indexing for AI applications:
CocoIndex Stack
### 1. Universal data connectivity
We offer seamless integration with various data sources, making it simple to ingest and process content regardless of where it lives.
### 2. Intelligent indexing strategy
Our platform makes it easy to experiment with and evaluate different indexing strategies, both during development and in production:
- Flexible content chunking with A/B testing support
- Plug-and-play embedding model switching
- Deduplication with content-based and metadata-based matching
- Flexible reconciliation with custom merge strategies and conflict resolution
- Multiple vector store backends with seamless migration
- Extensible knowledge graph schemas
Chunking and embedding are where most retrieval quality is won or lost, so CocoIndex ships these as built-in operations rather than leaving them as glue code. The text ops include a syntax-aware RecursiveSplitter that understands code structure and returns chunks with start/end positions (line, column, and offset) rather than blind character windows, plus a regex-based SeparatorSplitter when you want explicit control. Embedding is just as swappable: the sentence-transformers integration runs models locally with automatic model caching and thread-safe GPU access, and because it advertises its vector dimension and dtype to the connector, switching embedding models doesn't mean hand-editing your target schema. Trying a different chunk size or a different model is a one-line change, which is what makes systematic A/B evaluation practical instead of aspirational.
### 3. Robust pipeline management
CocoIndex manages the entire indexing pipeline with:
- Built-in monitoring and observability
- Data lineage tracking
- Automatic updates and maintenance
- Performance optimization
- Error handling and recovery
- Robust out-of-order update handling
- Careful ordinal tracking across all pipeline stages
- Version-aware commit logic to prevent stale data
- Safe concurrent processing of updates
- Data freshness
- Transactional consistency across storage systems
- Clean state management for incremental processing
- Safe deletion of obsolete versions
- Framework-level complexity handling
- Users focus on pure transformations
- Framework manages storage and states
- Automatic delta processing and caching
The piece that ties all of this together is incremental processing. Sources are not static - documents get edited, rows get deleted, and the transformation logic itself evolves as you tune chunking or swap models. Reprocessing everything on every change is expensive, slow, and disruptive, but doing the delta by hand is genuinely hard: you have to determine what changed, compute exactly what to insert, update, or delete in the target, preserve intermediate state to avoid recomputation, and evolve the target schema when the logic changes. CocoIndex does this for you. It tracks fine-grained lineage, so when one source file changes only the chunks and embeddings derived from that file are recomputed, and only the corresponding target rows are touched. It also reuses prior computation through memoization - for example, a file that was merely touched or moved but whose content is unchanged validates against its modification time first and skips re-embedding entirely. The result is that an end-to-end refresh that would otherwise take hours drops to the cost of the actual change, while staying correct.
### 4. Standardized API access
We provide clean, standard APIs to access indexed data for:
- Semantic searches
- Context retrieval for RAG
- Knowledge graph navigation
## Focus on what matters
By handling the heavy lifting of data preparation and indexing, CocoIndex enables:
- Reduced time spent on infrastructure development
- Focus on core business logic
- Faster AI application development
- Maintained high data quality
- Confident scaling
## Getting started
Looking to streamline AI application data infrastructure? Check out our documentation to learn more about how CocoIndex can help build robust AI applications with properly indexed data. CocoIndex is open source and has grown to more than 10,000 stars on GitHub, where the project develops in the open.
Join our Discord community to connect with other developers and get support. Follow us on Twitter for the latest updates.
Let CocoIndex handle the complexity of data indexing while you focus on building amazing AI applications!
---
# Welcome to CocoIndex
URL: https://cocoindex.io/blogs/welcome/
Date: 2025-01-02
Summary: Welcome to the official CocoIndex blog! We're excited to share our journey in building high-performance indexing infrastructure for AI applications.
Welcome to the official CocoIndex blog! We're excited to share our journey in building high-performance indexing infrastructure for AI applications.
CocoIndex is designed to provide exceptional velocity for AI systems that need fast, reliable access to their data. Whether you're building large language models, recommendation systems, or other AI applications, our goal is to make data indexing and retrieval as efficient as possible.
## The problem we're solving
AI applications live and die by the freshness and structure of the data they retrieve. A retrieval system, a recommendation model, or an agent grounding itself in your documents is only as good as the index behind it. The hard part is rarely the first build — it's keeping that index correct as the world moves underneath it. Both sides change constantly: your source data (new records arrive, old ones get updated or deleted) and your processing logic (you add a field, swap an embedding model, refine a transformation). Most teams answer this by re-running the whole pipeline on a schedule, which is expensive, slow, and disruptive. Doing it properly by hand is worse: you have to compute the delta of what to insert, update, or delete, preserve intermediate state to avoid recomputation, and evolve the target schema when the code changes. With that many moving parts, a small mismatch turns into a subtle bug that's painful to debug.
## How it works at a high level
CocoIndex takes a declarative, state-driven approach. Instead of writing the *how* of incremental updates, you declare *what* your target should look like as a function of your source data — and the engine figures out what changed and applies only the necessary updates. If you've used spreadsheets, React, or materialized views, the mental model will feel familiar:
- **Spreadsheets:** you declare formulas in cells; when an upstream cell changes, downstream cells recompute.
- **React:** you declare your UI as a function of state; when state changes, React re-renders what changed.
- **CocoIndex:** you declare your target as a function of your source; CocoIndex syncs what changed.
CocoIndex core model: source data flows into a declarative flow where target = f(source), running on a Rust engine with Python transforms to produce a fresh vector index and knowledge graph; underneath, an incremental engine detects what changed, recomputes only the delta, and syncs the targets, taking updates from hours or days down to seconds.
A few properties make this practical at production scale:
- **A high-performance Rust engine.** Pipelines execute on a Rust runtime built for resilient, scalable data processing, while you write your transformations in plain Python — no new DSL to learn.
- **Incremental and low-latency.** The engine tracks fine-grained dependencies and only recomputes what changed, in the input data *or* the code. The same batch-style code runs incrementally in both batch and live mode.
- **Full lineage and explainability.** Every processing step, intermediate result, and execution path is inspectable — which matters for auditability and transparency requirements.
- **An open integration model.** Sources and targets plug in through a standard interface, so you're not locked into a particular database or vector store.
## Why incremental processing matters
It's worth being concrete about why this design pays off, because it's the heart of what CocoIndex does. When you store derived results in a target for retrieval — embeddings in a vector store, extracted entities in a graph, summaries in a table — both your data and your code keep evolving. Reprocessing everything on every change is the easy answer, and it's the one most pipelines reach for, but it scales badly: cost grows with the size of your corpus rather than the size of the change, latency climbs, and the index is disrupted while it rebuilds.
Doing incremental updates by hand is genuinely hard. You have to figure out what changed and what didn't, reason in the time dimension to compute the delta of inserts, updates, and deletes, track intermediate state so you can avoid recomputing work that's still valid, and evolve the target schema (and backfill it) when the code logic changes. Each of those is a place for a subtle bug to hide. CocoIndex absorbs all of it into the engine: you declare the target, and the framework tracks fine-grained dependencies and recomputes only what changed. In practice this can take end-to-end update times from hours or days down to seconds while preserving correctness — and because every step is inspectable, you can see exactly how a given output was produced.
## Who it's for
If you're an AI or data engineer wiring up retrieval, building a knowledge graph from documents, indexing a codebase, or feeding a recommendation system, CocoIndex is meant to take the incremental-update burden off your plate. You write the transformation once; the framework keeps the target in sync as data and logic evolve. It fits equally well for someone shipping their first embedding pipeline and for a team operating a large, continuously updated index.
CocoIndex is open source, and the community has been a big part of the journey — the project has crossed **10,000+ stars** on GitHub. To go deeper on the mental model, the overview and core concepts pages are the best place to start.
We'll be sharing:
- Technical deep dives into our architecture and design decisions
- Performance benchmarks and optimization techniques
- Best practices for implementing CocoIndex in your AI systems
- Updates on new features and improvements
- Case studies and success stories from our community
We believe that great infrastructure should be both powerful and accessible. That's why we're committed to:
- Building robust, scalable solutions
- Maintaining clear, comprehensive documentation
- Fostering an active, supportive community
- Continuously improving based on user feedback
Stay tuned for regular updates as we continue to enhance CocoIndex and share insights from our development journey. Whether you're a seasoned AI engineer or just getting started with data indexing, we're here to help you build faster, more efficient systems.
Follow our GitHub repository to stay up to date with the latest developments!
---