CocoIndex Blog

CocoIndex Changelog 1.0.8 - 1.0.16

George He — Tue, 07 Jul 2026 00:00:00 GMT

This is the second changelog since the v1.0 launch, covering nine releases (1.0.8–1.0.16). The focus shifts from initial hardening into new engine primitives for stateful, production-grade pipelines, a major expansion of cloud and enterprise connectors, and continued depth on agent ergonomics.

CocoIndex builds fresh, structured context for AI from sources such as PDFs, codebases, emails, screenshots, and meeting notes, and keeps it up to date with an incremental engine. The source is on GitHub.

Why this release cycle matters

For developers building retrieval, extraction, and knowledge-graph pipelines, the most significant changes are:

Persistent per-component state. coco.use_state() lets pipeline components carry state that survives across runs, opening the door to accumulators, counters, and stateful deduplication without an external store.
In-memory intermediate collections. LiveMap is a live, in-memory keyed collection that bridges a producing and a consuming stage of the same run: one side declares entries, the other consumes them through mount_each, which grows and shrinks one live component per entry as the data changes — a producer/consumer hand-off that pure declarative pipelines can't express.
Batched target writes. Row-level writes to the same target from different components can now be coalesced into one apply, collapsing N write transactions into one — a direct win for connectors like LanceDB and cloud warehouses where per-transaction cost dominates.
Token-bucket rate limiting. RateLimiter is now a first-class engine primitive: a consistent, cooperative throttle shared across pipeline components, replacing hand-rolled sleep pacing at each call site with one fair, budgeted rate.
Enterprise cloud connectors. BigQuery, Snowflake, and Azure Blob Storage land alongside sparse-vector Qdrant support and a new Valkey vector store connector, completing coverage of the major cloud data platforms.

Engine

`coco.use_state()`: persistent per-component state

coco.use_state() gives any pipeline component a persistent, typed state slot that survives across update cycles. The state is stored in the LMDB state store alongside other engine metadata, keyed by a StableKey (not just a plain string, as of #2085). A type_hint parameter added in v1.0.15 enables typed deserialization so components can round-trip structured state objects cleanly.

Why it matters: the incremental engine already knows what changed between runs. use_state lets components know what they have already done. This enables accumulators (running totals, seen-entity sets), watermark trackers, and stateful deduplication, patterns that previously required an external Redis or Postgres sidecar. The net effect is that a wider class of stateful logic can now live inside the pipeline declaration, subject to the same incremental reconciliation semantics as the rest of CocoIndex.

`LiveMap`: in-memory intermediate collections

LiveMap is a new in-memory keyed collection that bridges a producing and a consuming half of the same live run. Producing components declare entries into it as target states (LiveMap.declare_entry), and the consuming side reads that same in-memory dict as a live source through mount_each. Where coco.stats_group provides observability over a data slice, LiveMap provides computation over one: a mutable intermediate collection the pipeline both writes and reads, decoupled through the shared map.

Because LiveComponent classes can now be used inside mount_each(), the consuming side is a dynamic live component tree — one live component per entry, mounted and dropped as producers add and stop declaring keys.

Token-bucket `RateLimiter`

RateLimiter is now a first-class engine primitive implementing a token-bucket algorithm. It provides a cooperative throttle shared across pipeline components: any component that acquires a token from the same RateLimiter instance participates in the same budget, and concurrent callers are served FIFO so none starve. This replaces hand-rolled sleep pacing at each call site with one budgeted, coordinated rate.

Combined with the existing retry-on-429 (HTTP rate limit) behavior added in earlier cycles, RateLimiter gives pipelines deterministic, budget-aware flow control rather than reactive backpressure alone.

Preview mode for update actions

Update actions can now be previewed before they are applied. A pipeline run can describe the adds, reprocesses, and deletes it would perform, without writing to any target, enabling dry-run validation and change auditing before committing to a production store.

Batched target writes across components

Row-level writes to the same target from different components can now be coalesced into a single batched apply. The common shape is mount_each(..., table_target), where many child components each declare one row into the same target — but the coalescing is keyed by the target sink, not by any sibling relationship, so any components whose writes land on the same sink concurrently are batched together. Previously each component submitted its leaf actions directly to the sink after precommit, so the connector saw a stream of single-row calls even though its API already accepts a batch. In practice a 50-row ingest into LanceDB produced roughly 50 merge_insert commits — and 50 table versions — instead of one (#2219).

An app-scoped batcher, keyed by the target sink, merges concurrent commits that hit the same sink before the connector's apply call (#2221). It reuses CocoIndex's existing Batcher primitive rather than a bespoke queue, so leaf-target batching shares the same path as the engine's other batched work and adds no debounce latency to a lone write. Batching preserves order, each output mapping back to its input; container targets, which return per-action child handlers, stay on the direct apply path, while the batched path handles leaf writes. Sink lifetime drives cleanup: a keeper owns the batcher one-to-one and drops its registry entry the moment the sink goes away, so a pipeline with an unbounded number of sink functions cannot leak batcher entries.

Why it matters: for connectors with real per-operation cost, such as LanceDB and cloud warehouses, write cost is dominated by the number of write transactions — round-trips and commits — not by row count. Coalescing N single-row applies into one batched apply collapses N write transactions into one, cutting that overhead directly. For a store like LanceDB it also collapses N table versions into one, which keeps version history and compaction manageable under heavy fan-out.

Engine performance improvements

Prefetch fn-memos and user-states in one read transaction: the engine now batches all LMDB reads for function memos and user states into a single read transaction per cycle, replacing N individual reads (#2076).
Minimize serialization for UserStateCache: reduces round-trip cost for state reads during hot-path execution (#2127).
LMDB map auto-resize: the state store now automatically resizes the LMDB map when it fills, eliminating the manual LMDB_MAP_SIZE tuning that previously required operator intervention (#2231).
LMDB map size rounded to page size: map sizes are now aligned to the OS page boundary, preventing subtle allocation failures on some Linux kernels (#2109).

Other engine fixes

coco.fn callable outside a component context: coco.fn-decorated functions can now be called directly in scripts and tests without being inside a mounted pipeline component (#2101).
!-prefixed negation in excluded_patterns: source file filters now support !pattern to explicitly un-exclude a path that matched an earlier pattern, following the .gitignore-style rule ordering (#1901).
Retry transient LiteLLM embedding errors: transient failures (network blips, upstream 5xx) during embedding calls are retried automatically (#2090).
Reject rows with missing or null primary keys: the engine now fails fast rather than silently writing corrupt rows (#2239).

Connectors

Target: Snowflake (v1.0.15)

A native Snowflake target connector lands, bringing CocoIndex's declare-target-state model to one of the most widely deployed cloud data warehouses. Snowflake targets participate in the same incremental reconciliation as all other CocoIndex targets: rows for deleted sources are removed automatically, and only changed rows are upserted.

Target: BigQuery (v1.0.16)

A Google BigQuery target connector was added alongside the Snowflake connector in the same cycle. For organizations already running Google Cloud infrastructure, this completes cloud data warehouse coverage across the major platforms.

Source: Azure Blob Storage (v1.0.16)

An Azure Blob Storage source connector mirrors the OCI and S3 source APIs. For teams on Azure infrastructure, this enables the same live-mode, incremental bucket-watching behavior available to AWS and OCI users.

Target: Valkey vector store (v1.0.12)

Valkey, the open-source Redis fork under Linux Foundation governance, gains a target connector with vector store support. For organizations that have standardized on Valkey over Redis, this provides a fully supported vector index target without requiring a separate vector database.

Qdrant: native sparse vector support (v1.0.16)

The Qdrant connector was upgraded with native sparse vector support and eager point-ID validation. Sparse vectors are essential for hybrid search pipelines that combine dense (semantic) and sparse (BM25/TF-IDF) representations, making CocoIndex's Qdrant integration first-class for hybrid retrieval.

LanceDB: vector and FTS index support (v1.0.11)

LanceDB gained explicit vector and full-text search (FTS) index management. The drop_index operation was added alongside a dependency bump to LanceDB 0.33.0 and PyArrow 23.0.0. Nullable schema-only row rewrites (a correctness bug that appeared on partial schema changes) were fixed in both the Python and Rust layers. In v1.0.15, LanceDB table maintenance was tied to LanceDB's own stats API, making compaction scheduling driven by actual table metrics rather than a fixed interval.

zvec: FTS field support (v1.0.14)

The zvec target connector gained full-text search field support, allowing pipelines to write both vector and FTS fields to zvec in a single target export.

OCI: skip startup scan on unchanged logic (v1.0.11)

The OCI Object Storage source now detects when pipeline logic is unchanged since the last run and skips the expensive full-bucket startup scan, going directly to live-stream event processing. For large buckets, this can reduce cold-start time from minutes to seconds.

Local file system: periodic rescan in live mode (v1.0.15)

The local file system source in live mode now periodically rescans and recreates the watcher if it becomes stale. OS file-system watchers (inotify, kqueue) have known limitations under high churn, network mounts, and certain editors; periodic rescan provides a safety net that catches any events the watcher missed.

New connector summary

Connector	Type	Release
Snowflake	Data warehouse target	v1.0.15
BigQuery	Data warehouse target	v1.0.16
Azure Blob Storage	Object store source	v1.0.16
Valkey	Vector store target	v1.0.12
Qdrant sparse vectors	Enhanced vector target	v1.0.16
LanceDB vector + FTS indexes	Enhanced vector target	v1.0.11
zvec FTS fields	Enhanced target	v1.0.14

Agent & developer experience

`cocoindex show`: full-fledged inspection tool (v1.0.12)

cocoindex show was upgraded from a basic state dump into a full inspection tool for pipeline debugging. It provides structured, human-readable output of pipeline state, component configuration, and target schema: the production debugging workflow that previously required querying the LMDB or Postgres state store directly.

Python error messages

Runtime errors crossing the Rust-to-Python boundary now use Display (human-readable) rather than Debug (Rust internals) formatting, making error messages that surface in Python stack traces significantly more actionable.

Multiple GPU and fractional GPU allocation (v1.0.15)

GPUPool was added to support multiple GPUs and fractional GPU allocations within a pipeline. For embedding-heavy pipelines running on multi-GPU infrastructure, this allows work to be distributed across available GPU resources without hand-partitioning the pipeline.

Example walkthroughs

v1.0.13 shipped a major documentation expansion: 17 new example walkthroughs (10 full + 7 variants) covering the breadth of CocoIndex use cases:

Meeting Notes → Knowledge Graph, with Neo4j and FalkorDB variants
CSV → Kafka streaming pipeline
Semantic Search over PDFs with mixed embedding models
Search Images by Text, a multimodal CLIP walkthrough
Face Recognition, ported from v0 to v1
Docs to Knowledge graph construction

Each walkthrough was structured for agent followability, with consistent heading patterns, copyable commands, and explicit prerequisites, so AI coding agents can execute them as step-by-step procedures.

Bug fixes and correctness

Issue	Impact	Release
S3 ETag encoded to bytes for content fingerprint	Prevents false cache invalidation on binary ETags	v1.0.10
`StableKey::Bytes` serde round-trip via msgpack	Correct key serialization for binary primary keys	v1.0.14
Postgres: preserve data and NOT NULL on column type evolution	Schema evolution no longer drops non-null constraints	v1.0.15
Postgres: schema-qualify DROP INDEX for vector indexes	Avoids dropping wrong-schema indexes in multi-schema deployments	v1.0.15
Logic deps propagated across mount boundary	Dependency tracking correct for nested mounted components	v1.0.12
Orphaned child-existence rows when Directory→Component	Prevents stale rows when a directory node becomes a component node	v1.0.15
LanceDB: escape single quotes in delete filter predicates	Prevents filter injection on document paths containing apostrophes	v1.0.11
LanceDB: nullable schema-only row rewrites	Correct behavior on partial schema updates	v1.0.15
Idle batchers cleared to avoid stale accumulation	Prevents write amplification on quiet pipelines	v1.0.8
PEP 695 type aliases in numpy 2.5 NDArray	Fixes `analyze_type_info` crash on numpy 2.5+	v1.0.13

Summary

This cycle's theme is depth and breadth: depth in the engine (stateful components, in-memory intermediate collections, GPU pools, performance batching) and breadth in connectors (completing cloud warehouse coverage with Snowflake, BigQuery, and Azure Blob, plus Valkey and enhanced Qdrant). The through-line is the incremental engine growing new primitives — persistent state, live intermediate collections, cooperative rate limiting, batched writes — that bring a wider class of stateful, production-grade pipelines inside a single declarative CocoIndex app.

For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.

Thanks to the community

Thanks to everyone who shipped this cycle, core maintainers and community alike, including twelve first-time contributors.

@georgeh0

Thanks @georgeh0 for the bulk of this cycle's engine work: LiveMap, the token-bucket RateLimiter, LiveComponent support inside mount_each, and the shared-parse code infrastructure behind cocoindex-code.

@badmonster0

Thanks @badmonster0 for the documentation updates.

@lichuang

Thanks @lichuang for GPUPool, supporting multiple and fractional GPU allocations, and for adding type_hint to use_state for typed state deserialization.

@Haleshot

Thanks @Haleshot for the initial zvec target integration and adding FTS field support to it.

@nuthalapativarun

Thanks @nuthalapativarun for switching runtime errors crossing into Python to Display formatting, adding Qdrant and local file system connector unit tests, and fixing broken documentation links.

@daric93

Thanks @daric93 for the Valkey vector store target connector, a fully supported vector index target for teams standardized on Valkey.

@prithvi-moonshot

Thanks @prithvi-moonshot for upgrading cocoindex show into a full-fledged inspection tool for debugging.

@MrAnayDongre

Thanks @MrAnayDongre for preview mode for update actions, letting a run describe the adds, reprocesses, and deletes it would perform before touching a target.

@pyjuan91

Thanks @pyjuan91 for making the OCI source skip its full startup scan when pipeline logic is unchanged, cutting cold-start time on large buckets.

@tomz-alt

Thanks @tomz-alt for native sparse vector support and eager point-ID validation in the Qdrant connector, making it first-class for hybrid retrieval.

@jordigilh

Thanks @jordigilh for adding periodic rescan and watcher recreation to the local file system source in live mode, a safety net for events an OS watcher misses.

@MaxRong

Thanks @MaxRong for making the engine reject rows with missing or null primary keys instead of silently writing corrupt rows.

@slliland

Thanks @slliland for making the state store automatically resize the LMDB map when it fills, removing the manual LMDB_MAP_SIZE tuning operators used to need.

@SuperMarioYL

Thanks @SuperMarioYL for !-prefixed negation in excluded_patterns, so a path matched by an earlier rule can be explicitly un-excluded.

@davidmyriel

Thanks @davidmyriel for documenting Tigris alongside MinIO as a supported S3-compatible service.

@dashitongzhi

Thanks @dashitongzhi for improving the docs CI with proper triggers and build verification.

Index Your Codebase for AI Agents with CocoIndex V1

Linghua Jin — Wed, 10 Jun 2026 00:00:00 GMT

In this blog, we'll build a live semantic index over a codebase with CocoIndex V1. Point it at a repo, and you get a vector index you can search in natural language ("where do we embed chunks?") that updates itself as you edit. It's the kind of fresh, low-latency code context a coding agent needs, and it's about 100 lines of plain async Python.

CocoIndex has built-in support for codebase chunking, with native Tree-sitter parsing and live updates.

The heavy lifting (incremental processing, change tracking, managed targets) runs in a Rust engine underneath. With a source that watches for changes, like the local filesystem in live mode, the index tracks your edits as they happen: save a file, and only the chunks that changed get re-embedded and re-upserted.

Use case

Code context for agents: semantic context for Claude, Codex, OpenCode, Factory.
Code search: natural-language and semantic search over your repo.
Review & refactor agents: context for code review, security analysis, and large-scale refactoring.

Why CocoIndex for codebase indexing

A codebase is hard to keep indexed well, and it exercises most of what CocoIndex was built for:

Chunks follow real code structure (functions, classes, blocks) instead of arbitrary line windows. Tree-sitter parsing is built in for every major language.
Code changes constantly: every commit, save, and rebase mutates a handful of files. CocoIndex re-embeds only the chunks that changed, and with live=True plus cocoindex update -L it keeps watching and applies each edit as it happens.
The engine is Rust. The run below indexes 622 files in 43 seconds cold; after that, a save costs one file.
The pipeline is ordinary async Python and your own types. Pick your embedding model, chunking strategy, and vector database; if your agent can write Python, it can extend this flow.

Overview

From a high level, these are the steps:

Read code files from a local directory.
Split each file into syntax-aware chunks with Tree-sitter, then embed every chunk.
Store the chunks and their embeddings in Postgres (as target states).

With CocoIndex, you declare the transformation logic with native Python, without worrying about handling updates.

Think: target_state = transformation(source_state).

When your codebase changes, or your processing logic changes (a different embedding model, a new chunk size), only the difference is reprocessed. The Incremental updates section walks through exactly what gets recomputed when.

Setup

In this example we store the indexed rows in Postgres with the pgvector extension. Your metadata and the vector search index both live there. CocoIndex supports many targets, so you can pick the one that works best for you.

If you don't have Postgres running, use the Docker Compose file in the cocoindex repo to start one.
```
docker compose -f dev/postgres.yaml up -d
```

Point CocoIndex at it

export POSTGRES_URL="postgres://cocoindex:cocoindex@localhost/cocoindex"

Install CocoIndex and the dependencies

pip install -U "cocoindex[postgres,sentence_transformers]" asyncpg pgvector numpy python-dotenv

Define the App

Apps are the top-level runnable unit in CocoIndex. An app names your pipeline and binds a main function with its parameters.

Define the data and shared resources

This block sets up two things the rest of the code builds on. CodeEmbedding defines one row of the output table: each chunk of code becomes one row, with its text, location, and embedding vector. coco_lifespan provides the shared resources every step needs, the Postgres connection pool and the embedding model, built once at startup instead of per file.

import os
import pathlib
from dataclasses import dataclass
from typing import AsyncIterator, Annotated

import asyncpg
from numpy.typing import NDArray

import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter, detect_code_language
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.chunk import Chunk
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.id import IdGenerator

DATABASE_URL = os.getenv("POSTGRES_URL", "postgres://cocoindex:cocoindex@localhost/cocoindex")
TABLE_NAME = "code_embeddings"
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"

PG_DB = coco.ContextKey[asyncpg.Pool]("code_embedding_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)

_splitter = RecursiveSplitter()

@dataclass
class CodeEmbedding:
    id: int
    filename: str
    code: str
    embedding: Annotated[NDArray, EMBEDDER]
    start_line: int
    end_line: int

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    async with asyncpg.create_pool(DATABASE_URL) as pool:
        builder.provide(PG_DB, pool)
        builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
        yield

Define file processing

With the row schema and shared resources in place, we can write the actual work: turning a file into rows.

Process file

@coco.fn(memo=True)
async def process_file(
    file: FileLike,
    table: postgres.TableTarget[CodeEmbedding],
) -> None:
    text = await file.read_text()
    language = detect_code_language(filename=str(file.file_path.path.name))
    chunks = _splitter.split(
        text,
        chunk_size=1000,
        min_chunk_size=300,
        chunk_overlap=300,
        language=language,
    )
    id_gen = IdGenerator()
    await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)

process_file runs once per file. It reads the file, detects the language so Tree-sitter can parse it, splits the code along the syntax tree, and maps each chunk to process_chunk.

Each chunk is a coherent syntactic unit, so retrieval returns whole functions or blocks, never a fragment cut mid-statement. All major languages are supported (Python, Rust, JavaScript, TypeScript, Java, C++, and more); unknown types fall back to plain text.

Note that @coco.fn with memo=True is what makes this incremental: if a file's content and this function's code are both unchanged, the whole file is skipped on the next run. coco.map fans out to one process_chunk call per chunk.

Here is what chunking produces: each file is split into syntactic chunks, each with its own location and text.

Process chunk

Next, we embed the chunk with the shared embedder and declare the target row.

@coco.fn
async def process_chunk(
    chunk: Chunk,
    filename: pathlib.PurePath,
    id_gen: IdGenerator,
    table: postgres.TableTarget[CodeEmbedding],
) -> None:
    embedding = await coco.use_context(EMBEDDER).embed(chunk.text)
    table.declare_row(
        row=CodeEmbedding(
            id=await id_gen.next_id(chunk.text),
            filename=str(filename),
            code=chunk.text,
            embedding=embedding,
            start_line=chunk.start.line,
            end_line=chunk.end.line,
        ),
    )

We use SentenceTransformerEmbedder with all-MiniLM-L6-v2; there are 12k+ sentence-transformer models on Hugging Face, so swap in whichever you prefer. Note chunk.start.line and chunk.end.line: V1 carries line numbers through, so search results point straight at the lines that matched.

Define the main function

app_main wires the source to the target. It mounts the Postgres table (with a vector index), walks the codebase, and mounts one processing component per file.

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    target_table = await postgres.mount_table_target(
        PG_DB,
        table_name=TABLE_NAME,
        table_schema=await postgres.TableSchema.from_class(
            CodeEmbedding, primary_key=["id"],
        ),
    )
    target_table.declare_vector_index(column="embedding")

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(
            included_patterns=["**/*.py", "**/*.rs", "**/*.toml", "**/*.md", "**/*.mdx"],
            excluded_patterns=["**/.*", "**/target", "**/node_modules"],
        ),
        live=True,  # watch for changes; pass -L to `cocoindex update` to run live
    )
    await coco.mount_each(process_file, files.items(), target_table)

We walk the codebase from sourcedir (point this at the repository you want to index). We index files with the extensions .py, .rs, .toml, .md, .mdx, and skip dot-directories, target, and node_modules.

mount_table_target creates and manages the Postgres table for you: schema, the pgvector index, idempotent upserts, and orphan cleanup when a file disappears. live=True makes the filesystem source watch for changes, and mount_each runs one component per file so the engine can track and update them independently.

Create the App

Bind app_main into a coco.App and point it at the codebase root.

app = coco.App(
    coco.AppConfig(name="CodeEmbeddingV1"),
    app_main,
    sourcedir=pathlib.Path(__file__).parent / ".." / "..",  # index from repo root
)

That is the entire indexing path.

Run the pipeline

Run the cocoindex CLI to set up and update the index. Choose catch-up (scan, sync, exit) or live (catch up, then keep watching):

# Catch-up run
cocoindex update main

# Live run: keep watching for file changes
cocoindex update -L main

You'll see the index update state in the terminal:

✅ app_main: 1 total | 1 added
✅ declare_target_state_with_child: 1 total | 1 added
✅ _MountEachLiveComponent.process: 1 total | 1 added
✅ process_file: 622 total | 622 added
⏳ Elapsed: 43.4s

Query the index

We match user text against the index with a plain SQL query, reusing the same embedder from the indexing flow so indexing and querying stay consistent.

async def query_once(pool, embedder, query: str, *, top_k: int = 5) -> None:
    query_vec = await embedder.embed(query)
    async with pool.acquire() as conn:
        rows = await conn.fetch(
            f"""
            SELECT filename, code, embedding <=> $1 AS distance, start_line, end_line
            FROM "{TABLE_NAME}"
            ORDER BY distance ASC
            LIMIT $2
            """,
            query_vec, top_k,
        )
    for r in rows:
        score = 1.0 - float(r["distance"])
        print(f"[{score:.3f}] {r['filename']} (L{r['start_line']}-L{r['end_line']})")
        print(f"    {r['code']}")
        print("---")

The <=> operator is pgvector's cosine distance. We turn it into a similarity score and print the filename, the matched line range, and the code snippet. Run it against your index:

python main.py "embedding"

The search results print in the terminal:

Incremental updates

You never compute a diff or write update logic: you change something, and CocoIndex works out what to embed, upsert, and delete. @coco.fn(memo=True) decides what to recompute: a file is skipped when its content and the function's code are both unchanged, and an embedding is reused when its chunk text is unchanged. mount_table_target decides what to write: each row's id is derived from its chunk's content, so only rows that actually changed are upserted, and rows whose source is gone are deleted.

The same machinery covers two kinds of change: changes to your data (the code being indexed) and changes to your logic (the pipeline itself).

Data changes.

A file is added: only that file is chunked and embedded, and its rows are inserted. The rest of the repo is untouched.
A file is deleted: its rows are removed from the target.
A file is edited: the file is re-chunked, and the new chunks usually overlap the old ones. Chunks whose text is unchanged keep their id and embedding, so they are left as-is; genuinely new chunks are embedded and inserted; chunks that no longer exist are deleted. Edit one function and only that function's chunks are re-embedded, even though the whole file was re-read.

Logic changes. A pipeline change is reconciled the same way: CocoIndex compares the new output against what is already in Postgres and applies only the difference.

Change the file patterns (included_patterns / excluded_patterns): files that now match are added automatically; files that no longer match have their rows deleted automatically.
Tune the chunking (chunk size, overlap): files are re-chunked, producing the same partial-overlap diff shown above: unchanged chunks are no-ops, new chunks are embedded and inserted, dropped chunks are deleted.
Swap the embedding model: the vectors genuinely change, so all embeddings are recomputed, but row identity is stable: it is an in-place update of the embedding column, with no rows added or removed.

A catch-up run (cocoindex update main) does this once and exits; live mode (cocoindex update -L main) keeps watching and applies each change as you code.

CocoIndex Code

If you'd rather not wire the pipeline yourself, CocoIndex Code is an end-to-end implementation of exactly this indexing, packaged as a CLI and an MCP server. It does the same thing the example above does: AST-aware chunking, incremental re-index on file changes, local embeddings by default.

You can plug it straight into your coding agent or code-review agent:

Claude Code skill: npx skills add cocoindex-io/cocoindex-code, then invoke /ccc.
MCP server: claude mcp add cocoindex-code -- ccc mcp (Codex, OpenCode, Cursor, and any MCP client work the same way).
CLI: ccc index to build the index, ccc search "where we embed chunks" to query it.

The agent gets the whole repository as semantic context instead of reading one file at a time.

Next steps

The full, runnable example is in the CocoIndex repo: examples/code_embedding, with a walkthrough in the index-codebase example.

Point sourcedir at your own repository.
Swap the embedding model or the target store; the pipeline is plain Python.
Read the CocoIndex V1 docs for more on incremental processing, components, and targets.

Issues and contributions are welcome on GitHub.

CocoIndex Changelog 1.0.1 - 1.0.7

George He — Mon, 01 Jun 2026 00:00:00 GMT

CocoIndex reached v1.0 at the start of this cycle.

This is the first changelog since the v1 launch, covering seven releases (1.0.1–1.0.7). The focus is on making the engine easier to operate in production pipelines, then connecting it to more of the systems where teams already store data.

The short version

The engine work is the headline this cycle. If you run long-lived, expensive, or shared pipelines:

Per-argument memoization keys let you exclude clients, loggers, and debug flags from the cache key, so recomputation depends only on the inputs that matter.
coco.auto_refresh turns "poll this source every few minutes" into a live component with consistent error handling and target-state reconciliation.
coco.stats_group(...) breaks the adds / reprocesses / deletes counts down by data slice (per tenant, project, or folder) instead of one aggregate per processor.
New connectors: OCI Object Storage and Apache Iggy as sources; Turbopuffer, Neo4j, and FalkorDB as targets; LanceDB gained background compaction and in-place schema evolution.
Security and correctness fixes, including SQL identifier injection hardening and a concurrency race that only appeared under real database latency.

Engine

Per-argument memoization control

@coco.fn now accepts a per-argument memo_key, giving you fine-grained control over which arguments participate in the memoization cache key.

Production functions take more than data: clients, handles, config objects, loggers, tracing context, debug flags. Those values change across runs even when the meaningful input is identical, and each change could invalidate the cache and rerun expensive transforms, embeddings, or LLM calls for the wrong reason. memo_key keeps the cache keyed to the semantic input.

The API maps parameter names to either a callable (transform the value before fingerprinting) or None (exclude the parameter entirely):

@coco.fn(memo=True, memo_key={"entry": lambda e: (e.name, e.version), "extra": None})
def transform(entry: SourceDataEntry, extra: str) -> str:
    ...

Callable → applied to the argument; its return value is fingerprinted in place of the original.
None → the parameter is excluded from the memo key; changing it never invalidates the cache.
Not listed → fingerprinted normally.

`coco.auto_refresh` for live components

A common live-mode pattern is "run this processor on a fixed schedule." coco.auto_refresh now makes that pattern first-class.

Without an engine-level primitive, every pipeline hand-rolls the same loop and re-decides how errors propagate and how missing rows get reconciled. coco.auto_refresh wraps any processor function as a live component that re-runs on an interval and routes cycle failures through one error-handling channel.

import datetime
import cocoindex as coco

@coco.fn
async def app_main(db, target) -> None:
    await coco.mount(
        coco.auto_refresh(sync_users, interval=datetime.timedelta(minutes=5)),
        db, target,
    )

Each cycle reconciles target states against the previous cycle. If sync_users stops declaring a row, CocoIndex deletes the corresponding target automatically. See Live components → periodic refresh (#1995).

Scoped stats reports

By default, CocoIndex already breaks stats down by processor: each entry function (process_doc, process_code, …) reports its own counts: adds, reprocesses, deletes. What it can't do by default is break those counts down across data slices. coco.stats_group(title) opens a scope where everything mounted inside aggregates into a separate report under title, split out of the parent (no double counting).

Seeing that one tenant did 600 reprocesses while another did 1,150 fresh adds tells you who is churning, who is growing, and where an unexpected reprocess storm came from. One aggregate per processor hides all of that. The natural slice is the data: per tenant, per project, or per source folder.

@coco.fn
async def app_main(tenants, target):
    for tenant in tenants:
        # same processor, but stats scoped per tenant
        with coco.stats_group(f"tenant:{tenant.id}", report_to_stdout=True):
            files = localfs.walk_dir(tenant.docs_dir, ...)
            await coco.mount_each(process_doc, files.items(), target)

The block also yields a handle with the same stats() and watch() methods as UpdateHandle, so you can stream a single slice's counts to a dashboard. See Progress monitoring → Scoped reports (#2042).

Other engine improvements

Configurable logging: set the Python logger level directly via the COCOINDEX_LOG_LEVEL environment variable (#2035).
Cleaner Rust internals: the workspace was unified and the core SDK isolated from PyO3, separating the engine from its Python bindings and laying the groundwork for native Rust consumers of the engine (#1973).

Built-in operations

LiteLLM speech-to-text

CocoIndex added speech-to-text (STT) support through LiteLLM (#1889), so audio sources can be transcribed inside a pipeline using any LiteLLM-backed STT provider. A LiteLLMTranscriber wraps the transcription API:

from cocoindex.ops.litellm import LiteLLMTranscriber

transcriber = LiteLLMTranscriber("whisper-1")
transcript = await transcriber.transcribe(audio_file)

Code splitter: eight new languages

The RecursiveSplitter gained tree-sitter support for eight new languages this cycle, so syntax-aware chunking now covers a much wider slice of real codebases and config:

Svelte and Vue: component-aware chunking for frontend code (#1937)
Julia: for scientific and numerical codebases (#1942)
Elm (#1955) and Astro (#1984)
Bash, CMake, and HCL: shell scripts, build files, and Terraform/infra configs (#1954)

Pass the language name to language= (or let detect_code_language() infer it from a filename):

from cocoindex.ops.text import RecursiveSplitter

splitter = RecursiveSplitter()
chunks = splitter.split(source_code, chunk_size=2000, chunk_overlap=200, language="svelte")

Entity resolution: parallel by default

Entity resolution (dedup-and-canonicalize a set of extracted entity names) now resolves independent connected components of the candidate graph in parallel, speeding up LLM-backed deduplication on large entity sets (#2006).

Connectors

Source: OCI Object Storage with live bucket watching

A new Oracle Cloud Infrastructure (OCI) Object Storage source mirrors the S3 source's API for scanning a bucket, and adds an optional live mode.

It's a good illustration of the stream–state duality in CocoIndex's source model. A bucket is state: list_objects() scans it and establishes the full set of objects. A change feed is a stream: when you supply a live_stream, CocoIndex does the initial scan once and then keeps watching, applying createobject / updateobject / deleteobject events incrementally instead of rescanning. For OCI those events flow through OCI Streaming, consumed over the Kafka protocol; an event-time cutoff with a 5-second clock-skew tolerance decides which streamed events predate the scan. Under the hood it's built on a new lower-level LiveStream abstraction, a keyless stream of messages with in-memory watermark tracking and ack-on-completion (#1905).

This is the same model CocoIndex already uses for the local file system, where the change stream comes from OS file-system events rather than Kafka. More sources, such as S3, will gain live mode over time.

Source: Apache Iggy

CocoIndex now reads from Apache Iggy, a persistent message-streaming platform (#1969).

Target: Turbopuffer

A new Turbopuffer target connector landed, along with a runnable text-embedding example (#1934).

Target: Neo4j

A native Neo4j property-graph target landed alongside a meeting_notes_graph_neo4j example, mapping nodes and relationships into Neo4j without writing Cypher by hand (#1932).

Target: FalkorDB

The FalkorDB property-graph target was brought forward into the v1 connector set, with support for nodes, relationships, and vector/FTS indexes (#1908).

Target: LanceDB

LanceDB got two upgrades for production workloads:

The v1 target now optimizes (compacts) tables periodically in the background, keeping query performance steady as data churns (#2008, with a scheduling fix in #2013).
Table targets can add new columns in place, so schema evolution doesn't require a rebuild (#1951).

Bug fixes

Postgres correctness

halfvec op classes are now used for indexes on half-precision vectors (#2029).
U+0000 (NUL) bytes are stripped when writing text/jsonb, so Postgres no longer rejects otherwise-valid payloads (#2032).
The pgvector extension now installs into the default schema, avoiding search_path surprises (#1979).

Other fixes

SQL identifier validation: the Postgres and SQLite connectors now validate table and column identifiers before interpolating them into statements, closing a class of SQL-injection vectors at the connector boundary (#1947, #1965).
Ownership-transfer race: a pending-state protocol closes a preempt race so target-state ownership transfer stays correct under Postgres I/O latency, not just on microsecond-fast LMDB (#1994).
Clean cancellation: cancellation propagates through task spawn boundaries, tearing work down cleanly instead of leaking orphaned tasks (#1902).
on_error cascade: errors cascade through the Build-mode GC sweep (#1999).
numpy serde: _frombuffer is registered under both numpy 1.x and 2.x paths (#2012).
Responsive progress display: the PTY reader moved onto a dedicated OS thread, keeping the async runtime responsive under heavy logging (#2033, #2040).

Build with CocoIndex

Meeting notes → knowledge graph, now on Neo4j

The popular meeting-notes-to-knowledge-graph example now ships in a Neo4j flavor (meeting_notes_graph_neo4j). It watches a folder of meeting notes, uses LLM extraction to pull out structured entities (Meetings, People, Tasks) and the relationships between them, then maps everything into Neo4j as nodes and edges. Change one note and CocoIndex reprocesses only that note, keeping the graph continuously in sync.

Turbopuffer and OCI text-embedding examples

Two more end-to-end examples landed alongside the new connectors: text_embedding_turbopuffer shows a complete embed-and-query pipeline against Turbopuffer, and oci_object_storage_embedding demonstrates ingesting from OCI Object Storage with live bucket watching. Both are ready to clone and run.

Watch: build it end to end on FalkorDB

Follow along with the Build with CocoIndex walkthrough, which builds the meeting-notes-to-knowledge-graph pipeline end to end on FalkorDB, then clone the meeting_notes_graph_falkordb example to run it yourself.

For the complete list of changes, see the GitHub releases. If CocoIndex is useful to you, consider starring the repository.

Thanks to the community

Thanks to the contributors who shipped changes this cycle.

@prrao87

Thanks @prrao87 for fixing LanceDB optimize scheduling.

@zherendong

Thanks @zherendong for parallelizing entity resolution via candidate-graph components.

@countradooku

Thanks @countradooku for adding the Apache Iggy connector.

@Haleshot

Thanks @Haleshot for keeping the docs in shape after the v1 launch: removing the stale v1 mention from the install command, fixing a broken URL, and updating stale v1 branch links to main.

@galshubeli

Thanks @galshubeli for bringing the FalkorDB target connector into the v1 connector set.

@Gohlub

Thanks @Gohlub for adding LiteLLM speech-to-text support, CocoIndex's first audio operation.

@MrAnayDongre

Thanks @MrAnayDongre for adding per-argument memo_key support to @coco.fn.

@aaronjmars

Thanks @aaronjmars for validating SQL identifiers in the Postgres and SQLite connectors.

@tuanaiseo

Thanks @tuanaiseo for reporting and fixing a potential SQL injection in the Postgres connector, responsibly disclosed.

@nuthalapativarun

Thanks @nuthalapativarun for a prolific cycle of splitter work: adding Elm, Astro, and Bash, CMake, and HCL tree-sitter support, plus adding context to Rust→Python error messages and expanding the supported-languages docs.

@qWaitCrypto

Thanks @qWaitCrypto for making the LanceDB v1 target optimize tables periodically and for adding columns in place.

@shaiar

Thanks @shaiar for fixing serde _frombuffer registration across numpy 1.x and 2.x.

@octo-patch

Thanks @octo-patch for adding Python formatting and linting commands to CLAUDE.md.

@phuctoan123

Thanks @phuctoan123 for documenting gws for Google Drive setup.

Live CSV → Kafka with CocoIndex's New Kafka Target Connector

George He — Tue, 28 Apr 2026 00:00:00 GMT

CocoIndex just got a Kafka target connector. You can now declare Kafka topics as a target of your pipeline the same way you'd declare a Postgres table or a vector index, and CocoIndex will incrementally produce messages as your source data changes: no producer loop or bookkeeping, and no "did I already publish this row?" logic.

In this post we walk through a small but complete example: watch a local directory of CSV files, turn each row into a JSON message keyed by the row's primary key, and publish to a Kafka topic hosted on StreamNative. The whole pipeline is about 60 lines of Python, and it runs in live mode: edit a CSV, and within a second only the changed rows show up on the topic.

Getting data into Kafka has always been a pile of glue: producers, dedup state, deletes, schema re-bootstraps, file-watcher debouncing. Easy to get subtly wrong, never interesting to maintain.

The new connector lets you skip it. Hand CocoIndex an AIOProducer, a topic, and a function mapping each source row to a key and value. It handles the rest, producing messages only for rows that actually changed.

From static knowledge to streaming signals

Many agent stacks today are built around periodic snapshots of their knowledge sources. The wiki is re-indexed overnight, the codebase is re-embedded on a cron, the CRM is re-pulled on a schedule, and agents read those snapshots over and over, hoping to detect when something has changed. For long-running agents that may execute for hours at a time, a snapshot captured at the start of the run quickly drifts out of sync with the underlying data.

A common next step is to wire sources directly to consumers via point-to-point webhooks, but this approach has well-known limitations once more than one consumer is involved. There is no shared path for replay or backfill, no buffer to absorb bursts when many changes arrive at once, and no common schema describing what a "change" looks like across systems. Teams that go down this path tend to end up reimplementing pieces of a durable log, separately, in each integration.

The combination of CocoIndex and Kafka takes a different approach: it treats the knowledge layer the same way operational data has been handled for years: as a stream of change events rather than a snapshot to be re-read. Drives, repos, design files, wikis, PDFs, and file shares (the unstructured data that has traditionally lived outside the streaming world) can be published to the same event backbone that already carries orders, clicks, and CDC traffic. The benefits show up in several places:

More efficient AI workloads. Embeddings, retrievals, and agent context are refreshed only when something has actually changed, which reduces redundant work and improves freshness at the same time.
A single change reaches every consumer. A commit, a renamed Drive document, or a Notion edit can update the vector index, notify an agent, update search, feed a Flink job, and land in a BI tile, without any of those systems needing to know about each other.
Easier extensibility. A new agent, a rebuilt RAG layer, or a compliance tool can be added as another subscriber to the topic, with the log providing replay so it sees historical changes the same way it sees new ones.
Better auditability. Each change consumed by an agent is durably recorded with offsets and timestamps, which makes it possible to answer questions like "did the agent see the updated policy before it acted?" with concrete evidence.
A stable contract over time. The change-event schema on the topic provides a stable interface between sources and consumers. Detectors, sources, and models can evolve independently while the wire format stays consistent.

It is worth stating the contrast directly. A static unstructured knowledge graph is rebuilt on a schedule, drifts between rebuilds, and pushes freshness logic into each consumer. A stream of unstructured change events stays current by construction and gives every consumer (agents, indexes, analytics, auditors) a shared view of what has changed. Most AI stacks today are built around the former; production agents tend to benefit more from the latter.

Kafka: message in, message out — now from the unstructured world

Kafka's contract is famously simple: a message goes in, a message comes out. That single shape is what makes the rest of the streaming ecosystem composable: Flink, ksqlDB, vector stores, OLAP sinks, search backends, microservices, agent runtimes. Anything downstream gets a real-time, replayable, fan-out-by-default view of the world by just reading a topic.

The catch is that the messages going in have historically been the structured kind: orders, clicks, IoT telemetry, Debezium CDC off Postgres or MySQL. The other half of the business (meeting notes, codebases, design files, PDFs, file shares, wikis) has lived in a parallel universe of vendor-specific webhooks, nightly batch jobs, and one-off ETL scripts no one wants to own.

CocoIndex closes that gap. It treats dynamically-changing unstructured assets as first-class CDC sources and emits clean key/value change events into Kafka, with upsert / delete / no-op semantics described later in this post.

The point is not "another connector." It is that Kafka stays Kafka (message in, message out) and CocoIndex makes the in side cover the dynamic, unstructured majority of the data that previously couldn't get there cleanly. Every team that already knows how to consume from Kafka now gets the unstructured world for free.

Why choose CocoIndex with Kafka

Kafka is great at durable, high-throughput event streaming, but it expects you to solve data shaping, schema evolution, and incremental change detection yourself. CocoIndex turns that into a declarative problem:

Define transformations like formulas: “from this table or API, derive these messages and fields.” CocoIndex computes and re-computes derived data whenever sources change, similar to spreadsheet formulas reacting to edits.
Built-in incremental processing: CocoIndex keeps local state (checksums, primary keys, or version markers) so subsequent runs push only changed or deleted rows into Kafka topics.
Schema-aware updates: when schemas change upstream, CocoIndex re-derives downstream messages instead of forcing you to manually re-bootstrap topics or write ad-hoc migration scripts.
Pluggable sinks beyond Kafka: the same flow that feeds Kafka can also populate vector databases, OLTP/OLAP stores, or search indexes, so Kafka events and your AI/search backends stay in sync from a single definition.

Partnership with StreamNative

We're partnering with StreamNative to bring real-time data infrastructure to AI workloads. The two halves fit cleanly because they answer different questions: CocoIndex looks at noisy, dynamic, unstructured sources and produces a precise answer to "what's different since last time?" StreamNative's Ursa-powered Kafka service makes sure that answer reaches every system that cares, reliably, at scale, with history intact. With Kafka in the middle, sources don't have to behave like streams and agents don't have to behave like database clients; the topic absorbs the impedance mismatch and lets each side evolve on its own schedule.

The example: CSV files → JSON messages

To make all of this concrete, we'll build the smallest end-to-end pipeline that exercises the new connector: a local data/ folder of CSV files, watched in real time, with each row published as a JSON message to a Kafka topic on StreamNative. Edit a cell, and within a second exactly one message (for that one row) appears on the topic. Add a row, get one new message. Delete a file, and every row from it is tombstoned. The whole thing is around sixty lines of Python.

CSV is a deliberately humble starting point. It's the format that shows up everywhere and gets respect nowhere: exports from BI tools, dumps from vendors who won't give you a webhook, spreadsheets parked in a shared drive by a colleague, output from a nightly script no one wants to maintain. CSV files look structured but live like unstructured assets: dropped into a folder, edited at random, with no notifications and no schema contract. If we can turn a directory of them into a clean, row-keyed, live Kafka stream with diff-only writes, the same pattern carries over to the noisier sources mentioned earlier: PDFs, codebases, wikis, design files. CSV just lets us focus on the new connector and the live-update loop without getting tangled in a richer parser.

The full example lives in the CocoIndex repo. Here's the shape of it.

We have a data/ folder with a couple of CSV files:

# data/products.csv
sku,name,category,price
SKU001,Wireless Mouse,Electronics,29.99
SKU002,Mechanical Keyboard,Electronics,89.99
SKU003,USB-C Hub,Accessories,45.00

# data/employees.csv
emp_id,first_name,last_name,department,email
E101,Alice,Chen,Engineering,alice.chen@example.com
E102,Bob,Smith,Marketing,bob.smith@example.com

The goal: every row becomes a JSON message on a Kafka topic, keyed by the value of the row's first column (the primary key). When a CSV file is edited, only the rows that actually changed should be re-published.

The pipeline

First, the Kafka producer is set up once at app startup using a lifespan hook, and stashed in a ContextKey so the rest of the pipeline can grab it without passing it around:

import cocoindex as coco
from cocoindex.connectors import kafka, localfs
from confluent_kafka.aio import AIOProducer

KAFKA_PRODUCER = coco.ContextKey[AIOProducer]("kafka_producer", tracked=False)

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder):
    config = {
        "bootstrap.servers": KAFKA_BOOTSTRAP_SERVERS,
        "sasl.mechanism": "PLAIN",
        "security.protocol": "SASL_SSL",
        "sasl.username": KAFKA_SASL_USERNAME,
        "sasl.password": KAFKA_SASL_PASSWORD,
    }
    producer = AIOProducer(config)
    builder.provide(KAFKA_PRODUCER, producer)
    yield

ContextKey is how CocoIndex shares the producer across components without threading it through every function call; we'll come back to it later. The SASL block is what StreamNative (or any production broker) wants. For a local broker you can drop those four lines and just point bootstrap.servers at localhost:9092.

Next, the per-file processor. This is where the new Kafka API shows up:

@coco.fn(memo=True)
async def process_csv(file: FileLike, topic_target: kafka.KafkaTopicTarget) -> None:
    text = await file.read_text()
    reader = csv.DictReader(io.StringIO(text))

    headers = reader.fieldnames
    if not headers:
        return
    first_col = headers[0]

    for row in reader:
        key_value = row.get(first_col)
        if key_value is not None:
            value = json.dumps(row)
            topic_target.declare_target_state(key=key_value, value=value)

The @coco.fn(memo=True) decorator means the per-file work itself is memoized too: if a file's contents haven't changed, process_csv doesn't even run.

Declare states, not messages

topic_target.declare_target_state(key=key, value=value)

It's deliberately not called send_message() or produce() or declare_message(). It's declare_target_state.

CocoIndex is a state-driven data framework. The mental model is the same one you'd use for a spreadsheet, a React component tree, or a SQL materialized view: you describe what the target should look like as a function of the source, and the framework figures out the transitions. You don't compute deltas. You don't track "what did I send last time." You don't handle insert vs. update vs. delete as separate code paths. You just say, "given this CSV row, the target state for key SKU001 is this JSON blob," and that's it.

Kafka makes that distinction unusually visible because Kafka's wire model is the opposite: a topic is a log of events, not a snapshot. Producers send change events; consumers (or compacted topics) reconstruct state from the log. So the question is: who's responsible for the gap between "I have desired states" and "the broker needs to receive change events"?

In CocoIndex, the framework owns that gap. When you call declare_target_state(key=k, value=v):

If k is new or v differs from the last value the framework remembers for k, it emits an upsert message: (k, v) on the wire.
If k was previously declared but isn't declared this time, it emits a delete message: (k, None), or (k, deletion_value_fn(k)) if you supplied a tombstone constructor.
If k was previously declared with the same v, nothing is sent to the broker, and no consumer wakes up.

Messages are derived from state transitions. You only ever talk about states. This is exactly the same pattern as the Postgres target (declare_target_state → INSERT / UPDATE / DELETE) and the vector index targets: the wire-level operations differ, but the user-facing API is the same shape, because the semantics are the same.

The reason this matters in practice: it means the same process_csv function works correctly on the first run and every run after, whether a row is edited or removed, a file is deleted, or the whole pipeline crashes and restarts. There is no separate "initial load" code path versus "incremental update" code path. There's just "given the source, here's what the target should look like," and that statement is true whether the target is empty, half-populated, or already in sync.

Finally, wire it all together:

@coco.fn
async def app_main() -> None:
    topic_target = await kafka.mount_kafka_topic_target(KAFKA_PRODUCER, KAFKA_TOPIC)

    files = localfs.walk_dir(
        localfs.FilePath(path="./data"),
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.csv"]),
        live=True,
    )
    await coco.mount_each(process_csv, files.items(), topic_target)

app = coco.App(coco.AppConfig(name="CsvToKafka"), app_main)

Two things to notice here:

mount_kafka_topic_target(...) resolves the producer from the context key and gives back a handle. The topic itself is user-managed: CocoIndex never creates or deletes topics; it just produces into one you already own.
localfs.walk_dir(..., live=True) turns the directory walker into a live source: it does an initial scan, then keeps watching the filesystem and pushes incremental updates downstream. Combined with cocoindex update -L, the whole pipeline runs continuously instead of one-shot.

Live mode: one flag, everything else is the same

So far we've described what happens on a single run. But in reality, source files get edited, rows get added and removed, and you usually want the topic to keep up. The same process_csv runs as a catch-up run (scan once, reconcile everything that's changed since last time, exit) or as a continuously-running pipeline that keeps watching for changes. The diff between the two is one keyword argument and one CLI flag:

Catch-up run:

files = localfs.walk_dir(
    localfs.FilePath(path="./data"),
    path_matcher=PatternFilePathMatcher(included_patterns=["**/*.csv"]),
)
await coco.mount_each(process_csv, files.items(), topic_target)

cocoindex update main.py

Live:

files = localfs.walk_dir(
    localfs.FilePath(path="./data"),
    path_matcher=PatternFilePathMatcher(included_patterns=["**/*.csv"]),
    live=True,                 # ← +1 line
)
await coco.mount_each(process_csv, files.items(), topic_target)

cocoindex update -L main.py   # ← +1 flag

That's the entire diff. process_csv doesn't change. The Kafka target doesn't change. There's no separate "streaming" code path to maintain.

Run the live version:

cocoindex update -L main.py

CocoIndex does a full scan first, publishes one message per row, then sits there watching data/:

Edit one cell in products.csv: exactly one Kafka message is produced for that one row (modulo broker retries; the producer is at-least-once by default). The other four rows are silent.
Add a new row: one new message.
Delete a row: one delete message (no value, since this example doesn't supply a deletion_value_fn).
Add a brand new CSV file: process_csv runs once for that file, publishing its rows.
Delete a CSV file: every row from that file gets a delete message.

What the flag actually does (and doesn't do)

It's worth being precise about this, because it's easy to assume the flag is doing more magic than it is.

Reconciliation happens in both modes. Whether you run cocoindex update or cocoindex update -L, you declare target states and CocoIndex computes deltas against the previous run, emitting only the upsert and delete messages needed to bring the topic in line. To do that, CocoIndex maintains an internal state store that remembers, for every declared key, the fingerprint of the value last sent. That store survives restarts, so when you stop and restart the pipeline it picks up where it left off: no re-broadcast of unchanged rows.

This is also where the ContextKey from earlier earns its keep. The state store doesn't identify "this Kafka topic" by SASL credentials or bootstrap address. It identifies it by the ContextKey it was anchored to (here, KAFKA_PRODUCER) plus the topic name. If you later rotate the SASL password, swap the broker endpoint, or replace the AIOProducer with a differently configured one, the same ContextKey is enough for CocoIndex to recognize the backend as the same one, and the state store carries over.

The flag controls when reconciliation runs. That's it. In a catch-up run, CocoIndex scans sources once, reconciles up to the moment, and exits. In live mode, it does the same initial catch-up, then keeps watching the sources for changes and reconciles incrementally as they arrive. The code path, reconciliation logic, and target API are identical: just catch-up-and-exit vs. catch-up-and-keep-watching.

The "watching" half comes from the source itself, not the flag. localfs.walk_dir(..., live=True) returns a live source backed by watchfiles; it knows how to deliver filesystem events. A non-live source (or live=False) just enumerates current state and stops. The CLI -L flag tells the runtime to keep the app alive so the watcher's events have something to drive.

One Kafka detail worth mentioning since the connector doesn't change it: each declared key is hashed to a partition by Kafka's default partitioner, so the same key always lands on the same partition. Log compaction and key-based consumers behave exactly the way they'd behave with any hand-rolled producer.

What do the messages look like on the topic?

We pointed this at a Kafka cluster on StreamNative Cloud. It gave us a real SASL_SSL endpoint in one click, with a hosted console for inspecting messages without writing a consumer. (Plain localhost:9092 works too if you skip the SASL fields in the producer config.)

Here's what shows up in the console for the cocoindex-csv-rows topic after running the example:

Keys are the row's primary-key column (SKU001, E101, …); values are the JSON-encoded rows. Edit a CSV locally, refresh, and a new message with the same key appears: same key, latest value wins, log-compacted consumers get exactly the current state.

Try it

The Kafka connector ships as an optional dependency group: pip install cocoindex[kafka] if you're pulling it into an existing project. To run the example directly:

git clone https://github.com/cocoindex-io/cocoindex
cd cocoindex/examples/csv_to_kafka
cp .env.example .env  # fill in your Kafka bootstrap + SASL creds
pip install -e .
cocoindex update -L main.py

No cloud broker handy? Point KAFKA_BOOTSTRAP_SERVERS at localhost:9092 and leave the SASL fields empty. The example works against any Kafka the confluent_kafka client can connect to.

Then go edit data/products.csv and watch the messages land.

Support our work

If this project helps you, we'd appreciate a GitHub star at the CocoIndex project.

CocoIndex V1 is Live!

Linghua Jin, George He — Wed, 22 Apr 2026 00:00:00 GMT

CocoIndex is an incremental engine for long-horizon agents. It continuously transforms data from any source (codebase, meeting notes) and builds the context for agents deployed in production.

CocoIndex V1 is now live. It is a fundamental redesign of how you write incremental data pipelines, built from a year of watching what people actually wanted to do with CocoIndex. CocoIndex V1 is built for AI engineers and agent builders: people building context, RAG, memory, knowledge graphs that live agents depend on. The design choice that matters most to that audience is deliberately small: the framework meets you in the language you already write, with the data types you already use. If Claude or Cursor can already fluently write it, it works here.

The headline change is simple: there is no DSL anymore. Your whole pipeline is now regular async Python, growing organically as functions call each other. The engine still does what CocoIndex has always done (managed targets, incremental processing, change tracking in Rust), but it now lives behind a Python-native surface, not behind a DSL with its own type system and its own rules about what you can and can't do.

The mental model is unchanged, and it's the clearest way to describe what CocoIndex does: you declare what the target should look like as a function of the source, and the engine figures out the transitions. CocoIndex calls this state-driven programming, the same shape as React, spreadsheets, or SQL materialized views. Declare the state; Coco handles the rest.

Why CocoIndex: the incremental engine for long-horizon agents

Agents run roughly 50× faster than humans, but the tools they rely on were built for human speed.

At GTC 2026, Jeff Dean and Bill Dally named a bottleneck that's about to reshape every piece of infrastructure around AI. Amdahl's law takes over: no matter how fast the model gets, end-to-end throughput is capped by the slowest thing in the loop. Jeff's specific emphasis was on ultra-low-latency inference for agents "operating in the background," running tasks that "take hours or perhaps even days, independently doing a bunch of things, correcting themselves, doing some more things."

Data infrastructure is one of those tools, and it matters beyond inference. An agent reasoning over a codebase, a conversation graph, a document corpus, or a stream of events needs that data fresh, organized, and cheap to query, not just on the first call, but throughout the run, because:

Agents write code. The artifacts they produce become source data for the next reasoning step.
Agents make decisions and record them. Traces, plans, audit logs: written by agents, read by other agents.
Agents update data while they run. A long-horizon agent isn't a one-shot query; it's a process that edits its own context as it proceeds.
Source data arrives faster, too. Codebases, logs, Slack, PRs, tickets, sensor feeds: all updating in the background, all expected to be visible to whatever the agent is doing next.

The default pattern for indexing that data ("batch-rebuild overnight") was built for humans who check a dashboard every morning. For agents running 50x faster, that cadence is the bottleneck. What agents need is an engine that treats derived data as a function of source data and keeps it in sync incrementally: only what actually changed gets re-embedded, re-upserted, or re-published, with no full rebuild and no waiting for the next batch window.

That has always been CocoIndex's job. V1 makes it the right shape for agent-era workloads: the same incremental, state-driven guarantees, but now expressive enough to cover the pipeline shapes agents actually produce (entity resolution, clustering, multi-phase reduction, per-tenant topologies, conditional targets) instead of just "chunk → embed → upsert." Every pattern in the examples gallery is something a long-horizon agent might want to run itself, and have its outputs become fresh source data for the next agent, without a human babysitting the refresh job.

What is an incremental engine, and why it's hard to build for production

An incremental engine does one job: keep derived data in sync with source data without redoing work that's already been done. The source data is changing and the processing code is dynamic.

To do it right, you need: reliable change detection across heterogeneous sources (filesystems, databases, queues, APIs), content fingerprints stable enough to diff but cheap enough to compute every run, persistent state that records what the last run produced so this run can compare against it, memoization keyed on both data and code fingerprints so that editing a helper function invalidates only the callers that actually depended on it, managed target lifecycles (schema evolution, orphan cleanup when a source disappears, idempotent upserts), transactional behavior when you're writing to multiple targets at once, and recovery logic for every partial failure in between, and many more.

Teams that take this on seriously typically allocate 10 – 20 engineers for at least six months to land the first production-worthy version, and then keep paying for maintenance indefinitely as sources, targets, and schemas evolve. CocoIndex ships all of this in the engine, so the code you write is the pipeline itself, not the scaffolding around it.

Notion has recently published their engineering work around the tip of the iceberg on maintaining an in-house data pipeline with incremental processing on simpler data transformation (e.g., without clustering, etc.) for live agents.

The V1 mental model: declare states, not steps

The mental model didn't change. The execution model did.

CocoIndex has always been a state-driven framework. You don't write code that says "compute this delta and apply it"; you describe what the target should look like as a function of the source, and the engine works out the transitions. If you've used React, spreadsheets, or SQL materialized views, you already know the shape of this:

React: declare UI as a function of state → React re-renders what changed.
Spreadsheets: declare formulas → cells recompute when inputs change.
CocoIndex: declare target states as a function of source → engine syncs what changed.

What's new in V1 is how you express that declaration. The whole pipeline is now plain async Python, without FlowBuilder, DataScope, or a two-phase "define, then run" lifecycle. Functions call other functions. Loops are loops. if statements are if statements. Behind the scenes, CocoIndex tracks every target state you declare and every memoized intermediate value, so the next run only does what changed.

V1 is deliberately not intrusive. You can start with a single @coco.fn on a plain Python function and opt into more of the engine as the pipeline grows: flip on memo=True when re-running starts to cost real time, add declare_* targets when you want the engine to own the output lifecycle, move to live mode when you want the pipeline to keep watching for changes. There's no "set up the framework first" step and no ceremony you have to write before you get value. You turn the incremental knobs on when you actually need them.

That model rests on a small vocabulary: an App is the top-level entity you run; inside it, each processing component groups one item's work with its target states and runs independently; incremental processing means only what changed gets reapplied to the target; and function memoization skips any function whose input and code haven't changed since the last run.

What that means in practice is code you don't have to write: a bookkeeping table tracking which file was last processed, insert / update / delete branches, "did I already embed this chunk?" checks, migration scripts when your schema changes. The engine owns the target state, diffs against the previous run, and only does what changed. V0 made that tractable. V1 makes it feel like writing native Python.

A complete V1 app (a PDF-to-Markdown converter) is short enough to read at a glance:

import pathlib
import cocoindex as coco
from cocoindex.connectors import localfs
from cocoindex.resources.file import PatternFilePathMatcher
from docling.document_converter import DocumentConverter

_converter = DocumentConverter()

@coco.fn(memo=True)
def process_file(file: localfs.File, outdir: pathlib.Path) -> None:
    markdown = _converter.convert(file.file_path.resolve()).document.export_to_markdown()
    outname = file.file_path.path.stem + ".md"
    localfs.declare_file(outdir / outname, markdown, create_parent_dirs=True)

@coco.fn
async def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:
    files = localfs.walk_dir(
        sourcedir, path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
    )
    await coco.mount_each(process_file, files.items(), outdir)

app = coco.App("PdfToMarkdown", app_main,
               sourcedir=pathlib.Path("./pdf_files"),
               outdir=pathlib.Path("./out"))
app.update_blocking()

Three things to notice:

process_file is just a function: you can set a breakpoint in it, call it directly, run it under pytest, whatever. The @coco.fn(memo=True) decorator hooks it into the engine's change-detection and memoization, but doesn't change its calling convention.
localfs.declare_file(...) is a declaration, not a write. It tells CocoIndex "the target state for this path is this content." If the target state hasn't changed since the last run, nothing happens on disk. If the source file disappears, the declaration disappears too, and CocoIndex deletes the corresponding output file automatically.
coco.mount_each(...) mounts one processing component per file. Each component is its own unit of incremental change detection: when one file's content changes, only that component re-runs.

That's the whole programming model. Everything below is consequences.

Why V1: the V0 ceiling

V0 worked (developers shipped real pipelines in production), but four constraints kept coming back, all consequences of the DSL-first design:

Two worlds, one program. FlowBuilder / DataScope / DataSlice for topology, regular Python for leaves. The DSL world couldn't if its way into a different shape, couldn't pass values into normal Python and back. Clustering, entity resolution, reduction across items, conditional sources: every one of them hit the seam.
A separate type system. Like a database, the engine had its own types distinct from Python's, so every Python type you wanted to use needed a bi-directional conversion wired in. dataclass and NDArray were covered; a PIL Image or a pyarrow array was not: if no conversion existed, you couldn't hand it to a function, and what did cross the boundary got marshaled on the way in and out.
Postgres as a hard dependency. The engine used Postgres for its own bookkeeping, so pip install cocoindex && python script.py wasn't a real path: you stood up a database first, even for a tiny local pipeline.
Static topology. add_source() and .export() were declared before execution, so multi-tenancy, config-driven topologies, and runtime target selection all required scaffolding around CocoIndex rather than being expressible in it.

Fixing any one of these meant changing the design.

Four shifts that change what you can build

1. No more DSL — the flow grows during execution

In V0, the flow had to be fully described before any data moved through it. FlowBuilder returned a frozen graph; the engine then interpreted that graph at execution time. In V1, the flow is the execution. app_main runs as the root processing component. Every await coco.mount(...) it does adds a child component. Every declare_* call records a target state. The component tree is built by running your code, not by parsing a graph beforehand.

This unlocks shapes that were awkward or impossible in V0:

Reduction patterns: "for each project, look at all its files, then aggregate." The multi-codebase summarization example does exactly this:

@coco.fn(memo=True)
async def process_project(project_name, files, output_dir):
    file_infos = await coco.map(extract_file_info, files)         # fan out
    project_info = await aggregate_project_info(project_name, file_infos)  # reduce
    markdown = generate_markdown(project_name, project_info, file_infos)
    localfs.declare_file(output_dir / f"{project_name}.md", markdown)

Multi-phase pipelines with cross-item logic: entity resolution across sessions, clustering, deduplication. The conversation-to-knowledge example uses three phases (per-session extraction, cross-session entity resolution, knowledge-base assembly) all expressed as function calls in a single app_main.
Conditional topology: if config.enable_kafka: await mount_kafka_target(...). No flag-pattern, no separate flow definitions.

And because everything is real Python code, debugging is real Python debugging. Set a breakpoint. Step through. Print things. The engine's not interpreting an opaque graph; it's calling your functions.

2. Native Python types — no separate type system

V1 doesn't have a separate type system: it uses Python's directly. The types v0 supported through bindings (dataclass, Pydantic, NDArray, dict, list, tuple) still work with nothing to wire up, and types v0 couldn't express (a PIL Image, a pyarrow array, an arbitrary class from a library you just pulled in) can be passed as function arguments the same way:

from PIL import Image

@coco.fn(memo=True)
async def caption_image(img: Image) -> str:
    return await vlm.describe(img)

That Image is just PIL.Image: no wrapper, no conversion. Pass the function a pyarrow Table, a torch.Tensor, or any class from a library you happen to have imported; the signature is whatever you'd write in plain Python.

Targets, function arguments, return types, memoization keys: everything is described in the type hints you'd write anyway. For memoized return values that persist between runs, the same type hint drives serialization; we wrote a long post on how the type-guided serializer works underneath.

3. Embedded LMDB — `pip install` and go

V0's "Postgres for engine bookkeeping" requirement is gone. V1 stores its internal state in LMDB, an embedded key-value store that lives in a single local file. There's no server process to run, schema to migrate, or port to expose.

pip install cocoindex
cocoindex update main.py

That's the whole setup. The LMDB file holds the target-state ledger, the memoization cache, and the component-path tree across runs. On a 64-bit system the default 4 GiB map size is virtual address space, not physical memory. You can bump it to 8 GiB or more without paying the cost upfront.

Postgres is still a first-class target (with pgvector support); it's just no longer required for the engine to function.

4. Dynamic sources and targets

In V0, sources and targets were declared in the flow definition, before execution. In V1, they're created during execution, by regular function calls, and the component-path tree tracks their identity across runs. So you can:

Mount a target conditionally based on a config row, an environment variable, or a feature flag.
Create different targets per item: e.g., one Postgres schema per tenant, one Kafka topic per category, one S3 prefix per dataset.
Build the topology from a database: query the list of active tenants at startup and mount one component per tenant, each with its own set of targets. When a tenant disappears from the list, CocoIndex automatically cleans up their target states (because the path is no longer mounted).

The conversation-to-knowledge example does this for entity types: it iterates over a list of entity configurations and mounts one SurrealDB target per type, all from inside a normal for loop in app_main.

entity_tables = {
    cfg.name: await surrealdb.mount_table_target(SURREAL_DB, cfg.name, entity_schema)
    for cfg in ENTITY_TYPES
}

Add a new entity type to ENTITY_TYPES, restart, and a new table appears, managed by CocoIndex from then on. Remove a type, and its table goes away. No migration, no orphaned data.

The real leap over v0 is when that list isn't a constant at all (a changing file, a database row, a live tenant registry), since v0 could already generate topology from static inputs at flow-creation time, but couldn't react when those inputs changed.

What we kept: the core promise

The redesign was the means, not the goal. The reason V1 exists is to make the things V0 was already good at more available. Those things are still here, and now they apply across a much wider set of pipeline shapes.

Fully managed targets

You declare what your target should look like; CocoIndex makes sure it's in that state, on any change, whether to source data or to your code. This is the lifecycle, top to bottom:

Layer	Upsert (first or changed declaration)	Delete (no longer declared)
Container: schema (tables · directories · other containers)	Create or alter	Drop
Leaf: data (rows · files · embeddings · other leaves)	Insert or update	Delete

No manual migrations. No orphaned data. No "did I already write this row?" code paths. The target is fully owned by CocoIndex: if you stop declaring something, it goes away. This is the same contract whether the target is a Postgres table, a LanceDB collection, a Neo4j graph node, a Kafka topic, an S3 prefix, or a directory of files on disk.

Incremental processing, two levels deep

Incremental processing is still the core of the engine, and V1 exposes it through two concepts:

Components own target states. When a component finishes, CocoIndex diffs its declared target states against the last run and applies only the changes (create / alter / drop for containers, insert / update / delete for contents), including recursive cleanup of sub-paths no longer mounted. When a component's inputs and code haven't changed, the whole component is skipped.
@coco.fn functions participate in change detection. Every decorated function's code is fingerprinted, and that fingerprint propagates up the call chain. Add memo=True and the return value is also cached, keyed on the function's arguments plus its code fingerprint, so the body is skipped when neither has changed.

Edit a helper function and only the components and memoized callers that actually depend on it are invalidated.

Rust-powered engine

The change-detection, fingerprinting, target-state diffing, and persistence are all written in Rust on top of Tokio. That's still true: Python is the user-facing API, but the hot paths run in Rust. You get pythonic ergonomics without giving up the performance characteristics that made V0 feel snappy.

What it looks like end-to-end

Here's a complete text-embedding pipeline that walks markdown files, chunks them, embeds the chunks with a SentenceTransformer model, and writes everything (with a vector index) to Postgres. It's the text_embedding example trimmed down: every line of pipeline logic is below.

import pathlib
from dataclasses import dataclass
from typing import AsyncIterator, Annotated

import asyncpg
from numpy.typing import NDArray

import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.chunk import Chunk
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from cocoindex.resources.id import IdGenerator

PG_DB = coco.ContextKey[asyncpg.Pool]("text_embedding_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)

_splitter = RecursiveSplitter()

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    async with await postgres.create_pool("postgres://...") as pool:
        builder.provide(PG_DB, pool)
        builder.provide(EMBEDDER, SentenceTransformerEmbedder("sentence-transformers/all-MiniLM-L6-v2"))
        yield

@dataclass
class DocEmbedding:
    id: int
    filename: str
    chunk_start: int
    chunk_end: int
    text: str
    embedding: Annotated[NDArray, EMBEDDER]

@coco.fn
async def process_chunk(chunk, filename, id_gen, table):
    table.declare_row(row=DocEmbedding(
        id=await id_gen.next_id(chunk.text),
        filename=str(filename),
        chunk_start=chunk.start.char_offset,
        chunk_end=chunk.end.char_offset,
        text=chunk.text,
        embedding=await coco.use_context(EMBEDDER).embed(chunk.text),
    ))

@coco.fn(memo=True)
async def process_file(file: FileLike, table) -> None:
    text = await file.read_text()
    chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language="markdown")
    id_gen = IdGenerator()
    await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    table = await postgres.mount_table_target(
        PG_DB, table_name="doc_embeddings",
        table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=["id"]),
    )
    table.declare_vector_index(column="embedding")

    files = localfs.walk_dir(
        sourcedir, recursive=True,
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
    )
    await coco.mount_each(process_file, files.items(), table)

app = coco.App(coco.AppConfig(name="TextEmbedding"), app_main,
               sourcedir=pathlib.Path("./markdown_files"))

Worth pausing on what's not in there:

No flow definition phase. app_main runs and the topology emerges.
No custom column types. embedding: Annotated[NDArray, EMBEDDER] is the schema; from_class reads it.
No insert / update / delete branches. declare_row(...) describes the desired state; the engine diffs.
No "did this chunk already get embedded?" check. Memoization handles it.
No bookkeeping table for which file was last processed. LMDB has the previous run's fingerprints.
No manual migration if you change DocEmbedding. Schema evolution is part of the managed-target contract.

Add a markdown file → its chunks get embedded and inserted. Edit one paragraph → only the affected chunks re-embed; the others reuse cached embeddings. Delete a file → its rows vanish from Postgres. Change the splitter's chunk_size → every file re-chunks, but embed(chunk) calls whose chunk text didn't change still hit the memo cache. Change the embedder model → because EMBEDDER is detect_change=True, every embedding gets recomputed and the column re-syncs.

That's the whole thing. None of those scenarios need a separate code path.

Living shapes V0 couldn't reach

A short tour of pipeline shapes the V1 model makes natural, drawn from the example gallery:

Multi-codebase summarization: fan out per project, fan out per file with coco.map, reduce back up to a project summary, write a markdown file per project. Pure Python control flow; the engine handles which projects need re-summarizing.
Conversation-to-knowledge: three phases in one app_main: per-session extraction (mount per YouTube ID), cross-session entity resolution (mount per entity type), and knowledge-base assembly (one final mount). Each phase reads the outputs of the prior one through normal function returns.
CSV → Kafka live: a live filesystem watcher feeding a Kafka topic target with at-most-once-per-change semantics, in about 60 lines. File-level source, row-level change events: one Kafka message per changed CSV row.
HN trending topics, image search, PDF embedding, paper metadata extraction, Amazon S3 + embeddings, Google Drive + embeddings, code embedding to LanceDB: the connector ecosystem that landed during V0 came over to V1, and a few new ones (SurrealDB, SQLite, Kafka, S3) joined it.

Try it

pip install cocoindex

Then either follow the Quickstart (PDF → Markdown in about thirty lines), or clone the repo and run any of the examples directly:

git clone https://github.com/cocoindex-io/cocoindex
cd cocoindex/examples/text_embedding
pip install -e .
cocoindex update main.py

The docs for V1 live at cocoindex.io/docs. The pieces most worth reading first:

Core Concepts: state-driven processing, target states, processing components, memoization
App: top-level entry point, lifespan, CLI
Processing Component: mount, use_mount, mount_each, component paths
Function: @coco.fn, memoization, change detection
Live Mode: pipelines that keep watching for changes

Bug reports, feature requests, and "this section of the docs is unclear" notes are all very welcome on GitHub issues, and we hang out in Discord. ⭐ us on GitHub if you want to follow where this goes. There's a lot more shipping over the next few months.

Support us

If you found this useful, starring CocoIndex on GitHub is the most direct way to help. It's how other developers find the project :)

Turn Podcasts into a Knowledge Graph with LLM and CocoIndex

George He — Thu, 02 Apr 2026 00:00:00 GMT

Podcasts are one of the richest sources of expert knowledge on the internet. A single Lex Fridman or Dwarkesh Patel episode can contain dozens of substantive claims about people, technologies, and organizations, but it's all locked inside hours of audio. You can't query any of it. You can't cross-reference what two different guests said about the same topic.

In this post, we'll build a CocoIndex pipeline that turns YouTube podcast episodes into a queryable knowledge graph. The pipeline downloads audio, transcribes with speaker diarization, uses an LLM to extract structured statements and entities, resolves duplicates across episodes, and stores everything in SurrealDB as a graph.

We use CocoIndex to build the pipeline. CocoIndex is a data indexing framework for building incremental data transformation pipelines: it tracks what's been processed, so re-running the pipeline only processes new or changed episodes. It also made it simple to take out any unwanted podcast content that gets merged into the knowledge graph.

The full source code is available at CocoIndex Examples: podcast_to_knowledge.

What We're Building

Here's the knowledge graph schema, five node types connected by four relationship types:

A session is one podcast episode. A statement is a thematic claim extracted from the conversation, e.g., "Scaling laws suggest that larger models will continue to improve." Each statement is linked to who said it and what entities it mentions.

Person, tech, and org are named entities. The tricky part is that the same entity can appear under different names across episodes ("GPT-4", "GPT4", "OpenAI's GPT-4"). We handle this with entity resolution, more on that later.

Pipeline Overview

The pipeline runs in three phases:

Phase 1 processes each episode independently: download audio, transcribe, and use an LLM to extract metadata, speakers, and statements. Sessions and statements are written to the database immediately since they don't need cross-episode deduplication.

Phase 2 collects all raw entity names (persons, techs, orgs) from every episode and deduplicates them using embedding similarity + LLM confirmation.

Phase 3 writes the deduplicated entities and all relationships to the database.

Phase 1: Per-Session Processing

Each session goes through a multi-step pipeline, starting from a YouTube URL.

Fetch Transcript

We download the audio with yt-dlp and transcribe it with AssemblyAI, which gives us speaker-diarized utterances (labeled "Speaker A", "Speaker B", etc.) along with YouTube metadata (channel name, title, description, upload date):

@coco.fn(memo=True)
async def fetch_transcript(youtube_id: str) -> SessionTranscript:
    """Download audio via yt-dlp, transcribe with speaker diarization via AssemblyAI."""
    url = f"https://www.youtube.com/watch?v={youtube_id}"
    with tempfile.TemporaryDirectory() as tmpdir:
        # 1. Download audio via yt-dlp, convert to mp3
        audio_path = os.path.join(tmpdir, "audio.mp3")
        ydl_opts = {"format": "bestaudio/best", "outtmpl": audio_path, "quiet": True,
                    "postprocessors": [{"key": "FFmpegExtractAudio",
                                        "preferredcodec": "mp3"}]}
        with yt_dlp.YoutubeDL(ydl_opts) as ydl:
            info = ydl.extract_info(url, download=True)

        # 2. Transcribe with AssemblyAI (speaker diarization)
        config = aai.TranscriptionConfig(speaker_labels=True)
        transcript = aai.Transcriber().transcribe(audio_path, config)

    # 3. Convert diarized output to structured utterances
    utterances = [Utterance(speaker=u.speaker, text=u.text)
                  for u in transcript.utterances]
    return SessionTranscript(
        utterances=utterances,
        yt_channel=info["channel"],
        yt_title=info["title"],
        yt_description=info.get("description"),
        yt_upload_date=info.get("upload_date"),
    )

The @coco.fn(memo=True) decorator is a CocoIndex feature that memoizes the function: if you've already fetched and transcribed a video, re-running the pipeline skips it entirely. This is essential when you're iterating on the downstream extraction logic and don't want to re-download hours of audio every time.

Two-Step LLM Extraction

Here's where it gets interesting. We can't extract everything in a single LLM call because of a bootstrapping problem: to extract good statements, the LLM needs to know who the speakers are (so it can attribute statements correctly). But the raw transcript only has generic labels like "Speaker A". We need to figure out who's who first.

So we split extraction into two passes. Both steps use a shared format_transcript() function that replaces raw diarization labels with names: Step 1 passes an empty map (no names known yet, so all speakers show as "Speaker A", "Speaker B"), and Step 2 passes the mapping from Step 1.

Step 1: Identify speakers and extract metadata. We format the transcript with generic labels and give the LLM the YouTube metadata (channel name, title, description) as context. The LLM returns who each speaker is, plus session metadata:

@coco.fn(memo=True)
async def extract_metadata(
    reformatted_transcript: str, transcript: SessionTranscript
) -> SessionMetadata:
    client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
    return await client.chat.completions.create(
        model=coco.use_context(LLM_MODEL),
        response_model=SessionMetadata,
        messages=[
            {"role": "system", "content": METADATA_PROMPT},
            {"role": "user", "content":
                f"YouTube channel: {transcript.yt_channel}\n"
                f"Video title: {transcript.yt_title}\n"
                f"Description: {transcript.yt_description or 'N/A'}\n\n"
                f"Transcript:\n{reformatted_transcript}"},
        ],
    )

The LLM output is a Pydantic model, enforced by instructor:

class SpeakerIdentification(pydantic.BaseModel):
    label: str      # e.g. "A", "B"
    name: str       # e.g. "Lex Fridman" — unidentifiable speakers are excluded

class SessionMetadata(pydantic.BaseModel):
    name: str
    description: str | None
    date: str | None
    speakers: list[SpeakerIdentification]

Step 2: Extract statements with real names. Now that we know "Speaker A" is "Lex Fridman", we reformat the transcript with real names and ask the LLM to extract thematic statements:

# Reformat: replace "Speaker A" with "Lex Fridman", etc.
speaker_map = {s.label: s.name for s in metadata.speakers}
step2_text = format_transcript(transcript.utterances, speaker_map)

# Extract statements from the named transcript
stmt_extraction = await extract_statements(step2_text)

Each extracted statement includes the speakers who made it and the entities it mentions:

class RawStatement(pydantic.BaseModel):
    statement: str              # "Scaling laws suggest larger models will improve"
    speakers: list[str]         # ["Lex Fridman"]
    mentioned_person: list[str] # ["Ilya Sutskever"]
    mentioned_tech: list[str]   # ["Large language model"]
    mentioned_org: list[str]    # ["OpenAI"]

All entity names must be self-contained. The extraction prompt explicitly instructs the LLM: "Never use pronouns, speaker labels, or contextual references. Every name must be a clear, unambiguous identifier that stands on its own." This matters because statements from different episodes will later be cross-referenced: an entity name like "he" or "the host" would be meaningless outside its original transcript.

Declaring Results with CocoIndex

After extraction, we declare the session and its statements as records in SurrealDB. CocoIndex uses a declarative model: you describe what the target state should look like, and the framework handles inserts, updates, and deletes.

IDs are generated using CocoIndex's IdGenerator, which produces stable IDs: the same inputs always yield the same ID, so re-running the pipeline doesn't create duplicates. Calling next_id() without arguments generates a sequential ID; calling next_id(content) incorporates the content into the ID so it remains stable even if the order of statements changes:

id_gen = IdGenerator(youtube_id)
session_id = await id_gen.next_id()

session_table.declare_record(row=Session(
    id=session_id,
    youtube_id=youtube_id,
    name=metadata.name,
    transcript=step2_text,
    # ... description, date
))

for stmt in stmt_extraction.statements:
    stmt_id = await id_gen.next_id(stmt.statement)
    statement_table.declare_record(row=Statement(id=stmt_id, statement=stmt.statement))
    session_statement_rel.declare_relation(from_id=session_id, to_id=stmt_id)

Mounted Components

Each session is processed as an independent mounted component via coco.use_mount(). A mounted component is a self-contained unit of work that CocoIndex tracks independently: each YouTube video ID gets its own component, so adding a new episode only processes that episode:

for youtube_id in video_ids:
    raw = await coco.use_mount(
        coco.component_subpath("session", youtube_id),
        process_session, youtube_id,
        session_table, statement_table, session_statement_rel,
    )
    all_session_raw.append(raw)

Each process_session call returns a SessionRawEntities, the raw entity names and statement linkages that Phases 2 and 3 will need:

@dataclass
class SessionRawEntities:
    session_id: int
    raw_entities: dict[str, list[str]]       # e.g. {"person": ["Lex Fridman", ...]}
    statements: list[IdentifiedStatement]     # statements with generated IDs

Sessions and statements are already written to SurrealDB at this point. The raw entities are carried forward for cross-session deduplication.

Phase 2: Entity Resolution

After processing all sessions, we have a pile of raw entity names from every episode. The same entity often appears under different names: "GPT-4" vs "GPT4", "Google" vs "Google LLC", "Sam Altman" vs "Samuel Altman". We need to collapse these into canonical names.

The approach uses embedding similarity to find candidates, then an LLM to confirm:

Compute an embedding for each entity name (using a small sentence transformer model).
For each entity, search a FAISS index for the most similar names already processed.
If any are close enough (cosine distance < 0.3, i.e. similarity > 0.7), ask an LLM: "Is GPT4 the same thing as GPT-4?"
The LLM picks the canonical name. Build a deduplication map.

async def resolve_entities(all_raw_entities: set[str]) -> dict[str, str | None]:
    index = faiss.IndexFlatIP(dim)

    for entity in entity_list:
        vec = embeddings[entity].reshape(1, -1).copy()
        faiss.normalize_L2(vec)

        # Find similar entities already in the index
        candidates = []
        if index.ntotal > 0:
            sims, idxs = index.search(vec, k=min(TOP_N, index.ntotal))
            for sim, idx in zip(sims[0], idxs[0]):
                if sim >= 1.0 - MAX_DISTANCE and idx >= 0:
                    canonical = resolve_canonical(index_names[idx], dedup)
                    if canonical != entity:
                        candidates.append(canonical)

        # Deduplicate candidate list
        unique_candidates = list(dict.fromkeys(candidates))

        if unique_candidates:
            resolution = await resolve_entity_pair(entity, unique_candidates)
            # Update dedup map based on LLM decision
            ...

        index.add(vec)
        index_names.append(entity)

    return dedup  # e.g. {"Apple Inc.": None, "Apple": "Apple Inc.", "AAPL": "Apple Inc."}

The embedding step is fast and cheap: it filters out the vast majority of entity pairs that are obviously different. The LLM call only happens for the small set of near-matches, keeping the cost down. We use a smaller, cheaper model for these resolution calls since the task is simple (configurable via the RESOLUTION_LLM_MODEL environment variable).

Entity resolution runs independently per entity type (person, tech, org), so CocoIndex processes them concurrently:

entity_dedup = dict(zip(
    [cfg.name for cfg in ENTITY_TYPES],
    await asyncio.gather(*(
        coco.use_mount(
            coco.component_subpath("resolve", cfg.name),
            resolve_entities,
            collect_all_raw(all_session_raw, cfg.name),
        )
        for cfg in ENTITY_TYPES
    )),
))

Phase 3: Knowledge Base Creation

With the deduplication maps ready, we write the final knowledge graph. Canonical entities become nodes, and all relationships use the resolved canonical names. A helper resolve_canonical(name, dedup) chases the dedup chain to find the root, e.g., resolve_canonical("AAPL", dedup) → "Apple Inc.":

@coco.fn
async def create_knowledge_base(all_session_raw, entity_dedup, ...):
    # Declare canonical entity nodes (name IS the id)
    for cfg in ENTITY_TYPES:
        dedup = entity_dedup[cfg.name]
        table = entity_tables[cfg.name]
        for name, upstream in dedup.items():
            if upstream is None:  # this name is canonical
                table.declare_record(row=Entity(id=name, name=name))

    # Declare relationships using canonical names
    person_dedup = entity_dedup["person"]
    for session_raw in all_session_raw:
        for person_name in session_raw.raw_entities.get("person", []):
            canonical = resolve_canonical(person_name, person_dedup)
            person_session_rel.declare_relation(from_id=canonical, to_id=session_raw.session_id)

        for stmt in session_raw.statements:
            # person_statement: who made the statement
            for speaker in stmt.raw.speakers:
                canonical = resolve_canonical(speaker, person_dedup)
                person_statement_rel.declare_relation(from_id=canonical, to_id=stmt.id)

            # statement_mentions: what the statement is about (polymorphic)
            for cfg in ENTITY_TYPES:
                dedup = entity_dedup[cfg.name]
                for entity_name in getattr(stmt.raw, f"mentioned_{cfg.name}"):
                    canonical = resolve_canonical(entity_name, dedup)
                    statement_mentions_rel.declare_relation(
                        from_id=stmt.id, to_id=canonical,
                        to_table=entity_tables[cfg.name])  # polymorphic target

The statement_mentions relationship is polymorphic: the target can be a person, tech, or org table. The to_table parameter tells CocoIndex which table the target ID belongs to.

Incremental Updates

A pipeline like this isn't a one-shot job. You'll add new podcast episodes over time, and you'll want to evolve the schema, maybe adding Product, Event, or Place as new entity types, or refining the extraction prompts. CocoIndex's memoization and component model make both scenarios efficient.

Adding new episodes

When you add a new YouTube URL and re-run the pipeline, only the new episode is processed. Existing episodes are untouched:

Fetch transcript: Skipped for existing episodes (memo=True on fetch_transcript).
LLM extraction (Step 1 + Step 2): Skipped for existing episodes (both are memoized).
Entity resolution: Re-runs to incorporate new entity names, but compute_entity_embedding and resolve_entity_pair are memoized: existing entities' embeddings and resolution decisions are reused. Only the new names trigger fresh LLM calls.
Knowledge base: CocoIndex's declarative targets handle the diff: new records are inserted, removed episodes are cleaned up.

Removing an episode works the same way: its mounted component is gone, so CocoIndex deletes its session, statements, and relationships from SurrealDB.

Evolving the schema

Say you want to add a new entity type, like Product. You'd add it to ENTITY_TYPES, update the extraction prompt, and re-run. Here's what happens:

Pipeline step	What happens	Why
Fetch transcript	Reused for all episodes	Memoized, no input changed
Step 1: Speaker identification	Reused for all episodes	Memoized, prompt unchanged
Step 2: Statement extraction	Re-runs for all episodes	Extraction prompt changed (new entity type)
Entity resolution (person, tech, org)	Reused	Raw entities unchanged, memoized
Entity resolution (product)	Runs fresh	New entity type
Knowledge base creation	Re-declared	New entities and relationships added

The expensive operations (downloading audio, transcribing, and identifying speakers) are fully reused. Only the statement extraction LLM calls re-run (because the prompt changed), plus entity resolution for the new type. If you have 50 episodes and add one entity type, you save most of the compute cost.

This is the payoff of CocoIndex's memo and use_mount design: each function is memoized by its inputs, and each session is an independent component. Changes propagate only to what's actually affected.

Running the Pipeline

Prerequisites

Python 3.11+, FFmpeg, Docker
An AssemblyAI API key (for transcription)
An OpenAI API key (for LLM extraction)

Setup

Start SurrealDB:

docker run -d --name surrealdb --user root -p 8787:8000 \
  -v surrealdb-data:/data surrealdb/surrealdb:latest \
  start --user root --pass root surrealkv:/data/database

Set environment variables and install:

export ASSEMBLYAI_API_KEY="..."
export OPENAI_API_KEY="sk-..."
pip install -e .

Add YouTube URLs to input/sample.txt:

# AI podcasts
https://www.youtube.com/watch?v=VIDEO_ID_1
https://www.youtube.com/watch?v=VIDEO_ID_2

Build the Knowledge Graph

cocoindex update podcast_knowledge.app

This is incremental: re-running skips episodes that have already been processed.

Exploring the Results

Once the pipeline completes, you can explore the knowledge graph in SurrealDB's built-in explorer (Surrealist). Connect to ws://localhost:8787, namespace cocoindex, database yt_conversations.

The graph view shows the relationships between persons and statements: each pink node is a statement, each blue node is a person:

You can also run analytical queries. For example, finding which technologies are mentioned by the most people across all episodes:

SELECT
  name,
  array::distinct(
    <-statement_mentions<-statement<-person_statement<-person.name
  ) AS persons,
  array::len(array::distinct(
    <-statement_mentions<-statement<-person_statement<-person.id
  )) AS person_count
FROM tech
ORDER BY person_count DESC
LIMIT 15;

Other useful queries:

-- All statements a person made
SELECT <-person_statement<-person.name AS speaker, statement FROM statement;

-- Full graph around a person
SELECT name,
  ->person_session->session.name AS sessions,
  ->person_statement->statement.statement AS statements
FROM person;

-- All entities involved in statements
SELECT statement,
  ->statement_mentions->person.name AS persons,
  ->statement_mentions->tech.name AS techs,
  ->statement_mentions->org.name AS orgs
FROM statement;

Summary

We built a pipeline that turns YouTube podcasts into a queryable knowledge graph:

Fetch: Download audio and transcribe with speaker diarization.
Extract (two-step): First identify speakers using YouTube metadata, then extract statements with real names, solving the bootstrapping problem of needing to know who's speaking before you can attribute what they said.
Resolve: Deduplicate entities across episodes using embedding similarity + LLM confirmation: cheap embeddings filter candidates, expensive LLM calls only happen for near-matches.
Store: Write the knowledge graph to SurrealDB with CocoIndex's declarative target model.

Because every expensive step is memoized and each session is an independent component, the pipeline is fully incremental, whether you're adding episodes or evolving the schema. See Incremental Updates for details.

The full source code is at CocoIndex Examples: podcast_to_knowledge. To stay updated on CocoIndex, star us on GitHub.

Support us ❤️

If this walkthrough was useful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub. It's how other developers discover the project, and every star keeps us motivated to ship better tools for building knowledge graphs, RAG pipelines, and live data infrastructure for AI.

Got a podcast, a meeting archive, or any other corpus you want to turn into a graph? We'd love to hear what you're building: drop by our Discord or open an issue on GitHub.

From pickle to type-guided, safer Python serialization

George He — Fri, 27 Mar 2026 00:00:00 GMT

CocoIndex is a framework for building incremental data pipelines. It has a Rust core for performance and exposes a Python SDK for users to define their pipelines. Under the hood, the engine needs to serialize and deserialize Python objects constantly, caching function results, persisting pipeline state, tracking records for change detection, with the serialized data crossing the Rust/Python boundary and stored by the Rust core.

In CocoIndex v0, we had a closed type system: a fixed set of supported data types, with serialization handled entirely in Rust. Safe and fast, but too rigid. Users couldn't use their own data types without converting them into our type system first.

When we started building v1, we wanted to natively support any Python data type. Pickle was the obvious first choice.

Pickle: Fast to Prototype, Painful to Live With

Pickle is Python's built-in serialization. It handles virtually any Python object out of the box: dataclasses, NamedTuples, Pydantic models, nested containers, custom classes. For early prototyping under prerelease, it was perfect: one line of code, and everything just worked.

But pickle has two fundamental problems that make it unsuitable for production.

Security. Pickle executes arbitrary code during deserialization. A crafted payload can run any Python code on your machine. In CocoIndex's case, the serialized data lives in internal storage (a database managed by the framework), so the risk requires an attacker to have access to that storage, which is unlikely for most deployments. Still, defense in depth matters: if the storage is ever exposed through a misconfiguration or a compromised database, pickle becomes an escalation path from data access to code execution. We didn't want that risk in the foundation of our persistence layer.

Overhead. Pickle encodes type information into every value: each value is a sequence of opcodes that tells the deserializer what type to reconstruct and how. This makes pickle self-describing, but at a cost. The integer 1024 takes 15 bytes in pickle, while msgpack encodes it in just 3. Msgpack can be this compact because it relies on the deserializer already knowing the expected type:

Value	pickle	msgpack	Ratio
`""` (empty string)	15 bytes	1 byte	15x
`1024`	15 bytes	3 bytes	5x
`"hello"`	20 bytes	6 bytes	3.3x
`{"key": [1,2,3]}`	31 bytes	9 bytes	3.4x
`Point(42, "origin")`	63 bytes	16 bytes	3.9x

For a pipeline that caches thousands of intermediate results, this overhead adds up.

Whitelisted Pickle: Secure but Tedious

Our first attempt to fix the security problem was a restricted unpickler. We maintained an allowlist of types that were safe to deserialize, and rejected everything else. Users could opt their types into the allowlist with a @coco.unpickle_safe decorator:

@coco.unpickle_safe
@dataclass
class DocumentChunk:
    text: str
    embedding: list[float]

This solved the security problem. But it introduced a new one: annotation burden.

Every type that appeared anywhere in a serialized object graph needed the decorator, not just the top-level type, but every type nested inside it. Miss one, and you get a runtime error.

We experienced this firsthand when building the conversation_to_knowledge example. It needed 9 @coco.unpickle_safe annotations across the codebase. Getting there took 4 rounds of run-fail-annotate-repeat: run the pipeline, hit an error about an unregistered type, add the decorator, run again, hit the next error. Four times.

For our own example. Imagine what this is like for users building real pipelines with dozens of model types.

Type-Guided Deserialization: Letting Type Hints Do the Work

We stepped back and asked: what do we actually have that describes the shape of users' data? The answer was already in their code: type hints.

In CocoIndex, there are three places where data gets serialized and deserialized: memoized function return values (the most common case), memo state values for change detection, and target tracking records. For memoized return values, the type hint is right there in the function signature:

@coco.fn(memo=True)
async def embed_chunk(chunk: str) -> list[float]:
    return await embedding_model.embed(chunk)

The return type list[float] tells us exactly how to deserialize the cached result. No extra annotation needed: the type hint that users already write for readability and IDE support doubles as the deserialization schema.

This led us to a type-guided serialization system built on msgspec, a fast msgpack-based serialization library for Python. Msgspec natively handles dataclasses, NamedTuples, and standard types (str, int, list, dict, datetime, UUID, etc.), and it does so by relying on type information at deserialization time.

The routing byte

Different types need different serialization engines. We use a single routing byte at the start of each payload to select the engine:

Byte	Engine	Used for
`0x01`	msgspec (msgpack)	Dataclasses, NamedTuples, primitives, collections
`0x02`	Pydantic	`BaseModel` subclasses
`0x80`	Pickle	Opted-in types, legacy data

Serialization checks types in priority order: explicit pickle opt-in first (the user specifically requested it), then Pydantic models, then msgspec as the default. Deserialization reads the routing byte and dispatches accordingly.

For Pydantic models, we serialize via model_dump(mode="json") into msgpack, and deserialize via TypeAdapter.validate_python(). This avoids pickle entirely while preserving Pydantic's validation semantics.

For types that msgspec can't handle natively, like numpy arrays or pathlib paths, users can opt in to pickle serialization with @coco.serialize_by_pickle. These types get the pickle routing byte (0x80), still through a restricted unpickler for safety.

What changed for users

The conversation_to_knowledge example? All 9 @coco.unpickle_safe annotations, gone. Users define their data types with standard Python type hints, and serialization just works:

@dataclass
class DocumentChunk:
    text: str
    embedding: list[float]
    metadata: dict[str, str]

No decorators. No registration. No run-fail-annotate-repeat cycle. The framework reads the type hint and picks the right serializer and deserializer automatically.

The Edge Cases: When Round-Trip Isn't Perfect

We'd be lying if we said this approach is perfect for every type. There are edge cases with union types that users should be aware of, and these aren't specific to CocoIndex. They're fundamental to how msgspec and Pydantic handle ambiguous types.

msgspec: loud failure

msgspec rejects ambiguous unions at construction time:

import msgspec.msgpack

# These all raise TypeError immediately:
msgspec.msgpack.Decoder(str | datetime.date)     # two str-like types
msgspec.msgpack.Decoder(list[int] | tuple[int, ...])  # two array-like types
msgspec.msgpack.Decoder(bytes | bytearray)        # two bytes-like types

This is actually a feature. msgspec tells you upfront that these types can't be distinguished on the wire, rather than silently corrupting your data.

Pydantic: silent surprise

Pydantic is more permissive: it accepts date | str, but the behavior after a serialization-deserialization round-trip can be surprising.

Consider a Pydantic model with a date | str field:

class Event(BaseModel):
    ts: datetime.date | str

e = Event(ts=datetime.date(2024, 1, 15))

If you serialize this to JSON (or msgpack), date becomes the string "2024-01-15". On deserialization, Pydantic sees a string input and needs to decide: is it a str or a date?

With Pydantic v2's default smart mode, the answer is always str. Smart mode scores candidates by exactness: a string input is an exact match for str but only a lax match (requires coercion) for date. Exact wins. Ordering doesn't matter: date | str and str | date both produce str.

Left-to-right mode behaves differently. With date | str ordering, Pydantic tries date first, the lax coercion from "2024-01-15" succeeds, and you get your date back:

from pydantic import Field
from typing import Annotated

class Event(BaseModel):
    # Left-to-right mode: more specific type first
    ts: Annotated[datetime.date | str, Field(union_mode='left_to_right')]

But this requires users to explicitly opt in to left-to-right mode AND put the more specific type first.

The fundamental issue

This isn't a bug in any library. It's inherent to the wire format. Msgpack (like JSON) has no date type: dates are serialized as strings. Once the type information is erased on the wire, no amount of clever decoding can reliably recover it when multiple types map to the same wire representation.

The practical advice: avoid union types where members share a wire representation in data objects that go through serialization. Use unambiguous types, or use Pydantic's discriminated unions with a literal tag field.

What We Learned

Type hints are an underutilized source of truth. Python developers already annotate their data types. Using those annotations for serialization is a natural extension: no new concepts, decorators, or registration APIs.

Make ambiguity loud, not silent. msgspec's approach of rejecting ambiguous unions at construction time is better than silently producing wrong results. When building a serialization layer, fail early and clearly.

Don't fight the wire format. If the wire format can't distinguish two types, your serialization layer can't either. Design your data types around what the serialization format can express, not the other way around.

Start simple, but have a plan. Pickle was right for prototyping. The whitelisted pickle was right for the security fix. The type-guided approach was right for production. Each step taught us what the next step needed to be.

Learn More

CocoIndex on GitHub: the open-source data indexing framework
CocoIndex v1 Documentation: getting started guides and API reference
Building an Invisible Daemon: how we built the local daemon architecture behind CocoIndex

Support us

⭐ Star CocoIndex on GitHub and share with your community if you find it useful!

Invisible Daemon: architecture patterns for local dev tools

George He — Tue, 24 Mar 2026 00:00:00 GMT

Building a Python CLI tool with a background daemon? The five patterns that eliminate manual start/restart, connection leaks, and 15-second shutdown times are:

Auto-start on first use via a Unix socket probe (no daemon start command).
Version handshake on every connection for transparent upgrades after pipx upgrade.
Per-request connections instead of persistent sessions: connection leaks become impossible by construction.
Resource closure instead of thread signaling: close the listener; don't poll a threading.Event.
PID file removal as the exit signal: os.kill(pid, 0) returns True for zombies.

This is how we built the daemon behind cocoindex-code (ccc), an embedded AST-based semantic code search tool for Claude Code, Codex, and Cursor.

Why this post exists. cocoindex-code is a local MCP server and CLI that gives coding agents semantic understanding of your whole codebase via AST-based chunking: Tree-sitter parses code by syntax (functions, classes, methods), and only changed chunks are re-embedded on update. The daemon described here is the background process that makes that possible: it loads the ML model once, watches for file changes, and serves searches in under a second. If you're building something similar (a local AI context layer, a code-aware MCP server, any dev tool with a heavy background process), the patterns below are what we landed on after shipping to production.

We moved cocoindex-code from a monolithic per-session process (load model, index, search, exit) to a persistent daemon with a thin CLI (ccc) and a lightweight MCP client as frontends. This unlocked a benefit beyond performance: humans can call ccc search "auth logic" directly from the terminal and pipe it into other tools.

The journey from "obvious daemon" to production-ready was full of design decisions, and the best ones were usually about what to remove. Here's what the user sees:

ccc init                    # one-time setup: creates settings files (no daemon)
ccc index                   # build the search index (daemon auto-starts here)
ccc search "auth logic"     # semantic search (daemon handles it)
ccc search --refresh "auth" # re-index then search (two daemon requests)

ccc init is purely local: it creates configuration files and a .gitignore entry. Every other command talks to the daemon. But the user never starts, stops, or restarts the daemon manually. It just works.

The initial daemon was largely built by Claude Code: persistent connections, async handler loops, multi-step shutdown. It worked, but it was complex in ways we didn't appreciate until we looked closely. The simplifications came from stepping back and questioning the design rather than fixing individual bugs.

This post covers the patterns we landed on, in the hope that they're useful to anyone building a local daemon for developer tools.

How do you make a local daemon invisible to the user?

The most important design constraint: users should never think about the daemon. No ccc daemon start in the setup guide. No "please restart the daemon after upgrading." No stale config surprises.

How do you auto-start a Python daemon on first use without a manual command?

The first time a user runs ccc index or ccc search, the client tries to connect to the daemon's Unix socket. If it fails, it starts the daemon as a detached background process and waits for the socket to appear.

This logic is built into the lowest-level connection function, not into a separate "ensure daemon" step:

_daemon_ensured = False

def _connect_and_handshake() -> Connection:
    global _daemon_ensured

    if _daemon_ensured:
        return _raw_connect_and_handshake()

    # First connection — auto-start/restart as needed.
    try:
        conn = _raw_connect_and_handshake()
        _daemon_ensured = True
        return conn
    except DaemonVersionError:
        stop_daemon()
    except (ConnectionRefusedError, OSError):
        pass

    start_daemon()
    _wait_for_daemon()
    # ... retry and return

A module-level _daemon_ensured flag means the first call in a CLI session pays the startup cost. Every subsequent call connects directly, with no separate "ensure" step or throwaway probe connection.

How do you auto-restart a daemon when the CLI tool upgrades?

When a user runs pipx upgrade cocoindex-code, the next CLI command should just work, using the new version. No manual restart.

Every connection starts with a version handshake. The client sends its version; the daemon responds with its version:

# In the client
conn.send_bytes(encode_request(HandshakeRequest(version=__version__)))
resp = decode_response(conn.recv_bytes())
if not resp.ok or _needs_restart(resp):
    raise DaemonVersionError(resp)

On the first call (_daemon_ensured = False), a version mismatch triggers a restart: stop the old daemon, start a new one. On subsequent calls, a mismatch raises an error (the daemon was replaced mid-session, which is unusual and worth surfacing).

How should a local daemon handle config — restart, reload, or re-read fresh?

We have two levels of configuration:

Global settings (~/.cocoindex_code/global_settings.yml): embedding model, API keys, environment variables. These affect daemon-wide state: the ML model is loaded once at startup, environment variables are set once. When these change, the daemon must restart.

Project settings ($PROJECT/.cocoindex_code/settings.yml): include/exclude patterns, language overrides. These only affect per-operation behavior: which files to index, how to parse them.

Both files are created automatically by ccc init with sensible defaults. Users only edit them if they need to customize.

For global settings, the daemon records the file's mtime (as integer microseconds) at startup and reports it in the handshake response. The client compares it with the current mtime. Different? Restart.

For project settings, we do nothing special. Each indexing pass reads the settings file fresh from disk, so there is no staleness check or cache to invalidate.

The principle: minimize what the daemon holds as state. Global settings are unavoidable state (the loaded model), so we detect staleness. Project settings can be stateless, so we make them stateless.

Per-request vs. persistent connections — which is better for local IPC?

Per-request. Each request opens a fresh Unix socket connection, sends, receives, closes. Setup overhead is ~0.1ms, negligible for human-initiated CLI commands. Persistent connections require idle handler tasks, shutdown signaling across async/thread boundaries, and connection lifecycle management at every call site. We tried persistent connections first. We removed them. This was the single biggest simplification, and it came from a bug.

Why we tried persistent connections first

Our first design followed the obvious pattern: the client connects, handshakes, then sends multiple requests on the same connection. The daemon handler loops, reading and dispatching requests until the client disconnects:

# The original handler — persistent connection with recv loop
async def handle_connection(conn, registry, ...):
    loop = asyncio.get_event_loop()

    def _recv():
        while not shutdown_event.is_set():
            if conn.poll(0.5):          # wake every 0.5s to check shutdown
                return conn.recv_bytes()
        raise EOFError("shutdown")

    try:
        while not shutdown_event.is_set():
            data = await loop.run_in_executor(None, _recv)
            req = decode_request(data)
            # ... handshake, dispatch, respond ...
    except (EOFError, OSError, asyncio.CancelledError):
        pass
    finally:
        conn.close()

This handler had to deal with a lot:

A _recv helper that polls every 0.5 seconds, checking a threading.Event for shutdown
An executor thread per connection, blocked in conn.poll()
A while loop waiting for the next request that may never come
Cancellation handling for shutdown

Two problems emerged from this design:

Leaked connections. CLI commands didn't always close their connections. Each leaked connection meant a handler task stuck in _recv, waiting forever. We added __enter__/__exit__, with blocks, .close() calls, but it was whack-a-mole.

Shutdown was broken. Our test suite took 15 seconds per test teardown. The stop_daemon() function had a 5-step escalation (StopRequest, wait 5 seconds, SIGTERM, wait 2 seconds, SIGKILL) and always fell through to SIGKILL. Three bugs were stacked on top of each other:

asyncio.Event.is_set() called from executor threads, but asyncio.Event is not thread-safe. The _recv threads never saw the shutdown signal.
asyncio.run() cleanup hanging: it calls shutdown_default_executor(wait=True), which waits for the _recv threads that would never exit (because of bug #1).
os.kill(pid, 0) returning True for zombie processes, so the client kept waiting for a process that had already exited.

We fixed each bug individually. But the real insight came when we stepped back: all three bugs existed because of persistent connections. Persistent connections required idle handler tasks. Idle tasks required a shutdown signal. The signal had to cross async-to-thread boundaries. And every layer of that chain had a bug.

We fixed three stacked shutdown bugs with careful threading, event types, and process liveness checks. Then we removed persistent connections and all three bugs disappeared. The debugging was correct. The design was wrong.

The per-request model: read two messages, respond, close

Instead of fixing the plumbing, we removed the need for it. What if each request just opens a fresh connection?

connect → handshake → one request → response(s) → close

The handler becomes straight-line code:

# The current handler — per-request, no loop, no polling
async def handle_connection(conn, registry, ...):
    loop = asyncio.get_event_loop()
    try:
        # 1. Handshake
        data = await loop.run_in_executor(None, conn.recv_bytes)
        req = decode_request(data)
        # ... validate handshake, send response ...

        # 2. Single request
        data = await loop.run_in_executor(None, conn.recv_bytes)
        req = decode_request(data)
        result = await _dispatch(req, registry, ...)

        # 3. Send response(s) and done
        if isinstance(result, AsyncIterator):
            async for resp in result:
                conn.send_bytes(encode_response(resp))
        else:
            conn.send_bytes(encode_response(result))
    except (EOFError, OSError, asyncio.CancelledError):
        pass
    finally:
        conn.close()

Read two messages. Respond. Close. That's it.

What per-request connections eliminated

The diff wasn't just shorter. Entire concepts disappeared:

The _recv polling loop (0.5-second wakeups checking shutdown_event): gone
shutdown_event (threading.Event): gone entirely
The DaemonClient class, __enter__/__exit__, .close(): gone
with client: blocks in every CLI command: gone
Connection leak bugs: impossible by construction
All three shutdown bugs: gone (no idle handler tasks means nothing to signal)

The client module went from a stateful class to module-level functions:

# No class, no state, no close() to forget
def index(project_root, on_progress=None, on_waiting=None):
    conn = _connect_and_handshake()
    try:
        conn.send_bytes(encode_request(IndexRequest(project_root=project_root)))
        # ... read streaming responses ...
    finally:
        conn.close()

Compound operations — open a second connection, not a second protocol

What about operations that need multiple requests? ccc search --refresh needs to index first, then search. Simple. Two connections:

def search_cmd(project_root, refresh=False):
    if refresh:
        client.index(project_root)       # connection 1
    client.search(project_root, query)   # connection 2

MCP background indexing? Just call client.index() from a thread. It opens its own connection, with no shared state or connection-isolation bugs to reason about.

The overhead is negligible: Unix socket setup is ~0.1ms, the handshake is a tiny msgspec payload, and CLI commands are human-initiated. Streaming responses (like indexing progress) still work: the connection lives for one request's full response stream. It just doesn't linger afterwards.

Per-request connections can't leak by construction. The overhead of Unix socket setup is ~0.1ms, negligible for developer tools.

How do you cleanly shut down a Python daemon without falling through to SIGKILL?

Per-request connections solved the hardest shutdown problems (no idle tasks to signal). The remaining lesson: design shutdown around closing resources, not coordinating threads.

The accept loop runs in a background thread, blocking on listener.accept(). To stop it, we don't set a flag and hope the thread notices: we close the listener. accept() raises OSError, the thread exits immediately:

def _accept_loop():
    while True:
        try:
            conn = listener.accept()
            loop.call_soon_threadsafe(_spawn_handler, conn)
        except OSError:
            break  # listener was closed — exit

The same principle applies to the full shutdown sequence:

try:
    loop.run_forever()
finally:
    listener.close()                    # 1. stop accepting
    for task in tasks: task.cancel()    # 2. cancel handlers
    if tasks:
        loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
    registry.close_all()                # 3. release resources
    loop.close()
    # 4. remove socket and PID file ...
    os._exit(0)                         # 5. skip slow Python teardown

A few details worth noting:

PID file as the exit signal. The client needs to know when the daemon has finished cleaning up.

:::note os.kill(pid, 0) returns True for zombie processes: they're still in the process table. The fix: the daemon removes its PID file as the last cleanup step, and the client polls for its absence instead. :::

os._exit(0) for fast teardown.

:::note Python's normal exit takes seconds when torch is loaded (atexit handlers, GC finalizers, module teardown). os._exit(0) skips all of that, safe because we've already cleaned up explicitly. :::

(One caveat: os._exit(0) terminates the entire process, so we guard it with threading.current_thread() is threading.main_thread() for tests that run the daemon in a thread.)

Event loop on the main thread. The accept loop runs in a background thread, but the asyncio event loop stays on the main thread. This is required for loop.add_signal_handler() to work: SIGTERM and SIGINT call loop.stop(), which causes run_forever() to return into the finally block above.

The Result

                    per-request connections
  CLI / MCP  ──────────────────────────────►  Daemon
  (thin)        connect → handshake →         (persistent)
                request → response →          - event loop (main thread)
                close                         - accept loop (background thread)
                                              - project registry
                                              - ML model (loaded once)

Before and after:

	Before	After
Connection handler	~70 lines, recv loop, polling, shutdown event	~40 lines, straight-line read-respond-close
Client	`DaemonClient` class, `__enter__`/`__exit__`, `.close()`	Module-level functions, no state
Shutdown concepts	`threading.Event`, executor polling, 5-step escalation	`listener.close()`, `loop.stop()`, `os._exit(0)`
Daemon stop time	~15 seconds (fell through to SIGKILL)	<1 second
Impact for coding-agent users	Per-session ML model reload on every `ccc search`	Model loaded once; search returns in <1 second

Patterns for Local Daemon Architecture

If you're building a local daemon for developer tools, here's what we'd suggest:

Make the daemon invisible. Auto-start on first use. Auto-restart on version mismatch. Auto-reload config where possible. The user's mental model should be "I run CLI commands". The daemon is an implementation detail.

Prefer stateless IPC. Per-request connections can't leak by construction. The overhead of Unix socket setup (~0.1ms) is negligible for developer tools. The handshake on every connection catches staleness faster.

Categorize your state. What must live in the daemon (loaded ML model, open databases) vs what can be read fresh (config files, file lists)? Only restart for the former. The less state the daemon holds, the fewer staleness bugs you'll have.

Design shutdown around resource closure, not signaling. Close the listener to interrupt accept(). Remove the PID file to signal completion. Don't poll threads to check if they've noticed an event. Close the thing they're waiting on.

When debugging gets complex, question the design. We fixed three stacked shutdown bugs with careful threading, event types, and process liveness checks. Then we removed persistent connections and all three bugs disappeared. The debugging was correct. The design was wrong.

Support the Project

If you find this article helpful, please check out cocoindex-code for the implementation and star the project if you like it!

CocoIndex Changelog 0.3.27 - 0.3.34

George He — Tue, 10 Mar 2026 00:00:00 GMT

Since the last release, CocoIndex shipped 8 releases (0.3.27–0.3.34), and this cycle was focused on one clear goal: making CocoIndex the context engine that adapts to the knowledge ecosystems (vector stores, graph databases, or analytics backends your agents need) while getting even smarter about keeping itself healthy underneath.

We are super excited to build with the amazing infrastructure projects together on the path of agents going to production in 2026.

Agents don't just need fresh data: they need it wherever it lives. Different teams pick different databases by their use cases. Some want LanceDB for multi-modal capability. Some want ChromaDB for local-first development. Graph-heavy workloads need FalkorDB or Ladybug. Analytics & Search workloads want Apache Doris. And… a lot more great projects in the ecosystems coming soon. This cycle, we made sure CocoIndex speaks all of them, so you pick the best target for your use case, and CocoIndex handles the rest.

CocoIndex helps you ace the data for agents without heavy data engineering work. Continuous fresh, structured and programmable context for AI, from PDFs, Codebase, Emails, Screenshots, Meeting Notes, ... Always up-to-date, at any scale, ultra-performant, with a smart incremental engine. ⭐ Star the repo

Here is what we shipped since the last release.

0.3.27–0.3.34 was about target reach and engine hardening: five new target connectors, filesystem-level change detection, Python 3.14 free-threading, and smarter pipeline lifecycle management.

Core capability

CocoIndex's core engine got sharper this cycle. The runtime now automatically cleans up stale derived metadata and data when you rename or remove a source: no manual intervention, no orphaned rows. Tighter validation catches bad table and column names before they hit your database. A new print_stats option surfaces pipeline statistics directly in the update() API. And for the first time, CocoIndex supports Python 3.14 free-threaded mode, running without the GIL for true parallelism in Python-heavy pipelines.

Automatic stale source cleanup

When CocoIndex detects that a source has been renamed or removed, it now automatically streams through tracking metadata, deletes corresponding target rows, and removes stale state records, all during flow setup with controlled parallelism to avoid overloading your database (#1468).

Python 3.14 free-threaded support

CocoIndex now runs natively in Python 3.14's free-threaded mode (no GIL). CI builds ship free-threaded wheels on manylinux, so you can leverage true multi-threaded Python without any config changes (#1357, #1609, #1638).

Tighter target validation

Table names and column names are now validated before target setup, catching naming issues early instead of surfacing cryptic database errors downstream (#1714).

Execution plan initialization fix

Fixed a race condition where the execution plan could be used before target setup completed, ensuring correct pipeline initialization order (#1715).

Pipeline stats in update() API

Added a print_stats option to the update() API, giving you runtime statistics directly at the call site, useful for monitoring, benchmarking, and debugging pipeline performance (#1525).

Custom database schema for internal tables

CocoIndex's internal metadata and tracking tables now support custom Postgres schemas via explicit schema.table qualification. This means cleaner isolation in shared databases: no more search_path side effects (#1459).

Slow HTTP request warnings

The runtime now warns when HTTP requests (e.g., to LLM providers or external APIs) are running slow, making it easier to spot bottlenecks in your pipeline (#1617).

Enhanced tracing

Improved tracing coverage across the engine for better observability and debugging (#1618).

ColPali/ColQwen organization prefix fix

Fixed a crash when using ColPali or ColQwen models with organization-prefixed model names (e.g., org/model-name) (#1624).

Together, these changes make the core engine more self-healing, more observable, and ready for the next generation of Python runtimes.

Integrations

This cycle is a massive expansion of where CocoIndex can send data: five brand-new target connectors, filesystem-level change detection for local files, and deeper LanceDB indexing support.

Sources

Local file change notifications

The LocalFile source now supports native filesystem change notifications via the notify crate. Enable the new watch_changes option and CocoIndex will detect file changes through OS-level events instead of polling: faster reaction times, less CPU overhead, and your include/exclude patterns are respected automatically (#1669).

Targets

Apache Doris

Full-featured native support for Apache Doris 4.0 (and VeloDB Cloud) as a vector database target. Includes HNSW and IVF vector indexes, inverted full-text search indexes with Unicode/English/Chinese parsers, Stream Load ingestion with batching and exponential backoff, and upsert/delete support. A complete text embedding example ships alongside (#1475, #1642).

ChromaDB

Initial ChromaDB integration with support for PersistentClient, CloudClient (with SPANN+SPFresh), and HttpClient. Includes HNSW configuration options, explicit document_field specification, and early validation for vector fields (#1548, #1663).

FalkorDB

New property graph target powered by FalkorDB, a high-performance graph database built on Redis. Full support for nodes, relationships, vector and FTS indexes, and batch operations. Ships with a complete docs-to-knowledge-graph example using LLM extraction (#1481).

Ladybug (successor to Kuzu)

Kuzu was archived in October 2025. Ladybug is the active community fork and a drop-in replacement with an identical Cypher API. CocoIndex now targets Ladybug natively, with backward-compatible Kuzu* type aliases so existing pipelines migrate seamlessly (#1487).

LanceDB

Added VectorIndexMethod support for HNSW and IVF Flat indexes in LanceDB, giving you fine-grained control over index configuration (#1409).

Pinecone

Pinecone is now a built-in target connector: bring your managed Pinecone indexes directly into CocoIndex pipelines with a text embedding example included out of the box (#1556).

These integrations bring CocoIndex's supported target ecosystem to 10+ databases, covering vector stores, graph databases, relational systems, and analytics engines.

Build with CocoIndex

SEC EDGAR financial analytics with CocoIndex + Apache Doris

A full multi-source ETL pipeline demonstrating how to build a financial analytics system from SEC EDGAR data using CocoIndex and Apache Doris.

The example ingests three data sources: TXT filings (10-K/10-Q risk factors), JSON company facts from the SEC XBRL API, and PDF proxy statements. It then indexes them into Apache Doris with hybrid search (vector + keyword via Reciprocal Rank Fusion), topic filtering with JSON arrays, and portfolio-level search with per-company aggregation.

It also demonstrates PII scrubbing before indexing and incremental updates with caching, a production-ready pattern for financial data pipelines.

Check out the tutorial for more details.

Slides-to-Speech: Turn slide decks into searchable, narrated knowledge

Static slide decks sitting in Google Drive are one of the most underutilized knowledge sources in any organization. We published a full walkthrough showing how to build a continuously updating slides-to-speech pipeline using CocoIndex and LanceDB.

The pipeline watches a Drive folder for new or modified PDF decks, converts each slide to an image with PyMuPDF, extracts structured speaker notes using BAML + Gemini Vision, synthesizes narration with Piper-TTS, generates embeddings, and indexes everything into LanceDB: images, transcripts, audio bytes, and metadata all in one place.

What's cool about this example:

Incremental by design: Only changed slides get reprocessed. Update one slide in a 200-page deck? CocoIndex touches only that slide, regenerating its audio, transcript, and embedding while leaving the rest untouched.
True multimodal search: Semantic queries return the top matching slides with images, speaker notes, and synced audio, not just filenames.
Local-first TTS: Neural text-to-speech runs locally via Piper, so there are no API costs for audio generation.
Living knowledge base: What used to be a static PDF becomes a searchable, listenable, always-fresh dataset ready for agents, accessibility tools, or video generation.

Check out the blog post for more details.

Thanks to the community 🤗🎉

Welcome new contributors to the CocoIndex community! We are so excited to have you!

@tomz-alt

Thanks @tomz-alt for adding the Apache Doris 4.0 vector database connector (a full-featured target with vector indexes, full-text search, and Stream Load ingestion) and for building the SEC EDGAR financial analytics example demonstrating multi-source ETL with hybrid search, plus improving the Doris text embedding example and cleaning up the SEC EDGAR tutorial for an end-to-end demo experience.

@prrao87

Thanks @prrao87 for adding Ladybug (the Kuzu successor) as a target, ensuring CocoIndex users have a smooth migration path to the actively maintained graph database fork.

@galshubeli

Thanks @galshubeli for adding the FalkorDB target, a complete property graph integration with nodes, relationships, vector/FTS indexes, and a docs-to-knowledge-graph example.

@Geoff-Robin

Thanks @Geoff-Robin for adding custom database schema support for CocoIndex's internal tables, enabling cleaner isolation in shared Postgres databases with explicit schema.table qualification.

@wuxiaobai24

Thanks @wuxiaobai24 for adding Python 3.14 free-threaded support, configuring PyO3 for GIL-free operation and bringing CocoIndex to the cutting edge of Python parallelism.

@ceshine

Thanks @ceshine for fixing the ColPali/ColQwen crash when using models with organization prefixes, unblocking multimodal embedding workflows.

@LED-0102

Thanks @LED-0102 for adding filesystem change notifications to the LocalFile source using the notify crate, enabling OS-level file watching for faster live updates.

@prithvi-moonshot

Thanks @prithvi-moonshot for adding Pinecone as a target connector with a text embedding example, bringing managed vector search into the CocoIndex ecosystem.

@Haleshot

Thanks @Haleshot for building the initial ChromaDB target integration with support for PersistentClient, CloudClient, and HttpClient, fixing ChromaDB collection creation on first run, adding ChromaDB documentation, adding llms.txt to docs, adding short flag aliases for common CLI options, clarifying .env usage across examples, and migrating CI from pre-commit to prek.

@thisisharsh7

Thanks @thisisharsh7 for adding automatic cleanup of stale source data, ensuring renamed or removed sources don't leave orphaned tracking metadata or target rows.

@majiayu000

Thanks @majiayu000 for adding VectorIndexMethod support for HNSW and IVF Flat indexes in LanceDB, expanding vector index configuration options.

Summary

CocoIndex continues evolving into a universal, resilient data engine for the modern AI stack.

This cycle focuses on target ecosystem breadth, engine self-healing, and Python 3.14 readiness, ensuring your pipelines can write to any database your agents need, clean up after themselves, and run on the latest Python runtime without compromise.

If you like this project, please support us with a star ⭐ at https://github.com/cocoindex-io/cocoindex !

🔥🔥 We are cooking something big! v1 is coming soon. Stay tuned! If you read all the way here, you are a true fan!

CocoIndex joins the GitHub Secure Open Source Fund

Linghua Jin — Tue, 17 Feb 2026 00:00:00 GMT

At CocoIndex, we've always believed that the strength of open source lies in its community and in building transparent, reliable infrastructure together. CocoIndex is a data transformation framework that sits between raw data and the AI systems that depend on it, and that position makes security a core requirement.

When developers trust CocoIndex to process sensitive documents, codebases, and enterprise data, we owe them the highest standards. We are honored to be selected to join the GitHub Secure Open Source Fund, to formalize our commitment and accelerate our security practices.

What is GitHub's Secure Open Source Fund?

When the Log4j "Log4Shell" vulnerability hit in December 2021, it showed that a bug in one small, underfunded open-source project can ripple through the entire global tech stack. Most apps today rely on countless dependencies, many maintained by unpaid volunteers, and a weakness in any one of them reaches everyone downstream.

Recognizing this, GitHub launched the Secure Open Source (SOS) Fund in late 2024 to strengthen the foundation of open-source security. The program pairs direct funding with a focused three-week security sprint that combines expert guidance, better tooling, and a community of security-conscious maintainers.

In its first two cohorts, 125 maintainers from 71 projects participated, collectively fixing over 1,100 vulnerabilities, disclosing 50+ new CVEs, and resolving nearly 270 secret exposures.

Participants also exchanged knowledge, leveraged AI-assisted tools like GitHub Copilot for security tasks, and developed incident response plans that many have since published for others to reuse.

What is CocoIndex?

Dynamic context engineering for AI.

As AI systems evolve from chatbots to autonomous agents, one hard problem keeps coming back: keeping context accurate and up to date as the underlying data changes.

Enterprises sit on massive, dynamic datasets (documents, codebases, emails, APIs), yet most data infra is still batch-oriented and blind to change. Every shift in data or logic forces engineers to rebuild from scratch, breaking the feedback loop between real-time world state and AI decision-making. Dynamic context engineering addresses this: maintaining an AI system's "mental model" of the world, continuously, reliably, and incrementally. CocoIndex is our take on making this simple, with a declarative Python-native data transformation engine designed for AI workloads, and you will never need to handle change.

Think of it as React for data processing.

The Cohort & Thanks to our batchmates

We were in the program alongside Pandas, Apache Airflow, Fabric.js, PyPI, and others. We heard from teams dealing with similar challenges: how they handle dependency updates, structure security reviews, and think about trust boundaries.

Open source can be isolating. You're deep in your own codebase, solving your own problems. Hearing how other maintainers answer the same security questions was one of the most useful parts of the program.

Strengthening CocoIndex: Security Improvements Implemented

During the program, the CocoIndex team ran a focused security sprint to harden both our codebase and the way we build and ship it. With guidance from GitHub Security Lab and community experts, we concentrated on four areas that matter most for a framework sitting directly in AI data paths.

Threat modeling & AI-specific risk assessment

We systematically threat-modeled the CocoIndex framework, examining how an attacker might exploit a data transformation pipeline that feeds directly into AI systems. We mapped out attack vectors including adversarial document content that could poison embeddings, prompt injection through source data that could manipulate LLM-based extraction, and compromised dependencies that could silently corrupt downstream indexes. Resources like the Adversarial AI Reading List and Embrace The Red were particularly useful in shaping this analysis. By identifying these failure points upfront, we developed mitigation strategies before they could be exploited in production pipelines.

Secure coding & automated checks

We wired GitHub’s security tooling directly into CocoIndex’s development loop. CodeQL now runs on every pull request, scanning both our Rust core and Python SDK for vulnerability patterns. For a system where data flows from sources through transformations to targets, CodeQL’s data flow analysis maps naturally onto our architecture, tracing untrusted input from ingestion all the way to exported artifacts. We also enabled secret scanning to keep credentials and API keys out of the repository, and we started using GitHub Copilot as a guardrail during development to surface potential issues before code is even committed. Together, these checks act as a continuous security layer in our CI pipeline, catching problems early instead of after a release.

Dependency & supply chain hardening

CocoIndex integrates with embedding models, vector databases, document parsers, and cloud storage backends, each one pulling in its own dependency chain. We audited our dependencies across both the Rust core and Python SDK, going beyond direct dependencies to examine the transitive tree underneath. We tightened our dependency review process with more systematic tracking of what we depend on and why, and adopted the OpenSSF Scorecard to continuously benchmark our project's security posture across branch protection, dependency updates, CI/CD practices, and more. Following OpenSSF standards, we're working toward a comprehensive software bill of materials (SBOM) for each release, giving users visibility into exactly what ships with CocoIndex.

Policy, documentation & vulnerability response

We formalized our SECURITY.md with clear instructions for responsible disclosure and defined response commitments. Vulnerabilities should be reported to security@cocoindex.io, not filed as public issues. We also established an internal process for triaging and patching reported vulnerabilities: who gets involved, how we communicate, and how quickly we ship fixes. Every new connector or transformation function now gets reviewed through the lens of: what's the worst-case input, and what happens if a dependency in this path is compromised?

CodeQL is already catching issues that manual review missed, and the disclosure process gives users a clear channel when something does slip through. Because CocoIndex sits in the data path of the AI systems built on it, these improvements carry downstream to every application that depends on those pipelines.

Thank You to Our Contributors

None of this would have been possible without the people who build CocoIndex with us every day. Everyone who has submitted a PR, reported a bug, reviewed code, or helped another user in Discord: you're the reason this project exists and the reason it was recognized.

We especially want to thank those doing the work that doesn't show up in commit logs: writing documentation, answering questions, and building examples that help new users get started.

Looking Forward

The sprint is over, but the checks it added run on every pull request, and the review habits stay with us for every new connector and integration.

The GitHub Secure Open Source Fund gave us dedicated time, expert guidance, and a community of peers to learn from. The improvements we've made will benefit every developer who builds an AI pipeline with CocoIndex.

We're grateful to GitHub and the entire cohort for this experience, and we plan to keep sharing what we've learned with the broader AI infrastructure community.

CocoIndex is open source on GitHub. Questions or feedback: join our Discord.

SEC EDGAR financial analytics with Apache Doris

Tom Zhang, Linghua Jin — Mon, 09 Feb 2026 00:00:00 GMT

SEC filings are the backbone of financial transparency. Every public company in the United States files 10-Ks, 10-Qs, proxy statements, and exhibits with the SEC: thousands of documents each quarter across text, structured data, and PDF formats.

Searching across all of these effectively requires more than keyword matching. You need semantic understanding, structured metadata filtering, and the ability to combine multiple document formats into a single searchable index.

In this post, we walk through the SEC EDGAR Financial Analytics example: a CocoIndex v1 pipeline that ingests two source formats (TXT 10-K filings and XBRL company-facts JSON), scrubs PII, extracts topic tags, generates embeddings, and loads everything into Apache Doris for hybrid search combining vector similarity with full-text matching using Reciprocal Rank Fusion (RRF). Both formats fan into one shared path, so adding another format like PDF is the same handful of lines.

The project is open-sourced in the CocoIndex v1 repo.

Why CocoIndex?

CocoIndex is a Rust-based, open-source data transformation framework for AI workloads, combining high performance with flexibility. It supports incremental processing, data lineage, and customizable logic, allowing teams to build efficient and intelligent data pipelines. CocoIndex makes transformation pipelines modular, transparent, and easy to maintain.

Why Apache Doris?

Apache Doris is an open-source, Apache-licensed real-time MPP data warehouse built for lightning-fast analytics. It supports high-concurrency workloads with real-time data ingestion and querying, handles both structured and semi-structured (VARIANT) data, includes full-text inverted indexes, and native vector storage with approximate nearest neighbor (ANN) search.

Together, CocoIndex and Apache Doris form a powerful agentic data infrastructure stack. CocoIndex transforms and indexes unstructured data through modular, lineage-tracked pipelines with built-in incremental processing, while Apache Doris delivers real-time analytics at scale with sub-second ingestion latency, sub-100ms query response, and 10k QPS concurrency: capabilities purpose-built for AI agents that need to make fast, data-driven decisions. The combination bridges the gap from raw, unstructured data to ultra-performant real-time search and analytics, while CocoIndex's data lineage and transparency ensure the auditability and compliance that regulated industries demand.

Architecture overview

The pipeline follows a multi-format, shared-path pattern:

┌─────────────────────────────────────────────────────────────────────────┐
│                     CocoIndex Multi-Format Pipeline                       │
│                                                                           │
│        ┌──────────────────┐          ┌──────────────────┐                 │
│        │ TXT 10-K Filings │          │ XBRL JSON Facts  │                 │
│        │  (risk factors)  │          │ (company facts)  │                 │
│        └────────┬─────────┘          └────────┬─────────┘                 │
│                 │                             │                           │
│                 ▼                             ▼                           │
│        ┌────────────────────────────────────────────────┐                 │
│        │            Shared _index_text path              │                 │
│        │   Scrub PII → Chunk → Embed → Tag → declare_row │                 │
│        └────────────────────────────────────────────────┘                 │
│                             │                                             │
│                             ▼                                             │
│                      Apache Doris                                         │
│         (vector ANN index + inverted full-text index)                     │
└─────────────────────────────────────────────────────────────────────────┘

Two file formats feed into a single table. Each source extracts its own metadata, then converges on one shared function that scrubs PII, splits, embeds, and tags. The output lands in a single Doris table with both a vector index and a full-text index.

Define the pipeline

CocoIndex v1 is declarative and Python-native: you write target_state = transformation(source_state) in plain async Python, and the Rust engine handles incremental processing. The whole example lives in one main.py with no DSL and no separate flow-definition file.

The entry point is an app_main function wrapped in a coco.App:

import cocoindex as coco
from cocoindex.connectors import doris, localfs
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

app = coco.App(coco.AppConfig(name="SECFilingAnalytics"), app_main)

The row schema

Each chunk becomes one typed row. In v1 you declare the output as a plain dataclass; the embedding field is annotated with the embedder context so the engine knows its vector shape:

@dataclass
class FilingChunk:
    chunk_id: str            # primary key: uuid5 of (filename, chunk offsets)
    source_type: str         # "filing" | "facts"
    doc_filename: str
    cik: str
    filing_date: str
    form_type: str
    text: str
    topics: list[str]
    embedding: Annotated[NDArray, EMBEDDER]

The dual-index target

app_main mounts one Doris table target that carries both a vector index and a full-text inverted index, then walks each source directory and mounts a per-file processor:

@coco.fn
async def app_main() -> None:
    table = await doris.mount_table_target(
        DORIS_DB,
        TABLE,
        await doris.TableSchema.from_class(FilingChunk, primary_key=["chunk_id"]),
        vector_indexes=[
            doris.VectorIndexDef(field_name="embedding", metric_type="l2_distance")
        ],
        inverted_indexes=[doris.InvertedIndexDef(field_name="text", parser="unicode")],
    )

    txt = localfs.walk_dir(
        localfs.FilePath(path="./data/filings"),
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.txt"]),
    )
    await coco.mount_each(process_filing, txt.items(), table)

    facts = localfs.walk_dir(
        localfs.FilePath(path="./data/company_facts"),
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.json"]),
    )
    await coco.mount_each(process_facts, facts.items(), table)

A single mount_table_target declares two indexes on one table:

Vector (ANN) index on the embedding field for l2_distance semantic similarity search
Inverted index on the text field for keyword matching with MATCH_ANY

This dual-index setup is what makes hybrid search possible without maintaining separate stores.

Each source uses localfs.walk_dir with pattern filtering, and coco.mount_each runs the per-file processor once per matched file.

Process text filings

process_filing parses the structured filename convention {CIK}_{date}_{form}.txt to pull out the company identifier (CIK), filing date, and form type (10-K, 10-Q, etc.), metadata needed for filtering and aggregation at query time. It then hands the text to the shared indexing path:

@coco.fn(memo=True)
async def process_filing(
    file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
    """10-K text filing: metadata from the {CIK}_{date}_{form}.txt filename."""
    name = file.file_path.path.name
    parts = name.rsplit(".", 1)[0].split("_")
    cik = parts[0] if parts else "unknown"
    filing_date = parts[1] if len(parts) > 1 else "2024-01-01"
    form_type = parts[2] if len(parts) > 2 else "10-K"
    await _index_text(
        await file.read_text(), "filing", name, cik, filing_date, form_type, table
    )

memo=True memoizes the function: re-running skips files whose content and code are unchanged, so only new or edited filings are reprocessed.

We intentionally parse metadata with a deterministic string split rather than an LLM. When you know the file format upfront, a regex or string parser is faster, cheaper, and more reliable than calling an LLM on every document. CocoIndex still supports native LLM extraction for formats you don't know in advance, and tools like CocoInsight help validate the output either way.

The shared indexing path

Both sources converge on one _index_text helper that does the real work: scrub PII, split into overlapping chunks, embed, tag, and declare one row per chunk.

async def _index_text(text, source_type, filename, cik, filing_date, form_type, table):
    embedder = coco.use_context(EMBEDDER)
    for chunk in _splitter.split(
        _scrub_pii(text), chunk_size=1000, chunk_overlap=200, language="markdown"
    ):
        table.declare_row(row=FilingChunk(
            chunk_id=_chunk_id(filename, chunk.start.char_offset, chunk.end.char_offset),
            source_type=source_type,
            doc_filename=filename,
            cik=cik,
            filing_date=filing_date,
            form_type=form_type,
            text=chunk.text,
            topics=_extract_topics(chunk.text),
            embedding=await embedder.embed(chunk.text),
        ))

1. Scrub PII before chunking. PII (Personally Identifiable Information) includes things like Social Security numbers, phone numbers, and email addresses. SEC filings sometimes contain these inadvertently (there are even SEC rules about this: Reg S-T Rule 83). Scrubbing happens on the full document before chunking, so a phone number or SSN split across a chunk boundary can't slip through into the search index.

2. Split into chunks. Chunk size is a tradeoff: too large and you lose search resolution (a 10,000-character chunk matches broadly but vaguely); too small and each chunk lacks enough context to be meaningful on its own. We use 1,000 characters with 200-character overlap as a practical middle ground for SEC filings. RecursiveSplitter splits text by respecting document structure: it tries to break at markdown headings, then paragraphs, then sentences, before falling back to character boundaries. A flat splitter would just cut every N characters regardless of where a sentence or section ends, often splitting mid-thought.

3. Embed, tag, and declare a row. For each chunk we generate an embedding vector (for semantic search) and extract topic tags (for structured filtering), then declare_row assembles one FilingChunk combining the chunk's text, embedding, and topics with the document-level metadata (CIK, filing date, form type). chunk_id is a stable uuid5 of the filename and chunk offsets, so re-running reconciles rows in place instead of duplicating.

Topics are extracted as string arrays, enabling Doris array filtering:

def _extract_topics(text: str) -> list[str]:
    low = text.lower()
    return [t for t, kws in _TOPIC_KEYWORDS.items() if any(k in low for k in kws)]

_TOPIC_KEYWORDS maps labels like RISK:CYBER, RISK:CLIMATE, TOPIC:AI, and TOPIC:FINANCIAL to keyword lists. This produces arrays like ["RISK:CYBER", "TOPIC:AI"], stored as a JSON column you can filter in Doris with json_contains(topics, '"RISK:CYBER"').

Process XBRL company facts

The JSON facts path is the same shape: process_facts reads the file, renders the XBRL metrics to searchable natural-language text, then calls the same _index_text.

@coco.fn(memo=True)
async def process_facts(
    file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
    """XBRL company-facts JSON: render to text, then index."""
    name = file.file_path.path.name
    content = await file.read_text()
    cik = name.replace("CIK", "").replace(".json", "")
    filing_date = (
        json.loads(content).get("filingDate", "2024-01-01") if content else "2024-01-01"
    )
    await _index_text(
        _company_facts_to_text(content), "facts", name, cik, filing_date, "FACTS", table
    )

_company_facts_to_text converts structured financial metrics (revenue, net income, R&D expense) into natural-language text so they're discoverable via semantic search. Because both sources end at the same _index_text and declare_row into the same table, adding a third format such as PDF is just another process_* that converts to text and calls _index_text (see Manuals to Structured Data for the docling PDF path).

Providing shared resources

The Doris connection and the embedder are provided once in the app lifespan and retrieved anywhere with coco.use_context:

DORIS_DB = coco.ContextKey[doris.ManagedConnection]("sec_doris")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    builder.provide(DORIS_DB, doris.connect(doris.DorisConnectionConfig(...)))
    builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
    yield

Hybrid search with RRF

search.py combines semantic and lexical ranking using Reciprocal Rank Fusion, all in one Doris SQL query over the MySQL protocol:

def search(query: str, source_type: str | None = None, limit: int = 5) -> None:
    vec = asyncio.run(_embedder.embed(query))
    vstr = "[" + ",".join(f"{x:.6f}" for x in vec) + "]"
    keywords = " ".join(re.findall(r"[A-Za-z]{3,}", query.lower()))
    where = f"source_type = '{source_type}'" if source_type else "1 = 1"

    sql = f"""
    WITH semantic AS (
        SELECT chunk_id, doc_filename, source_type, topics, text,
               ROW_NUMBER() OVER (ORDER BY l2_distance(embedding, {vstr})) AS rk
        FROM filing_chunks WHERE {where}
    ),
    lexical AS (
        SELECT chunk_id,
               ROW_NUMBER() OVER (ORDER BY CASE WHEN text MATCH_ANY '{keywords}'
                                  THEN 0 ELSE 1 END) AS rk
        FROM filing_chunks WHERE {where}
    )
    SELECT s.doc_filename, s.source_type, s.topics, s.text,
           1.0/(60 + s.rk) + 1.0/(60 + l.rk) AS rrf
    FROM semantic s JOIN lexical l USING (chunk_id)
    ORDER BY rrf DESC LIMIT {limit}
    """

The RRF formula 1/(k + rank) with k=60 is a standard approach for combining rankings from different signals without needing to normalize scores. A chunk that ranks #1 in both semantic and lexical search gets 1/61 + 1/61 = 0.0328. A chunk that's #1 semantically but #100 lexically gets 1/61 + 1/160 = 0.0226. The formula naturally balances both signals.

Pass --source filing or --source facts to restrict the search to one document format.

Running the example

Prerequisites

Python 3.11+
Docker and Docker Compose (for Apache Doris 4.0+, which is required for vector-index support)

Quick start

cd examples/sec_edgar_analytics

# 1. Start Doris (FE + BE)
docker compose up -d fe be
# Wait ~90 seconds for Doris to initialize

# 2. Fetch sample data, configure, and install
python download.py          # writes data/filings/*.txt + data/company_facts/*.json
cp .env.example .env         # Doris host/ports
pip install -e .

# 3. Build the index
cocoindex update main

cocoindex update main loads 4 chunks (2 filings + 2 company-facts) into Doris, creating both the vector ANN index and the inverted full-text index. Topics come out as you'd expect: Apple tagged RISK:CYBER, RISK:CLIMATE, RISK:SUPPLY, RISK:REGULATORY, TOPIC:AI; Microsoft RISK:CYBER, RISK:REGULATORY, TOPIC:AI, TOPIC:CLOUD.

Hybrid search

python search.py "cloud computing and AI risk"

Hybrid search: "cloud computing and AI risk"

[0.0328] 0000789019_2025-10-15_10-K.txt (filing)  topics=["RISK:CYBER","RISK:REGULATORY","TOPIC:AI","TOPIC:CLOUD"]
    MICROSOFT CORPORATION FORM 10-K ANNUAL REPORT. Our cloud computing business (Azure) faces intense competition...

[0.0320] 0000320193_2025-11-01_10-K.txt (filing)  topics=["RISK:CYBER","RISK:CLIMATE","RISK:SUPPLY","RISK:REGULATORY","TOPIC:AI"]
    APPLE INC. FORM 10-K ANNUAL REPORT. Cybersecurity threats are a material risk...

On the sample data that ranks Microsoft's cloud-and-AI filing first (it carries both TOPIC:CLOUD and TOPIC:AI), Apple's second, and the company-facts rows below. Restrict to one format with python search.py "cloud revenue" --source facts.

Direct Doris queries

You can also query the index directly via MySQL protocol:

mysql -h localhost -P 9030 -u root

Array field filtering

-- Find all chunks tagged with cybersecurity risk
SELECT doc_filename, text, topics
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"');

-- Chunks matching any of multiple topics
SELECT doc_filename, text
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"')
   OR json_contains(topics, '"RISK:CLIMATE"');

Portfolio aggregation

-- Top 3 relevant chunks per company
WITH ranked AS (
    SELECT cik, doc_filename, text,
           l2_distance(embedding, [...]) AS score,
           ROW_NUMBER() OVER (PARTITION BY cik ORDER BY score ASC) AS rank
    FROM sec_analytics.filing_chunks
    WHERE cik IN ('0000320193', '0000789019')
)
SELECT * FROM ranked WHERE rank <= 3;

Temporal trends

-- Cybersecurity mentions by filing year
SELECT LEFT(filing_date, 4) AS filing_year,
       COUNT(DISTINCT cik) AS num_companies,
       COUNT(*) AS total_mentions
FROM sec_analytics.filing_chunks
WHERE text MATCH_ANY 'cybersecurity risk'
GROUP BY filing_year
ORDER BY filing_year DESC;

What's next

The techniques in this example (multi-format ingestion, array field filtering, hybrid search, PII scrubbing) generalize beyond financial documents. The same patterns apply to healthcare records, legal documents, or any domain where you need to search across heterogeneous document formats with structured metadata filtering.

Check out the full source code in the CocoIndex v1 repo, including the sample-data generator and the hybrid-search CLI.

Build a Self-Updating Wiki for Your Codebases with an LLM

Linghua Jin — Thu, 05 Feb 2026 00:00:00 GMT

Documentation tends to drift out of sync with the code it describes. In this tutorial, we build a pipeline that generates a wiki page for each project in your codebase and keeps it current with incremental processing.

The full source code is available at CocoIndex Examples - multi_codebase_summarization.

The source is on GitHub.

Here's an example of the generated documentation:

The Problem

Documentation drifts. A module gets refactored, and the wiki describing it is now wrong. It usually stays wrong until someone new asks whether the docs can still be trusted.

The alternative is to derive the documentation from the code itself: a pipeline that reads your source, extracts structure from it, and produces documentation that updates when the code changes.

What We'll Build

:::note This project uses CocoIndex v1. :::

A pipeline that:

Scans subdirectories, treating each as a separate project
Extracts structured information from each Python file using an LLM (classes, functions, relationships)
Aggregates file-level data into project-level summaries
Generates Markdown documentation with Mermaid diagrams

flowchart LR
    subgraph Input
        A[projects/] --> B[project_1/]
        A --> C[project_2/]
        A --> D[project_N/]
    end

    subgraph "Process (per project)"
        B --> E["extract_file_info\n(LLM + Pydantic)"]
        E --> F["aggregate_project_info"]
        F --> G["generate_markdown"]
    end

    subgraph Output
        G --> H["project_1.md"]
    end

    classDef extract fill:#FBE7DA,stroke:#BE5133,stroke-width:1.5px,color:#7A2E1A;
    classDef aggregate fill:#FCF3D8,stroke:#8F3B24,stroke-width:1.5px,color:#5A2417;
    classDef generate fill:#DDF5DE,stroke:#16A534,stroke-width:1.5px,color:#11472A;
    class E extract;
    class F aggregate;
    class G generate;

The model behind this is a single relationship:

target_state = transformation(source_state)

You declare the transformation. CocoIndex determines when to run it and which inputs need reprocessing.

The reality of modern codebases

Codebases change constantly. Teams merge many pull requests a day, and AI coding agents add and modify files continuously. A documentation pipeline has to keep up with that rate of change.

Batch regeneration handles this poorly. A nightly job is already stale by morning. An hourly job spends most of its time rebuilding files that did not change. The cost of a full rebuild scales with the size of the codebase, not with the size of the change.

Why not just write a script?

A script that loops through files, calls an LLM, and writes Markdown works for a one-time run. Keeping it correct over time raises several problems:

File changes: after editing one file, re-running the whole pipeline is slow and expensive.
Tracking state: to process only what changed, you have to track timestamps, checksums, or diffs yourself.
LLM costs: re-analyzing unchanged files wastes API calls, which adds up at scale.
Logic changes: changing the extraction prompt invalidates results for that transformation across every file.

These are caching, invalidation, and orchestration concerns, separate from the extraction logic you actually want to write.

The declarative difference

CocoIndex separates the transformation logic from the change tracking. You provide:

Functions that extract information from a file, aggregate it, and format it.
Dependencies between those functions, which CocoIndex records.

CocoIndex then handles incremental processing. When a source file is edited, or the processing logic changes (a different model or an updated prompt), only the affected outputs are recomputed, and the output stays consistent with the source.

Prerequisites

Install dependencies:

pip install --pre 'cocoindex>=1.0.0a6' instructor litellm pydantic

Set up environment:

export GEMINI_API_KEY="your-api-key"
export LLM_MODEL="gemini/gemini-2.5-flash"
echo "COCOINDEX_DB=./cocoindex.db" > .env

Create a projects/ directory with subdirectories for each Python project:

```bash
projects/
├── my_project_1/
│   ├── main.py
│   └── utils.py
├── my_project_2/
│   └── app.py
└── ...
```

Define the app

Define a CocoIndex App, the top-level runnable unit in CocoIndex.

The snippets below focus on the pipeline logic; the full import list (localfs, instructor, the Pydantic models, and the rest) is in main.py.

import os
import pathlib

import cocoindex as coco

LLM_MODEL = os.environ.get("LLM_MODEL", "gemini/gemini-2.5-flash")

app = coco.App(
    "MultiCodebaseSummarization",
    app_main,
    root_dir=pathlib.Path("./projects"),
    output_dir=pathlib.Path("./output"),
)

The app scans projects/ and outputs documentation to output/

Define the main function

In the main function, we walk through each project in the subdirectories and process it.

It is up to you to declare the process granularity. It can be

at a directory level per project. For example, code_embedding is a project, containing multiple files,
or at file level,
or at even smaller units (e.g., page level, or semantic unit level).

In this example, we have a projects folder containing 20+ projects. It is natural to pick granularity at the directory level for each project, because we want to create a wiki page per project.

@coco.function
def app_main(
    root_dir: pathlib.Path,
    output_dir: pathlib.Path,
) -> None:
    """Scan subdirectories and generate documentation for each project."""
    for entry in root_dir.resolve().iterdir():
        if not entry.is_dir() or entry.name.startswith("."):
            continue
        project_name = entry.name

        files = list(
            localfs.walk_dir(
                entry,
                recursive=True,
                path_matcher=PatternFilePathMatcher(
                    included_patterns=["*.py"],
                    excluded_patterns=[".*", "__pycache__"],
                ),
            )
        )

        if files:
            coco.mount(
                coco.component_subpath("project", project_name),
                process_project,
                project_name,
                files,
                output_dir,
            )

The main function does two things:

Find all projects. Loop through each subdirectory in root_dir, treating each as a separate project.
Mount a processing component for each project. For each project with Python files, coco.mount() sets up a processing component. CocoIndex handles execution and tracks dependencies.

A processing component groups an item's processing together with its target states. Each component runs independently and in parallel, so when project_a finishes, its results are applied to the external system without waiting for project_b or any other project.

To learn more about processing components, you can read the documentation.

Process each project

For each project, we will

use an LLM to extract info
aggregate all the extraction into a project-level summary
output the extraction to nice documentation with a Mermaid diagram.

@coco.function(memo=True)
async def process_project(
    project_name: str,
    files: Collection[localfs.File],
    output_dir: pathlib.Path,
) -> None:
    """Process a project: extract, aggregate, and output markdown."""
    # Extract info from each file concurrently using asyncio.gather
    file_infos = await asyncio.gather(*[extract_file_info(f) for f in files])

    # Aggregate into project-level summary
    project_info = await aggregate_project_info(project_name, file_infos)

    # Generate and output markdown
    markdown = generate_markdown(project_name, project_info, file_infos)
    localfs.declare_file(
        output_dir / f"{project_name}.md", markdown, create_parent_dirs=True
    )

Concurrent extraction. asyncio.gather() runs the file extractions concurrently, which is faster than sequential processing when each call waits on an LLM API response.

Extract file information with LLM

Now let's take a look at the details for each transformation. For file extraction, we define a structure using Pydantic and use Instructor to extract with LLMs.

Define the data models

The key to structured LLM outputs is defining clear Pydantic models.

class FunctionInfo(BaseModel):
    """Information about a public function."""
    name: str = Field(description="Function name")
    signature: str = Field(
        description="Function signature, e.g. 'async def foo(x: int) -> str'"
    )
    is_coco_function: bool = Field(
        description="Whether decorated with @coco.function"
    )
    summary: str = Field(description="Brief summary of what the function does")

class ClassInfo(BaseModel):
    """Information about a public class."""
    name: str = Field(description="Class name")
    summary: str = Field(description="Brief summary of what the class represents")

class CodebaseInfo(BaseModel):
    """Extracted information from Python code."""
    name: str = Field(description="File path or project name")
    summary: str = Field(description="Brief summary of purpose and functionality")
    public_classes: list[ClassInfo] = Field(default_factory=list)
    public_functions: list[FunctionInfo] = Field(default_factory=list)
    mermaid_graphs: list[str] = Field(
        default_factory=list,
        description="Mermaid graphs showing function relationships"
    )

Extract file info

The core extraction function uses memoization to cache LLM results:

_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)

@coco.function(memo=True)
async def extract_file_info(file: FileLike) -> CodebaseInfo:
    """Extract structured information from a single Python file using LLM."""
    content = file.read_text()
    file_path = str(file.file_path.path)

    prompt = f"""Analyze the following Python file and extract structured information...""" # see full prompt below

    result = await _instructor_client.chat.completions.create(
        model=LLM_MODEL,
        response_model=CodebaseInfo,
        messages=[{"role": "user", "content": prompt}],
    )
    return CodebaseInfo.model_validate(result.model_dump())

:::note See the full prompt in the repo. :::

memo=True caches the result by the function's inputs. When the file content and the code are unchanged, CocoIndex returns the previous result instead of calling the LLM again, so unchanged files skip the remote call on later runs.

Aggregate project information

For projects with multiple files, we aggregate into a unified summary:

@coco.function
async def aggregate_project_info(
    project_name: str,
    file_infos: list[CodebaseInfo],
) -> CodebaseInfo:
    """Aggregate multiple file extractions into a project-level summary."""
    if not file_infos:
        return CodebaseInfo(
            name=project_name, summary="Empty project with no Python files."
        )

    # Single file - just update the name
    if len(file_infos) == 1:
        info = file_infos[0]
        return CodebaseInfo(
            name=project_name,
            summary=info.summary,
            public_classes=info.public_classes,
            public_functions=info.public_functions,
            mermaid_graphs=info.mermaid_graphs,
        )

    # Multiple files - use LLM to create unified summary
    files_text = "\n\n".join(
        f"### {info.name}\n"
        f"Summary: {info.summary}\n"
        f"Classes: {', '.join(c.name for c in info.public_classes) or 'None'}\n"
        f"Functions: {', '.join(f.name for f in info.public_functions) or 'None'}"
        for info in file_infos
    )

    # Collect all mermaid graphs from files
    all_graphs = [g for info in file_infos for g in info.mermaid_graphs]

    prompt = f"""Aggregate the following Python files into a project-level summary...""" # see full prompt in repo

    result = await _instructor_client.chat.completions.create(
        model=LLM_MODEL,
        response_model=CodebaseInfo,
        messages=[{"role": "user", "content": prompt}],
    )
    result = CodebaseInfo.model_validate(result.model_dump())

    # Keep original file-level graphs if LLM didn't generate a unified one
    if not result.mermaid_graphs and all_graphs:
        result.mermaid_graphs = all_graphs

    return result

:::note See the full prompt in the repo. :::

This function combines file-level extractions into a single project summary:

Single-file project: use that file's info directly, with no extra LLM call.
Multi-file project: ask the LLM to synthesize the file summaries into one project overview.

The result is a unified CodebaseInfo that represents the entire project rather than individual files.

Generate the Markdown documentation

Create output Markdown for each project, including a Mermaid pipeline diagram.

@coco.function
def generate_markdown(
    project_name: str, info: CodebaseInfo, file_infos: list[CodebaseInfo]
) -> str:
    """Generate markdown documentation from project info."""
    lines = [
        f"# {project_name}",
        "",
        "## Overview",
        "",
        info.summary,
        "",
    ]

    if info.public_classes or info.public_functions:
        lines.extend(["## Components", ""])

        if info.public_classes:
            lines.append("**Classes:**")
            for cls in info.public_classes:
                lines.append(f"- `{cls.name}`: {cls.summary}")
            lines.append("")

        if info.public_functions:
            lines.append("**Functions:**")
            for fn in info.public_functions:
                marker = " ★" if fn.is_coco_function else ""
                lines.append(f"- `{fn.signature}`{marker}: {fn.summary}")
            lines.append("")

    if info.mermaid_graphs:
        lines.extend(["## CocoIndex Pipeline", ""])
        for graph in info.mermaid_graphs:
            graph_content = graph.strip()
            if not graph_content.startswith("```"):
                lines.append("```mermaid")
                lines.append(graph_content)
                lines.append("```")
            else:
                lines.append(graph_content)
            lines.append("")

    if len(file_infos) > 1:
        lines.extend(["## File Details", ""])
        for fi in file_infos:
            lines.extend([f"### {fi.name}", "", fi.summary, ""])

    lines.extend(["---", "", "*★ = CocoIndex function*"])
    return "\n".join(lines)

This function converts the structured CodebaseInfo into readable documentation:

Overview: the project summary at the top.
Components: classes and functions with descriptions (★ marks CocoIndex functions).
Pipeline diagram: Mermaid graphs showing how functions connect.
File details: per-file summaries, for multi-file projects.

Run the pipeline

cocoindex update main.py

CocoIndex will:

Scan each subdirectory in projects/
Extract structured information from Python files using the LLM
Aggregate file summaries into project summaries
Generate Markdown files in output/

Check the output:

ls output/
# project1.md project2.md ...

Incremental Updates

The real power shows when you make changes:

Modify a file:

# Edit a Python file in one of your projects
cocoindex update main.py
# Only the modified file is re-analyzed

Add a new project:

mkdir projects/new_project
# Add .py files
cocoindex update main.py
# Only the new project is processed

Output Example

Each generated Markdown file includes:

Overview: what the project does.
Components: public classes and functions with descriptions.
Pipeline diagram: a Mermaid graph showing how functions connect.
File details: per-file summaries for multi-file projects.

graph TD
    app_main[app_main] ==> process_project[process_project]
    process_project ==> extract_file_info[extract_file_info]
    process_project ==> aggregate_project_info[aggregate_project_info]
    process_project --> generate_markdown[generate_markdown]

    classDef extract fill:#FBE7DA,stroke:#BE5133,stroke-width:1.5px,color:#7A2E1A;
    classDef aggregate fill:#FCF3D8,stroke:#8F3B24,stroke-width:1.5px,color:#5A2417;
    classDef generate fill:#DDF5DE,stroke:#16A534,stroke-width:1.5px,color:#11472A;
    class extract_file_info extract;
    class aggregate_project_info aggregate;
    class generate_markdown generate;

Key Patterns

This example combines several patterns:

Structured LLM outputs: Pydantic models with Instructor return validated, typed data.
Memoized LLM calls: identical inputs return a cached result, which avoids most LLM calls on incremental runs.
Async concurrent processing: asyncio.gather() runs file extractions in parallel.
Hierarchical aggregation: extract at the file level, then aggregate to the project level.
Incremental processing: only changed inputs are reprocessed.

Beyond documentation: keeping derived knowledge current

The same pipeline applies wherever derived knowledge has to track a changing source, including the context that coding agents read. It is the same incremental pattern behind indexing a codebase for RAG, text embeddings, and knowledge graphs: declare the transformation once, and only changed inputs are reprocessed.

Long-horizon agents need current context

AI agents increasingly operate over long sessions, planning and acting across many steps. Their decisions depend on the state of the codebase as it is now: what changed recently, how modules interact, and the current structure of the system. Documentation that was accurate last week can lead an agent to act on assumptions that no longer hold.

Treating change as the unit of work

Processing change rather than reprocessing the whole corpus is what makes this practical:

Efficiency: only changed inputs are processed, not the entire corpus.
Latency: new information appears in minutes rather than after a full rebuild.
Cost: unchanged content does not incur repeated LLM calls.
Scalability: codebases too large to reprocess from scratch can still be kept current.

Declaring the transformation, rather than scripting the update logic by hand, is what lets the pipeline keep derived knowledge in sync with the source as the source changes.

Thanks to the community

@prrao87

Thanks @prrao87 for reviewing the example and providing detailed feedback on terminology, style, and conceptual clarity, which helped improve the developer experience.

If you have ideas, questions, or want to contribute, join us on Discord.

Next Steps

Try the full example, or read the example walkthrough in the docs
Join the CocoIndex Discord for questions

The source is on GitHub.

Slides-to-speech: Bring your slide decks to life with narrated audio

Prashanth Rao, Linghua Jin — Thu, 22 Jan 2026 00:00:00 GMT

Presentation slide decks are information-rich, dense sources of company knowledge, containing everything from quarterly updates, technical specs, sales reports, and training materials. These decks are locked away in PDF files that get revised constantly and are generally hard to consume in their static form.

What if we could turn those decks into a continuously updated, searchable, and narrated (audio) knowledge base? Doing that would result in a few immediate benefits:

Listen to any slide deck instead of reading every slide (useful for busy executives on the go, and for people with visual impairments).
Extract insights from slide content as structured speaker notes by using a VLM (vision language model).
Search slide content semantically, not just by keywords or filenames.

Producing those artifacts once is only half the task. When a deck changes, its speaker notes, audio, and embeddings also have to be kept fresh. Once everything is in sync, it can lay the foundation for more advanced user experiences later, including video generation from text.

In this post, we’ll show how to turn the slide deck's assets into a multimodal knowledge base stored in LanceDB: the same dataset contains the slide content, speaker notes, MP3 audio narration, and embeddings for vector search, all of which are orchestrated and kept fresh by CocoIndex.

Introducing CocoIndex

CocoIndex is an ultra-performant framework that keeps context continuously fresh for AI Agents. Think “React for data processing.” You declare state transformations, and CocoIndex handles the processing logic: it continuously watches for changes, recomputes only what’s necessary, and applies minimal updates downstream for end-to-end freshness. As AI systems become increasingly autonomous, the bottleneck is no longer model capability: it’s keeping context fresh and relevant.

CocoIndex follows an incremental update philosophy:

New slide decks appear in the source folder
Updated files get reprocessed automatically
Only affected rows get new embeddings and regenerated audio, if needed

Introducing LanceDB

LanceDB is a multimodal lakehouse for AI, available as an open-source, embedded, developer-friendly retrieval library that pairs well with CocoIndex. Under the hood, it’s powered by the Lance columnar format, which is designed for fast random access (great for search workloads) without sacrificing scan performance (great for large-scale aggregation queries).

For this Slides-to-Speech pipeline, that translates into a few concrete benefits:

Multimodal in one table: store speaker notes, MP3 audio bytes, embeddings, and metadata side by side as a single dataset.
Indexes are first-class: vector search, full-text search (FTS), and secondary indexes are stored alongside your dataset, so you don’t need to manage those using separate systems.
Data evolution: add or refine columns over time (backfill new embeddings, quality tags, moderation fields) without touching unmodified columns, which is great for feature engineering and experimentation.
Local-first simplicity: run it as an embedded database with minimal operational overhead.

CocoIndex's LanceDB connector automatically appends and updates rows as new results arrive via the CocoIndex pipeline, so you don’t have to rewrite your table (or rebuild your indexes) from scratch each time content changes.

What we are building: one slide, three derived modalities

To begin building our pipeline, let's first get the shape of the data clear. A PDF slide deck doesn’t just become one large database record. We first fan it out page by page, then turn each slide into three related outputs: speaker notes, MP3 narration (audio), and a text embedding for semantic search.

The speaker notes sit at the center of the pipeline, but they first need to be generated by a vision model that reasons over the slide content. Once they exist, audio generation and embedding can run on them concurrently. The completed outputs are then declared together as one SlideRecord, identified by its filename and page number.

Each tool has one focused job:

Component	Role in the pipeline
CocoIndex	Discovers PDFs, runs the transformations, and reconciles the declared slide rows with the source files.
PyMuPDF	Renders every PDF page as a PNG for the vision model.
DSPy + Gemini	DSPy lets us call the VLM via a concise signature, and the VLM (`gemini/gemini-2.5-flash`) reads each rendered slide PNG and writes the speaker notes.
Pocket TTS	Synthesizes those notes as MP3 audio locally on the CPU.
Sentence Transformers	Embeds the speaker notes locally with `sentence-transformers/all-MiniLM-L6-v2`.
LanceDB	Stores text, binary audio, metadata, and vectors together, then searches the notes by meaning.

Most of the pipeline runs locally on CPU: PyMuPDF renders the pages as images for the VLM, Pocket TTS synthesizes audio from the speaker notes, sentence-transformers generates text embeddings, and LanceDB runs as an embedded, local retrieval library.

Start backwards from the target

CocoIndex pipelines are easiest to reason about backwards from the target, i.e., where the data sits in its final form. Put simply, target_state = transformation(source_state), where transformation can be any function, or a set of functions. For this pipeline, the target state is one clean LanceDB row for each slide.

# Declare the LanceDB table shape via a strictly typed schema
# Pydantic or PyArrow schemas work well, but dataclasses are simpler for this example
@dataclass
class SlideRecord:
    id: str
    filename: str
    page: int
    speaker_notes: str
    voice: bytes
    embedding: Annotated[NDArray, EMBEDDER]

# Mount a LanceDB table as the target
table = await lancedb.mount_table_target(
    LANCE_DB,
    table_name="slides_to_speech",
    table_schema=await lancedb.TableSchema.from_class(
        SlideRecord,
        primary_key=["id"],
    ),
)

Multimodal data is a first-class citizen in LanceDB. Each row can hold the slide’s metadata, generated notes, MP3 narration bytes, and vector embedding side by side. SlideRecord is the typed contract that keeps those values together, via a dataclass.

Each slide needs a stable primary key so CocoIndex can match it to the same LanceDB row across runs, updating or removing the correct record as the source changes. This example derives that key from the filename and page number: page 3 of quarterly_review.pdf becomes quarterly_review.pdf#3.

Functions vs. processing components

Incremental processing is only useful when its boundaries match the unit of change. We want our pipeline to run every time a change is detected in the source files, without wastefully recomputing what didn't change.

In CocoIndex, functions define the work, while processing components group that work into independently synchronized units. Where we draw those component boundaries determines how much of the pipeline runs each time a change is detected, and which target states CocoIndex reconciles afterward.

Functions are callable Python functions decorated with @coco.fn. Setting memo=True lets CocoIndex reuse results for unchanged inputs and code. Several functions can run inside the same processing component.
Processing components are created by grouping functions and mounting them to the target. They manage the necessary function calls and the target states they declare. Processing components help decide the granularity at which CocoIndex reruns work and synchronizes the result.

In this example, we define the processing component at the per-slide level, through the process_file function. It takes a PDF file, renders each slide as an image, and mounts one process_slide function for each slide.

@coco.fn(memo=True)
async def process_file(
    file: FileLike,
    table: lancedb.TableTarget[SlideRecord],
) -> None:
    slides = await pdf_to_slides(await file.read())
    await coco.mount_each(
        process_slide,
        ((slide.page_number, slide) for slide in slides),
        str(file.file_path.path),
        table,
    )

process_file is memoized too, so CocoIndex can skip reading and rendering an unchanged PDF. If the file does change, process_slide provides the finer-grained memoization boundary: slide images with unchanged inputs reuse their prior results, while changed slides go through note generation, TTS, embedding, and the declaration of SlideRecord in the LanceDB target based on its stable key.

Changing a slide in a PDF still reruns page rendering for that entire file (still, PyMuPDF is a cheap CPU-based operation, so no VLM calls happen yet). Once the images are available, CocoIndex fingerprints files with a content-based hash, so it processes only slide images whose contents changed from the previous state. This is the power of memoization and the reason we mount the component at the slide level.

If a PDF is deleted from the source folder, CocoIndex unmounts its per-file component and recursively cleans up the target states it owns. In this pipeline, that means deleting the PDF's slide rows from LanceDB.

Let's now go over each stage of the pipeline below.

Fan a PDF out into slides

The function pdf_to_slides turns the PDF bytes into typed SlidePage values, each carrying the page number and the rendered PNG that the vision model will inspect.

@dataclass
class SlidePage:
    page_number: int
    image: bytes

@coco.fn.as_async(runner=coco.GPU, memo=True)
def pdf_to_slides(content: bytes) -> list[SlidePage]:
    import pymupdf

    slides: list[SlidePage] = []
    doc = pymupdf.open(stream=content, filetype="pdf")
    for i, page in enumerate(doc):
        pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
        slides.append(
            SlidePage(page_number=i + 1, image=pix.tobytes("png"))
        )
    doc.close()
    return slides

PyMuPDF renders each page at 2× scale, so the vision model sees a legible PNG rather than extracted PDF text. The image is an intermediate input: Gemini uses it to generate notes, but the pipeline discards it afterward. If you want to keep the rendered image for reuse, you can add an image field and store its bytes in the same LanceDB table.

Turn a slide image into narration

Extracting speaker notes from the slide images gives the Pocket TTS model the text it needs to turn the script into an audio file that narrates the insights from that slide. Text embeddings are also generated from the speaker notes, so they can support semantic search, allowing you to easily find slides based on their meaning rather than just keywords in their titles or filenames.

class SlideNotes(dspy.Signature):
    """Write natural spoken narration for a slide, as a presenter would say it aloud."""

    slide: dspy.Image = dspy.InputField(desc="the rendered slide image")
    speaker_notes: str = dspy.OutputField(
        desc="a few sentences of spoken narration — no markdown or bullet symbols"
    )

_speaker_notes = dspy.Predict(SlideNotes)

@functools.cache
def _get_lm(model: str) -> dspy.LM:
    return dspy.LM(model, max_tokens=8192)

@coco.fn(memo=True)
async def extract_speaker_notes(image: bytes) -> str:
    data_url = "data:image/png;base64," + base64.b64encode(image).decode()
    with dspy.context(lm=_get_lm(coco.use_context(LLM_MODEL))):
        result = await _speaker_notes.acall(slide=dspy.Image(url=data_url))
    return result.speaker_notes

SlideNotes is a DSPy signature, a typed contract that tells the VLM what we need. When a slide image comes in, speaker notes come out. The CocoIndex function extract_speaker_notes does the actual work of turning the PNG bytes into a text output.

This extraction function is memoized because speaker-note generation is an expensive dependency: it calls a capable VLM to reason over the image. Its memoization key includes the rendered image and the change-detected LLM_MODEL context, so an unchanged slide can reuse its notes if the image bytes and the model remain the same. If either of them changes, it's tracked by CocoIndex and the work is rerun for that slide.

Turn speaker notes into audio and embeddings

We can now inspect the function process_slide and the work it does, by reading it top to bottom. The speaker notes must first exist, so they are computed and awaited via extract_speaker_notes. Once they exist, audio synthesis and embedding have no dependency on each other, so we call asyncio.gather to run them concurrently. CocoIndex commits them to the LanceDB table only after both are ready.

@coco.fn(memo=True)
async def process_slide(
    slide: SlidePage,
    filename: str,
    table: lancedb.TableTarget[SlideRecord],
) -> None:
    notes = await extract_speaker_notes(slide.image)
    voice, embedding = await asyncio.gather(
        text_to_speech(notes, coco.use_context(TTS_VOICE)),
        coco.use_context(EMBEDDER).embed(notes),
    )
    table.declare_row(
        row=SlideRecord(
            id=f"{filename}#{slide.page_number}",
            filename=filename,
            page=slide.page_number,
            speaker_notes=notes,
            voice=voice,
            embedding=embedding,
        )
    )

The target state is declared as a SlideRecord dataclass (per the LanceDB schema) keyed by its unique ID that combines the filename and page number. This is how CocoIndex reconciles the target state with the source files, updating or removing rows as needed.

The expensive nested operations are memoized as well. If only the voice changes, CocoIndex can reuse the notes and embedding while regenerating the audio. If the embedding model changes, it can reuse the notes and audio while computing a new vector. This gives us reuse at the file, slide, and individual transformation levels.

Define the app

Apps are the top-level runnable unit in CocoIndex. They name your pipeline and bind a main function with its parameters. When you call app.update(), CocoIndex runs that main function as the root processing component that can mount processing components (like our process_file and its child component process_slide) that do work and declare target states.

app_main mounts the SlideRecord table we defined earlier, uses localfs.walk_dir in live mode to discover PDFs, then calls the outer mount_each. The app is configured with a name, the main function, and the source directory to watch for changes.

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    table = await lancedb.mount_table_target(
        LANCE_DB,
        table_name="slides_to_speech",
        table_schema=await lancedb.TableSchema.from_class(
            SlideRecord,
            primary_key=["id"],
        ),
    )
    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(
            included_patterns=["**/*.pdf"]
        ),
        live=True,
    )
    await coco.mount_each(process_file, files.items(), table)

app = coco.App(
    coco.AppConfig(name="SlidesToSpeech"),
    app_main,
    sourcedir=pathlib.Path("./slides"),
)

Run the pipeline and run semantic search

Prerequisites

Python 3.11+
A Gemini API key for the default vision model
ffmpeg for MP3 export

Pocket TTS is open source and runs locally on the CPU. Its weights download on the first run, so it needs neither a TTS API key nor a GPU.

Quick start

From the example directory, copy the environment file, set GEMINI_API_KEY, and ensure LLM_MODEL=gemini/gemini-2.5-flash to match the example’s default configuration. Then install and run:

cp .env.example .env
uv sync
cocoindex update main

# Keep watching slides/ for changes instead:
cocoindex update -L main

Search the deck

python main.py "reducing latency and reliability"

The query helper embeds the question with the same sentence-transformer, then asks LanceDB for its nearest neighbors. The results already arrive ranked, so we can print the matching slide and its notes directly:

for row in table.search(vec).limit(5).to_list():
    print(f"{row['filename']} slide {row['page']}")

The slide deck we use here is an example "quarterly-review" deck, so the query about "reducing latency and reliability" brings the slide about "Engineering Priorities" to the top because its notes discuss "reducing p95 latency and multi-region resilience".

The returned row can also fetch the voice bytes, so a downstream application can use those bytes as audio/mpeg, write them to an MP3 file for playback, and play them on demand if needed.

Here is the narration the pipeline generated for the sample deck, one MP3 per slide:

Slide 1 Your browser doesn't support MP3 playback.

Slide 2: Engineering Priorities Your browser doesn't support MP3 playback.

Slide 3 Your browser doesn't support MP3 playback.

How CocoIndex tracks changes

In this demo, CocoIndex watches the ./slides directory for changed PDFs. It then identifies the changed slides, and only those slides regenerate notes, audio, and embeddings. As the table above shows, it can track much more than file changes.

However, source files changing is only one such scenario. The table below shows the other changes that CocoIndex tracks, and what they mean for other pipelines like this one.

Change	What it means in this pipeline
Nothing	Memoized calls reuse their prior results and retain their rows.
One slide changes inside a PDF	The PDF renders again, then only the slide with different PNG bytes reruns notes, TTS, embedding, and its row declaration.
A PDF is added or deleted	CocoIndex creates or removes the file component and all of its owned slide rows.
LLM model, embedder, or voice changes	The change-detected context refreshes the functions that consume it; unrelated memoized calls can be reused.
Decorated function logic changes	A changed `@coco.fn` fingerprint invalidates memoized calls that depend on that function, then CocoIndex reconciles the resulting rows.
`SlideRecord` fields or primary key change	The target contract has changed. Treat compatibility, backfill, and identity changes as a deliberate migration decision.

Bring your slides to life (and beyond)

The pipeline shown here is a concrete example of how to bring slide decks to life via narration. But don’t just think of organizational data as structured tables and flat text. A large amount of company intelligence is locked away in richer artifacts like PDF reports, dashboards, and video recordings.

With CocoIndex and LanceDB, we can turn all of that raw multimodal content into a living, breathing, contextually rich knowledge base that agents can retrieve from:

Unify multimodal content in one storage layer: text, audio, vectors, and metadata in the same table.
Add structure that supports more questions: speaker notes, extracted fields, embeddings, tags, and other derived signals.
Keep everything synchronized over time: incremental recomputation when source files change, with updated rows in LanceDB.
Build user-facing experiences on top: semantic search, narrated slide playback, assistants, and video-generation pipelines.

Try the complete slides-to-speech example to bring your own deck to life! Then, star CocoIndex and LanceDB on GitHub.

CocoIndex Changelog 0.3.11 - 0.3.26

George He — Sun, 18 Jan 2026 00:00:00 GMT

CocoIndex is moving at insane speed! 🚀 Since the last release, CocoIndex shipped 15 releases (0.3.11–0.3.26) and crossed a major milestone: #1 GitHub Global Trending across all languages and #1 on Rust. This cycle was focused on one clear goal: making fresh, structured, programmable context reliable enough for agents running in production.

Agents are only as good as the data they reason over. Thinking about codebases, meeting notes, emails, relationships, commerce ... anything that may be subject to change. If you have ever done index-swap or had to keep multiple copies of the target to serve live agents on data or code change, you'll never have to do it again.

Glad this project resonates with the community! 2026 will see more autonomous agents going into production, and we're excited to see CocoIndex help in that frontier.

Here's what we shipped since the last release. 0.3.11–0.3.26 was about production trust: resilient pipelines, structured errors, better observability, and always-fresh structured context for agents operating in the real world.

Core capability

CocoIndex just got a significant upgrade in runtime resilience and developer ergonomics. The core engine now also features a fully structured error system, enabling unified error tracing from Rust to Python so you can finally see human‑readable stack traces across boundaries. Combined with smarter runtime warnings for slow live updates, simplified progress bar rendering, and cleaner internal API boundaries, CocoIndex continues to evolve into the most transparent and trustworthy data‑processing core in its class.

Failure-tolerant target deletion

Target drop failures no longer block pipeline setup updates: an environment toggle COCOINDEX_IGNORE_TARGET_DROP_FAILURES lets you log and continue gracefully (#1454).

Simplified progress experience

Streamlined progress bar updates for a more predictable runtime visualization experience (#1467).

Structured error system overhaul

A new foundation for error definition and propagation:

Introduced structured error framework with host exception tunneling
End-to-end Python stack trace propagation now supported
New unified error types improve exception messaging and logging at both Rust and Python layers (#1380, #1413).

Smart runtime alerts

CocoIndex now warns when live updates exceed the configured refresh interval, ideal for maintaining freshness guarantees (#1408).

Cleaner internal API boundaries

Refactored datatype.py with a dedicated _internal module, decluttering the public API surface (#1402).

Force reprocessing option for updates

Added explicit control for full recomputation during incremental updates for safer state resets. Run cocoindex update --full-reprocess ... to force a reprocessing (#1385).

Flow correctness improvements

Fixed target diffing logic for user-managed setup states
General FTS fixes for LanceDB (#1369, #1403)

Together, these changes strengthen the core runtime's fault tolerance, improve observability, and lay the groundwork for a richer developer debugging experience.

Integrations

This release massively expands what CocoIndex can plug into.

Sources & targets

Qdrant

Full support for VectorIndexMethod HNSW configuration in the Qdrant connector (#1410).

Postgres

Native support for pgvector types in Postgres source (#1387).

Local files

Automatic symlink resolution for easier graph ingestion (#1463).

LanceDB

Added optimize() for post-ingest table compaction
Extended full-text search integration in the LanceDB connector (#1466, #1358)

Functions & LLM providers

Unified OpenAI/Azure architecture

Shared logic between openai.rs and azureopenai.rs for cleaner code reuse (#1376).

Azure OpenAI integration

Bring your Azure LLM accounts seamlessly into CocoIndex pipelines (#1362).

OpenRouter embeddings

Added embedding model support across OpenRouter's expanding provider network (#1359).

Embedding dimension validation

Introduced expected_output_dimension param to enforce vector consistency (#1373).

Direct JSON output in functions

GeneratedOutput enum now supports structured JSON returns for downstream consumption (#1395).

Gemini API key fixes

Cleaner, more consistent propagation of credentials across function calls (#1375).

These integrations make CocoIndex a more open, extensible, and multi-LLM-ready data processing cortex.

Build with CocoIndex

We are seeing more projects built with CocoIndex.

Build fresh data with LanceDB and CocoIndex

🚀 If agents are going to search, reason, and act over long periods of time, we need infrastructure that treats data freshness and multimodal as a first-class primitive.

Our friends at LanceDB just published a full walkthrough with us showing how to build a live-updating multimodal search index using LanceDB and CocoIndex, where text, images, embeddings, and LLM-derived features stay continuously in sync as data and logic evolve.

Check out the tutorial for more details.

🔥 What's cool about this example:

Incremental-by-design processing: Only new or changed records are reprocessed. Add one recipe? Update one image? Fix one ingredient? CocoIndex touches only what changed, not the other 100k rows.
True multimodal indexing: Text, images, embeddings, and metadata live together in LanceDB, optimized for both search and analytics, without splitting data across systems.
Schema evolution without rewrites: We add new LLM-extracted feature columns (vegetarian, dairy, category, etc.) after data is already live. CocoIndex updates the schema and backfills safely, with no table rebuilds.
Structured LLM extraction (no brittle prompts): DSPY signatures define input/output types, so LLMs behave like programs, not JSON-generating slot machines.
Upserts + deletes handled automatically: Primary keys + incremental flows mean no duplicates or stale rows, and no "did this record change?" logic to maintain.
End-to-end lineage & observability: With CocoInsight, you can see exactly why a record changed and which transformation produced each field.
Live mode for agent-ready systems: Run the pipeline continuously. As data arrives or logic changes, your search layer updates in near real time.

Think PDFs, slides, screenshots, images, videos, and support tickets across massive enterprise data lakes, all kept fresh, indexed, searchable, and generating real insights.

Building a real-time HackerNews trending topics detector with CocoIndex

In the age of information overload, understanding what's trending (and why) is crucial for developers, researchers, and data engineers. HackerNews is one of the most influential tech communities, but manually tracking emerging topics across thousands of threads and comments is practically impossible.

In this post, we'll explore how to build a production-ready, real-time trending topics pipeline on top of HackerNews using CocoIndex. It shows how unstructured, high-volume community data can be incrementally ingested, enriched with LLMs, and queried as structured, up-to-date knowledge.

Each HackerNews thread and comment is then processed through an LLM-powered extraction step to identify normalized topics (technologies, products, companies, people, etc.). These topics, along with the original messages, are indexed into Postgres as queryable tables. Weighted scoring distinguishes primary topics in threads from secondary mentions in comments, enabling higher-quality trend rankings.

The result is a continuously updating system that turns raw HackerNews activity into structured, queryable intelligence, ready for dashboards, analytics, or AI agents.

Check out the blog for more details.

Building a knowledge graph from meeting notes that automatically updates

Most companies sit on an ocean of meeting notes and treat them like static text files. But inside those documents are decisions, tasks, owners, and relationships: basically an untapped knowledge graph that is constantly changing. We just published a full walkthrough showing how to turn meeting notes in Drive into a live-updating knowledge graph using CocoIndex + LLM extraction.

Check out the tutorial for more details.

What's cool about this example:

Incremental processing: Only changed documents get reprocessed. If you have thousands of meeting notes, but only 1% change each day, CocoIndex only touches that 1%, saving 99% of LLM cost and compute.
Structured extraction with LLMs: We use a typed Python dataclass as the schema, so the LLM returns real structured objects, not brittle JSON prompts.
Graph-native export: CocoIndex maps nodes (Meeting, Person, Task) and relationships (ATTENDED, DECIDED, ASSIGNED_TO) without writing Cypher, directly into Neo4j.
Real-time updates: If a meeting note changes (task reassigned, typo fixed, new discussion added), the graph updates automatically.

Extracting structured data from patient intake forms with DSPy and CocoIndex

Patient intake forms are indeed a rich source of structured clinical data, but traditional OCR + regex pipelines fail to reliably capture their nested, conditional, and variable structure, leaving most of that value locked in unstructured text or manual entry.

This project demonstrates a production-ready approach to extracting clean, validated, structured patient data directly from PDF intake forms using DSPy and CocoIndex, without brittle OCR pipelines, regex rules, or Markdown intermediates.

The result is a scalable, explainable, and compliant pipeline well-suited for healthcare and other regulated domains, turning unstructured intake forms into always-fresh, queryable patient models ready for downstream systems, analytics, or AI agents.

Check out the tutorial for more details.

Thanks to the community 🤗🎉

Welcome new contributors to the CocoIndex community! We are so excited to have you!

@1fanwang

Thanks @1fanwang for adding Azure OpenAI support as an LLM provider in #1362, enabling enterprise users to leverage CocoIndex with their Azure deployments.

@vipulaSD

Thanks @vipulaSD for adding field-level descriptions for data classes in the Product Recommendation example in #1327, making the example more informative and easier to understand.

@prrao87

Thanks @prrao87 for adding support for embedding models in OpenRouter in #1359, expanding LLM provider options, and fixing typos and consistency issues in the docs in #1360, making the documentation clearer and more polished.

@Haleshot

Thanks @Haleshot for improving the README note for better GitHub rendering in #1288, shifting to the tracing crate for logging in #1363, adding Postgres server reminder to quickstart docs in #1368, fixing the broken workflow badge in README in #1370, introducing a structured error system with host exception tunneling in #1380, adding llms.txt file to docs in #1455, adding short flag aliases for common CLI options in #1458, and adding env file sourcing tip to meeting_notes_graph example in #1470.

@thisisharsh7

Thanks @thisisharsh7 for making openai.rs generic to share logic with azureopenai.rs in #1376, including mypy type checking for the examples directory in #1383, and adding the GeneratedOutput enum for direct JSON returns in #1395, enhancing type safety and code reusability.

@prithvi-moonshot

Thanks @prithvi-moonshot for adding onBrokenAnchors config to report documentation errors in #1325, providing an option to force reprocessing during update in #1385, and making target deletion (drop) tolerate failures in #1454, improving robustness of the data pipeline.

@skprasadu

Thanks @skprasadu for wrapping cargo test to avoid embedded Python stdlib discovery crash in #1398, moving Query Support above Built-in Sources in the docs for better organization in #1397, and adding a warning when live update pass exceeds refresh_interval in #1408.

@Gohlub

Thanks @Gohlub for adding with_context to user‑configured operations in #1275, enabling more powerful context‑aware flows for sources, functions, and targets.

@wxh06

Thanks @wxh06 for adding expected_output_dimension parameter to embed_text in #1373, providing more control over embedding output dimensions.

@samojavo

Thanks @samojavo for fixing mypy errors in example entry points and adding a local type-check script in #1248, improving code quality and developer experience.

@aryasoni98

Thanks @aryasoni98 for enabling programmatic API key passing besides reading from environment variables in #1134, providing more flexibility in configuration.

@shinobiwanshin

Thanks @shinobiwanshin for enhancing the Button component with hover effects and styling improvements in #1393, making the UI more polished and interactive.

@majiayu000

Thanks @majiayu000 for adding VectorIndexMethod support for HNSW configuration in Qdrant in #1410, enabling more fine-grained control over vector indexing parameters.

Summary

CocoIndex continues evolving into a resilient, intelligent data engine for the modern AI stack.

This cycle focuses on error transparency, LLM integration depth, and frictionless developer operations, ensuring your pipelines stay alive, observable, and blazing fast even under adverse conditions.

If you like this project, please support us with a star ⭐ at https://github.com/cocoindex-io/cocoindex !

🔥🔥 We are cooking something big! v1 is coming soon. Stay tuned! If you read all the way here, you are a true fan!

Extract patient intake forms with DSPy and CocoIndex

Linghua Jin — Mon, 15 Dec 2025 00:00:00 GMT

In today's example, we are going to show how to extract clean, typed, Pydantic-validated structured data directly from PDFs, using:

DSPy (for multimodal structured extraction with Gemini 2.5 Flash vision)
CocoIndex (for incremental processing, caching, and database storage)

No manual text extraction, no brittle markdown conversion: just connect to the source, transform the PDFs, and get validated patient models out, ready to go in production.

The entire code is open sourced with Apache 2.0 license. To see more examples built with CocoIndex, you could refer to the examples page. ⭐ Star the project if you find it helpful!

Why DSPy + CocoIndex?

Before jumping in, here’s what each component contributes:

DSPy: A programming framework for LLMs

Traditional LLM apps rely on prompt engineering: you write a prompt with instructions, few‑shot examples, and formatting, then call the model and parse the raw text. This approach is fragile:

Small changes in the prompt, model, or data can break the output format or quality.
Logic is buried in strings, making it hard to test, compose, or version.

DSPy replaces this with a programming model: you define what each LLM step should do (inputs, outputs, constraints), and the framework figures out how to prompt the model to satisfy that spec.

CocoIndex: An ultra performant data processing engine for AI workloads

CocoIndex is an ultra performant compute framework for AI workloads, with incremental processing. Users write simple in-memory computations in Python and coco runs them as a resilient, scalable data pipeline (with Rust Engine) – with fresh data always ready for serving, built on a small set of core concepts. The same flow definition you use in a notebook can be lifted easily into production.

With CocoIndex, changes in sources or transformation logic only trigger minimal recompute, cutting cold-start “backfill” latencies from hours to seconds while reducing GPU/API spend. In production, this manifests as always-fresh targets: you run in “live” mode with change data capture or polling, and CocoIndex keeps derived stores in sync with complex unstructured sources like codebases, PDFs, and multi-hop API compositions.

Because every transformation step is observable with lineage, teams get auditability and explainability out of the box, which helps with regulated scenarios like healthcare extraction or financial workflows.

DSPy & CocoIndex synergy

The synergy shows up most clearly in end-to-end AI data products: DSPy defines robust, typed extractors or decision modules, and CocoIndex wires them into a resilient, incremental pipeline that can meet SLOs and compliance needs. Any change in documents, code, or business rules is reflected quickly and explainably in the targets and features those agents consume.

Flow overview

Prerequisites

Before getting started, make sure you have the following set up:

Install Postgres if you don't have one, and ensure you can connect to it from your development environment.
Python dependencies

pip install -U cocoindex dspy-ai pydantic pymupdf

Create a .env file:

# Postgres database address for cocoindex
COCOINDEX_DATABASE_URL=postgres://cocoindex:cocoindex@localhost/cocoindex

# Gemini API key
GEMINI_API_KEY=YOUR_GEMINI_API_KEY

Pydantic models: Define the structured schema

We defined Pydantic-style classes (Contact, Address, Insurance, etc.) to match a FHIR-inspired patient schema, enabling structured and validated representations of patient data. Each model corresponds to a key aspect of a patient's record, ensuring both type safety and nested relationships.

1. Contact model

class Contact(BaseModel):
    name: str
    phone: str
    relationship: str

Represents an emergency or personal contact for the patient.
Fields:
- name: Contact's full name.
- phone: Contact phone number.
- relationship: Relation to the patient (e.g., parent, spouse, friend).

2. Address model

class Address(BaseModel):
    street: str
    city: str
    state: str
    zip_code: str

Represents a postal address.
Fields:
- street, city, state, zip_code: Standard address fields.

3. Pharmacy model

class Pharmacy(BaseModel):
    name: str
    phone: str
    address: Address

Represents the patient’s preferred pharmacy.
Fields:
- name: Pharmacy name.
- phone: Pharmacy contact number.
- address: Uses the Address model for structured address information.

4. Insurance model

class Insurance(BaseModel):
    provider: str
    policy_number: str
    group_number: str | None = None
    policyholder_name: str
    relationship_to_patient: str

Represents the patient’s insurance information.
Fields:
- provider: Insurance company name.
- policy_number: Unique policy number.
- group_number: Optional group number.
- policyholder_name: Name of the person covered under the insurance.
- relationship_to_patient: Relationship to patient (e.g., self, parent).

5. Condition model

class Condition(BaseModel):
    name: str
    diagnosed: bool

Represents a medical condition.
Fields:
- name: Condition name (e.g., Diabetes).
- diagnosed: Boolean indicating whether it has been officially diagnosed.

6. Medication model

class Medication(BaseModel):
    name: str
    dosage: str

Represents a current medication the patient is taking.
Fields:
- name: Medication name.
- dosage: Dosage information (e.g., "10mg daily").

7. Allergy model

class Allergy(BaseModel):
    name: str

Represents a known allergy.
Fields:
- name: Name of the allergen (e.g., peanuts, penicillin).

8. Surgery model

class Surgery(BaseModel):
    name: str
    date: str

Represents a surgery or procedure the patient has undergone.
Fields:
- name: Surgery name (e.g., Appendectomy).
- date: Surgery date (as a string, ideally ISO format).

9. Patient model

class Patient(BaseModel):
    name: str
    dob: datetime.date
    gender: str
    address: Address
    phone: str
    email: str
    preferred_contact_method: str
    emergency_contact: Contact
    insurance: Insurance | None = None
    reason_for_visit: str
    symptoms_duration: str
    past_conditions: list[Condition] = Field(default_factory=list)
    current_medications: list[Medication] = Field(default_factory=list)
    allergies: list[Allergy] = Field(default_factory=list)
    surgeries: list[Surgery] = Field(default_factory=list)
    occupation: str | None = None
    pharmacy: Pharmacy | None = None
    consent_given: bool
    consent_date: str | None = None

Represents a complete patient record with personal, medical, and administrative information.
Key fields:
- name, dob, gender: Basic personal info.
- address, phone, email: Contact info.
- preferred_contact_method: How the patient prefers to be reached.
- emergency_contact: Nested Contact model.
- insurance: Optional nested Insurance model.
- reason_for_visit, symptoms_duration: Visit details.
- past_conditions, current_medications, allergies, surgeries: Lists of nested models for comprehensive medical history.
- occupation: Optional job info.
- pharmacy: Optional nested Pharmacy model.
- consent_given, consent_date: Legal/administrative consent info.

Why use Pydantic here?

Validation: Ensures all fields are the correct type (e.g., dob is a date).
Structured Nested Models: Patient has nested objects like Address, Contact, and Insurance.
Default Values & Optional Fields: Handles optional fields and defaults (Field(default_factory=list) ensures empty lists if no data).
Serialization: Easily convert models to JSON for APIs or databases.
Error Checking: Automatically raises errors if invalid data is provided.

DSPy vision extractor

DSPy signature

Let’s define PatientExtractionSignature. A Signature describes what data your module expects and what it will produce. Think of it as a schema for an AI task.

PatientExtractionSignature is a dspy.Signature, which is DSPy's way of declaring what the model should do, not how it does it.

# DSPy Signature for patient information extraction from images
class PatientExtractionSignature(dspy.Signature):
    """Extract structured patient information from a medical intake form image."""

    form_images: list[dspy.Image] = dspy.InputField(
        desc="Images of the patient intake form pages"
    )
    patient: Patient = dspy.OutputField(
        desc="Extracted patient information with all available fields filled"
    )

This signature defines the task contract for patient information extraction.

Inputs: form_images – a list of images of the intake form.
Outputs: patient – a structured Patient object.

From DSPy's point of view, this Signature is a "spec": a mapping from an image-based context to a structured, Pydantic-backed semantic object that can later be optimized, trained, and composed with other modules.

PatientExtractor module

PatientExtractor is a dspy.Module, which in DSPy is a composable, potentially trainable building block that implements the Signature.

class PatientExtractor(dspy.Module):
    """DSPy module for extracting patient information from intake form images."""

    def __init__(self) -> None:
        super().__init__()
        self.extract = dspy.ChainOfThought(PatientExtractionSignature)

    def forward(self, form_images: list[dspy.Image]) -> Patient:
        """Extract patient information from form images and return as a Pydantic model."""
        result = self.extract(form_images=form_images)
        return result.patient  # type: ignore

In __init__, ChainOfThought is a DSPy primitive module that knows how to call an LLM with reasoning-style prompting to satisfy the given Signature. In other words, it is a default "strategy" for solving the "extract patient from images" task.
The forward method is DSPy's standard interface for executing a module. You pass form_images into self.extract(). DSPy then handles converting this call into an LLM interaction (or a trained program) that produces a patient field as declared in the Signature.

Conceptually, PatientExtractor is an ETL operator: the Signature describes the input/output types, and the internal ChainOfThought module is the function that fills that contract.

Single-step extraction

Now let’s wire the DSPy Module to extract from a single PDF. From a high level,

The extractor receives PDF bytes directly
Internally converts PDF pages to DSPy Image objects using PyMuPDF
Processes images with vision model
Returns Pydantic model directly

@cocoindex.op.function(cache=True, behavior_version=1)
def extract_patient(pdf_content: bytes) -> Patient:
    """Extract patient information from PDF content."""

    # Convert PDF pages to DSPy Image objects
    pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")

    form_images = []
    for page in pdf_doc:
        # Render page to pixmap (image) at 2x resolution for better quality
        pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
        # Convert to PNG bytes
        img_bytes = pix.tobytes("png")
        # Create DSPy Image from bytes
        form_images.append(dspy.Image(img_bytes))

    pdf_doc.close()

    # Extract patient information using DSPy with vision
    extractor = PatientExtractor()
    patient = extractor(form_images=form_images)

    return patient  # type: ignore

This function is a CocoIndex custom function (decorated with @cocoindex.op.function) that takes PDF bytes as input and returns a fully structured Patient Pydantic object.

cache=True allows repeated calls with the same PDF to reuse results.
behavior_version=1 ensures versioning of the function for reproducibility.

Create DSPy image objects

We open the PDF from bytes using PyMuPDF (pymupdf), then we iterate over each page.

Useful trick: Render the page as a high-resolution image (2x) for better OCR/vision performance.
Convert the rendered page to PNG bytes.
Wrap the PNG bytes in a DSPy Image object.

DSPy extraction

The list of form_images is passed to the DSPy module:

ChainOfThought reasoning interprets each image.
Vision + NLP extract relevant text fields.
Populate Pydantic Patient object with structured patient info.

CocoIndex flow

Loads PDFs from local directory as binary
For each document, applies single transform: PDF bytes → Patient data
Exports the results to a PostgreSQL table

Declare flow

Declare a CocoIndex flow, connect to the source, add a data collector to collect processed data.

@cocoindex.flow_def(name="PatientIntakeExtractionDSPy")
def patient_intake_extraction_dspy_flow(
    flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
    data_scope["documents"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(path="data/patient_forms", binary=True)
    )

    patients_index = data_scope.add_collector()

@cocoindex.flow_def tells CocoIndex that this function is a flow definition, not regular runtime code.
add_source() registers a LocalFile source that traverses the data/patient_forms directory and creates a logical table named documents.

You can connect to various sources, or even custom sources with CocoIndex if native connectors are not available. CocoIndex is designed to keep your indexes synchronized with your data sources. This is achieved through a feature called live updates, which automatically detects changes in your sources and updates your indexes accordingly. This ensures that your search results and data analysis are always based on the most current information. You can read more in the live updates documentation.

Process documents

with data_scope["documents"].row() as doc:
    # Extract patient information directly from PDF using DSPy with vision
    # (PDF->Image conversion happens inside the extractor)
    doc["patient_info"] = doc["content"].transform(extract_patient)

    # Collect the extracted patient information
    patients_index.collect(
        filename=doc["filename"],
        patient_info=doc["patient_info"],
    )

This iterates over each document. We transform doc["content"] (the bytes) by our extract_patient function. The result is stored in a new field patient_info.

Then we collect a row with the filename and extracted patient_info.

Export to Postgres

patients_index.export(
    "patients",
    cocoindex.storages.Postgres(table_name="patients_info_dspy"),
    primary_key_fields=["filename"],
)

We export the collected index to Postgres. This will create/maintain a table patients keyed by filename, automatically deleting or updating rows if inputs change.

Because CocoIndex tracks data lineage, it will handle updates/deletions of source files incrementally.

Configure CocoIndex settings

Define a CocoIndex settings function that configures the AI model for DSPy:

@cocoindex.settings
def cocoindex_settings() -> cocoindex.Settings:
    # Configure the model used in DSPy
    lm = dspy.LM("gemini/gemini-2.5-flash")
    dspy.configure(lm=lm)

    return cocoindex.Settings.from_env()

It returns a cocoindex.Settings object initialized from environment variables, enabling the system to use the configured model and environment settings for all DSPy operations.

Running the pipeline

Update the index:

cocoindex update main

CocoInsight

I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with zero pipeline data retention.

cocoindex server -ci main

Scalable open ecosystem, not a closed box

CocoIndex is intentionally "composable by default": it gives you a fast, incremental data engine and clean flow abstraction, but never locks you into a specific model, vector DB, processing module or orchestration stack.

CocoIndex treats everything (sources, ops, and storages) as pluggable interfaces rather than proprietary primitives. You can read from local files, S3, APIs, or custom sources, call any data transformation logic (beyond SQL, DSPy modules, any complex Python transformations, generated parsers, etc.), and export to relational databases, vector databases, search engines, or custom sinks through its storage layer.

Why DSPy + CocoIndex fits this philosophy

DSPy is itself a compositional framework for LLMs: you define typed Signatures and Modules, and it learns how to implement them, making the LLM layer programmable, testable, and optimizable.

CocoIndex treats these modules as first-class operators in the flow, so you get a clean separation of concerns: DSPy owns “how the model thinks,” while CocoIndex owns “how data moves, is cached, and is served” across changing PDFs, code, or APIs.

This pairing is powerful because neither system tries to be the entire stack: CocoIndex does not prescribe a prompt framework, and DSPy does not prescribe a data pipeline engine. Instead, they interlock: DSPy modules become composable building blocks inside CocoIndex flows, and CocoIndex gives those modules a production context with retries, batching, caching, and live updates.

Comparison of DSPy vs BAML examples

For this DSPy and BAML example with CocoIndex, there are some differences in the data flow and the schema enforcement.

Aspect	This tutorial (DSPy)	Previous intake tutorials (OpenAI/BAML)
Model interaction	DSPy Signatures + Modules, trainable and optimizable.	Direct prompts or BAML functions with generated clients.
Input format	PDF bytes → images → Gemini Vision (no Markdown step).	PDF/DOCX → Markdown via parsers (e.g., MarkItDown) → text LLM.
Schema enforcement	Pydantic models used directly as DSPy output types.	Dataclasses or BAML types mapped into CocoIndex collectors.
Pipeline engine	CocoIndex incremental flow with LocalFile source + Postgres sink.	Same engine

DSPy and BAML both help build LLM applications with structured outputs, but they emphasize different things: DSPy focuses on modular Python workflows and optimizer-driven prompt tuning, while BAML focuses on a typed DSL, schema guarantees, and multi-language client generation.

Support CocoIndex ❤️

If this example was helpful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub.

Your stars help us grow the community, stay motivated, and keep shipping better tools for real-time data ingestion and transformation.

A knowledge graph from meeting notes that auto-updates

Linghua Jin — Mon, 08 Dec 2025 00:00:00 GMT

Meeting notes capture decisions, action items, who was in the room, and how people and tasks connect. Most teams keep them as static documents, reachable only through text search. A knowledge graph lets you query them like a database instead: "Who attended the budget planning meetings?" or "What tasks did Sarah get assigned across every meeting?"

To answer questions like these you extract structured information from unstructured notes and store it as a graph, which makes relationship queries direct instead of impossible.

This post walks through the Meeting Notes Graph example: a CocoIndex v1 pipeline that reads Markdown meeting notes from Google Drive, extracts structure with an LLM, deduplicates person names with embedding-based entity resolution, and builds a queryable graph in Neo4j.

The project is open-sourced in the CocoIndex v1 repo.

Why a graph, and why incremental

Meeting notes are living documents. Attendee names get corrected, tasks get reassigned, and decisions get revised. A batch pipeline that rebuilds the whole graph on every edit is wasteful once you have more than a handful of files, and it makes near-real-time freshness expensive.

CocoIndex processes incrementally: it detects which files changed and reprocesses only those. Unchanged notes never hit the LLM, and the graph write is a no-op for anything that did not change. The heavy LLM extraction step is memoized, so as long as a section's text and the model are unchanged, the cached result is reused.

The graph model

We build a property graph with three node types and three edge types:

Nodes

Meeting: one per meeting section, with its date and notes
Person: a canonical individual (organizer, participant, or task assignee)
Task: an action item decided in a meeting

Edges

ATTENDED: Person → Meeting, carrying an is_organizer flag
DECIDED: Meeting → Task
ASSIGNED_TO: Person → Task

To learn more about property graphs, see CocoIndex's Property Graph Targets documentation.

Define the pipeline

CocoIndex v1 is declarative and Python-native: you write plain async Python and the Rust engine handles incremental processing. The whole example lives in one main.py, wrapped in a coco.App:

import cocoindex as coco
from cocoindex.connectors import google_drive, neo4j
from cocoindex.ops.entity_resolution import ResolvedEntities, resolve_entities
from cocoindex.ops.entity_resolution.llm_resolver import LlmPairResolver
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.id import IdGenerator

app = coco.App(coco.AppConfig(name="MeetingNotesGraphNeo4j"), app_main)

Shared resources

The Neo4j connection, the two LLM models, and the embedder are provided once in the app lifespan and retrieved anywhere with coco.use_context. The neo4j.ConnectionFactory reads standard connection settings from the environment, defaulting to a local Neo4j:

KG_DB = coco.ContextKey[neo4j.ConnectionFactory]("kg_db")
LLM_MODEL = coco.ContextKey[str]("llm_model", detect_change=True)
RESOLUTION_LLM_MODEL = coco.ContextKey[str]("resolution_llm_model", detect_change=True)
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    builder.provide(
        KG_DB,
        neo4j.ConnectionFactory(
            uri=os.environ.get("NEO4J_URI", "bolt://localhost:7687"),
            auth=(
                os.environ.get("NEO4J_USER", "neo4j"),
                os.environ.get("NEO4J_PASSWORD", "cocoindex"),
            ),
            database=os.environ.get("NEO4J_DATABASE", "neo4j"),
        ),
    )
    builder.provide(LLM_MODEL, os.environ.get("LLM_MODEL", "openai/gpt-5-mini"))
    builder.provide(
        RESOLUTION_LLM_MODEL,
        os.environ.get("RESOLUTION_LLM_MODEL", "openai/gpt-5-mini"),
    )
    builder.provide(
        EMBEDDER,
        SentenceTransformerEmbedder("Snowflake/snowflake-arctic-embed-xs"),
    )
    yield

detect_change=True means swapping a model or the embedder re-runs only the affected steps.

Node and edge schemas

Node rows are plain dataclasses. The primary key of each becomes the node's identity in Neo4j, so re-runs reconcile existing nodes in place instead of duplicating them:

@dataclass
class Meeting:
    id: int
    note_file: str
    time: datetime.date
    note: str

@dataclass
class Person:
    name: str  # canonical

@dataclass
class Task:
    description: str

@dataclass
class AttendedRel:
    is_organizer: bool

AttendedRel is the edge payload for ATTENDED. DECIDED and ASSIGNED_TO carry no payload, so they are declared without a record. Relation primary keys are auto-derived by the Neo4j connector from (from_id, to_id), giving exactly one edge per endpoint pair.

Each dataclass maps directly to the graph: a record becomes a node keyed by its primary key, and a declared relation becomes an edge between two endpoint keys.

The extraction schema

The LLM extraction target is a set of Pydantic models. instructor uses these as the response schema, so the model returns structured data that matches the Python types instead of free-form text:

class ExtractedPerson(pydantic.BaseModel):
    name: str = pydantic.Field(
        description="Full name of the person, as written in the note."
    )

class ExtractedTask(pydantic.BaseModel):
    description: str = pydantic.Field(
        description="Concise, standalone description of the task or action item."
    )
    assigned_to: list[ExtractedPerson] = pydantic.Field(default_factory=list)

class ExtractedMeeting(pydantic.BaseModel):
    time: datetime.date
    note: str
    organizer: ExtractedPerson
    participants: list[ExtractedPerson] = pydantic.Field(default_factory=list)
    tasks: list[ExtractedTask] = pydantic.Field(default_factory=list)

Split, then extract

A single note file often holds several meetings. _split_meetings breaks the text at Markdown headings (# or ##) so each section becomes one meeting:

_HEADING_RE = re.compile(r"\n\n##?\s+")

def _split_meetings(text: str) -> list[str]:
    parts = _HEADING_RE.split("\n\n" + text)
    return [p.strip() for p in parts if p.strip()]

Each section goes to extract_meeting, which calls the model through LiteLLM and instructor. The model comes from the LLM_MODEL context (default openai/gpt-5-mini), and memo=True caches the result so an unchanged section is never re-extracted:

@coco.fn(memo=True)
async def extract_meeting(section_text: str) -> ExtractedMeeting:
    client = instructor.from_litellm(litellm.acompletion, mode=instructor.Mode.JSON)
    result = await client.chat.completions.create(
        model=coco.use_context(LLM_MODEL),
        response_model=ExtractedMeeting,
        messages=[
            {"role": "system", "content": EXTRACT_PROMPT},
            {"role": "user", "content": section_text},
        ],
    )
    return ExtractedMeeting.model_validate(result.model_dump())

Three phases

The pipeline runs in three phases. Phase 1 extracts each file and declares the nodes and edges it can build on its own. Phase 2 resolves person names across the whole corpus. Phase 3 declares the canonical Person nodes and the edges that touch people.

Phase 1: per-file extraction

process_file reads a Drive file, splits it, and for each meeting declares a Meeting node and its Task nodes plus DECIDED edges. Meeting IDs come from an IdGenerator keyed on the meeting date. Person names are not resolved yet, so the raw names are collected into a MeetingExtraction and returned for later phases:

@coco.fn(memo=True)
async def process_file(
    file: google_drive.DriveFile,
    meeting_table: neo4j.TableTarget[Meeting],
    task_table: neo4j.TableTarget[Task],
    decided_rel: neo4j.RelationTarget[Any],
) -> list[MeetingExtraction]:
    text = await file.read_text()
    note_file = file.file_path.path.as_posix()
    id_generator = IdGenerator()
    extractions = []
    for section in _split_meetings(text):
        extracted = await extract_meeting(section)
        meeting_id = await id_generator.next_id(extracted.time)

        meeting_table.declare_record(
            row=Meeting(
                id=meeting_id,
                note_file=note_file,
                time=extracted.time,
                note=extracted.note,
            )
        )

        for task in extracted.tasks:
            task_table.declare_record(row=Task(description=task.description))
            decided_rel.declare_relation(from_id=meeting_id, to_id=task.description)

        extractions.append(
            MeetingExtraction(
                meeting_id=meeting_id,
                organizer=extracted.organizer.name,
                participants=[p.name for p in extracted.participants],
                task_assignees=[
                    (t.description, [a.name for a in t.assigned_to])
                    for t in extracted.tasks
                ],
            )
        )
    return extractions

Phase 2: person entity resolution

The same person shows up under different names across files: "Sarah" in one note, "Sarah Chen" in another. If you create a Person node per raw string, the graph fragments and relationship queries miss edges.

_resolve_persons deduplicates raw names with embedding-based entity resolution. Candidate matches are found by embedding similarity, then confirmed pairwise by an LLM. The result maps every raw name to a canonical one:

@coco.fn(memo=True)
async def _resolve_persons(raw_persons: set[str]) -> ResolvedEntities:
    return await resolve_entities(
        entities=raw_persons,
        embedder=coco.use_context(EMBEDDER),
        resolve_pair=LlmPairResolver(model=coco.use_context(RESOLUTION_LLM_MODEL)),
    )

ResolvedEntities exposes .canonicals() for the deduplicated set and .canonical_of(raw_name) to look up which canonical a raw name folds into. So "Sarah" and "Sarah Chen" collapse to a single Person node.

Phase 3: canonical people and their edges

With names resolved, create_person_relations declares one Person node per canonical name, then the person-touching edges. ATTENDED aggregates organizer and participants so a person listed as both gets a single edge with is_organizer=True. ASSIGNED_TO dedupes per canonical person and task:

@coco.fn
async def create_person_relations(
    meetings: list[MeetingExtraction],
    persons: ResolvedEntities,
    person_table: neo4j.TableTarget[Person],
    attended_rel: neo4j.RelationTarget[Any],
    assigned_rel: neo4j.RelationTarget[Any],
) -> None:
    for canonical_name in persons.canonicals():
        person_table.declare_record(row=Person(name=canonical_name))

    for m in meetings:
        attendees: dict[str, bool] = {persons.canonical_of(m.organizer): True}
        for p in m.participants:
            attendees.setdefault(persons.canonical_of(p), False)

        for canonical, is_organizer in attendees.items():
            attended_rel.declare_relation(
                from_id=canonical,
                to_id=m.meeting_id,
                record=AttendedRel(is_organizer=is_organizer),
            )

        for task_desc, assignees in m.task_assignees:
            seen: set[str] = set()
            for raw in assignees:
                canonical = persons.canonical_of(raw)
                if canonical in seen:
                    continue
                seen.add(canonical)
                assigned_rel.declare_relation(from_id=canonical, to_id=task_desc)

Wiring it together

app_main mounts the three node tables and three relation targets, then runs the phases. Node tables take a TableSchema and a primary key; relations mount without a schema so the connector derives their PK from the endpoints. The Google Drive source is iterated with async for, and each file is processed under its own component path so change detection is per-file:

@coco.fn
async def app_main() -> None:
    meeting_table = await neo4j.mount_table_target(
        KG_DB, "Meeting",
        await neo4j.TableSchema.from_class(Meeting, primary_key="id"),
        primary_key="id",
    )
    person_table = await neo4j.mount_table_target(
        KG_DB, "Person",
        await neo4j.TableSchema.from_class(Person, primary_key="name"),
        primary_key="name",
    )
    task_table = await neo4j.mount_table_target(
        KG_DB, "Task",
        await neo4j.TableSchema.from_class(Task, primary_key="description"),
        primary_key="description",
    )

    attended_rel = await neo4j.mount_relation_target(
        KG_DB, "ATTENDED", person_table, meeting_table
    )
    decided_rel = await neo4j.mount_relation_target(
        KG_DB, "DECIDED", meeting_table, task_table
    )
    assigned_rel = await neo4j.mount_relation_target(
        KG_DB, "ASSIGNED_TO", person_table, task_table
    )

    # Phase 1: per-file extraction
    source = google_drive.GoogleDriveSource(
        service_account_credential_path=os.environ["GOOGLE_SERVICE_ACCOUNT_CREDENTIAL"],
        root_folder_ids=[
            f.strip()
            for f in os.environ["GOOGLE_DRIVE_ROOT_FOLDER_IDS"].split(",")
            if f.strip()
        ],
    )
    file_coros = []
    async for path_key, file in source.items():
        file_coros.append(
            coco.use_mount(
                coco.component_subpath("file", path_key),
                process_file, file, meeting_table, task_table, decided_rel,
            )
        )
    per_file = list(await asyncio.gather(*file_coros))
    all_meetings = [m for ms in per_file for m in ms]

    # Phase 2: person entity resolution
    raw_persons: set[str] = set()
    for m in all_meetings:
        raw_persons.add(m.organizer)
        raw_persons.update(m.participants)
        for _task_desc, assignees in m.task_assignees:
            raw_persons.update(assignees)
    persons = await coco.use_mount(
        coco.component_subpath("resolve_persons"), _resolve_persons, raw_persons
    )

    # Phase 3: declare Person nodes + person-touching relations
    await coco.mount(
        coco.component_subpath("person_relations"),
        create_person_relations,
        all_meetings, persons, person_table, attended_rel, assigned_rel,
    )

When a component finishes, CocoIndex compares its declared nodes and edges against the previous run and applies only the differences: new items are inserted, changed ones updated, and items whose source file disappeared are removed. Nothing that stayed the same is rewritten.

Run

Prerequisites

Neo4j running locally. Default connection bolt://localhost:7687, browser at http://localhost:7474, username neo4j, password cocoindex.
An OpenAI API key (OPENAI_API_KEY). The default models are openai/gpt-5-mini for both extraction and resolution.
Google Drive service-account credentials. Create a Google Cloud service account, download its JSON key, share the source folders with the service-account email, and collect the folder IDs. See Setup for Google Drive.

Set the environment variables:

export OPENAI_API_KEY=sk-...
export GOOGLE_SERVICE_ACCOUNT_CREDENTIAL=/absolute/path/to/service_account.json
export GOOGLE_DRIVE_ROOT_FOLDER_IDS=folderId1,folderId2

GOOGLE_DRIVE_ROOT_FOLDER_IDS accepts a comma-separated list.

Build the graph

pip install -e .
cocoindex update main

Use cocoindex update -L main for live mode, which keeps watching Drive and updates the graph as notes change.

Browse the graph

Open Neo4j Browser at http://localhost:7474 and run Cypher:

// All relationships
MATCH p=()-->() RETURN p

// Who attended which meetings (including organizer)
MATCH (p:Person)-[:ATTENDED]->(m:Meeting)
RETURN p, m

// Tasks decided in meetings
MATCH (m:Meeting)-[:DECIDED]->(t:Task)
RETURN m, t

// Task assignments
MATCH (p:Person)-[:ASSIGNED_TO]->(t:Task)
RETURN p, t

Where else this applies

The same pattern (ingest → split → extract → resolve entities → build graph) fits any text-heavy domain where the entities and their relationships matter more than the raw text:

Research papers: build a graph of concepts, authors, and citations across a repository, and keep it current as papers are added or revised.
Support tickets: link issues, solutions, customers, and recurring patterns across a large, frequently edited ticket base.
Compliance documents: extract requirements from policies and trace how a policy change cascades through related obligations.
Competitive intelligence: connect companies, products, and market moves extracted from public filings and news.

Summary

This example turns unstructured meeting notes into a queryable Neo4j graph: an LLM extracts structure, embedding-based entity resolution keeps one node per real person, and CocoIndex's incremental processing keeps the graph in sync as notes change without reprocessing the whole corpus. The same source → split → extract → resolve → graph template carries over to any domain where relationships are the thing you want to query.

Support CocoIndex ❤️

If this example was helpful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub.

Your stars help us grow the community, stay motivated, and keep shipping better tools for real-time data ingestion and transformation.

What HackerNews is talking about: trending topics with an LLM

Linghua Jin — Tue, 02 Dec 2025 00:00:00 GMT

The tech community's attention is hiding in plain sight, scattered across story titles and thousands of comments. This post builds a pipeline that reads recent HackerNews threads and their comments, asks an LLM to pull the topics out of every message, and ranks what's trending in Postgres, rewritten for CocoIndex v1.

Two pieces do the work: the LLM extraction that turns free-form text into normalized topics, and a SQL score that turns those topics into a leaderboard. This post focuses on both. The source here is a custom one (two async functions over the Algolia HN API); the custom-source walkthrough covers that side in detail, so we won't repeat it here.

The full example is in the CocoIndex v1 repo.

How it works

Each recent story becomes its own processing component. The component fetches the story and its comments, extracts topics from each message with an LLM, and declares rows into two Postgres tables:

hn_messages: one row per message (the story and each comment), for full-text search
hn_topics: one row per topic mention, keyed on (topic, message_id)

Because the same topic mentioned in many messages becomes many hn_topics rows, ranking becomes a simple SQL aggregate. Re-running catches up on new stories, and because per-message extraction is memoized, only content CocoIndex hasn't seen hits the LLM.

The topic extraction is the product

The interesting part is not calling an LLM; it's getting topics back in a shape you can rank. Raw model output would give you "postgres" in one comment and "PostgreSQL" in another, or a whole phrase like "books for autistic kids" as a single blob. Those never aggregate.

The fix lives entirely in the prompt, expressed as the description of a Pydantic field. The model normalizes three ways: it splits combined phrases into parts, keeps proper nouns canonical, and emits aliases for the same thing:

class TopicsResponse(BaseModel):
    """Response containing a list of extracted topics."""

    topics: list[str] = Field(
        description="""List of extracted topics.

Each topic can be a product name, technology, model, people, company name, business domain, etc.
Capitalize for proper nouns and acronyms only.
Use the form that is clear alone.

For topics that are a phrase combining multiple things, normalize into multiple topics. Examples:
- "books for autistic kids" -> "book", "autistic", "autistic kids"

When there're multiple common ways to refer to the same thing, use multiple topics. Examples:
- "John Kennedy", "JFK"
"""
    )

That normalization is what makes the trending ranking meaningful later. Get it wrong and "PostgreSQL", "postgres", and "psql" split their votes three ways.

Extraction itself is one CocoIndex function: text in, a list of topics out, through litellm so any provider works. memo=True caches the result, so an unchanged message is never sent to the model twice:

@coco.fn(memo=True)
async def extract_topics(text: str | None) -> list[str]:
    if not text or not text.strip():
        return []

    response = await acompletion(
        model=LLM_MODEL,
        messages=[
            {"role": "user", "content": f"Extract topics from the following text:\n\n{text[:4000]}"}
        ],
        response_format=TopicsResponse,
    )
    content = response.choices[0].message.content
    return TopicsResponse.model_validate_json(content).topics

Declare a row per message and per topic

The two target tables are plain dataclasses. A thread and each of its comments both become an HnMessage (distinguished by content_type), and every extracted topic becomes an HnTopic keyed on (topic, message_id):

@dataclass
class HnMessage:
    id: str
    thread_id: str
    content_type: str  # "thread" or "comment"
    author: str | None
    text: str | None
    url: str | None
    created_at: datetime | None

@dataclass
class HnTopic:
    topic: str
    message_id: str
    thread_id: str
    content_type: str
    created_at: datetime | None

process_thread fetches one story, extracts topics for the story and every comment, and declares the rows. You declare what should exist; CocoIndex works out the inserts, updates, and deletes:

@coco.fn
async def process_thread(thread_id: str, targets: TableTargets) -> None:
    async with aiohttp.ClientSession() as session:
        thread = await fetch_thread(session, thread_id)
    thread_topics = await extract_topics(thread.text)

    targets.messages.declare_row(row=HnMessage(
        id=thread.id, thread_id=thread.id, content_type="thread",
        author=thread.author, text=thread.text, url=thread.url, created_at=thread.created_at,
    ))
    for topic in thread_topics:
        targets.topics.declare_row(row=HnTopic(
            topic=topic, message_id=thread.id, thread_id=thread.id,
            content_type="thread", created_at=thread.created_at,
        ))

    for comment in thread.comments:
        comment_topics = await extract_topics(comment.text)
        targets.messages.declare_row(row=HnMessage(..., content_type="comment", ...))
        for topic in comment_topics:
            targets.topics.declare_row(row=HnTopic(..., content_type="comment", ...))

app_main mounts the two Postgres tables and fans out one process_thread per recent story with coco.mount_each. The two async fetch functions that make up the custom source are covered in the custom-source post.

Ranking what's trending

Trending is a SQL query over hn_topics. Each row contributes a weight to its topic's score: a thread-level mention counts for more than a comment-level one, because a topic in the story title is the subject, while a topic in a comment is supporting discussion. Group by topic, order by score:

SELECT
    topic,
    SUM(CASE WHEN content_type = 'thread' THEN 5 ELSE 1 END) AS score,
    MAX(created_at) AS latest_mention,
    COUNT(DISTINCT thread_id) AS thread_count
FROM coco_examples.hn_topics
GROUP BY topic
ORDER BY score DESC, latest_mention DESC
LIMIT 20

Weighting thread mentions 5 to 1 over comment mentions is what makes the ranking read as "what HackerNews is about" rather than "what people mentioned in passing." The same score, filtered to one topic and grouped by thread_id, ranks which threads are driving a given topic.

Run it

Install the example and build the index:

pip install -e .
cocoindex update main

This fetches the most recent stories, runs topic extraction on every message, and writes coco_examples.hn_messages and coco_examples.hn_topics. This source is a one-shot catch-up per run; re-run cocoindex update main anytime to pick up new stories, and only new messages hit the LLM.

Then explore from the command line, top trending topics or a single topic:

python main.py            # top 20 trending topics + interactive search
python main.py "rust"     # search content for one topic

What you can build on top

Once topics are structured and ranked, the pipeline is a foundation:

Trending dashboards. Serve the leaderboard and per-topic threads to a live page that shows what's hot and which discussions are driving it.
Sentiment-aware trends. Add a second extraction step for sentiment to track not just what's trending, but how people feel about it.
Contributor tracking. Every message stores its author, so you can find who drives conversation around a given topic.
A knowledge graph. Feed the topics into a graph linking companies, products, and people mentioned across HackerNews.

Each of these is a new transform or target on the same declared source, not a rewrite. The extraction and the ranking, the two pieces this post focused on, stay exactly as they are.

Support us ❤️

If this was useful, a star on GitHub helps more developers find CocoIndex and keeps the project growing.

CocoIndex Changelog 0.2.21 - 0.3.10

George He — Tue, 25 Nov 2025 00:00:00 GMT

CocoIndex is moving at insane speed! 🚀 This release is packed with some of our biggest upgrades yet, from automatic batching, to robust async execution, schema and type-system upgrades, and the long-awaited launch of Custom Sources.

CocoIndex is rapidly becoming the best framework for persistent-state–driven data processing, where it continuously transforms and updates the target data (e.g. AI context, feature stores, knowledge graphs and beyond) upon source change with zero manual orchestration.

We’re incredibly excited to keep building in the open with our community. Together, we’re redefining how AI systems maintain, evolve, and reason over long-lived state. Onward! 🎉

Core capability

Batching support for CocoIndex functions

CocoIndex now supports automatic batching for all CocoIndex functions. When embedding the CocoIndex codebase with sentence-transformers/all-MiniLM-L6-v2, batching delivered ~5× higher throughput (≈80% lower runtime) compared to processing items one-by-one.

Why it’s fast

Each call normally pays:

Fixed overhead (GPU kernel launches, Python↔C transitions, memory allocator work)
Data-dependent compute (FLOPs, bytes moved, tokens processed)

Batching amortizes the fixed cost and lets GPUs run larger, denser GEMMs with fewer data transfers, dramatically increasing utilization.

What’s new

CocoIndex introduces adaptive, knob-free batching:

Requests queue while the GPU is busy
As soon as a batch completes, all queued requests are flushed as the next batch
No timers, target batch sizes, or tuning: batching automatically adapts to workload traffic

Batching significantly enhances processing speed by amortizing fixed overhead across multiple items, enabling more efficient GPU operations, and reducing data transfer. CocoIndex simplifies this by offering automatic batching for several built-in functions and an easy batching=True parameter for custom functions.

You can read more about batching and the benchmark in the blog post.

Changes: #1229, #1230, #1232, #1233, #1236, #1238, #1261

Custom source support

We're thrilled to introduce Custom Sources in CocoIndex, a feature that lets you pull data from any system: APIs, databases, file systems, cloud storage, or other services. CocoIndex now ingests data incrementally, tracks changes efficiently, and integrates seamlessly into your workflows.

With Custom Sources, you're no longer limited by prebuilt connectors or targets. Use CocoIndex for anything, leveraging its robust incremental computing to continuously build fresh knowledge for AI.

A custom source defines how CocoIndex reads data from an external system.

Custom sources are defined by two components:

A source spec that configures the behavior and connection parameters for the source.

class CustomSource(cocoindex.op.SourceSpec):
    """
    Custom source for my external system.
    """
    param1: str
    param2: int | None = None

A source connector that handles the actual data reading operations. It provides the following required methods:
- create(): Create a connector instance from the source spec.
- list(): List all available data items. Return keys.
- get_value(): Get the full content for a specific data item by a given key.

@cocoindex.op.source_connector(
    spec_cls=CustomSource,
    key_type=DataKeyType,
    value_type=DataValueType
)
class CustomSourceConnector:
    @staticmethod
    async def create(spec: CustomSource) -> "CustomSourceConnector":
        """Initialize connection, authenticate, and return connector instance."""
        ...

    async def list(self, options: SourceReadOptions) -> AsyncIterator[PartialSourceRow[DataKeyType, DataValueType]]:
        """List available data items with optional metadata (ordinal, content)."""
        ...

    async def get_value(self, key: DataKeyType, options: SourceReadOptions) -> PartialSourceRowData[DataValueType]:
        """Retrieve full content for a specific data item."""
        ...

    def provides_ordinal(self) -> bool:
        """Optional: Return True if the source provides timestamps or version numbers."""
        return False

Check out the article on Bring your own data: Index any data with Custom Sources for more details. Get started and read the documentation now.

Changes: #1195, #1197, #1198, #1199, #1201

Execution robustness / debugability enhancement

Robust async runtime, better error propagation and observability for HTTP calls, function‑level timeouts and contextual execution in CocoIndex.

Runtime and async safety

The runtime now detects misuse of the sync API from async code and guards it explicitly. Sync entrypoints now safely handle calls from different event loops, avoiding deadlocks.
Async cancellation is now propagated correctly through CocoIndex's async contexts, so task cancellations (for example, from higher‑level orchestration or timeouts) reliably unwind work instead of silently hanging.
Async custom functions are fully supported and wired into the runtime so user‑defined async operations behave like first‑class citizens, including proper scheduling and error handling.

Changes: #1224, #1228, #1255

HTTP utility improvements

A new http::request() utility centralizes HTTP calls, providing:
- Better, more actionable error messages for network and protocol failures.
- Built‑in retry behavior so transient errors are handled consistently across integrations.
This utility is intended as the recommended path for HTTP interactions in sources, functions and sinks, improving both debuggability and resilience.

Changes: #1235

Function timeout support

User‑configured operations (sources, functions, etc.) now support a function timeout to prevent unbounded or long‑running executions.
The timeout is fully wired through the batching executor so batched executions respect per‑function limits instead of letting a single slow item stall the entire batch.
Timeouts integrate with the new cancellation propagation so timed‑out work is properly cancelled and cleaned up, rather than left in an indeterminate state.

Changes: #1241, #1284

Clear context in error messages

CocoIndex now attaches clear context information (like source, function and target names) with error messages.

Changes: #1275

Schema & type system

This set of changes improves how CocoIndex handles JSON Schemas, type metadata, and schema merging, with a focus on configurability, correctness, and compatibility across tools and providers.

Collector schema alignment

Collectors now automatically merge and align schemas when collect() is called multiple times with different shapes, producing a unified schema instead of requiring manual pre‑alignment.
The new collector implementation has been simplified in a follow‑up change while preserving the automatic merge‑and‑align behavior for easier maintenance.

Changes: #1153, #1265

Improved schema support

additionalProperties is now configurable via supports_additional_properties on ToJsonSchemaOptions, so providers that support it (OpenAI, Anthropic, Bedrock, Ollama, etc.) still get strict schemas while Gemini (AI Studio, Vertex) receives schemas without this keyword, removing the previous Gemini‑specific strip workaround.
Forward‑referenced field types are now resolved before export, improving compatibility with BAML and other tooling that expects fully concrete types, and type descriptions are correctly encoded so documentation survives serialization.
A previously incorrect assertion condition in the type/schema logic has been fixed to avoid spurious failures in valid configurations.

Changes: #1254, #1272, #1294, #1312

More accurate logic change detection

CocoIndex now detects logic changes with much finer granularity, so incremental runs only reprocess data that is truly affected.

New per‑source SourceLogicFingerprint and field‑level FieldDefFingerprint track exactly which sources and operations each export/field depends on.
Execution and indexing now use these fingerprints (including target schema IDs) to avoid unnecessary recomputation while still catching real logic changes.

Changes: #1292

Building blocks

Builtin sources

File size management across sources

CocoIndex’s built-in file-based sources now support consistent file size limits to protect pipelines from unexpectedly large inputs.

max_file_size is now supported on Azure Blob, Amazon S3, LocalFile, and Google Drive sources.
Files larger than the configured max_file_size are treated as non-existent in both list() and get_value() calls, letting you cap resource usage and skip oversized blobs uniformly across all these sources.

Changes: #1257, #1259, #1260, #1269

GoogleDrive source enhancements

GoogleDrive now supports included_patterns and excluded_patterns, using glob-style patterns to decide exactly which Drive files to index. This makes it easy to focus on specific file types or paths (for example, only *.md docs) while excluding noisy locations like temp folders or logs.

Changes: #1263

S3 configuration enhancements

Amazon S3 sources gain a force_path_style configuration flag, improving compatibility with S3-compatible object stores and strict networking setups that require path-style URLs instead of virtual-hosted–style.

Changes: #1290

S3 event notifications via Redis queue

CocoIndex can now consume S3 event notifications via a Redis queue, enabling push-based, near–real-time updates from S3 buckets into your indexing flows. This reduces the need for frequent full scans or tight polling loops while still keeping downstream indexes fresh.

Changes: #1189

Postgres source stability

The Postgres source’s change‑listening connection is now more stable, reducing disconnects and flaky behavior when capturing updates via LISTEN/NOTIFY.
This makes long‑running live update pipelines against Postgres more resilient, especially under network hiccups or database restarts.

Changes: #1264

UTF-16/UTF-32 file support

File decoding now auto‑detects BOMs and supports UTF‑16 and UTF‑32 in addition to UTF‑8, so mixed‑encoding corpora can be indexed without pre‑normalizing everything.
The internal bytes_to_string() helper was optimized to return a Cow instead of always allocating, avoiding unnecessary copies when data is already valid UTF‑8.

Changes: #1185, #1186

Builtin functions

Performance optimization for SentenceTransformerEmbed

SentenceTransformerEmbed now sorts inputs by length before batching, which reduces padding, improves GPU utilization, and lowers per-batch compute cost for sentence-transformer models. This change is transparent to users but can significantly improve throughput for large embedding workloads.

Changes: #1245

Ollama embedding support

The Ollama embedding integration now correctly parses the nested embeddings array returned by the /api/embed endpoint. This fixes shape/format mismatches so multi-input embedding calls against Ollama work reliably in CocoIndex.

Changes: #1227

Operations & monitoring

CocoIndex now exposes basic server health checks and clearer runtime progress reporting.

HTTP server & health checks

A new /healthz endpoint on the CocoIndex HTTP server returns a lightweight JSON payload with status and version, making it easy to plug the server into Kubernetes, load balancers, and external monitoring.
Documentation has been added for the HTTP server, including how to start it via CLI or Python and how to wire it up with CocoInsight or custom frontends.

Changes: #1270, #1271

Statistics & progress reporting

Processing commands now report elapsed time, so you can quickly see how long an update or run took.
Progress reporting has been reworked: there is a consolidated, clearer progress bar, continuous stats updates during batch operations, and better handling of errors so counts and statuses remain accurate throughout the run.

Changes: #1204, #1223, #1231, #1240, #1247

CLI simplification

CocoIndex simplifies the CLI around flow setup by making setup the default and deprecating the extra flag.

The --setup flag on cocoindex update and cocoindex server is now enabled by default and marked as deprecated, since these commands will automatically perform any required setup before running.
When --setup is used explicitly, the CLI shows a deprecation warning, and docs have been updated to remove references to manually passing this flag in common workflows.

Changes: #1212, #1237

Claude skills integration

CocoIndex now ships a dedicated Claude Code skill so developers can work with CocoIndex flows directly from their IDE-like Claude environment.

A CocoIndex Claude skill has been introduced and moved into its own cocoindex-claude repository, giving Claude richer, structured knowledge of CocoIndex concepts, data types, and workflows.
Documentation was added showing how to install and enable the CocoIndex skill in Claude Code, so Claude can help scaffold flows, edit pipelines, and navigate CocoIndex projects more effectively.

New tutorials

Index PDF elements

PDFs are rich with both text and visual content, from descriptive paragraphs to illustrations and tables. This example builds an end-to-end flow that parses, embeds, and indexes both, with full traceability to the original page.

Check out the blog for more details.

Extracting intake forms with BAML and CocoIndex

This tutorial shows how to use BAML together with CocoIndex to build a data pipeline that extracts structured patient information from PDF intake forms. The BAML definitions describe the desired output schema and prompt logic, while CocoIndex orchestrates file input, transformation, and incremental indexing.

Check out the blog for more details.

Special edition

CocoIndex recently hit 3K GitHub stars and became the #1 trending Rust repo globally!

🎉 We shared an article reflecting on why we built CocoIndex, the journey so far, and some fresh thoughts:

Data for AI (CocoIndex Blog)

For a look back at when we reached 1K stars, see:

CocoIndex at 1K (Reflection)

We're just getting started and can't wait to see what comes next on our way to 5K stars!

Thanks to the community

Welcome new contributors to the CocoIndex community! We are so excited to have you!

@Haleshot

Thanks @Haleshot for making additionalProperties configurable in JSON schemas in #1312, improving compatibility with Gemini while keeping richer schema support for other providers, and for improving the README note for better GitHub rendering in #1288, making the docs clearer and more readable.

@Gohlub

Thanks @Gohlub for adding with_context to user‑configured operations in #1275, enabling more powerful context‑aware flows, and for adding included_patterns and excluded_patterns to the GoogleDrive source in #1263, making it easier to control which files are indexed.

@dcbark01

Thanks @dcbark01 for adding the force_path_style parameter to S3 config in #1290, improving support for S3‑compatible storage.

@prabhath004

Thanks @prabhath004 for adding max_file_size support to the AzureBlob source in #1259, LocalFile source in #1260, and AmazonS3 source in #1257, making large‑file handling safer and more configurable.

@xuzijan

Thanks @xuzijan for adding a README with CocoIndex project examples in #1273, giving users concrete starting points for building projects.

@AdwitaSingh1711

Thanks @AdwitaSingh1711 for adding function timeout support in #1241, preventing long‑running functions from blocking pipelines.

@skalwaghe-56

Thanks @skalwaghe-56 for teaching the collector to automatically merge and align multiple collect() calls with different schemas in #1153, simplifying complex aggregation flows.

@ansu86d

Thanks @ansu86d for enhancing UpdateStats with a progress bar and better error handling in #1223, making batch updates more transparent.

@samojavo

Thanks @samojavo for guarding sync APIs inside async contexts in #1224, improving runtime safety.

@CAPsMANyo

Thanks @CAPsMANyo for correctly parsing nested embeddings arrays from the Ollama /api/embed endpoint in #1227, ensuring robust embedding ingestion.

@belloibrahv

Thanks @belloibrahv for adding Redis queue support for S3 event notifications in #1189 and improving CLI --help/Markdown docstring formatting in #1210, making event‑driven updates and CLI usage smoother.

@GooglyBlox

Thanks @GooglyBlox for deprecating the setup flag in #1212, simplifying the CLI surface and guiding users toward the preferred setup flow.

Extract HackerNews into Postgres with a custom source

Linghua Jin — Tue, 25 Nov 2025 00:00:00 GMT

Custom sources are one of the most useful ideas in CocoIndex. They let you turn any API, internal or external, into a first-class incremental data stream that the framework can diff, track, and sync.

Think of it as React for data flows: you declare what your target should look like as a function of the source, and CocoIndex handles incremental updates, state persistence, and downstream sync. In CocoIndex v1 this needs no special connector API. A custom source is ordinary async Python that fetches data, plus processing components that declare the rows that should exist downstream.

This post walks through a custom source for HackerNews built that way. It fetches recent stories and their nested comments through the Algolia HN API, extracts topics from every message with an LLM, and keeps two Postgres tables in sync for search and analytics.

The project is open-sourced in the CocoIndex v1 repo.

Why use a custom source?

In many scenarios, pipelines don't just read from clean tables. They depend on:

Internal REST services
Partner APIs
Legacy systems
Non-standard data models that don't fit traditional connectors

CocoIndex makes these integrations declarative and incremental without a plugin interface. You fetch data with plain async def functions, each item runs as its own processing component, and the component declares the target rows it owns. The engine takes it from there: change detection, syncing to the target, and cleanup when an item disappears from the source.

Project walkthrough: building a HackerNews index

Goals

Call the HackerNews search API
Fetch nested comments for each story
Extract topics from every message with an LLM
Store messages and topics in Postgres
Search the index by topic

CocoIndex handles change detection, idempotency, and state sync automatically.

Overview

The pipeline consists of three parts:

Two async functions over the Algolia HN API
- fetch_thread_list lists recent story IDs
- fetch_thread pulls one story with its full comment tree
One processing component per thread
- process_thread extracts topics from the story and every comment
- Declares rows into two Postgres tables (hn_messages, hn_topics)
Query utilities
- Rank trending topics in SQL
- Search messages by topic

Each cocoindex update catches up on the latest stories and only does the work it hasn't seen before.

Prerequisites

Postgres running locally. The example repo ships a compose file: docker compose -f dev/postgres.yaml up -d.
An LLM API key. The default model is gemini/gemini-2.5-flash through litellm; set GEMINI_API_KEY, or set LLM_MODEL to any litellm model id and the matching provider key.

Define the data models

Everything the pipeline handles is a plain dataclass. Thread and Comment model what we scrape from the API:

@dataclass
class Comment:
    id: str
    author: str | None
    text: str | None
    created_at: datetime | None

@dataclass
class Thread:
    id: str
    author: str | None
    text: str
    url: str | None
    created_at: datetime | None
    comments: list[Comment]

Each source item is one thread: a story plus all of its comments, flattened out of HackerNews' nested reply tree into a single list.

HnMessage and HnTopic are the schemas of the two target tables:

@dataclass
class HnMessage:
    """Schema for hn_messages table."""

    id: str
    thread_id: str
    content_type: str
    author: str | None
    text: str | None
    url: str | None
    created_at: datetime | None

@dataclass
class HnTopic:
    """Schema for hn_topics table."""

    topic: str
    message_id: str
    thread_id: str
    content_type: str
    created_at: datetime | None

A thread and each of its comments both become an HnMessage row, distinguished by content_type. Every extracted topic becomes an HnTopic row keyed on (topic, message_id), so the same topic mentioned in many messages shows up many times, which is exactly what makes ranking meaningful.

Primary keys are the identity of a row, the same role the key type played for a v0 custom source: they must be stable over time so re-runs reconcile rows in place instead of duplicating them.

Fetch from the HackerNews API

The source itself is two small async functions over the Algolia HN API. No decorators, no connector class.

Listing recent threads

fetch_thread_list returns the most recent story IDs. CocoIndex uses this list to know which threads exist right now:

async def fetch_thread_list(
    session: aiohttp.ClientSession, max_results: int = MAX_THREADS
) -> list[str]:
    """Fetch list of recent thread IDs from HackerNews."""
    search_url = "https://hn.algolia.com/api/v1/search_by_date"
    params: dict[str, str | int] = {"tags": "story", "hitsPerPage": max_results}

    async with session.get(search_url, params=params) as response:
        response.raise_for_status()
        data = await response.json()
        thread_ids = [
            hit["objectID"] for hit in data.get("hits", []) if hit.get("objectID")
        ]
        return thread_ids

Fetching a full thread

fetch_thread pulls one story with all its comments and parses the raw JSON into the structured Thread. The inner parse_comments walks the reply tree recursively and flattens it; the story title and body are combined into the thread text:

async def fetch_thread(session: aiohttp.ClientSession, thread_id: str) -> Thread:
    """Fetch a single thread with all its comments."""
    item_url = f"https://hn.algolia.com/api/v1/items/{thread_id}"

    async with session.get(item_url) as response:
        response.raise_for_status()
        data = await response.json()

        comments: list[Comment] = []

        # Parse comments recursively
        def parse_comments(parent: dict[str, Any]) -> None:
            for child in parent.get("children", []):
                if comment_id := child.get("id"):
                    ctime = child.get("created_at")
                    comments.append(
                        Comment(
                            id=str(comment_id),
                            author=child.get("author"),
                            text=child.get("text"),
                            created_at=datetime.fromisoformat(ctime) if ctime else None,
                        )
                    )
                parse_comments(child)

        parse_comments(data)

        ctime = data.get("created_at")
        text = data.get("title", "")
        if more_text := data.get("text"):
            text += "\n\n" + more_text

        return Thread(
            id=thread_id,
            author=data.get("author"),
            text=text,
            url=data.get("url"),
            created_at=datetime.fromisoformat(ctime) if ctime else None,
            comments=comments,
        )

These are ordinary async def functions. Any HTTP API, message queue, or third-party SDK can be a source the same way: fetch the data in plain Python and hand it to the pipeline.

Extract topics with an LLM

The core transformation is a single CocoIndex function: text in, a list of topics out. The response schema is a small Pydantic model, so the LLM returns structured data instead of free-form text:

class TopicsResponse(BaseModel):
    """Response containing a list of extracted topics."""

    topics: list[str] = Field(
        description="""List of extracted topics.

Each topic can be a product name, technology, model, people, company name, business domain, etc.
Capitalize for proper nouns and acronyms only.
Use the form that is clear alone.
..."""
    )

@coco.fn
async def extract_topics(text: str | None) -> list[str]:
    """Extract topics from text using LLM."""
    if not text or not text.strip():
        return []

    response = await acompletion(
        model=LLM_MODEL,
        messages=[
            {
                "role": "user",
                "content": f"Extract topics from the following text:\n\n{text[:4000]}",
            }
        ],
        response_format=TopicsResponse,
    )

    content = response.choices[0].message.content
    return TopicsResponse.model_validate_json(content).topics

The prompt in TopicsResponse does the heavy lifting: it normalizes phrases into separate topics ("books for autistic kids" → "book", "autistic", "autistic kids"), keeps proper nouns canonical ("PostgreSQL", "Claude"), and emits aliases for the same thing ("JFK", "John Kennedy").

Process each thread

Each thread runs as its own processing component. process_thread fetches the thread, extracts topics for the story and every comment, and declares the resulting rows:

@coco.fn
async def process_thread(
    thread_id: str,
    targets: TableTargets,
) -> None:
    """Fetch and process a single thread and its comments."""
    async with aiohttp.ClientSession() as session:
        thread = await fetch_thread(session, thread_id)
    thread_topics = await extract_topics(thread.text)

    # Declare thread message row
    targets.messages.declare_row(
        row=HnMessage(
            id=thread.id,
            thread_id=thread.id,
            content_type="thread",
            author=thread.author,
            text=thread.text,
            url=thread.url,
            created_at=thread.created_at,
        ),
    )
    # Declare thread topic rows
    for topic in thread_topics:
        targets.topics.declare_row(
            row=HnTopic(
                topic=topic,
                message_id=thread.id,
                thread_id=thread.id,
                content_type="thread",
                created_at=thread.created_at,
            ),
        )
    # Process comments
    for comment in thread.comments:
        comment_topics = await extract_topics(comment.text)

        # Declare comment message row
        targets.messages.declare_row(
            row=HnMessage(
                id=comment.id,
                thread_id=thread.id,
                content_type="comment",
                author=comment.author,
                text=comment.text,
                url="",
                created_at=comment.created_at,
            ),
        )
        # Declare comment topic rows
        for topic in comment_topics:
            targets.topics.declare_row(
                row=HnTopic(
                    topic=topic,
                    message_id=comment.id,
                    thread_id=thread.id,
                    content_type="comment",
                    created_at=comment.created_at,
                ),
            )

You declare what rows should exist; you don't write inserts or deletes. When the component finishes, CocoIndex diffs the declared rows against the previous run at the same component path and applies only the creates, updates, and deletes needed. If a thread drops out of the feed, the rows it owned are cleaned up automatically.

Why a component per thread? A processing component groups one thread's work together with the target rows it owns. Each component runs independently and in parallel, and its rows are committed to Postgres as soon as that thread is done, without waiting for the rest of the batch.

Wire up the app

The Postgres pool is provided once in the app lifespan and referenced by a context key:

PG_DB = coco.ContextKey[asyncpg.Pool]("hn_db")

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    builder.settings.db_path = pathlib.Path("./cocoindex.db")
    async with asyncpg.create_pool(DATABASE_URL) as pool:
        builder.provide(PG_DB, pool)
        yield

app_main mounts the two table targets, fetches the recent thread IDs, and fans out one process_thread component per thread with coco.mount_each:

@dataclass
class TableTargets:
    """Container for table targets."""

    messages: postgres.TableTarget[HnMessage]
    topics: postgres.TableTarget[HnTopic]

@coco.fn
async def app_main() -> None:
    """Main pipeline function."""
    # Set up table targets
    messages_table = await postgres.mount_table_target(
        PG_DB,
        table_name="hn_messages",
        table_schema=await postgres.TableSchema.from_class(
            HnMessage, primary_key=["id"]
        ),
        pg_schema_name="coco_examples",
    )
    topics_table = await postgres.mount_table_target(
        PG_DB,
        table_name="hn_topics",
        table_schema=await postgres.TableSchema.from_class(
            HnTopic, primary_key=["topic", "message_id"]
        ),
        pg_schema_name="coco_examples",
    )
    targets = TableTargets(messages=messages_table, topics=topics_table)

    # Fetch thread IDs from HackerNews
    async with aiohttp.ClientSession() as session:
        thread_ids = await fetch_thread_list(session)

    # Process threads (each component fetches its own thread data)
    await coco.mount_each(process_thread, ((tid, tid) for tid in thread_ids), targets)

app = coco.App(
    coco.AppConfig(name="HNTrendingTopics"),
    app_main,
)

mount_each takes one (component_key, *args) tuple per item, so each thread gets a stable component path keyed on its ID. The TableSchema.from_class calls derive the SQL columns straight from the dataclasses. That's the whole app: plain Python on top, the Rust engine handling incremental sync underneath.

Query and search the index

The index is two plain Postgres tables, so you can query them with any library or framework. The example ships two query helpers over asyncpg. Trending topics are ranked in SQL, weighting a thread-level mention higher than a comment-level one:

SELECT
    topic,
    SUM(CASE WHEN content_type = 'thread' THEN 5 ELSE 1 END) AS score,
    MAX(created_at) AS latest_mention,
    COUNT(DISTINCT thread_id) AS thread_count
FROM coco_examples.hn_topics
GROUP BY topic
ORDER BY score DESC, latest_mention DESC
LIMIT 20

And topic search joins the two tables to return the underlying messages:

SELECT m.id, m.thread_id, m.author, m.content_type, m.text, m.created_at, t.topic
FROM coco_examples.hn_topics t
JOIN coco_examples.hn_messages m ON t.message_id = m.id
WHERE LOWER(t.topic) LIKE LOWER($1)
ORDER BY m.created_at DESC

Run it

Install the project in editable mode from the example directory:

pip install -e .

Then build the index:

cocoindex update main

This fetches the most recent MAX_THREADS (default 10) stories, runs LLM topic extraction on every message, and writes coco_examples.hn_messages and coco_examples.hn_topics. This source is a one-shot catch-up per run: each cocoindex update syncs the tables to the current feed.

Re-running is where the incremental model pays off:

New threads in the feed get processed; threads already in the database reuse their committed rows, so the expensive LLM calls only run for content CocoIndex hasn't seen.
Change the extraction prompt or switch LLM_MODEL, and the next update re-extracts to keep hn_topics consistent with the new logic. No manual migration.

Explore the results from the command line:

python main.py            # top 20 trending topics + interactive search
python main.py "rust"     # search content for one topic

What you can build next

This example opens the door to a lot more:

Run LLM summarization on top of indexed threads
Add embeddings and vector search
Mirror HN into your internal data warehouse
Build a real-time HN dashboard
Extend to other news sources (Reddit, Lobsters, etc.)

Because the whole pipeline is declarative and incremental, extending it is a matter of declaring more rows.

Since a custom source is just Python code that fetches data, the best use cases are usually "hard-to-reach" data: systems that don't have standard database connectors, have complex nesting, or require heavy pre-processing.

The knowledge aggregator for LLM context

Building a context engine for an AI bot often requires pulling from non-standard documentation sources: wikis, ticket systems, internal APIs. Each becomes a fetch function and a component, and the index stays current as the sources change.

The composite entity (data stitching)

Most companies have user data fragmented across multiple microservices. A custom source can act as a virtual join before the data ever hits your index. For example, a component:

Fetches a user ID from an auth service (Okta, Auth0)
Uses that ID to fetch billing status from the Stripe API
Uses that ID to fetch usage logs from an internal Redis

Instead of managing ETL joins downstream, the component declares a single User360 row. CocoIndex tracks the state of that composite object; if the user upgrades in Stripe or changes their email in Auth0, the index updates automatically.

The legacy wrapper (modernization layer)

Enterprises often have valuable data locked in systems that are painful to query (SOAP, XML, mainframes). Wrap the legacy read path in a fetch function and you get a modern, queryable SQL interface via the CocoIndex target, without rewriting the legacy system itself.

Public data monitor (competitive intelligence)

Tracking changes on public websites or APIs that don't offer webhooks: competitor pricing pages, regulatory feeds, market data APIs. Because CocoIndex diffs declared rows against the previous run, downstream tables only change when the data actually changes, rather than filling up with identical polling results.

Why this matters

Custom sources extend the same declarative model to any API: internal, external, legacy, or real-time.

This unlocks a simple pattern:

If you can fetch it, CocoIndex can index it, diff it, and sync it.

Whether you're indexing HackerNews or stitching together dozens of enterprise services, the framework gives you a stable backbone: persistent state, deterministic incremental updates, automatic cleanup, and flexible target exports with minimal infrastructure overhead.

Support CocoIndex ❤️

If this example was helpful, the easiest way to support CocoIndex is to give the project a ⭐ on GitHub.

Your stars help us grow the community, stay motivated, and keep shipping better tools for real-time data ingestion and transformation.

Extracting Intake Forms with BAML and CocoIndex

Linghua Jin — Fri, 21 Nov 2025 00:00:00 GMT

This tutorial shows how to use BAML together with CocoIndex to build a data pipeline that extracts structured information with an LLM from PDF intake forms. The BAML definitions describe the desired output schema and prompt logic, while CocoIndex orchestrates file input, transformation, and incremental indexing.

We’ll walk through setup, defining the BAML schema, generating the Python client, writing the CocoIndex flow, and running the pipeline. Throughout, we follow best practices (e.g. caching heavy steps) and cite documentation for key concepts.

The full project is open sourced here ⭐. To see more examples built with CocoIndex, you could refer to the examples page.

BAML

BAML, created by BoundaryML, is a typed prompt engineering language that makes LLM workflows predictable, testable, and production-safe. Instead of treating prompts as fragile strings, BAML lets developers define clear input parameters, output schemas, and model configurations, transforming prompts into strongly typed functions.

CocoIndex

CocoIndex is a unified data processing engine built for AI-native applications. It lets you define transformations in one declarative workflow, then keeps everything continuously up to date with real-time, incremental processing. Designed for reliability and scale, CocoIndex ensures that every derived artifact (embeddings, metadata, extractions, models) always reflects the latest source data, making it the foundation for fast, consistent RAG, analytics, and automation pipelines.

Flow overview

Read PDF files from a directory.
For each file, call the BAML function to get a structured Patient.
Collect results and export to Postgres.

Prerequisites

Install Postgres if you don't have one.
Install dependencies
```
pip install -U cocoindex baml-py
```
Create a .env file. You can copy it from .env.example first:
```
cp .env.example .env
```
Then edit the file to fill in your GEMINI_API_KEY.

Structured extraction component with BAML

Create a baml_src/ directory for your BAML definitions. We’ll define a schema for patient intake data (nested classes) and a function that prompts Gemini to extract those fields from a PDF. Save this as baml_src/patient.baml

Define patient schema

Classes: We defined Pydantic-style classes (Contact, Address, Insurance, etc.) to match the FHIR-inspired patient schema. These become typed output models. Required fields are non-nullable; optional fields use ?.

class Contact {
  name string
  phone string
  relationship string
}

class Address {
  street string
  city string
  state string
  zip_code string
}

class Pharmacy {
  name string
  phone string
  address Address
}

class Insurance {
  provider string
  policy_number string
  group_number string?
  policyholder_name string
  relationship_to_patient string
}

class Condition {
  name string
  diagnosed bool
}

class Medication {
  name string
  dosage string
}

class Allergy {
  name string
}

class Surgery {
  name string
  date string
}

class Patient {
  name string
  dob string
  gender string
  address Address
  phone string
  email string
  preferred_contact_method string
  emergency_contact Contact
  insurance Insurance?
  reason_for_visit string
  symptoms_duration string
  past_conditions Condition[]
  current_medications Medication[]
  allergies Allergy[]
  surgeries Surgery[]
  occupation string?
  pharmacy Pharmacy?
  consent_given bool
  consent_date string?
}

Define the BAML function to extract patient info from a PDF

function ExtractPatientInfo(intake_form: pdf) -> Patient {
  client Gemini
  prompt #"
    Extract all patient information from the following intake form document.
    Please be thorough and extract all available information accurately.
    {{ _.role("user") }}
    {{ intake_form }}

    Fill in with "N/A" for required fields if the information is not available.

    {{ ctx.output_format }}
  "#
}

We specify client Gemini and a prompt template. The special variable {{ intake_form }} injects the PDF, and {{ ctx.output_format }} tells BAML to expect the structured format defined by the return type. The prompt explicitly asks Gemini to extract all fields, filling “N/A” if missing.

:::tip[BAML PDF Extraction: Crucial Prompt Role Gotcha] When using BAML to extract structured data (like a Patient record) from PDFs, it is absolutely critical to ensure the PDF content is injected as part of the user message in the prompt. Specifically, you need to include {{ _.role("user") }} before you insert your file data with {{ intake_form }}: :::

Why role("user") matters

For OpenAI models (e.g., GPT-4, GPT-4o), if the file's content is not presented in the user message, the model won't "see" the PDF at all; your extraction will fail or be empty.
For Gemini and Anthropic, it's more forgiving and can sometimes work anyway, which makes this confusing to debug across providers.

We only discovered this after a discussion on the BAML repo and our own investigations. If you skip the explicit role("user"), you might waste hours debugging inconsistent extractions.

Takeaway:
When building extraction flows with BAML, always set the role to "user" before adding file content to your prompt. That makes your workflow robust and portable across LLM providers.

Thanks to Deepu and Prashanth from our Discord community for working with us on this issue. You can see a real-world debugging journey in our Discord thread.

Configure the LLM client to use Google’s Gemini model

client<llm> Gemini {
  provider google-ai
  options {
    model gemini-2.5-flash
    api_key env.GEMINI_API_KEY
  }
}

Configure BAML generator

In baml_src folder add generator.baml:

generator python_client {
  output_type python/pydantic
  output_dir "../"
  version "0.213.0"
}

The generator block tells baml-cli to create a Python client with Pydantic models in the parent directory.

When we run baml-cli generate, this will compile the .baml definitions into a baml_client/ Python package in your project root. It contains:

baml_client/types.py with Pydantic classes (Patient, etc.).
baml_client/sync_client.py and async_client.py with a callable b object. For example, b.ExtractPatientInfo(pdf) will return a Patient.

Continuous data transformation flow with incremental processing

Next we will define a data transformation flow with CocoIndex. Once you declare the state and transformation logic, CocoIndex will take care of all the state change for you from source to target.

CocoIndex flow

Declare flow

Declare a CocoIndex flow, connect to the LocalFile source, add a data collector to collect processed data.

@cocoindex.flow_def(name="PatientIntakeExtractionBaml")
def patient_intake_extraction_flow(
    flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
    data_scope["documents"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(
            path=os.path.join("data", "patient_forms"), binary=True
        )
    )
    
    patients_index = data_scope.add_collector()

This iterates over each document. We transform doc["content"] (the bytes) by our extract_patient_info function. The result is stored in a new field patient_info. Then we collect a row with the filename and extracted patient info.

Define a custom function to use BAML extraction to transform a PDF

@cocoindex.op.function(cache=True, behavior_version=1)
async def extract_patient_info(content: bytes) -> Patient:
    pdf = baml_py.Pdf.from_base64(base64.b64encode(content).decode("utf-8"))
    return await b.ExtractPatientInfo(pdf)

The extract_patient_info function is a custom function decorated with @cocoindex.op.function(cache=True, behavior_version=1). Setting cache=True causes CocoIndex to cache outputs of this function for incremental runs (so unchanged inputs skip rerunning the LLM). We increase behavior_version (start at 1) so that any prompt or logic changes will force a refresh.
Inside the function, we convert bytes to a BAML Pdf (via base64) and then call await b.ExtractPatientInfo(pdf). This returns a Patient dataclass instance (mapped from the BAML output).

Process each document

Transform each doc with BAML
Collect the structured output

with data_scope["documents"].row() as doc:
    doc["patient_info"] = doc["content"].transform(extract_patient_info)

    patients_index.collect(
        filename=doc["filename"],
        patient_info=doc["patient_info"],
    )

It is common to have heavily nested data, and CocoIndex is natively designed to handle heavily nested data structures.

Export to Postgres

patients_index.export(
    "patients",
    cocoindex.storages.Postgres(),
    primary_key_fields=["filename"],
)

We export the collected index to Postgres. This will create/maintain a table patients keyed by filename, automatically deleting or updating rows if inputs change. Because CocoIndex tracks data lineage, it will handle updates/deletions of source files incrementally.

Running the pipeline

Generate BAML client code (required step, in case you didn’t do it earlier).

baml generate

This generates the baml_client/ directory with Python code to call your BAML functions.

Update the index:

cocoindex update main

CocoInsight

cocoindex server -ci main

Composable by default: use the best components for your use case

While CocoIndex provides a rich set of building blocks for building LLM pipelines, it is fundamentally designed as an open system. Developers can bring in their preferred transformation components tailored to their domain, from document parsers to structured extractors like BAML.

This flexibility enables deep composability with other open ecosystems. The synergy between CocoIndex and BAML highlights this philosophy: BAML brings powerful prompt-driven schema extraction, while CocoIndex orchestrates and maintains the flow at scale. There’s no lock-in: developers and enterprises experimenting at the frontier can adapt, extend, and integrate freely.

Summary

By combining BAML and CocoIndex, we get a robust, schema-driven workflow: BAML ensures the prompt-to-schema mapping is correct and type-safe, while CocoIndex handles data ingestion, transformation, and incremental storage. This example extracted patient intake information (names, insurance, medications, etc.) from PDFs, but the pattern applies to any structured data extraction task.

For more details on CocoIndex flows, see its documentation.
For BAML usage and prompt engineering, refer to the BoundaryML docs.

Adaptive Batching - 5x throughput on your data pipelines

George He — Mon, 10 Nov 2025 00:00:00 GMT

CocoIndex now batches GPU and ML workloads automatically: 5x throughput on text embeddings and AI ops, with zero configuration required.

AI-Native Data Pipeline - Why We Made It

Linghua Jin — Wed, 29 Oct 2025 00:00:00 GMT

Why the next wave of AI needs open-source, scalable, AI-native data infrastructure, and how CocoIndex is building the foundation for intelligent data pipelines.

Index PDF elements with mixed embedding models

Linghua Jin — Mon, 27 Oct 2025 00:00:00 GMT

Extract, embed, and store multimodal PDF elements (text with SentenceTransformers, images with CLIP) for unified semantic search with traceable metadata.

Bring your own data: Index any data with Custom Sources

Linghua Jin — Tue, 21 Oct 2025 00:00:00 GMT

Read data from any system with CocoIndex: in v1 a custom source is plain async Python plus processing components. No connector API to implement.

CocoIndex Changelog 2025-10-19

George He — Sun, 19 Oct 2025 00:00:00 GMT

Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks.

Automated invoice processing with AI and Snowflake

Dhilip Subramanian — Sat, 11 Oct 2025 00:00:00 GMT

Incremental ETL from Azure Blob Storage to Snowflake with CocoIndex v1: LLM invoice extraction from PDFs, processing only new or changed files.

Thinking in Rust: Ownership, Access, and Memory Safety

George He — Fri, 10 Oct 2025 00:00:00 GMT

A mental framework for Rust's memory safety concepts. Think systematically about ownership, references, Send, Sync, and Rc, Arc, RefCell, Mutex, etc.

Trace search results back to source data

Linghua Jin — Sun, 21 Sep 2025 00:00:00 GMT

Define query handlers in CocoIndex and trace search results back to source data in CocoInsight to close the loop on indexing strategy.

Turn a Postgres table into a semantic index

Linghua Jin — Mon, 01 Sep 2025 00:00:00 GMT

Use an existing PostgreSQL table as a CocoIndex source: derive fields, embed each row, and store the vectors in Postgres with pgvector, incrementally.

Index PDFs, images, and slides with ColPali, no OCR

Linghua Jin — Wed, 20 Aug 2025 00:00:00 GMT

Build a unified visual document index from multiple file formats (including PDFs, images, and slides) using CocoIndex and ColPali. No OCR needed.

CocoIndex Changelog 2025-08-18

Linghua Jin — Mon, 18 Aug 2025 00:00:00 GMT

CocoIndex updates: production readiness, scalability, and reliability, plus more customization, native integrations, and multi-modal pipeline features.

Control Processing Concurrency in CocoIndex

George He — Wed, 13 Aug 2025 00:00:00 GMT

How CocoIndex's layered concurrency controls optimize data-processing performance, prevent system overload, and keep pipelines stable and efficient at scale.

Index Images with ColPali: multi-vector visual search

Linghua Jin — Tue, 12 Aug 2025 00:00:00 GMT

Index images with ColPali multi-vector patch embeddings and a Qdrant MaxSim collection using CocoIndex v1: incremental, live, in plain async Python.

Multi-Dimensional Vector Support in CocoIndex

Linghua Jin — Sun, 10 Aug 2025 00:00:00 GMT

CocoIndex natively handles typed multi-dimensional vectors, from simple arrays to multi-vector embeddings, unlocking multimodal AI pipelines at scale.

Custom Targets: export your data anywhere

Linghua Jin — Sun, 03 Aug 2025 00:00:00 GMT

Export CocoIndex data anywhere: implement a TargetHandler with a tracking record and an action sink; the engine handles diffing, syncing, and cleanup.

Index faces for visual search: your own Google Photos

Linghua Jin — Thu, 24 Jul 2025 00:00:00 GMT

Build a scalable face detection and recognition pipeline with CocoIndex: embed faces, structure for search, and export to a vector DB.

Index academic papers and extract metadata for AI agents

Linghua Jin — Wed, 09 Jul 2025 00:00:00 GMT

Index academic papers with CocoIndex: read the first page of each PDF, extract title, authors, and abstract with an LLM, and embed them for semantic search.

CocoIndex Changelog 2025-07-07

CocoIndex Team — Mon, 07 Jul 2025 00:00:00 GMT

CocoIndex updates: in-process setup/drop API, EmbedText building block, SplitRecursively improvements, union/NumPy types, and the Kuzu graph target.

Introducing CocoInsight

Linghua Jin — Tue, 24 Jun 2025 00:00:00 GMT

Introducing CocoInsight, a data lineage and observability tool that lets you inspect, trace, and debug every step of a CocoIndex pipeline in real time.

Flow-based schema inference for Qdrant

Linghua Jin — Sun, 08 Jun 2025 00:00:00 GMT

CocoIndex sets up Qdrant collections automatically by inferring the target schema from your indexing flow: no manual config, vector sizes kept in sync.

CocoIndex + Kuzu: Real-time knowledge graph with Kuzu

Linghua Jin — Tue, 03 Jun 2025 00:00:00 GMT

Build a real-time knowledge graph with Kuzu as a native CocoIndex target: incremental updates, high-performance graph queries.

CocoIndex Changelog 2025-05-31

CocoIndex Team — Sat, 31 May 2025 00:00:00 GMT

CocoIndex updates: Amazon S3 as a data source, improved query handling, a standalone runtime mode, and more connector and performance improvements.

Incremental ETL on Amazon S3 with CocoIndex

Linghua Jin — Thu, 29 May 2025 00:00:00 GMT

Index Markdown from Amazon S3 into Postgres pgvector with CocoIndex v1: incremental processing re-embeds only the files and chunks that changed.

Image search in natural language with CLIP

Linghua Jin — Tue, 20 May 2025 00:00:00 GMT

Search a folder of photos by meaning with CocoIndex v1: CLIP embeds images and text into one vector space, the index runs live inside a FastAPI app, and vectors live in Qdrant.

How to build an index with text embeddings

Linghua Jin — Mon, 19 May 2025 00:00:00 GMT

Text embeddings 101: what they are, why you chunk and embed, and how to build a semantic search index with CocoIndex v1 and Postgres pgvector.

Story of CocoIndex, at 1k stars 🎉

Linghua Jin — Thu, 08 May 2025 00:00:00 GMT

The story of CocoIndex at 1,000 GitHub stars: the open-source engine that combines custom transformation logic with incremental processing for data indexing.

Build a product recommendation engine with LLM + Neo4j

Linghua Jin — Wed, 07 May 2025 00:00:00 GMT

An LLM extracts each product's taxonomy and complements; CocoIndex turns the labels into a Neo4j knowledge graph that answers 'bought this, also need…'.

CocoIndex Changelog 2025-04-30

CocoIndex Team — Wed, 30 Apr 2025 00:00:00 GMT

CocoIndex updates: knowledge graph support, Qdrant and Supabase targets, KTable and LTable data types, additional LLM providers, and more.

Build Real-Time Knowledge Graph For Documents with LLM

Linghua Jin — Tue, 29 Apr 2025 00:00:00 GMT

Turn a folder of Markdown docs into a Neo4j knowledge graph: an LLM extracts subject-predicate-object triples, and CocoIndex keeps the graph in sync as the docs change.

CocoIndex Changelog 2025-04-07

CocoIndex Team — Mon, 07 Apr 2025 00:00:00 GMT

CocoIndex updates: incremental live update mode, evaluation utilities, date/time types, a Google Drive source, and core performance improvements.

Keep derived data in sync with changing sources

CocoIndex Team — Mon, 07 Apr 2025 00:00:00 GMT

CocoIndex continuously watches source changes and applies incremental updates to keep derived data in sync, with low latency and no full reindexing.

Incremental Processing with CocoIndex

Linghua Jin — Sun, 06 Apr 2025 00:00:00 GMT

What incremental processing is, who needs it, and how CocoIndex keeps an index in sync with source changes through caching, lineage tracking, and change data capture.

Structured Extraction from Patient Intake Form with LLM

CocoIndex Team — Wed, 26 Mar 2025 00:00:00 GMT

Extract typed Patient records from PDF and DOCX intake forms with an LLM and CocoIndex v1: the nested schema is the whole prompt; results land in Postgres.

Search your Google Drive by meaning

CocoIndex Team — Sun, 23 Mar 2025 00:00:00 GMT

Index the documents in a shared Google Drive folder as text embeddings in Postgres with CocoIndex v1, then search them by meaning instead of by filename.

CocoIndex Changelog 2025-03-20

CocoIndex Team — Thu, 20 Mar 2025 00:00:00 GMT

First release of CocoIndex Changelog: LLM support, codebase indexing, custom functions, and assorted core/performance improvements

Build Real-Time Codebase Indexing for AI Code Generation

CocoIndex Team — Tue, 18 Mar 2025 00:00:00 GMT

Indexing codebase for RAG with CocoIndex and Tree-sitter in real-time: chunking, embedding, semantic search, and build vector index for efficient retrieval.

On-premise structured extraction from PDFs with Ollama

CocoIndex Team — Mon, 17 Mar 2025 00:00:00 GMT

Extract structured data from PDF manuals locally with Ollama and CocoIndex: docling converts PDFs to Markdown, a local LLM fills typed Postgres rows.

We are officially open sourced! 🎉

CocoIndex Team — Mon, 03 Mar 2025 00:00:00 GMT

CocoIndex is now open source: the first engine to combine custom transformation logic with incremental processing built specifically for data indexing.

Customizable Data Indexing Pipelines

CocoIndex Team — Thu, 20 Feb 2025 00:00:00 GMT

What customizable data indexing pipelines are and why custom transformation logic matters, with practical CocoIndex examples.

How indexing pipelines differ from other data pipelines

CocoIndex Team — Thu, 30 Jan 2025 00:00:00 GMT

What makes indexing pipelines different from other data systems, and why they need special handling for incremental processing and persistence.

System updates and automatic schema inference

CocoIndex Team — Mon, 20 Jan 2025 00:00:00 GMT

How CocoIndex handles system updates in indexing flows: automatic schema inference and managing data + logic evolution without downtime.

Processing Large Files in Data Indexing Systems

CocoIndex Team — Fri, 10 Jan 2025 00:00:00 GMT

Handle large files in data indexing: processing granularity, fan-in/fan-out, and memory pressure, walked through a patent XML example in CocoIndex.

Data Consistency in Indexing Pipelines

CocoIndex Team — Mon, 06 Jan 2025 00:00:00 GMT

Data consistency in indexing pipelines: concurrent updates, exposure risks, and how CocoIndex's data-driven approach keeps indexes converging.

Data Indexing and Common Challenges

CocoIndex Team — Sun, 05 Jan 2025 00:00:00 GMT

Fundamentals of data indexing pipelines for RAG: what makes a good one, common production pitfalls, and how CocoIndex addresses them.

CocoIndex - A Data Indexing Platform for AI Applications

CocoIndex Team — Sat, 04 Jan 2025 00:00:00 GMT

CocoIndex is a data indexing platform for AI: ingestion, chunking, embedding, and pipeline management for RAG, semantic search, and knowledge graphs.

Welcome to CocoIndex

Linghua Jin — Thu, 02 Jan 2025 00:00:00 GMT

Welcome to the official CocoIndex blog! We're excited to share our journey in building high-performance indexing infrastructure for AI applications.

CocoIndex Blog

CocoIndex Changelog 1.0.8 - 1.0.16

Why this release cycle matters

Engine

coco.use_state(): persistent per-component state

LiveMap: in-memory intermediate collections

Token-bucket RateLimiter

Preview mode for update actions

Batched target writes across components

Engine performance improvements

Other engine fixes

Connectors

Target: Snowflake (v1.0.15)

Target: BigQuery (v1.0.16)

Source: Azure Blob Storage (v1.0.16)

Target: Valkey vector store (v1.0.12)

Qdrant: native sparse vector support (v1.0.16)

LanceDB: vector and FTS index support (v1.0.11)

zvec: FTS field support (v1.0.14)

OCI: skip startup scan on unchanged logic (v1.0.11)

Local file system: periodic rescan in live mode (v1.0.15)

New connector summary

Agent & developer experience

cocoindex show: full-fledged inspection tool (v1.0.12)

Python error messages

Multiple GPU and fractional GPU allocation (v1.0.15)

Example walkthroughs

Bug fixes and correctness

Summary

Thanks to the community

@georgeh0

@badmonster0

@hakuuww

@sdhilip200

@Sujit-1509

@prrao87

@lichuang

@Haleshot

@nuthalapativarun

@daric93

@prithvi-moonshot

@MrAnayDongre

@pyjuan91

@tomz-alt

@jordigilh

@MaxRong

@slliland

@SuperMarioYL

@davidmyriel

@dashitongzhi

Index Your Codebase for AI Agents with CocoIndex V1

Use case

Why CocoIndex for codebase indexing

Overview

Setup

Define the App

Define the data and shared resources

Define file processing

Process file

Process chunk

Define the main function

Create the App

Run the pipeline

Query the index

Incremental updates

CocoIndex Code

Next steps

CocoIndex Changelog 1.0.1 - 1.0.7

The short version

Engine

Per-argument memoization control

coco.auto_refresh for live components

Scoped stats reports

Other engine improvements

Built-in operations

LiteLLM speech-to-text

Code splitter: eight new languages

Entity resolution: parallel by default

Connectors

Source: OCI Object Storage with live bucket watching

`coco.use_state()`: persistent per-component state

`LiveMap`: in-memory intermediate collections

Token-bucket `RateLimiter`

`cocoindex show`: full-fledged inspection tool (v1.0.12)

`coco.auto_refresh` for live components

3. Embedded LMDB — `pip install` and go