# CocoIndex Docs — Rust SDK

> CocoIndex is an ultra-performant framework for building data pipelines for AI, with built-in incremental processing. Declare what your target should look like as a function of your source — the Rust engine keeps it in sync, reprocessing only what changed.

> This index lists the Rust SDK pages (/docs/rust/). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are indexed at https://cocoindex.io/docs/llms.txt.

> Full docs text in one file: https://cocoindex.io/docs/rust/llms-full.txt. Docs pages and example walkthroughs have raw Markdown twins by replacing the trailing slash with `.md`, e.g. https://cocoindex.io/docs/rust/programming_guide/core_concepts.md — and they are all included in llms-full.txt.

> Coding agents: install the CocoIndex skill before writing v1 code — https://cocoindex.io/docs/skill.md (teaches the correct v1 API; without it, LLMs tend to hallucinate the deprecated v0 flow-builder DSL).

## Key pages
- [Core Concepts](https://cocoindex.io/docs/rust/programming_guide/core_concepts/): Declare your outputs. CocoIndex applies changes and reuses cached results.
- [CLI Reference](https://cocoindex.io/docs/rust/cli/): Reference for the `cocoindex` CLI for Rust projects — how it builds and drives your crate, the app target format, the three attributes an app registers with, and the ls, show, update, drop, init and auth subcommands.
- [FAQ](https://cocoindex.io/docs/rust/faq/): Common questions about CocoIndex behavior — how change detection propagates through logic vs inputs, runtime invocation tracking, hidden dependencies, and crash recovery vs cross-target atomicity for target states.

## Getting Started
- [Overview](https://cocoindex.io/docs/rust/getting_started/overview/): CocoIndex is a real-time data transformation framework for AI with incremental processing, with a Rust SDK that builds pipelines into your own binary: only the data that actually changed is reprocessed, and lineage is observable end to end.
- [Installation](https://cocoindex.io/docs/rust/getting_started/installation/): Add the CocoIndex Rust SDK to a Cargo project, choose its feature flags, and install the cocoindex CLI for Rust projects.
- [Quickstart](https://cocoindex.io/docs/rust/getting_started/quickstart/): Build an incremental Rust pipeline that chunks Markdown, batches and caches local embeddings, and keeps a Postgres/pgvector table in sync.
- [Use with AI coding agents](https://cocoindex.io/docs/rust/getting_started/ai_coding_agents/): The official CocoIndex agent skill covers the Python SDK only. For the Rust SDK, point Claude Code, Cursor, and other AI coding agents at the Rust SDK showcase and the Rust examples.

## Programming Guide
- [Overview](https://cocoindex.io/docs/rust/programming_guide/): Build a pipeline from Apps, processing components, functions, and target states.
- [App](https://cocoindex.io/docs/rust/programming_guide/app/): Apps are the top-level runnable unit in CocoIndex. Learn how to create an App, trigger updates, configure the database path, and wire up lifespan-managed resources.
- [Target state](https://cocoindex.io/docs/rust/programming_guide/target_state/): Target states describe what should exist in an external system — files, rows, embeddings. Learn how to declare them through connector APIs, nest container targets, and rely on CocoIndex to sync minimal creates, updates, and deletes.
- [Processing component](https://cocoindex.io/docs/rust/programming_guide/processing_component/): Processing components are the units of incremental execution and the sync boundaries for target states. Covers mounting APIs (mount!, use_mount!, mount_each!, mount_target), component paths, and granularity tradeoffs.
- [Functions](https://cocoindex.io/docs/rust/programming_guide/function/): The #[cocoindex::function] attribute makes Rust functions participate in CocoIndex's change detection, memoization, and execution pipeline — with batching and GPU-aware runners included.
- [Context](https://cocoindex.io/docs/rust/programming_guide/context/): Use context_key!, EnvironmentBuilder::provide_key() and ctx.get_key() to share connections, models, and config across processing components. Covers change detection on context values and the query path outside an update.
- [Live mode](https://cocoindex.io/docs/rust/programming_guide/live_mode/): Keep an App running after its initial sweep so it reacts to source changes continuously. Covers UpdateOptions { live: true }, ctx.mount_each_live, and live-aware connectors like the local filesystem and Kafka.
- [Serialization](https://cocoindex.io/docs/rust/programming_guide/serialization/): How CocoIndex serializes memoized function returns with serde — the Serialize and Deserialize bounds, MemoInput for memo keys, and SchemaFields for connector rows.
- [SDK overview](https://cocoindex.io/docs/rust/programming_guide/sdk_overview/): A tour of the CocoIndex Rust SDK — crate layout (core, prelude, connectors, ops, resources), feature flags, common types like StableKey, and its async-only API.

## Common Resources
- [Overview](https://cocoindex.io/docs/rust/common_resources/): Shared types and utilities used across connectors and operations: data models, vector schema, stable ID generation, and the in-memory LiveMap.
- [Data types](https://cocoindex.io/docs/rust/common_resources/data_types/): Data types shared across connectors and built-in operations — FileLike, FilePath, FilePathMatcher, Chunk with text positions, and the Embedder trait for single-text async embedding.
- [Vector schema](https://cocoindex.io/docs/rust/common_resources/vector_schema/): Describe vector columns with VectorSchema and VectorSchemaProvider. Covers the three ways to size a vector field (an embedder from the context, an embedder instance, an explicit schema) and MultiVectorSchema for models like ColBERT.
- [ID generation](https://cocoindex.io/docs/rust/common_resources/id_generation/): Generate IDs and UUIDs that stay stable across incremental runs — generate_id and generate_uuid for pure derivation, and IdGenerator / UuidGenerator for stateful sequences that still reproduce across runs.
- [LiveMap](https://cocoindex.io/docs/rust/common_resources/live_map/): LiveMap is an in-memory keyed collection that bridges live producing and consuming logic — declare (key, value) entries as target states on one side, consume it as a LiveMapView with ctx.mount_each_live on the other.
- [Rate limiting](https://cocoindex.io/docs/rust/common_resources/rate_limiting/): Throttle outbound work with a token-bucket RateLimiter — a sustained rate plus a burst allowance, shared by reference so several callers run under one budget.

## Connectors
- [Overview](https://cocoindex.io/docs/rust/connectors/): The built-in connectors that move data in and out of a CocoIndex flow — object stores, message streams, and relational, vector, and graph databases. Many work as a source, a target, or both.
- [Amazon S3](https://cocoindex.io/docs/rust/connectors/amazon_s3/): Read objects from Amazon S3 buckets and S3-compatible services (e.g. MinIO, Tigris) — with prefix filters, path matchers, size limits, and stable S3FilePath for memoization.
- [Azure Blob Storage](https://cocoindex.io/docs/rust/connectors/azure_blob/): Read blobs from Azure Blob Storage containers with prefix filters, path matchers, size limits, and stable AzureBlobFilePath values for memoization.
- [BigQuery](https://cocoindex.io/docs/rust/connectors/bigquery/): Write CocoIndex target rows into BigQuery tables with managed DDL, MERGE upserts, deletes, and automatic mapping from Rust row structs.
- [Apache Doris](https://cocoindex.io/docs/rust/connectors/doris/): Write rows to Apache Doris tables via stream load with automatic upserts and deletions, HNSW/IVF vector indexes, and inverted indexes for full-text search — with optional schema enforcement.
- [FalkorDB](https://cocoindex.io/docs/rust/connectors/falkordb/): Write to FalkorDB (a Redis-backed graph database) with support for node tables, relationship tables (edges), per-graph multitenancy, and vector indexes with cosine / euclidean / inner-product distances.
- [Google Drive](https://cocoindex.io/docs/rust/connectors/google_drive/): Read files recursively from Google Drive folders, via a service account or a personal account signed in with OAuth. Supports MIME-type filtering and auto-exports Google Docs, Sheets, and Slides to text or CSV.
- [Iggy](https://cocoindex.io/docs/rust/connectors/iggy/): Consume Apache Iggy topics as live streams/maps and produce messages as target states.
- [Kafka](https://cocoindex.io/docs/rust/connectors/kafka/): Consume Kafka topics as live keyed maps (topic_as_map → LiveMapFeed) or as raw event streams (topic_as_stream → LiveStream), and produce messages as target states — with automatic offset commits and partition rebalancing at at-least-once semantics.
- [LanceDB](https://cocoindex.io/docs/rust/connectors/lancedb/): Write and upsert rows to LanceDB tables with schemas derived from Rust structs, vector columns via VectorSchemaProvider, and both system-managed and user-managed table lifecycles.
- [Local filesystem](https://cocoindex.io/docs/rust/connectors/localfs/): Walk local directories with path-matcher filtering, watch for changes in live mode via the platform's native watcher, and write outputs with declare_file() and DirTarget — with memo-stable FilePath across project moves.
- [Neo4j](https://cocoindex.io/docs/rust/connectors/neo4j/): Write to Neo4j (a property graph database) with support for node tables, relationship tables (edges), per-database multitenancy, atomic-batch writes via Bolt transactions, real CREATE CONSTRAINT uniqueness, and vector indexes with cosine / euclidean similarity.
- [OCI Object Storage](https://cocoindex.io/docs/rust/connectors/oci_object_storage/): Read objects from Oracle Cloud Infrastructure (OCI) Object Storage buckets — with prefix filters, path matchers, size limits, and an optional live mode driven by OCI Object Storage events delivered through OCI Streaming.
- [Postgres](https://cocoindex.io/docs/rust/connectors/postgres/): Read Postgres tables as a row stream via read_table and write with declare_table_target() — schema derivation from Rust structs, pgvector index management (ivfflat / hnsw), column type overrides, and SQL attachments for triggers and indexes.
- [Qdrant](https://cocoindex.io/docs/rust/connectors/qdrant/): Write points to Qdrant collections with single or named dense vectors, sparse vectors, multi-vector support, and schemaless JSON payloads — configured via CollectionSchema, QdrantVectorDef and QdrantSparseVectorDef.
- [Snowflake](https://cocoindex.io/docs/rust/connectors/snowflake/): Write CocoIndex target rows into Snowflake tables with managed DDL, MERGE upserts, deletes, and automatic mapping from Rust row types.
- [SQLite](https://cocoindex.io/docs/rust/connectors/sqlite/): Write rows to SQLite databases with optional sqlite-vec vector support via float[N] columns — covers regular tables and vec0 virtual tables for optimized vector similarity search, with auto-loading of the vec extension.
- [SurrealDB](https://cocoindex.io/docs/rust/connectors/surrealdb/): Write to SurrealDB with support for normal and relation (graph edge) tables, typed values, array record IDs, bounded transactions, and HNSW vector indexes with cosine / euclidean / manhattan distances.
- [Turbopuffer](https://cocoindex.io/docs/rust/connectors/turbopuffer/): Write rows to Turbopuffer namespaces with single or named vectors, per-namespace distance metric, and schemaless attributes — configured via VectorDef and NamespaceSchema.
- [Valkey](https://cocoindex.io/docs/rust/connectors/valkey/): Write documents to Valkey search indexes with vector similarity search — configured via VectorDef for dimension, distance metric, and index algorithm (HNSW or FLAT).
- [zvec](https://cocoindex.io/docs/rust/connectors/zvec/): Write documents to zvec, an embedded in-process vector database. Covers collection setup, declaring documents with dense vectors plus scalar, array and full-text fields, schema-from-row type mapping, and lifecycle management.

## Built-in Operations
- [Overview](https://cocoindex.io/docs/rust/ops/): Overview of CocoIndex's built-in operations: text chunking and splitting, local and hosted embeddings, audio transcription, and entity resolution, with links to each operation module's reference.
- [Entity resolution](https://cocoindex.io/docs/rust/ops/entity_resolution/): Deduplicate entity names via embedding similarity and a pluggable LLM pair-resolver that confirms matches and chooses canonical forms — with Pinned / Preferred policies for existing canonicals and resolution callbacks.
- [Model APIs](https://cocoindex.io/docs/rust/ops/model_apis/): Embed text, transcribe audio and run JSON-mode chat completions over any OpenAI-compatible HTTP API with ApiEmbedder, ApiTranscriber and ApiChatClient, memoized and batched by the engine, plus the LlmPairResolver for entity resolution.
- [Sentence transformers](https://cocoindex.io/docs/rust/ops/sentence_transformers/): Run sentence-transformers ONNX models locally for text embeddings through fastembed, memoized and batched by the engine, with GPU execution providers and VectorSchemaProvider integration for connectors.
- [Text ops](https://cocoindex.io/docs/rust/ops/text/): Built-in text utilities — code language detection, regex-based SeparatorSplitter, and syntax-aware RecursiveSplitter that returns position-tracked Chunks with optional custom domain-specific splitting rules.

## Advanced Topics
- [Overview](https://cocoindex.io/docs/rust/advanced_topics/): Tune concurrency and storage, recover cleanly from failures, see deep into a running update, customize memoization, build live components, and connect CocoIndex to systems it doesn't ship with.
- [Concurrency control](https://cocoindex.io/docs/rust/advanced_topics/concurrency_control/): Tune how many processing components can run at the same time with `max_inflight_components` or the `COCOINDEX_MAX_INFLIGHT_COMPONENTS` environment variable, and understand how the limit bounds a hierarchy of components without deadlocking it.
- [Timeouts](https://cocoindex.io/docs/rust/advanced_topics/timeouts/): Use `Ctx::with_timeout` to put a time budget around a piece of processing — one file, one LLM call — with cooperative checks, deadline-aware retries, and clean time-boxing of whole updates.
- [Progress monitoring](https://cocoindex.io/docs/rust/advanced_topics/progress_monitoring/): Beyond report_to_stdout — read structured update stats from an UpdateHandle with detailed_stats_snapshot() and changed(), and split a run into separately-reported scopes with Ctx::stats_group().
- [Memoization keys](https://cocoindex.io/docs/rust/advanced_topics/memoization_keys/): Customize how CocoIndex fingerprints memoized inputs through the MemoInput trait, layer state validation (e.g. mtime, then content hash) on top with memo_state, and declare dependencies beyond the arguments with deps, LogicDep and DataDep.
- [Error handling](https://cocoindex.io/docs/rust/advanced_topics/exception_handlers/): How CocoIndex isolates component failures, recovers from interrupted updates via two-phase processing, and lets you register exception handlers globally on the environment builder or for one scope with ctx.with_exception_handler to observe background errors.
- [Internal storage](https://cocoindex.io/docs/rust/advanced_topics/internal_storage/): CocoIndex persists target states and memo results in LMDB. The database grows automatically when a write runs out of space; tune the initial map_size and max_dbs via environment variables or programmatic settings for large-scale deployments.
- [Multiple environments](https://cocoindex.io/docs/rust/advanced_topics/multiple_environments/): Run isolated CocoIndex environments side by side with explicit Environment builders and separate db_paths — for multi-tenancy, library development, and testing.
- [Live components](https://cocoindex.io/docs/rust/advanced_topics/live_component/): Implement the LiveComponent trait with process() for full scans and process_live(operator) for incremental updates — covers LiveComponentOperator (update_full, update, delete, mark_ready), the LiveMapFeed / LiveMapView / LiveStream traits over event sources, and the in-process LiveMap.
- [Custom target connector](https://cocoindex.io/docs/rust/advanced_topics/custom_target_connector/): Connect CocoIndex's declarative target-state system to external systems. Covers TargetHandler, tracking records, actions, sinks, root vs container targets, attachment providers, and idempotent operations.

## Contributing
- [Setup dev environment](https://cocoindex.io/docs/rust/contributing/setup_dev_environment/): Install Rust, uv, and prek hooks, then build the native extension with `maturin develop` to get a local CocoIndex checkout ready for development.
- [Contributing guide](https://cocoindex.io/docs/rust/contributing/guide/): How to find good first issues, claim work, run prek checks, and submit a pull request to CocoIndex — with a one-business-day review target and squash-merge workflow.

## About
- [Community](https://cocoindex.io/docs/rust/about/community/): Find the Discord, GitHub, Twitter, LinkedIn, and YouTube channels where the CocoIndex community asks questions, shares ideas, and gets support.
- [Anonymous usage telemetry](https://cocoindex.io/docs/rust/about/telemetry/): What CocoIndex's anonymous usage tracking collects, what it doesn't, and how to opt out.

## Examples
> Runnable Rust SDK examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust — each is a standalone Cargo workspace; run it with `cargo run` or the Rust CLI (`cocoindex update`).