--- title: "CocoIndex Changelog 2025-10-19" description: "Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks." last_updated: 2025-10-19 doc_version: "2025-10-19" canonical: https://cocoindex.io/blogs/cocoindex-changelog-2025-10-19/ --- # CocoIndex Changelog 2025-10-19 > Production-ready upgrades: durable execution, faster incremental processing over large datasets, GPU isolation, and richer native building blocks. Published: 2025-10-19 · Canonical: https://cocoindex.io/blogs/cocoindex-changelog-2025-10-19/ :rocket: Over the 20+ releases, CocoIndex continued to advance its ultra-performant AI-native ETL infrastructure with a series of targeted features and enhancements focused on reliable [incremental processing](https://cocoindex.io/docs/programming_guide/core_concepts/), error resilience, and extensibility. We are constantly improving CocoIndex to make it easier to use, more powerful, and more reliable. :star: Star us on [GitHub](https://github.com/cocoindex-io/cocoindex) to stay updated! ## Core capability ### Durable execution CocoIndex is designed for massive, ever-changing datasets. Reliability and efficiency are critical. With durable execution, CocoIndex now automatically captures failures and retries only the affected rows, ensuring progress is never lost and avoiding full reprocessing. - **Automatic recovery on failure**: Failed rows are automatically retried in the next update [#995](https://github.com/cocoindex-io/cocoindex/pull/995), with optional fine-grained control through configurable retry lists [#997](https://github.com/cocoindex-io/cocoindex/pull/997). CocoIndex also maintains precise processing order tracking to guarantee consistent reprocessing sequences [#1001](https://github.com/cocoindex-io/cocoindex/pull/1001). - **Stable transient authentication**: Improved handling of short-lived auth keys ensures reliability and prevents drift across retries [#1002](https://github.com/cocoindex-io/cocoindex/pull/1002). These enhancements make incremental processing **more fault-tolerant, efficient, and self-healing**, allowing CocoIndex to continuously move forward - no wasted computation, no data left behind. ### Avoid unnecessary reprocessing In CocoIndex, reprocessing can be triggered by changes in either **source data** or **flow definitions**. We improved our flow change detection mechanism, significantly reducing false positives and avoiding unnecessary reprocessing. - **Improved change detection logic**: Refined how CocoIndex evaluates flow and dependency changes, cutting down false positives and ensuring reprocessing only happens when it’s truly needed. This leads to faster, more predictable runs and reduced compute costs. [#1182](https://github.com/cocoindex-io/cocoindex/pull/1182) - **Stable transient auth keys**: Enhanced retry behavior now keeps authentication keys consistent across transient failures, preventing drift and improving reliability for long-running flows. [#1002](https://github.com/cocoindex-io/cocoindex/pull/1002) Together, these updates make CocoIndex’s incremental engine more **deterministic, efficient, and resilient**. So your flows stay lean even as your data and logic evolve. ### Enhanced build & update target data We’ve expanded how CocoIndex builds and updates target data, giving users more control over replication and recovery workflows. #### Supported Modes - **One-time update**: Builds or updates target data based on the current state of the source. - **Live update**: Starts with a one-time update, then continuously captures source changes to keep targets up to date. #### New Option: Reexport Targets with `--reexport` We’ve added a new **`--reexport`** option that can be applied to both modes. When enabled, CocoIndex **ignores the previous state of the target** and rebuilds it entirely, even if neither the source data nor the flow definition has changed. This is especially useful for **data recovery**, **corrections**, or **full target refreshes** after data loss. In live update mode, reexport applies only to the initial build phase. Example: ```sh cocoindex update --reexport main ``` This enhancement broadens pipeline flexibility and improves data reliability in recovery or replay scenarios. Read more in [Build and update data](https://cocoindex.io/docs/programming_guide/target_state/). #### New option: `--reset` for setup, update, and server commands We’ve added a new `--reset` flag that makes it easier to start fresh by automatically clearing existing setups before running key commands. When you use `--reset`, CocoIndex will **drop the existing setup or state** before performing the new operation, equivalent to running `cocoindex drop` first. Example: ```sh cocoindex setup --reset main ``` This ensures a clean environment for rebuilds, testing, or reinitializing workflows, without needing to manually clear prior state. ### Incremental engine optimizations with **fast fingerprint collapsing** CocoIndex has smart incremental processing out of the box that only processes what’s changed. In the latest version, we’ve included more **engine-level optimizations** to improve efficiency for large datasets: - **Source content fingerprinting**: the engine tracks a lightweight “fingerprint” of each source row or file, allowing it to detect when content hasn’t changed. [*#892*](https://github.com/cocoindex-io/cocoindex/pull/892) - **Fast collapse**: unchanged data can be skipped without reading the full source, significantly reducing processing time and resource usage. [*#895*](https://github.com/cocoindex-io/cocoindex/pull/895) This optimization is particularly useful for sources that provide a **content hash, version tag, or fingerprint** (e.g., GitHub files, S3 objects, or custom sources). CocoIndex can save a **large number of external API calls in addition to compute costs**, making workflows faster and more cost-efficient. While primarily designed for versioned sources like GitHub, this approach can benefit **any source that exposes change indicators**, improving incremental processing across diverse pipelines. ### None handling CocoIndex supports *None* values. A *None* value represents the absence of data or an unknown value, distinct from empty strings, zero numbers, or false boolean values. None input values passed to required arguments of CocoIndex functions will short-circuit the function and result in a None output value. See [the documentation](https://cocoindex.io/docs/common_resources/data_types/) for more details. In this release, None value handling in transformations was improved to better manage required and nullable fields, increasing robustness. ## Robustness ### Isolate GPU workloads in subprocess Added full subprocess support for GPU workloads, improving stability and isolation while removing the need for global locks. These enhancements mark a strong step forward in robustness and performance for GPU‐driven workloads within CocoIndex. They reduce the risk of GPU jobs hanging, and provide better tolerance to GPU failures. For users leveraging GPU processing, the update is highly recommended and should bring a smoother, more resilient operational experience. #### What to expect for users / developers If you’re using CocoIndex’s GPU-capable workloads, you may notice improved stability. Temporary failures caused by the GPU will be isolated in the subprocess, which will be restarted gracefully. ### Error tolerance improvements Fixed live-mode behaviour so that source-level errors are now logged instead of aborting the process. Also improved coverage of retrying for several external APIs, e.g. OpenAI and Vertex AI. This reduces failures and improves stability. ## Query handler and CocoInsight support ### Query handler [Query handlers](https://cocoindex.io/docs/programming_guide/processing_component/) let you expose a simple function that takes a query string and returns structured results. They are discoverable by tools like CocoInsight so you can query your indexes without writing extra glue code. ```python @my_flow.query_handler(name="run_query") # Name is optional, use the function name by default def run_query(query: str) -> cocoindex.QueryOutput: # 1) Perform your query against the input `query` ... # 2) Return structured results return cocoindex.QueryOutput(results=[{"filename": "...", "text": "..."}]) ``` ### CocoInsight We are launching a major feature in both CocoIndex and CocoInsight to help users iterate quickly on the indexing strategy, and trace back all the way to the data. :tada: Announcement: https://cocoindex.io/blogs/query-support Demo: [https://www.youtube.com/watch?v=crV7odEVYTE](https://www.youtube.com/watch?v=crV7odEVYTE) ## Python SDK ### Support pydantic with field-level description for structured extraction `ExtractByLlm` is one of the native building blocks in CocoIndex to make it easier to extract structured information from unstructured documents with an LLM. This change allows users to directly define the description at the field level when doing structured extraction. For example, a user could define the following class: ```python class ProductTaxonomyInfo(BaseModel): """ Taxonomy information for the product. """ taxonomies: list[ProductTaxonomy] = Field( ..., description="Taxonomies for the current product." ) complementary_taxonomies: list[ProductTaxonomy] = Field( ..., description="Think about when customers buy this product, what else they might need as complementary products. Put labels for these complentary products." ) ``` And you can plug it into any `ExtractByLlm` transformation. ```python taxonomy = data["detail"].transform( cocoindex.functions.ExtractByLlm( llm_spec=cocoindex.LlmSpec( api_type=cocoindex.LlmApiType.OPENAI, model="gpt-4.1" ), output_type=ProductTaxonomyInfo, ) ) ``` ### support `@cocoindex.settings` and make `init()` optional A more flexible approach is to provide a setting function that returns a `cocoindex.Settings` dataclass object. The setting function can have any name, and needs to be decorated with the `@cocoindex.settings` decorator, for example: ```python @cocoindex.settings def cocoindex_settings() -> cocoindex.Settings: return cocoindex.Settings( database=cocoindex.DatabaseConnectionSpec( url="postgres://cocoindex:cocoindex@localhost/cocoindex" ) ) ``` This setting function will be called once when CocoIndex is initialized, whenever you're running the CLI or your own main script, hence you can provide settings in a single way within your code. Once the settings function is provided, environment variables will be ignored. ## Building blocks ### Postgres source enhancements The [Postgres source](https://cocoindex.io/docs/connectors/postgres/) allows users to take data from any PostgreSQL table as input. The schema of the imported data is automatically inferred from the source table. It supports change-capture based on `LISTEN`/`NOTIFY`. - **Composite-key Postgres source**: Auto-detects schema, supports single or multi-column keys, and maps Postgres → CocoIndex types. [#910](https://github.com/cocoindex-io/cocoindex/pull/910) - **Faster small-row performance**: Refactored Postgres source and `list()` API for high-volume, many-small-rows workloads. [#948](https://github.com/cocoindex-io/cocoindex/pull/948) - **Real-time Postgres updates**: New `LISTEN/NOTIFY` change-capture mode with auto channel naming and full source wiring. [#952](https://github.com/cocoindex-io/cocoindex/pull/952), [#953](https://github.com/cocoindex-io/cocoindex/pull/953), [#954](https://github.com/cocoindex-io/cocoindex/pull/954), [#955](https://github.com/cocoindex-io/cocoindex/pull/955) ### Postgres target enhancements - **Postgres filtering support**: added the ability to apply filters directly on Postgres sources, enabling more efficient data selection and ingestion. [#1178](https://github.com/cocoindex-io/cocoindex/pull/1178) - **Half-precision vector (`halfvec`) support**: added `halfvec` support for Postgres targets, reducing storage overhead for embeddings while maintaining compatibility with vector operations. [#1171](https://github.com/cocoindex-io/cocoindex/pull/1171) - **Schema support for Postgres targets**: users can now specify and write to custom schemas, allowing better organization in multi-schema database setups [#1138](https://github.com/cocoindex-io/cocoindex/pull/1138) ### Target enhancements - new & improved built-ins #### Support for LanceDB target We are officially supporting LanceDB as a target. Get started with [LanceDB target](https://cocoindex.io/docs/connectors/lancedb/). #### Postgres target enhancements **Target Attachment / PostgreSQL SQL Command Attachment**: The core engine now supports attachments on targets, including a `PostgresSqlCommand` type that allows you to execute arbitrary SQL (setup/teardown) on the target side [#1131](https://github.com/cocoindex-io/cocoindex/issues/1131). Example (create a custom index): ```python collector.export( "doc_embeddings", cocoindex.targets.Postgres(table_name="doc_embeddings"), primary_key_fields=["id"], attachments=[ cocoindex.targets.PostgresSqlCommand( name="fts", setup_sql=( "CREATE INDEX IF NOT EXISTS doc_embeddings_text_fts " "ON doc_embeddings USING GIN (to_tsvector('english', text));" ), teardown_sql= "DROP INDEX IF EXISTS doc_embeddings_text_fts;", ) ], ) ``` Check the full documentation [here](https://cocoindex.io/docs/connectors/postgres/). #### **Neo4j improvements** - Support vector index method - Changed the KEY constraint to UNIQUE in Neo4j mappings for compatibility with the Community Edition. [*#983*](https://github.com/cocoindex-io/cocoindex/pull/983) #### New LLM integrations - add AWS Bedrock LLM support CocoIndex provides builtin functions integrating with various LLM APIs, for various inference tasks. In the new release we’ve added AWS Bedrock LLM Support in addition to all major commercial and open source LLMs / APIs supported. See the [full documentation](https://cocoindex.io/docs/ops/litellm/) here. ## Native functions CocoIndex’s native building blocks for **codebase indexing** received major upgrades this month, driven by extensive community feedback. We’ve refined language detection, splitting strategies, and parsing robustness for diverse codebases. - `DetectProgrammingLanguage` automatically identifies programming languages from file content or extensions, improving indexing precision across mixed-language repositories. - Improved [`SplitRecursively`](https://cocoindex.io/docs/ops/text/) function. Improved its `tree-sitter` based chunking method with better markdown handling, richer punctuation handling for plain text, configurable chunk overlap, and Solidity support. - `SplitBySeparators` is a lightweight regex-only splitter for simpler, high-performance text segmentation workflows. To see how to build customized codebase indexing, you can [get started](https://cocoindex.io/examples/code_index) here. ## New blogs and examples: https://cocoindex.io/blogs/etl-to-snowflake - Incrementally Transform Structured + Unstructured Data from Postgres with AI [This blog](https://cocoindex.io/blogs/postgres-source) introduces the new PostgreSQL source and shows how to take data from a PostgreSQL table as a source, transform with both AI models and non-AI calculations, and write them into a new PostgreSQL table for semantic + structured search. - Automated invoice processing with AI, Snowflake and CocoIndex - with incremental processing [This blog](https://cocoindex.io/blogs/etl-to-snowflake) shows how to automate invoice processing using CocoIndex, an open-source ETL framework built for AI-powered data transformation. It walks through how to extract supplier invoices from Azure Blob Storage, use LLMs to read and structure the data, and then load everything into Snowflake, all with incremental processing for scalability and data freshness. - Build a Visual Document Index from multiple formats all at once - PDFs, Images, Slides - with ColPali [This blog](https://cocoindex.io/blogs/multi-format-indexing) shows how to build a visual document index from multiple formats all at once - PDFs, Images, Slides - with ColPali. It walks through how to use ColPali to extract text from multiple formats, and then use CocoIndex to index the text into a visual document index. ## **Thanks to the Community 🤗🎉** Welcome new contributors to the CocoIndex community! We are so excited to have you! ### @MrAnayDongre Thanks [@MrAnayDongre](https://github.com/MrAnayDongre) for the work on SplitBySeparators regex-only splitter [#1010](https://github.com/cocoindex-io/cocoindex/pull/1010), enabling simpler and high-performance text segmentation workflows. ### @lemorage Thanks [@lemorage](https://github.com/lemorage) for adding explicit type hints in engine value encoding for the Python SDK [#807](https://github.com/cocoindex-io/cocoindex/pull/807), making value serialization more robust and type-safe. ### @banrovegrie Thanks [@banrovegrie](https://github.com/banrovegrie) for fixing the Gemini embedding configuration and exposing Postgres index tuning [#1050](https://github.com/cocoindex-io/cocoindex/pull/1050), improving integration reliability. ### @thisisharsh7 Thanks [@thisisharsh7](https://github.com/thisisharsh7) for the work on: These updates improve usability, reliability, and cross-platform compatibility, making workflows smoother and more robust for all users. ### @Davda-James Thanks [@Davda-James](https://github.com/Davda-James) for the work on: These updates improve developer productivity, maintainability, and observability, making the Python SDK and CLI easier to use, extend, and monitor for large-scale workflows. ### @princyballabh Thanks [@princyballabh](https://github.com/princyballabh) for auto-applying safe defaults for missing fields in load_engine_object [#1104](https://github.com/cocoindex-io/cocoindex/pull/1104), improving reliability and reducing errors when loading engine objects. ### @shresthashim Thanks [@shresthashim](https://github.com/shresthashim) for implementing __str__ and __repr__ methods for Python schema classes [#1095](https://github.com/cocoindex-io/cocoindex/pull/1095), enhancing developer experience when working with schema objects. ### @belloibrahv Thanks [@belloibrahv](https://github.com/belloibrahv) for the work on: These updates expand integration capabilities, improve graph database support, and enhance schema clarity. ### @mensonones Thanks [@mensonones](https://github.com/mensonones) for updating documentation to use cocoindex update --setup main [#1093](https://github.com/cocoindex-io/cocoindex/pull/1093), making setup instructions clearer. ### @esther-anierobi Thanks [@esther-anierobi](https://github.com/esther-anierobi) for improving documentation navigation [#1136](https://github.com/cocoindex-io/cocoindex/pull/1136). ### @TheVijayVignesh Thanks [@TheVijayVignesh](https://github.com/TheVijayVignesh) for moving built-in sources to top-level navigation in the docs [#1119](https://github.com/cocoindex-io/cocoindex/pull/1119), improving documentation structure. ### @aryasoni98 Thanks [@aryasoni98](https://github.com/aryasoni98) for adding tests for json_schema.rs [#1133](https://github.com/cocoindex-io/cocoindex/pull/1133), improving code reliability. ### @skalwaghe-56 Thanks [@skalwaghe-56](https://github.com/skalwaghe-56) for adding the legacy-states-v0 Cargo feature for backward compatibility [#1135](https://github.com/cocoindex-io/cocoindex/pull/1135), making it easier to maintain legacy state logic. ### @siddharthbaleja7 Thanks [@siddharthbaleja7](https://github.com/siddharthbaleja7) for flattening recursion in SplitRecursively to prevent stack overflow [#1127](https://github.com/cocoindex-io/cocoindex/pull/1127), improving stability. ## Support us We are constantly improving CocoIndex, and more features are coming soon! Stay tuned and follow us by starring our [GitHub repo](https://github.com/cocoindex-io/cocoindex). ## Sitemap - [Blog index](https://cocoindex.io/blogs/) - [Site index (llms.txt)](https://cocoindex.io/llms.txt) - [Full blog corpus](https://cocoindex.io/llms-full.txt) - [Markdown sitemap](https://cocoindex.io/sitemap.md) - [XML sitemap](https://cocoindex.io/sitemap.xml) - [RSS feed](https://cocoindex.io/blogs/rss.xml)