An indexing pipeline builds indexes derived from source data. The index should always be converging to the current version of source data. In other words, once a new version of source data is processed by the pipeline, all data derived from previous versions should no longer exist in the target index storage. This is called the data consistency requirement for an indexing pipeline.
The challenge of concurrent updates
Compared to building single-process in-memory transformation logic, an indexing pipeline has additional challenges in two aspects, to maintain data consistency:
-
Temporally, it’s long-running, i.e. the execution may span across multiple processes. e.g. an early execution may terminate in the middle, due to a SIGTERM, or a failure to commit to the target storage or a reboot. When the process restarts later, it should pick up existing states, and keep moving forward to update the target storage to the desired status. This is both tedious and error-prone: for example, if source data is only partially processed, and it gets updated before the restarted pipeline starts to process it again, how can we make sure states derived from the old version are properly cleared?
-
Spatially, execution happens concurrently, sometimes distributed across multiple machines for large workloads. This brings up the problem caused by out-of-order processing. Consider a scenario where your source data gets updated rapidly:
- Version 1 (V1) exists in the system
- Version 2 (V2) and Version 3 (V3) arrive within a second of each other
- V2 takes longer to process than V3 Without proper handling, you might end up with V2 data in your index even though V3 is the latest version - effectively rolling back to stale data.
Common issues and gaps
Data exposure risks
Data inconsistency in indexing systems can lead to several critical issues:
System integrity
- Stale data persists in indexes despite source updates
- Inconsistent states across different storage systems
- Mixed versions of the same source entry coexisting in the index
Security and compliance
- Deleted sensitive information remains accessible through search interfaces
- Outdated personal data violates privacy requirements (e.g., GDPR)
- Regulatory non-compliance due to improperly retained records
AI system reliability
- RAG systems incorporating outdated information into responses
- Semantic search surfacing deprecated or retracted content
- AI agents making decisions based on stale data
Business operations
- Customer-facing applications showing incorrect information
- Loss of trust due to inconsistent data presentation
- Financial impact from outdated pricing or product details
Approaches for maintaining data consistency
1. Tracking source-to-target key mappings
If all your state management and storage systems are within a single transactional system (e.g., PostgreSQL), you can leverage database transactions for consistency. However, this is often not the case in real-world scenarios. For example, you may use an internal storage (e.g. PostgreSQL) to track per-source metadata, and use an external vector store (e.g. Pinecone or Milvus) to store vector embeddings.
There are extra challenges around this:
-
We have to keep track of all the keys in target vector stores derived from each source key, in the internal storage. This is necessary when the old version of source data is updated or deleted: we need to locate previously derived keys and delete them if they no longer exist in the new version after processing.
-
However, the internal storage is not transactional with the external vector store, as they’re two independent storage systems. So if the process terminates abnormally in the middle, states in the two systems will be out of sync.
So we need to carefully design the order of write operations during commit, to avoid data leakage even when the process is only partially executed:
- Add new target keys to the internal store
- Write to target storage
- Create new records
- Delete old records Note that for any target storage that supports atomic writing, this allows all updates derived from the same source key to be written atomically. This avoids mixed-version data at query time.
- Remove no-longer-exist target keys from the internal store
This ensures a key invariant: keys tracked in the internal storage are always a superset of those that really exist in the target store. This makes sure all the data in the target store is always tracked so never leaked.
2. Ordinal bookkeeping
Ordinals are unique identifiers that establish a strict ordering of data updates in your pipeline. They help bookkeep the sequence and timing of changes to ensure consistency. Common examples include:
- Timestamps (e.g., 2024-01-06T12:34) - Precise time-based ordering
- Version numbers (e.g., v1.2.3, 20240106.1) - Sequential numbering
Bookkeep ordinals carefully in two key stages:
- During ingestion: Record and preserve the source ordinal (e.g., timestamp or version) from the original data
- During commit: Verify the source ordinal to prevent newer versions from being overwritten by older ones
Ordinal bookkeeping happens in Phase 1, which rejects out-of-order commits - they won’t cause any updates in Phase 1, and Phase 2 and 3 won’t be executed at all.
3. Versioned unique key assignment with soft deletion
When there are multiple ongoing processings on the same source key, updates writing to the target storage (Phase 2 above) may be out of order, which causes results derived from old versions to overwrite newer ones.
To avoid this, we need to:
- For data in the target storage, make sure the keys are unique even across different versions.
- Leverage soft deletion in the target storage, i.e. deleted versions are represented by a row with a
deletedfield set.
Specifically:
- In the transaction in Phase 1, we establish a full view of all derived keys for the current source key: collect existing keys, and generate new keys.
- In Phase 2, we do UPSERT for both new and deleted versions - the only difference is the
deletedfield is set for deleted versions, and not touched for new versions. - During query time, we filter out all records with the
deletedfield set.
Besides, we need an offline GC process to garbage collect rows in the target storage with the deleted field set. Because of this, the deleted field could be a timestamp of the deletion time, hence we can decide when to GC a specific version based on it.
The CocoIndex paradigm
The discussions above only cover a part of the complexity of building and maintaining long-running or distributed indexing pipelines. There are other additional complexities, compared to building single-process in-memory transformation logic, such as:
- CRUD across different APIs
- Incremental historical state management
- Reuse existing computations
- Data migration on processing logic changes
The CocoIndex framework aims at handling these complexities so users can focus on pure transformations:
-
Users write transformations as pure functions
- Output depends only on input
- No side effects or hidden states
- Easy to understand and maintain
-
Framework handles:
- Previous computation result management
- Incremental processing
- State management
- Data consistency
- Data upgrade on processing logic changes
By exposing a data-driven programming paradigm, CocoIndex allows developers to focus on their core business logic while maintaining data consistency and freshness.
Join our community!
Interested in learning more about CocoIndex? Join our community!
- Follow our GitHub repository to stay up to date with the latest developments.
- Check out our documentation to learn more about how CocoIndex can help build robust AI applications with properly indexed data.
- Join our Discord community to connect with other developers and get support.
- Follow us on Twitter for the latest updates.
About the author.
Posts from the CocoIndex team — product launches, release notes, and announcements.
Frequently asked questions.
What does data consistency mean for an indexing pipeline?
An indexing pipeline builds indexes derived from source data, and the index should always be converging to the current version of that source. Concretely: once a new version of the source is processed, no data derived from previous versions should still exist in the target index storage. That requirement is what the post calls data consistency.
See the opening definition and the challenge of concurrent updates.
Why are concurrent updates hard for indexing pipelines?
There are two dimensions. Temporally, the pipeline is long-running and may span multiple processes — an execution can die mid-way (SIGTERM, commit failure, reboot) and a restart must pick up existing state and still clear data derived from old source versions. Spatially, execution happens concurrently and sometimes across machines, creating out-of-order processing: if V2 and V3 arrive nearly together but V2 finishes last, you can end up with stale V2 data in the index.
What can go wrong when an index is inconsistent with its source?
The post groups the risks: system integrity (stale data persisting, mixed versions of an entry coexisting), security and compliance (deleted sensitive data still searchable, outdated personal data violating privacy rules like GDPR), AI system reliability (RAG and agents acting on stale or retracted content), and business operations (customer-facing apps showing wrong info, lost trust, outdated pricing).
See Data exposure risks.
How do you keep an internal store consistent with a non-transactional vector store?
When internal metadata storage (e.g. PostgreSQL) and an external vector store (e.g. Pinecone or Milvus) are not transactional with each other, you order the commit writes so no data leaks even on partial execution: (1) add the new target keys to the internal store, (2) write to target storage (create new records, delete old ones — atomically per source key where supported), then (3) remove no-longer-existing target keys from the internal store. This preserves the invariant that tracked keys are always a superset of stored keys, so nothing is ever leaked or left untracked.
What are ordinals and how do they prevent stale data overwrites?
Ordinals are unique identifiers that establish a strict ordering of updates — typically timestamps or version numbers. You record the source ordinal during ingestion and verify it during commit so a newer version can't be overwritten by an older one. This bookkeeping happens in Phase 1, which rejects out-of-order commits before Phases 2 and 3 run at all.
See Ordinal bookkeeping.
How does soft deletion with versioned keys avoid out-of-order overwrites?
When multiple processings of the same source key run, the Phase-2 writes to target storage can arrive out of order. To prevent old versions overwriting newer ones, you make target keys unique across versions and use soft deletion: Phase 1 builds a full view of derived keys (existing plus new), Phase 2 UPSERTs both new and deleted versions (deleted ones just have a deleted field set), and queries filter out rows with deleted set. An offline GC process later removes the soft-deleted rows, which is why deleted can hold the deletion timestamp.
How does CocoIndex let developers ignore these consistency concerns?
CocoIndex exposes a data-driven paradigm where users write transformations as pure functions (output depends only on input, no side effects or hidden state). The framework handles previous-computation-result management, incremental processing, state management, data consistency, and data upgrades when processing logic changes — so developers focus on business logic while the index stays consistent and fresh.
