CocoIndex core concepts
Declare your outputs. CocoIndex applies changes and reuses cached results.
Incremental processing
Data and code change over time. Incremental processing reruns affected work and updates targets, such as a search database.
Manual implementations must track changes and preserve intermediate results. They must also manage inserts, updates, deletes, schema changes, and backfills.
State-driven programming
Declare the target state you want from the current source data. CocoIndex computes and applies the changes needed to match it.
TargetState = Transform(SourceState)
The target state is a pure function of the source state, with no other side effects. Transformations read sources and produce target data, such as database rows or files.
Compare familiar programming models
- React: Declare UI from state. React updates the UI when state changes.
- Spreadsheets: Declare formulas. Upstream changes recompute dependent cells.
- Materialized views: Declare a query. Source changes refresh the view.
App
An app is CocoIndex’s top-level executable. It reads sources, transforms data, and declares target states. CocoIndex synchronizes those target states with external systems, such as Postgres or a vector database.
For example, here’s an app that reads PDFs from a drive, converts them to markdown, and outputs to a folder:
Processing component
A processing component owns the work and target states for one source item. It runs independently. When it finishes, CocoIndex applies its target states without waiting for other components.
For example, if you have many files in a drive and want to process them file by file, your processing component would operate at the file level:
One file can produce several target states, such as one embedding vector per text chunk. CocoIndex batches the inserts, updates, and deletes for each file after processing completes. Each target backend applies its batch atomically when supported, such as within a database transaction:
These examples show how target states change:
Adding c.md creates a processing component for it. After processing completes, CocoIndex inserts vector5 and vector6.
Updating b.md reduces its content to one chunk.
Its target states change from vector3 and vector4 to vector5.
CocoIndex deletes the old rows and inserts the new row in one transaction.
Deleting b.md removes its associated target states, vector3 and vector4, in one transaction.
Function memoization: skip unchanged computations
A memoized function reuses its previous result when its inputs and code remain unchanged. Both processing components and transforms are functions, so you can memoize either level:
- Processing component: Skip the whole file when its content and processing logic remain unchanged.
- Transform: Reuse an embedding when its text and embedding logic, including the model, remain unchanged.
See Function Memoization for more details.
Here’s how memoization behaves in different scenarios:
When b.md changes:
a.mdand its processing component remain cached.b.mdruns again and produceschunk3andchunk5.Embed(chunk3)reuses its cached result.Embed(chunk5)requires a new computation.
When the “Split into chunks” logic changes:
- Both processing components run again.
a.mdproduces the same chunks, so itsEmbedresults are reused.b.mdproduceschunk5andchunk6, which require newEmbedcomputations.
A code change can still reuse downstream results when their inputs and function logic remain unchanged.
Try incremental processing
Edit a file and compare the updated row with the cached work. This example uses simulated data.
Loading the playground…
Next steps
Use the Programming Guide to apply these concepts in code:
- App — creating and running pipelines
- Target State — declaring what should exist in external systems
- Processing Component — structuring work and mounting components
- Function — the
#[cocoindex::function]attribute, memoization, and change detection