# Entity resolution > **CocoIndex v1, Rust SDK (experimental).** This page documents the CocoIndex **v1** Rust SDK (the `cocoindex` crate). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are at https://cocoindex.io/docs/. > > Source: https://cocoindex.io/docs/rust/ops/entity_resolution/ · Rust docs index: https://cocoindex.io/docs/rust/llms.txt · Rust SDK tour: https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/SHOWCASE.md · Rust examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust The `cocoindex::entity_resolution` module resolves a set of raw entity names into a deduplicated canonical map. It finds near-duplicate names via embedding similarity, then asks a pair-resolver (typically an LLM) to confirm matches and pick the better canonical name. ```rust use cocoindex::entity_resolution; use cocoindex::ops::api::LlmPairResolver; ``` The candidate search is an in-process cosine scan over the entities' unit vectors, not a FAISS index, so there is nothing extra to install. **Note — Cargo features** `cocoindex::entity_resolution` is behind no Cargo feature, and its types are in `cocoindex::prelude`. The built-in [`LlmPairResolver`](#llmpairresolver) is in `cocoindex::ops::api`, behind the `embed_api` feature; the local [`SentenceTransformerEmbedder`](/docs/rust/ops/sentence_transformers/) is behind the `fastembed` feature. ## Basic usage ```rust use std::collections::HashMap; use cocoindex::entity_resolution; use cocoindex::ops::api::{ApiChatClient, LlmPairResolver}; use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder; use cocoindex::prelude::*; // LLM used to confirm entity-resolution pairs. Each decision's memo key holds // the model, so no change detection on the key is needed. cocoindex::context_key!(pub static RESOLVER_LLM: ApiChatClient); // Local embedder for entity-resolution similarity. cocoindex::context_key!( pub static EMBEDDER: SentenceTransformerEmbedder, detect_change ); #[cocoindex::function(memo)] pub async fn resolve_entities( ctx: &Ctx, kind: &String, names: &Vec, ) -> Result>> { let embedder = ctx.get_key(&EMBEDDER)?; let resolver = LlmPairResolver::with_client(ctx.get_key(&RESOLVER_LLM)?.clone()) .with_entity_type(kind.as_str()); let resolved = entity_resolution::resolve_entities( ctx, names, embedder, &resolver, None, entity_resolution::ResolveOptions::default(), ) .await?; Ok(resolved.to_map().into_iter().collect()) } ``` `resolve_entities(&ctx, entities, &embedder, &resolve_pair, is_existing_canonical, options)` takes every argument positionally: the entities (any iterable of strings), an `EntityEmbedder`, a `PairResolver`, an optional existing-canonical predicate, and a `ResolveOptions`. The `Ctx` is passed through to the embedder and the resolver, so both memoize and batch through the engine. ## `ResolvedEntities` `resolve_entities` returns a `ResolvedEntities` object — a read-only wrapper around the dedup map with safe chain-walking: ```rust let result = entity_resolution::resolve_entities(/* ... */).await?; result.canonical_of("Microsoft Corp.")?; // -> "Microsoft" result.canonical_of("Microsoft")?; // -> "Microsoft" (self-canonical) result.canonicals(); // -> BTreeSet<&str> result.groups()?; // -> BTreeMap> result.to_map(); // -> BTreeMap> ``` `canonical_of` and `groups` return a `Result`: an unknown name is an error. `to_map` returns the whole map: each name to its canonical, or `None` for a canonical name. ## Existing-canonical handling If some entity names are already established as canonical (e.g., they have on-disk files or database records), you can pass `is_existing_canonical` to influence how matches involving those names are resolved. ### Without `is_existing_canonical` (default) The resolver decides which side becomes canonical on every match. No special treatment for any name. ```rust let result = entity_resolution::resolve_entities( &ctx, raw_entities, &embedder, &resolver, None, ResolveOptions::default(), ) .await?; ``` ### With `is_existing_canonical` Pass a predicate that returns `true` for names you consider already-established canonicals. The `existing_policy` field of `ResolveOptions` controls how strongly that status is enforced: #### `Pinned` (default) Existing canonicals are immutable. They are seeded directly as canonicals without consulting the resolver. Two existings never merge. Non-existing entities are resolved in a second pass; if they match an existing canonical, the existing always wins. ```rust let is_existing = |name: &str| existing_files.contains(name); let result = entity_resolution::resolve_entities( &ctx, raw_entities, &embedder, &resolver, Some(&is_existing as &(dyn Fn(&str) -> bool + Sync)), ResolveOptions::default(), // existing_policy defaults to Pinned ) .await?; ``` #### `Preferred` A softer policy: existing-canonical status breaks ties, but the resolver is always consulted. When exactly one side of a match is existing-canonical, that side wins regardless of the resolver's verdict. When both or neither are existing, the resolver decides. ```rust let options = ResolveOptions { existing_policy: ExistingCanonicalPolicy::Preferred, ..ResolveOptions::default() }; let result = entity_resolution::resolve_entities( &ctx, raw_entities, &embedder, &resolver, Some(&is_existing as &(dyn Fn(&str) -> bool + Sync)), options, ) .await?; ``` ## Events Call `resolve_entities_with_events` with an `on_resolution` callback to receive a `ResolutionEvent` for each entity. Events are delivered once resolution finishes, in the same order the sequential implementation would have emitted them (`Preferred`: sorted by entity; `Pinned`: pass-1 existings first then pass-2 non-existings): ```rust let cb = |event: &ResolutionEvent| collected.lock().unwrap().push(event.clone()); let result = entity_resolution::resolve_entities_with_events( &ctx, ["C", "B", "A"], &embedder, &resolver, None, ResolveOptions::default(), Some(&cb), ) .await?; ``` The callback is the last argument of `resolve_entities_with_events`; `resolve_entities` is the same call without it. Each event includes: - `entity` / `canonical` — the entity and its resolved canonical - `candidates` — what was passed to the resolver (empty if no resolver call) - `decision` — the resolver's raw verdict (compare with `canonical` to detect policy overrides) - `repointed` — if a prior canonical was demoted, its name - `seeded` — `true` for pinned existing-canonical entities seeded without resolver ## `resolve_entities` parameters The parameters, in order: | Parameter | Type | Default | |-----------|------|---------| | `ctx` | `&Ctx` | *(required)* | | `entities` | `impl IntoIterator>` | *(required)*; duplicates collapsed, iterated in sorted order | | `embedder` | `&impl EntityEmbedder` (any [`Embedder`](/docs/rust/common_resources/data_types/#embedder)) | *(required)* | | `resolve_pair` | `&impl PairResolver` | *(required)* | | `is_existing_canonical` | `Option<&(dyn Fn(&str) -> bool + Sync)>` | `None` | | `options` | `ResolveOptions { existing_policy, max_distance, top_n }` | `ResolveOptions::default()`: `Pinned`, `0.3`, `5` | | `on_resolution` (`resolve_entities_with_events` only) | `Option<&(dyn Fn(&ResolutionEvent) + Sync)>` | `None` | `max_distance` is the cosine distance threshold (similarity >= 0.7 at the default `0.3`), and `top_n` is the maximum number of candidates surfaced per entity. `existing_policy` is ignored when `is_existing_canonical` is `None`. --- ## `LlmPairResolver` `cocoindex::ops::api` provides `LlmPairResolver`, a built-in resolver that uses an LLM to decide pair matches. It sends each `(entity, candidates)` pair to the model and returns a structured decision — with automatic validation and retry when the model hallucinates a name not in the candidate list. It calls an OpenAI-compatible `/chat/completions` endpoint through [`ApiChatClient`](/docs/rust/ops/model_apis/#chat-and-structured-extraction), in JSON mode. The resolver does not use litellm or `instructor`: it asks the endpoint for a JSON-object response and parses it. The prompt, the validation and the retry behavior are the same as in Python. ### Usage ```rust use cocoindex::entity_resolution; use cocoindex::ops::api::LlmPairResolver; // Defaults to the OpenAI base URL and the OPENAI_API_KEY environment variable. let resolver = LlmPairResolver::new("gpt-4o-mini"); // Or against another OpenAI-compatible endpoint: let resolver = LlmPairResolver::new("gpt-4o-mini") .with_base_url(base_url) .with_api_key("k"); let result = entity_resolution::resolve_entities( &ctx, ["Barack Obama", "Barack H. Obama", "OpenAI"], &embedder, &resolver, None, ResolveOptions::default(), ) .await?; ``` `LlmPairResolver::with_client(client)` builds a resolver on a pre-configured `ApiChatClient`, for example one held in a context key. ### Model names The resolver takes the bare model name the endpoint expects (`gpt-4o-mini`), with `with_base_url` for a provider other than OpenAI. See [Model APIs](/docs/rust/ops/model_apis/). ### Entity type hints Pass `with_entity_type` to tailor the prompt for specific entity categories: ```rust let person_resolver = LlmPairResolver::new("gpt-4o-mini").with_entity_type("person"); let tech_resolver = LlmPairResolver::new("gpt-4o-mini").with_entity_type("technology"); ``` This adds context to the system prompt, helping the model make better judgments (e.g., being more conservative with personal names). ### Extra guidance Append domain-specific rules to the default prompt via `with_extra_guidance`: ```rust let resolver = LlmPairResolver::new("gpt-4o-mini") .with_entity_type("organization") .with_extra_guidance( "A parent organization and its subsidiary/division are DISTINCT things. \ 'Amazon' is not the same as 'AWS'. 'Google' is not the same as 'YouTube'.", ); ``` The guidance is for **domain rules only** — do not include output-format instructions. ### Validation and retry After each LLM call, the response is validated: 1. **Structural**: the JSON response must parse into the expected schema 2. **Semantic**: `matched` (if non-null) must be in the supplied `candidates`. If not, the LLM is re-prompted with explicit feedback and the conversation continues The default retry budget is 2. If exhausted, no match is returned. ```rust let resolver = LlmPairResolver::new("gpt-4o-mini").with_retries(3); // more retries let resolver = LlmPairResolver::new("gpt-4o-mini").with_retries(1); // fewer retries ``` ### Memoization Each unique `(entity, candidates)` pair's decision is [memoized](/docs/rust/programming_guide/function/) through the `ctx` the resolver receives, and persists across runs. The memo key holds the model, base URL, entity type and extra guidance, so changing the model or prompt invalidates the cache; `retries` is absent, since it cannot change a valid decision. No additional memoization wrapper is needed. ### Parameters | Builder | Default | Description | |---------|---------|-------------| | `LlmPairResolver::new(model)` | *(required)* | The model name the endpoint expects (e.g. `"gpt-4o-mini"`); builds its own `ApiChatClient`. | | `LlmPairResolver::with_client(client)` | | Builds on a pre-configured `ApiChatClient` instead of `new`. | | `with_base_url(url)` | OpenAI's | The endpoint's base URL. | | `with_api_key(key)` | `OPENAI_API_KEY` | The bearer API key. | | `with_entity_type(t)` | none | Entity type hint woven into the prompt. | | `with_extra_guidance(text)` | none | Domain rules appended to the default prompt. | | `with_retries(n)` | `2` | Max retries on invalid `matched` output. | --- ## Custom resolvers For cases where the built-in `LlmPairResolver` doesn't fit (rule-based matching, different LLM framework, etc.), implement the `PairResolver` trait: ```rust use cocoindex::{CanonicalSide, Ctx, PairDecision, PairResolver, async_trait}; struct MyResolver; #[async_trait] impl PairResolver for MyResolver { async fn resolve_pair( &self, _ctx: &Ctx, entity: &str, candidates: &[String], ) -> cocoindex::Result { // `candidates` is a non-empty, de-duplicated list of canonical names // with cosine similarity to `entity` above the threshold. // No match: Ok(PairDecision::no_match()) // Match — existing candidate stays canonical (default): // Ok(PairDecision::matched("Microsoft")) // Match — new entity is the better canonical name: // Ok(PairDecision::matched_with("Microsoft Corp.", CanonicalSide::New)) } } ``` `resolve_pair` also receives the `Ctx`, so a resolver can memoize through the engine as `LlmPairResolver` does. It may run concurrently across components. `PairDecision.canonical` is advisory: the `existing_policy` may override it (see [Existing-canonical handling](#existing-canonical-handling)).