Entity resolution

Deduplicate entity names via embedding similarity and a pluggable LLM pair-resolver that confirms matches and chooses canonical forms — with Pinned / Preferred policies for existing canonicals and resolution callbacks.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The cocoindex::entity_resolution module resolves a set of raw entity names into a deduplicated canonical map. It finds near-duplicate names via embedding similarity, then asks a pair-resolver (typically an LLM) to confirm matches and pick the better canonical name.

use cocoindex::entity_resolution;
use cocoindex::ops::api::LlmPairResolver;

Basic usage

use std::collections::HashMap;

use cocoindex::entity_resolution;
use cocoindex::ops::api::{ApiChatClient, LlmPairResolver};
use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder;
use cocoindex::prelude::*;

// LLM used to confirm entity-resolution pairs. Each decision's memo key holds
// the model, so no change detection on the key is needed.
cocoindex::context_key!(pub static RESOLVER_LLM: ApiChatClient);

// Local embedder for entity-resolution similarity.
cocoindex::context_key!(
    pub static EMBEDDER: SentenceTransformerEmbedder,
    detect_change
);

#[cocoindex::function(memo)]
pub async fn resolve_entities(
    ctx: &Ctx,
    kind: &String,
    names: &Vec<String>,
) -> Result<HashMap<String, Option<String>>> {
    let embedder = ctx.get_key(&EMBEDDER)?;
    let resolver = LlmPairResolver::with_client(ctx.get_key(&RESOLVER_LLM)?.clone())
        .with_entity_type(kind.as_str());
    let resolved = entity_resolution::resolve_entities(
        ctx,
        names,
        embedder,
        &resolver,
        None,
        entity_resolution::ResolveOptions::default(),
    )
    .await?;
    Ok(resolved.to_map().into_iter().collect())
}

resolve_entities(&ctx, entities, &embedder, &resolve_pair, is_existing_canonical, options) takes every argument positionally: the entities (any iterable of strings), an EntityEmbedder, a PairResolver, an optional existing-canonical predicate, and a ResolveOptions. The Ctx is passed through to the embedder and the resolver, so both memoize and batch through the engine.

ResolvedEntities

resolve_entities returns a ResolvedEntities object — a read-only wrapper around the dedup map with safe chain-walking:

let result = entity_resolution::resolve_entities(/* ... */).await?;

result.canonical_of("Microsoft Corp.")?; // -> "Microsoft"
result.canonical_of("Microsoft")?;       // -> "Microsoft" (self-canonical)
result.canonicals();                     // -> BTreeSet<&str>
result.groups()?;                        // -> BTreeMap<String, BTreeSet<String>>
result.to_map();                         // -> BTreeMap<String, Option<String>>

canonical_of and groups return a Result: an unknown name is an error. to_map returns the whole map: each name to its canonical, or None for a canonical name.

Existing-canonical handling

If some entity names are already established as canonical (e.g., they have on-disk files or database records), you can pass is_existing_canonical to influence how matches involving those names are resolved.

Without is_existing_canonical (default)

The resolver decides which side becomes canonical on every match. No special treatment for any name.

let result = entity_resolution::resolve_entities(
    &ctx,
    raw_entities,
    &embedder,
    &resolver,
    None,
    ResolveOptions::default(),
)
.await?;

With is_existing_canonical

Pass a predicate that returns true for names you consider already-established canonicals. The existing_policy field of ResolveOptions controls how strongly that status is enforced:

Pinned (default)

Existing canonicals are immutable. They are seeded directly as canonicals without consulting the resolver. Two existings never merge. Non-existing entities are resolved in a second pass; if they match an existing canonical, the existing always wins.

let is_existing = |name: &str| existing_files.contains(name);
let result = entity_resolution::resolve_entities(
    &ctx,
    raw_entities,
    &embedder,
    &resolver,
    Some(&is_existing as &(dyn Fn(&str) -> bool + Sync)),
    ResolveOptions::default(), // existing_policy defaults to Pinned
)
.await?;

Preferred

A softer policy: existing-canonical status breaks ties, but the resolver is always consulted. When exactly one side of a match is existing-canonical, that side wins regardless of the resolver’s verdict. When both or neither are existing, the resolver decides.

let options = ResolveOptions {
    existing_policy: ExistingCanonicalPolicy::Preferred,
    ..ResolveOptions::default()
};
let result = entity_resolution::resolve_entities(
    &ctx,
    raw_entities,
    &embedder,
    &resolver,
    Some(&is_existing as &(dyn Fn(&str) -> bool + Sync)),
    options,
)
.await?;

Events

Call resolve_entities_with_events with an on_resolution callback to receive a ResolutionEvent for each entity. Events are delivered once resolution finishes, in the same order the sequential implementation would have emitted them (Preferred: sorted by entity; Pinned: pass-1 existings first then pass-2 non-existings):

let cb = |event: &ResolutionEvent| collected.lock().unwrap().push(event.clone());
let result = entity_resolution::resolve_entities_with_events(
    &ctx,
    ["C", "B", "A"],
    &embedder,
    &resolver,
    None,
    ResolveOptions::default(),
    Some(&cb),
)
.await?;

The callback is the last argument of resolve_entities_with_events; resolve_entities is the same call without it.

Each event includes:

• entity / canonical — the entity and its resolved canonical
• candidates — what was passed to the resolver (empty if no resolver call)
• decision — the resolver’s raw verdict (compare with canonical to detect policy overrides)
• repointed — if a prior canonical was demoted, its name
• seeded — true for pinned existing-canonical entities seeded without resolver

resolve_entities parameters

The parameters, in order:

max_distance is the cosine distance threshold (similarity >= 0.7 at the default 0.3), and top_n is the maximum number of candidates surfaced per entity. existing_policy is ignored when is_existing_canonical is None.

LlmPairResolver

cocoindex::ops::api provides LlmPairResolver, a built-in resolver that uses an LLM to decide pair matches. It sends each (entity, candidates) pair to the model and returns a structured decision — with automatic validation and retry when the model hallucinates a name not in the candidate list. It calls an OpenAI-compatible /chat/completions endpoint through ApiChatClient, in JSON mode.

Usage

use cocoindex::entity_resolution;
use cocoindex::ops::api::LlmPairResolver;

// Defaults to the OpenAI base URL and the OPENAI_API_KEY environment variable.
let resolver = LlmPairResolver::new("gpt-4o-mini");

// Or against another OpenAI-compatible endpoint:
let resolver = LlmPairResolver::new("gpt-4o-mini")
    .with_base_url(base_url)
    .with_api_key("k");

let result = entity_resolution::resolve_entities(
    &ctx,
    ["Barack Obama", "Barack H. Obama", "OpenAI"],
    &embedder,
    &resolver,
    None,
    ResolveOptions::default(),
)
.await?;

LlmPairResolver::with_client(client) builds a resolver on a pre-configured ApiChatClient, for example one held in a context key.

Model names

The resolver takes the bare model name the endpoint expects (gpt-4o-mini), with with_base_url for a provider other than OpenAI. See Model APIs.

Entity type hints

Pass with_entity_type to tailor the prompt for specific entity categories:

let person_resolver = LlmPairResolver::new("gpt-4o-mini").with_entity_type("person");
let tech_resolver = LlmPairResolver::new("gpt-4o-mini").with_entity_type("technology");

This adds context to the system prompt, helping the model make better judgments (e.g., being more conservative with personal names).

Extra guidance

Append domain-specific rules to the default prompt via with_extra_guidance:

let resolver = LlmPairResolver::new("gpt-4o-mini")
    .with_entity_type("organization")
    .with_extra_guidance(
        "A parent organization and its subsidiary/division are DISTINCT things. \
         'Amazon' is not the same as 'AWS'. 'Google' is not the same as 'YouTube'.",
    );

The guidance is for domain rules only — do not include output-format instructions.

Validation and retry

After each LLM call, the response is validated:

1. Structural: the JSON response must parse into the expected schema
2. Semantic: matched (if non-null) must be in the supplied candidates. If not, the LLM is re-prompted with explicit feedback and the conversation continues

The default retry budget is 2. If exhausted, no match is returned.

let resolver = LlmPairResolver::new("gpt-4o-mini").with_retries(3); // more retries
let resolver = LlmPairResolver::new("gpt-4o-mini").with_retries(1); // fewer retries

Memoization

Each unique (entity, candidates) pair’s decision is memoized through the ctx the resolver receives, and persists across runs. The memo key holds the model, base URL, entity type and extra guidance, so changing the model or prompt invalidates the cache; retries is absent, since it cannot change a valid decision. No additional memoization wrapper is needed.

Parameters

Custom resolvers

For cases where the built-in LlmPairResolver doesn’t fit (rule-based matching, different LLM framework, etc.), implement the PairResolver trait:

use cocoindex::{CanonicalSide, Ctx, PairDecision, PairResolver, async_trait};

struct MyResolver;

#[async_trait]
impl PairResolver for MyResolver {
    async fn resolve_pair(
        &self,
        _ctx: &Ctx,
        entity: &str,
        candidates: &[String],
    ) -> cocoindex::Result<PairDecision> {
        // `candidates` is a non-empty, de-duplicated list of canonical names
        // with cosine similarity to `entity` above the threshold.

        // No match:
        Ok(PairDecision::no_match())

        // Match — existing candidate stays canonical (default):
        // Ok(PairDecision::matched("Microsoft"))

        // Match — new entity is the better canonical name:
        // Ok(PairDecision::matched_with("Microsoft Corp.", CanonicalSide::New))
    }
}

resolve_pair also receives the Ctx, so a resolver can memoize through the engine as LlmPairResolver does. It may run concurrently across components.

PairDecision.canonical is advisory: the existing_policy may override it (see Existing-canonical handling).