Sentence Transformers embeddings

Run sentence-transformers ONNX models locally for text embeddings through fastembed, memoized and batched by the engine, with GPU execution providers and VectorSchemaProvider integration for connectors.

Version
v 1.0.14
Last reviewed
Nov 4, 2026
ExperimentalThe Rust SDK is experimental. It is outside CocoIndex's compatibility promise: any release, including a minor one, may change or remove its API. It is not yet published on crates.io.

The cocoindex::ops::sentence_transformers module runs sentence-transformers models locally for text embeddings. It runs their ONNX exports through fastembed, with no Python runtime.

use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder;
Cargo feature

The module is behind the fastembed feature of the cocoindex crate:

cocoindex = { version = "...", features = ["fastembed"] }

Overview

The SentenceTransformerEmbedder struct wraps a locally loaded fastembed model. It:

  • Implements VectorSchemaProvider to supply vector schema information to CocoIndex connectors
  • Implements the Embedder trait, so it plugs into entity resolution directly
  • Provides an embed(&ctx, text) method, memoized and batched by the engine
  • Returns each embedding as a Vec<f32>

It runs the model on the ONNX Runtime execution providers compiled into the binary (CPU by default), on a blocking thread.

Different from Python

The Rust embedder is built on fastembed, not on the sentence-transformers library, so its options differ. It has no prompt_name and no normalize_embeddings: fastembed has no prompt registry and always L2-normalizes its output. It has no device either: load_with takes ONNX Runtime execution providers instead.

Basic usage

Creating an embedder

use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder;

// Load a pre-trained model; this downloads and initializes the ONNX model.
let embedder = SentenceTransformerEmbedder::load("sentence-transformers/all-MiniLM-L6-v2").await?;

load is async and returns a Result, because it downloads and initializes the model. embedder.dimension() returns the model’s dimension and embedder.model_name() the name it was loaded with. The embedder is cheap to clone.

Embedding text

The embed(&ctx, text) method converts text into a Vec<f32>:

// In a CocoIndex function: memoized per text, batched across concurrent calls.
let embedding = embedder.embed(&ctx, "Hello, world!").await?;

// Outside an update (a search query): one raw batch, no memoization.
let query_vec = embedder.embed_batch(vec![query.to_string()]).await?.remove(0);

Outside an update there is no Ctx: use embed_batch(texts), which embeds one batch directly.

Built-in memoization

embed(&ctx, text) is memoized: inside a processing component, each distinct text’s vector is cached across that component’s runs, so a caller that re-executes doesn’t recompute unchanged text. The tradeoff is model compute vs. cache storage (one full vector per distinct text). It also batches: concurrent cache misses in one update are coalesced into batches of up to 64 texts. The memo key holds the model name and dimension, so a different model invalidates the cached vectors. Mind the memoization placement rule if your components are keyed by the embedded content.

Declaring the vector column

The SentenceTransformerEmbedder implements VectorSchemaProvider, so a connector reads the vector dimension and element type from it when it creates a table. The row struct marks the column with #[coco(vector)]:

use cocoindex::prelude::*;

#[derive(Clone, Serialize, Deserialize, SchemaFields)]
struct DocEmbedding {
    id: i64,
    filename: String,
    chunk_start: i32,
    chunk_end: i32,
    text: String,
    #[coco(vector)]
    embedding: Vec<f32>,
}

A row struct cannot name the embedder in a field attribute: the embedder supplies the dimension when the table is mounted, below.

When you mount the table, pass the embedder to with_vector_schema, and the connector takes the vector column’s dimension and element type from it. For example, with Postgres:

use cocoindex::connectors::postgres;

// The embedder reports the dimension of the `embedding` column.
let table = postgres::mount_table_target(
    &ctx,
    &DB,
    "doc_embeddings",
    postgres::TableSchema::from_row::<DocEmbedding>(["id"])?
        .with_vector_schema("embedding", ctx.get_key(&EMBEDDER)?)
        .await?,
    Some("my_schema"),
)
.await?;

TableSchema::from_row::<Row>(primary_key) builds the schema from the row struct; with_vector_schema(field, &embedder) resolves the #[coco(vector)] column from any VectorSchemaProvider. See Vector schema annotations.

The connector automatically creates the appropriate vector(384) column. See the Connectors docs for other supported backends (LanceDB, Qdrant, SQLite).

Example: text embedding pipeline

Here’s a complete example of a text embedding pipeline (based on the text_embedding example):

use std::path::PathBuf;
use std::time::Duration;

use cocoindex::connectors::postgres;
use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder;
use cocoindex::ops::text::{RecursiveChunkConfig, RecursiveSplitter};
use cocoindex::prelude::*;

const EMBED_MODEL: &str = "sentence-transformers/all-MiniLM-L6-v2";
const PG_SCHEMA: &str = "coco_examples";
const TABLE: &str = "doc_embeddings";
const CHUNK_SIZE: usize = 2000;
const CHUNK_OVERLAP: usize = 500;

cocoindex::context_key!(static DB: postgres::Database);
cocoindex::context_key!(
    static EMBEDDER: SentenceTransformerEmbedder,
    detect_change
);

#[derive(Clone, Serialize, Deserialize, SchemaFields)]
struct DocEmbedding {
    id: i64,
    filename: String,
    chunk_start: i32,
    chunk_end: i32,
    text: String,
    #[coco(vector)]
    embedding: Vec<f32>,
}

/// Chunk and embed one file. Memoized on the file, so an unchanged file
/// returns its rows without re-chunking.
#[cocoindex::function(memo)]
async fn process_file(ctx: &Ctx, file: &FileEntry) -> Result<Vec<DocEmbedding>> {
    let filename = file.key();
    let text = file.content_str()?;

    let splitter = RecursiveSplitter::new()?;
    let chunks = splitter.split_with(
        &text,
        RecursiveChunkConfig {
            chunk_size: CHUNK_SIZE,
            min_chunk_size: None,
            chunk_overlap: Some(CHUNK_OVERLAP),
            language: Some("markdown".to_string()),
        },
    );
    if chunks.is_empty() {
        return Ok(Vec::new());
    }

    let texts: Vec<String> = chunks.iter().map(|c| c.text(&text).to_string()).collect();
    let embedder = ctx.get_key(&EMBEDDER)?.clone();
    let embedding_ctx = ctx.clone();
    let embeddings = ctx
        .map(texts.clone(), move |chunk_text| {
            let embedder = embedder.clone();
            let ctx = embedding_ctx.clone();
            async move { embedder.embed(&ctx, chunk_text).await }
        })
        .await?;

    let mut id_gen = IdGenerator::new();
    let mut rows = Vec::with_capacity(texts.len());
    for ((chunk, chunk_text), embedding) in chunks.iter().zip(texts).zip(embeddings) {
        let id = id_gen.next_id(&ctx, &chunk_text).await?;
        let id =
            i64::try_from(id).map_err(|_| Error::engine("generated id does not fit in BIGINT"))?;
        rows.push(DocEmbedding {
            id,
            filename: filename.clone(),
            chunk_start: chunk.start.char_offset as i32,
            chunk_end: chunk.end.char_offset as i32,
            text: chunk_text,
            embedding,
        });
    }
    Ok(rows)
}

/// Environment setup: the state db, Postgres and the embedder.
#[cocoindex::lifespan]
async fn lifespan(builder: EnvironmentBuilder) -> Result<EnvironmentBuilder> {
    let db = postgres::Database::connect(&database_url()).await?;
    let pool = db.pool().clone();
    let embedder = SentenceTransformerEmbedder::load(EMBED_MODEL).await?;
    Ok(builder
        .db_path(PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(".cocoindex_db"))
        .provide_key(&DB, db)
        .provide_key(&EMBEDDER, embedder)
        .on_shutdown(move || async move { pool.close().await }))
}

#[cocoindex::app(name = "TextEmbeddingRust")]
async fn app_main(ctx: Ctx) -> Result<()> {
    index(ctx, source_dir()).await
}

async fn index(ctx: Ctx, sourcedir: PathBuf) -> Result<()> {
    // The embedder reports the dimension of the `embedding` column.
    let table = postgres::mount_table_target(
        &ctx,
        &DB,
        TABLE,
        postgres::TableSchema::from_row::<DocEmbedding>(["id"])?
            .with_vector_schema("embedding", ctx.get_key(&EMBEDDER)?)
            .await?,
        Some(PG_SCHEMA),
    )
    .await?;

    // One component per markdown file; each declares its own rows. In live
    // mode (`cocoindex update -L`) the walker keeps watching the directory.
    let files = walk_dir(sourcedir)
        .recursive(true)
        .path_matcher(PatternFilePathMatcher::new(
            ["**/*.md"],
            std::iter::empty::<&str>(),
        )?)
        .live()
        .poll_interval(Duration::from_secs(1));
    ctx.mount_each_live(&"files", files, move |ctx, file: FileEntry| {
        let table = table.clone();
        async move {
            for row in process_file(&ctx, &file).await? {
                table.declare_row(&ctx, &row)?;
            }
            Ok(())
        }
    })
    .await
}

The full program, with the database_url() and source_dir() helpers, a query command and the #[cocoindex::main] entry point, is examples/rust/text_embedding. Run it with the CLI: cocoindex update, or cocoindex update -L to keep watching the directory.

Configuration options

Model selection

SentenceTransformerEmbedder::load(name_or_path) accepts a name from the fastembed model registry (matched case-insensitively, and by the trailing model name, so sentence-transformers/all-MiniLM-L6-v2 and Xenova/all-MiniLM-L6-v2 resolve to the same model), a local directory holding a sentence-transformers ONNX export (onnx/model.onnx or model.onnx plus the tokenizer files), or the Hugging Face Hub repository of such an export. HF_HOME and HF_ENDPOINT apply to the download. A model with a pooling mode other than CLS or mean is rejected. Pooling follows the export’s 1_Pooling/config.json when it has one.

let embedder = SentenceTransformerEmbedder::load("sentence-transformers/all-MiniLM-L6-v2").await?;

Normalization and prompts

fastembed always L2-normalizes its output, so every embedding has unit length (suitable for cosine similarity). It has no prompt registry: for a model that expects a query or document prefix, prepend the prefix to the text yourself.

Execution providers

SentenceTransformerEmbedder::load_with(name_or_path, execution_providers) runs the model on the first of execution_providers that ONNX Runtime can start. A provider is registered when its ort feature is compiled into the binary, such as cuda for CUDAExecutionProvider; otherwise ONNX Runtime falls back to the CPU. load(name_or_path) is load_with with no providers.

use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder;
use ort::execution_providers::CUDAExecutionProvider;

let embedder = SentenceTransformerEmbedder::load_with(
    "sentence-transformers/all-MiniLM-L6-v2",
    vec![CUDAExecutionProvider::default().build()],
)
.await?;

Each batch runs under the GPU runner, so one inference runs at a time per GPU in the pool.

PreviousModel APIs
CocoIndex DocsEdit this pageReport issue