Turbopuffer connector

Write rows to Turbopuffer namespaces with single or named vectors, per-namespace distance metric, and schemaless attributes — configured via VectorDef and NamespaceSchema.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The turbopuffer connector provides utilities for writing rows to Turbopuffer namespaces, with support for both single and named vectors.

use cocoindex::connectors::turbopuffer::{
    self, DistanceMetric, NamespaceSchema, TurbopufferConnection,
};

Connection setup

Turbopuffer uses a single client object that owns the API key and region. Construct one using TurbopufferConnection:

fn connect() -> Result<TurbopufferConnection> {
    let api_key = std::env::var("TURBOPUFFER_API_KEY")
        .map_err(|_| Error::engine("TURBOPUFFER_API_KEY is required"))?;
    let region = std::env::var("TURBOPUFFER_REGION").unwrap_or_else(|_| "gcp-us-central1".into());
    Ok(TurbopufferConnection::new(&region, &api_key))
}

TurbopufferConnection::new(region, api_key) is synchronous: it builds an HTTP client for https://{region}.turbopuffer.com. with_base_url(url) points it elsewhere, for a mock server in tests.

As target

The turbopuffer connector provides target state APIs for writing rows to namespaces. CocoIndex tracks what rows should exist and automatically handles upserts and deletions. Turbopuffer creates namespaces implicitly on the first write, so there is no separate “create namespace” step — but the connector still tracks namespace-level configuration (vector schema and distance metric) and clears the namespace if it must be rebuilt.

Declaring target states

Setting up a connection

Create a ContextKey<TurbopufferConnection> to identify your client, then provide it in your lifespan:

use cocoindex::connectors::turbopuffer::TurbopufferConnection;

cocoindex::context_key!(static DB: TurbopufferConnection);

let conn = TurbopufferConnection::new(&region, &api_key);
// Provide it on the environment builder, or in a `#[cocoindex::lifespan]`:
let builder = Environment::builder().provide_key(&DB, conn);

context_key!(static DB: TurbopufferConnection) declares the key; its name is the stable identity.

Namespaces (parent state)

Declares a namespace as a target state. Returns a NamespaceTarget for declaring rows.

pub fn declare_namespace_target(
    ctx: &Ctx,
    conn: &ContextKey<TurbopufferConnection>,
    namespace: impl Into<String>,
    schema: NamespaceSchema,
) -> Result<NamespaceTarget>

pub async fn mount_namespace_target(
    ctx: &Ctx,
    conn: &ContextKey<TurbopufferConnection>,
    namespace: impl Into<String>,
    schema: NamespaceSchema,
) -> Result<NamespaceTarget>

managed_by is set with ManagedTargetOptions::user_managed() on the _with_options variants (declare_namespace_target_with_options, mount_namespace_target_with_options). declare_namespace_target returns a handle whose rows resolve when the current component commits; mount_namespace_target(...).await? returns a handle that accepts rows at once.

Parameters:

• ctx — The current component’s Ctx.
• conn — A ContextKey<TurbopufferConnection> identifying the connection to use.
• namespace — Name of the namespace: ASCII letters, digits, -, _ and ..
• schema — Schema definition specifying vector configuration and distance metric (see Namespace schema).
• options — ManagedTargetOptions on the _with_options variants; ManagedTargetOptions::user_managed() assumes the namespace exists.

Rows (child states)

Once a NamespaceTarget is resolved, declare rows to be upserted with declare_row or declare_named_row:

impl NamespaceTarget {
    /// A row for an unnamed-vector namespace.
    pub fn declare_row(
        &self,
        ctx: &Ctx,
        id: impl Into<RowId>,
        vector: Vec<f32>,
        attributes: Map<String, JsonValue>,
    ) -> Result<()>

    /// A row for a named-vector namespace.
    pub fn declare_named_row<I, K>(
        &self,
        ctx: &Ctx,
        id: impl Into<RowId>,
        vectors: I,
        attributes: Map<String, JsonValue>,
    ) -> Result<()>
    where
        I: IntoIterator<Item = (K, Vec<f32>)>,
        K: Into<String>
}

pub enum RowId {
    Str(String),
    Int(u64),
}

The id, the vector(s) and the attributes are separate arguments. The id is a RowId, built from a String, a &str or a u64; the attributes are a serde_json::Map<String, Value>. A vector with a NaN or infinite element is rejected.

• id — Document id (string or unsigned integer).
• vector / vectors — For an unnamed-vector schema, pass one Vec<f32> to declare_row. For a named-vectors schema, pass (name, vector) pairs to declare_named_row. Each vector must have its field’s dimension and finite elements; otherwise the declare call returns an error.
• attributes — Non-vector attributes (text, tags, metadata, etc.). Turbopuffer infers attribute types from the data.

Namespace schema

Define vector configuration and distance metric for a namespace using NamespaceSchema:

impl NamespaceSchema {
    // One unnamed f32 vector under Turbopuffer's `vector` field
    pub fn new(vector_size: usize, distance: DistanceMetric) -> Self
    pub fn from_vector_schema(schema: VectorSchema, distance: DistanceMetric) -> Result<Self>
    pub async fn from_vector_provider(
        provider: &(impl VectorSchemaProvider + ?Sized),
        distance: DistanceMetric,
    ) -> Result<Self>

    // Named vectors
    pub fn named<I, K>(vectors: I, distance: DistanceMetric) -> Result<Self>
    where
        I: IntoIterator<Item = (K, VectorDef)>,
        K: Into<String>

    // Named vectors from a `#[derive(SchemaFields)]` row
    pub fn from_row<T: SchemaFields>(distance: DistanceMetric) -> Result<Self>
    pub fn with_vector_dim(self, field_name: &str, dim: usize) -> Result<Self>
}

pub enum DistanceMetric { CosineDistance, EuclideanSquared }

from_row turns each #[coco(vector)] field of a row struct into a named vector; the other fields stay schemaless attributes. with_vector_dim sets the dimension of one such field.

Parameters:

• The vector shape — new, from_vector_schema and from_vector_provider declare an unnamed vector (stored under turbopuffer’s default "vector" field); named and from_row declare named vectors.
• distance — Distance metric applied to all vector columns in the namespace. Turbopuffer applies a single distance metric per namespace.

VectorDef

Specifies a vector field’s dimension and dtype:

pub struct VectorDef {
    pub schema: VectorSchema,
}

impl VectorDef {
    pub fn new(schema: VectorSchema) -> Result<Self>
    pub fn f32(size: usize) -> Result<Self>
    pub async fn from_provider(provider: &(impl VectorSchemaProvider + ?Sized)) -> Result<Self>
}

The schema field is an explicit VectorSchema. Its dtype controls turbopuffer’s vector type: VectorSchema::f32(n) gives [N]f32 and VectorSchema::f16(n) gives [N]f16. VectorDef::from_provider(&embedder).await? resolves the schema from a VectorSchemaProvider, typically the embedder.

Single (unnamed) vector

For namespaces with a single unnamed vector:

let target = turbopuffer::mount_namespace_target(
    &ctx,
    &DB,
    &namespace,
    NamespaceSchema::new(3, DistanceMetric::CosineDistance),
)
.await?;
target.declare_row(&ctx, "1", vec![1.0, 0.0, 0.0], attrs("alpha"))?;

NamespaceSchema::new(size, distance) takes the dimension as a number; NamespaceSchema::from_vector_provider(&embedder, distance).await? takes it from an embedder.

Named vectors

Namespaces can have multiple named vector columns (turbopuffer supports up to two per namespace). The name "id" is reserved for the row id and cannot be used as a vector field name.

let schema = NamespaceSchema::named(
    [
        ("text", VectorDef::f32(3)?),
        ("image", VectorDef::f32(2)?),
    ],
    DistanceMetric::CosineDistance,
)?;

Rows pass (name, vector) pairs to declare_named_row:

target.declare_named_row(
    &ctx,
    r.id.clone(),
    [("vector", r.vector.clone())],
    attributes,
)?;

Distance metrics

Turbopuffer applies a single distance_metric per namespace. Supported values:

• DistanceMetric::CosineDistance — Cosine distance.
• DistanceMetric::EuclideanSquared — Squared Euclidean distance.

Every constructor takes the metric explicitly; there is no default.

Example

use std::path::PathBuf;

use cocoindex::connectors::turbopuffer::{
    self, DistanceMetric, NamespaceSchema, TurbopufferConnection,
};
use cocoindex::ops::sentence_transformers::SentenceTransformerEmbedder;
use cocoindex::prelude::*;
use serde_json::json;

cocoindex::context_key!(static DB: TurbopufferConnection);
cocoindex::context_key!(
    static EMBEDDER: SentenceTransformerEmbedder,
    detect_change
);

#[derive(Clone, Serialize, Deserialize, SchemaFields)]
struct RowData {
    id: String,
    #[coco(vector)]
    vector: Vec<f32>,
    filename: String,
    chunk_start: i64,
    chunk_end: i64,
    text: String,
}

#[cocoindex::function]
async fn process_file(ctx: &Ctx, file: FileEntry) -> Result<Vec<RowData>> {
    // chunk and embed the file ...
}

async fn app_main(ctx: Ctx, sourcedir: PathBuf, namespace: String) -> Result<()> {
    let vector_dim = ctx.get_key(&EMBEDDER)?.dimension();
    let target = turbopuffer::mount_namespace_target(
        &ctx,
        &DB,
        &namespace,
        NamespaceSchema::from_row::<RowData>(DistanceMetric::CosineDistance)?
            .with_vector_dim("vector", vector_dim)?,
    )
    .await?;

    let files = walk_items(&sourcedir, &["**/*.md"])?;
    let rows_by_file = use_mount_each!(files, |file| process_file(ctx, file)).await?;

    for rows in &rows_by_file {
        for r in rows {
            let attributes = json!({
                "filename": r.filename,
                "chunk_start": r.chunk_start,
                "chunk_end": r.chunk_end,
                "text": r.text,
            })
            .as_object()
            .unwrap()
            .clone();
            target.declare_named_row(
                &ctx,
                r.id.clone(),
                [("vector", r.vector.clone())],
                attributes,
            )?;
        }
    }
    Ok(())
}

See examples/rust/text_embedding_turbopuffer for the full program.

Row IDs

Turbopuffer rows are identified by a string or an unsigned integer: the id argument is impl Into<RowId>, so a String or &str becomes RowId::Str and a u64 becomes RowId::Int. UUIDs should be passed as strings. An integer id must fit in an i64 (at most i64::MAX); a larger one is rejected when the row is declared.

Attributes

Row attributes are schemaless; turbopuffer infers attribute types from the values you write. Supported scalar types include string, int, uint, float, bool, uuid, and datetime, plus their array variants. See Turbopuffer’s schema reference for the full list. The attributes are a serde_json::Map<String, Value>.

Reserved attribute names depend on the schema; putting any reserved name in the attributes makes declare_row / declare_named_row return an error:

• id is always reserved — it’s the row id.
• For an unnamed-vector schema, vector is also reserved (it’s the wire-level vector field).
• For a named-vectors schema, each declared vector field name is reserved instead.

Vector search

The connector focuses on writing rows. For vector search, use the turbopuffer client directly or the connector’s search helpers:

// Unnamed vector
let hits = turbopuffer::vector_search(&conn, &namespace, vec![1.0, 0.0, 0.0], 1).await?;

// Named vector field
let hits =
    turbopuffer::vector_search_by_field(conn, namespace, "vector", query_vec, TOP_K).await?;

The connector has two search helpers, vector_search and vector_search_by_field; each returns Vec<TurbopufferHit> with a distance and the row’s attributes.