Multi-Dimensional Vector Support in CocoIndex

CocoIndex supports vectors and multi-vectors (a set of vectors per item). In CocoIndex v1, a vector schema describes each vector column: the dtype of its elements and its number of dimensions. The Qdrant connector maps a vector schema to a dense vector and a multi-vector schema to a MaxSim multi-vector.

This post is a reference for those two schema types. It covers:

• What a multi-vector is, and why it is not flattened
• How vectors and multi-vectors are declared in Python
• What each schema becomes in Postgres, LanceDB and Qdrant
• A worked Qdrant example with a MaxSim multi-vector and a dense vector

Where multi-vectors come from

Late-interaction models such as ColPali embed an image as one vector per patch and a query as about one vector per token; at search time, MaxSim adds up each query vector’s best match among the item’s vectors. Multi-vector image search with ColPali explains the model and MaxSim with an interactive demo, and the v1 ColPali example stores one multi-vector per image in Qdrant.

What is a multi-dimensional vector?

A multi-vector (multi-dimensional vector) is a vector whose elements are themselves vectors: a matrix or a nested list of M vectors, each of dimension N. In CocoIndex v1, MultiVectorSchema fixes N through its inner VectorSchema, and M can differ from item to item. Targets need a fixed inner dimension: a multi-vector exported to Qdrant declares the size of each inner vector.

Deep learning and multi-modal applications produce this shape, for example:

• An image might be represented as a collection of patch-level embeddings.
• A document could be broken down into paragraph-level vectors.
• A user session might be a sequence of behavioral vectors.

CocoIndex keeps these representations as they are, without flattening, through MultiVectorSchema.

For example, a Vision Transformer (ViT) outputs a vector per patch instead of one global image vector: ViT-B/16 splits a 224×224 input into 16×16-pixel patches, 14×14 = 196 of them. Each patch vector has 768 numbers, so the image is a multi-vector of 196 vectors with this schema:

MultiVectorSchema(vector_schema=VectorSchema(dtype=np.dtype(np.float32), size=768))

Patch vectors enable local feature matching and region-aware retrieval: a query can match one region of the image.

Why not flatten into one vector?

You could flatten a 2×3 multi-vector like [[1,2,3],[4,5,6]] into [1,2,3,4,5,6], but you lose the boundary between the 2 vectors.

Kept as a multi-vector:

• Each inner vector keeps its own meaning.
• A query can match individual inner vectors.
• Vector databases like Qdrant can score a query against every inner vector and keep the best match (e.g., closest patch, paragraph, or step).
• The inner vector can have a fixed dimension even if the outer dimension varies. The outer dimension usually varies: the number of patches in an image and the number of paragraphs in an article depend on the data.

Most vector databases require a fixed dimension to build a vector index.

What Python vector types does CocoIndex support?

In CocoIndex v1, a vector field in a row-oriented target (Postgres, LanceDB) is a NumPy array, NDArray, and its column type comes from a schema object in the field’s type annotation, from the cocoindex.resources.schema module. Qdrant points take vectors as plain lists of numbers, and the collection schema carries the vector schema (see below). See CocoIndex vector schema documentation for more details.

1. Vectors: VectorSchema

from dataclasses import dataclass
from typing import Annotated

import numpy as np
from numpy.typing import NDArray

import cocoindex as coco
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.schema import VectorSchema

EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder")


@dataclass
class DocEmbedding:
    id: int
    text: str
    embedding: Annotated[NDArray, EMBEDDER]  # dimension from the embedder (384 for all-MiniLM-L6-v2)


@dataclass
class ImageEmbedding:
    id: int
    embedding: Annotated[NDArray, VectorSchema(dtype=np.dtype(np.float32), size=768)]

A VectorSchema has two fields:

• dtype: the NumPy dtype of each element, such as np.float32. The Postgres connector stores float16 vectors as halfvec and other dtypes as vector.
• size: the number of dimensions. It is required: a vector column in v1 always has a fixed dimension.

The annotation can be an embedder instance, a ContextKey that resolves to one (the recommended pattern, because the embedder is configured once in the lifespan), or an explicit VectorSchema for a model that does not provide one.

v1 has no dynamic-dimension vector column. A NumPy array field without a vector schema raises ValueError in the Postgres and LanceDB connectors. To store a list of numbers whose length varies, declare it as a plain list[float], which the Postgres connector stores as jsonb.

2. Multi-vectors: MultiVectorSchema

from cocoindex.resources.schema import MultiVectorSchema, VectorSchema

patches = MultiVectorSchema(
    vector_schema=VectorSchema(dtype=np.dtype(np.float32), size=128)
)

A MultiVectorSchema wraps the VectorSchema of one inner vector. It declares multi-vectors (e.g., multiple embeddings per item): any number of vectors per item, each with the same fixed dimension.

• Python value: a list of vectors, such as list[list[float]]. The ColPali example returns one list[float] of 128 numbers per image patch or query token.
• Typical uses: sets of embeddings per item, multi-view representations, or batched encodings.

An object that implements __coco_multi_vector_schema__() is a MultiVectorSchemaProvider and can stand in for the schema, the same way an embedder stands in for a VectorSchema.

How the schema becomes a column

A vector column is fully described by two facts: the dtype of each element and the number of dimensions. A vector schema records exactly those two, and a multi-vector schema wraps that single-vector schema to say “a variable number of these.” When a connector sets up a target, it reads the schema (from the field’s type annotation for row-oriented targets, from QdrantVectorDef for Qdrant) and translates it into the type the destination requires:

The table shows the fixed-inner-dimension rule in practice. The outer count can stay dynamic: one image yields as many patch vectors as it has patches. Each inner vector has to declare a fixed dtype and size, because the vector index is built over that unit. The schema carries that guarantee from your Python code down to the storage layer, so the column is created with the right type the first time, instead of being inferred from whatever the first row happened to contain.

How do CocoIndex vector types map to Qdrant?

Qdrant is a vector database that supports dense vectors, multi-vectors and sparse vectors. In v1, you declare the vectors of a collection with qdrant.CollectionSchema.create(), and each QdrantVectorDef maps to a Qdrant vector:

In CocoIndex v1, each point is declared explicitly:

• Each target.declare_point(qdrant.PointStruct(...)) call creates a Qdrant point.
• Its vector is one vector, or a dict of named vectors that match the collection schema.
• Its payload is a JSON-serializable dict for everything else, such as filenames or page numbers to filter on.

Example: a multi-vector and a dense vector per image

This collection stores each image’s patch embeddings as a MaxSim multi-vector, patches, and their mean as a dense vector, mean. We ran it on CocoIndex 1.1.1 against an in-memory Qdrant with 3-dimensional toy vectors:

QDRANT_DB = coco.ContextKey[QdrantClient]("mv_db")

DOCS = {
    1: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]],                   # 2 patch vectors
    2: [[0.0, 0.0, 1.0], [0.6, 0.8, 0.0], [0.0, 0.6, 0.8]],  # 3 patch vectors
}


@coco.fn
async def process(doc_id: int, target: qdrant.CollectionTarget) -> None:
    target.declare_point(qdrant.PointStruct(
        id=doc_id,
        vector={"patches": DOCS[doc_id], "mean": np.mean(DOCS[doc_id], axis=0).tolist()},
        payload={"n_patches": len(DOCS[doc_id])},
    ))


@coco.fn
async def app_main() -> None:
    target = await qdrant.mount_collection_target(
        QDRANT_DB,
        collection_name="images",
        schema=await qdrant.CollectionSchema.create(
            vectors={
                "patches": qdrant.QdrantVectorDef(
                    schema=MultiVectorSchema(
                        vector_schema=VectorSchema(dtype=np.dtype(np.float32), size=3)
                    ),
                    distance="cosine",
                    multivector_comparator="max_sim",
                ),
                "mean": qdrant.QdrantVectorDef(
                    schema=VectorSchema(dtype=np.dtype(np.float32), size=3),
                ),
            }
        ),
    )
    await coco.mount_each(process, ((str(k), k) for k in DOCS), target)

The two items have 2 and 3 patch vectors; the inner size is 3 for both. A query with 2 vectors, [[0, 1, 0], [0, 0, 1]], against patches scores image 2 at 1.8 (best matches 0.8 and 1.0) and image 1 at 1.0 (best matches 1.0 and 0). Step through the same numbers:

Summary

A vector schema describes element dtype and dimensions. In CocoIndex v1, VectorSchema and MultiVectorSchema (or an embedder that provides one) define vector dimensions and tell the target connector how to store each field. Of these three targets, only Qdrant stores a MultiVectorSchema as a vector, and it scores it with MaxSim.

Further reading:

• Multi-vector image search with ColPali: ColPali, MaxSim and a full Qdrant example
• Vector schema documentation
• CocoIndex source on GitHub

About the author

Linghua JinCofounder & CEO of CocoIndex · @linghuaj

CEO and cofounder of CocoIndex. Writes about incremental data infrastructure, agents, and the engineering decisions behind the engine.