Multi-Dimensional Vector Support in CocoIndex

CocoIndex now provides robust and flexible support for typed vector data — from simple numeric arrays to deeply nested multi-dimensional vectors. This support is designed for seamless integration with high-performance vector databases such as Qdrant, and enables advanced indexing, embedding, and retrieval workflows across diverse data modalities.

In this post, we’ll break down:

• What types of vectors CocoIndex supports
• How they are represented in Python
• How we handle dimensionality
• How they map to Qdrant vectors or payloads
• Practical examples of usage

⭐ Star CocoIndex on GitHub if you like it!

Multi-Vector Embedding Search Architecture

This architecture enables fine-grained retrieval by leveraging multi-vector embeddings for both images and queries. The system supports high-resolution semantic matching by comparing dense sets of embeddings from both modalities using a similarity operator like MaxSim.

1. Input Image Preprocessing: Image is divided into patches (typically non-overlapping or slightly overlapping regions). Each patch is treated as a semantic unit and will be embedded individually.

2. Multi-Vector Image Embedding: A multi-vector embedding model processes each image patch independently. Instead of encoding the entire image into a single vector, the model produces a set of embedding vectors per image (one or more vectors per patch). This results in a multi-vector representation of the image, preserving local semantic information.

3. Query Embedding: A query (text or image) is passed through a similar multi-vector embedding model, producing a set of embedding vectors that capture different semantic aspects of the query.

4. MaxSim Similarity Matching: The MaxSim operation compares all query vectors with all image vectors. For each query vector, it finds the maximum similarity score with any image vector. These max scores are aggregated (e.g., summed or averaged) to yield a final relevance score. This method allows the query to flexibly match any relevant part of the image representation.

5. Retrieval Based on the final similarity scores, the system retrieves the most relevant images for the given query.

Supported Python Vector Types in CocoIndex

CocoIndex accepts a range of vector types with strong typing guarantees. See CocoIndex data type documentation for more details.

tip

CocoIndex automatically infers types, so when defining a flow, you don’t need to explicitly specify any types. You only need to specify types explicitly when defining custom functions.

1. One-dimensional vectors

from cocoindex import Vector, Float32
from typing import Literal

Vector[Float32]                     # dynamic dimension
Vector[Float32, Literal[384]]       # fixed dimension of 384

These types are interpreted as:

• Underlying Python type: numpy.typing.NDArray[np.float32] or list[float]
• Dimension-aware: When used with Literal[N], dimension size is explicitly enforced
• Supported number types:
  • Python native types: float, int
  • CocoIndex type aliases: Float32, Float64, Int64, they're aliases for Python native types but annotated with CocoIndex type information to indicate the exact bit size. Using them as custom function return types, you can control the exact type used in downstream, within engine and target and more
  • numpy number types: numpy.float32, numpy.float64, numpy.int64

2. Two-dimensional vectors (multi-vectors)

Vector[Vector[Float32, Literal[3]], Literal[2]]

This declares a vector with two rows of 3-element vectors. These are treated as multi-vectors (e.g., multiple embeddings per point).

• Underlying Python type:
  • numpy.typing.NDArray[np.float32, Literal[3, 2]] or
  • nested list[list[float]]
• Semantics: Useful for scenarios like comparing sets of embeddings per item, multi-view representations, or batched encodings

What is a Multi-Dimensional Vector?

A multi-dimensional vector is simply a vector whose elements are themselves vectors — essentially a matrix or a nested list. In CocoIndex, we represent this using Vector[Vector[T, N], M], meaning M vectors, each of dimension N. M and N are optional - CocoIndex doesn't require them to be fixed, while some targets have requirements, e.g. a multi-vector exported to Qdrant needs to have a fixed inner dimension, i.e. Vector[Vector[T, N]].

This concept is crucial in deep learning and multi-modal applications — for example:

• An image might be represented as a collection of patch-level embeddings.
• A document could be broken down into paragraph-level vectors.
• A user session might be a sequence of behavioral vectors.

Instead of flattening these rich representations, CocoIndex supports them natively through nested Vector typing.

Imagine a 1024x1024 image processed by a Vision Transformer (ViT). Instead of one global image vector, ViT outputs a vector per patch (e.g. 8x8 patches → 64 vectors).

Vector[Vector[Float32, Literal[768]]]

• 196 patches, each with a 768-dim embedding
• Enables local feature matching and region-aware retrieval

Why Not Just Use a Flat Vector?

You could flatten a 2x3 multi-vector like [[1,2,3],[4,5,6]] into [1,2,3,4,5,6] — but you’d lose semantic boundaries.

With true multi-vectors:

• Each sub-vector retains its meaning
• You can match/query at the sub-vector level
• Vector DBs like Qdrant can score across multiple embeddings, choosing the best match (e.g., closest patch, paragraph, or step)
• The inner vector can have a fixed dimension even if the outer dimension is variable - which is quite common, e.g. number of patches of images and number of paragraphs of articles depend on specific data. Vector having fix dimension is required by most vector databases, as it's essential for building vector index.

CocoIndex → Qdrant Type Mapping

Qdrant is a popular vector database that supports both dense vectors and multi-vectors. CocoIndex maps vector types to Qdrant-compatible formats as follows:

CocoIndex Type	Qdrant Type
Vector[Float32, Literal[N]]	Dense Vector
Vector[Vector[Float32, Literal[N]]]	MultiVector
Anything else	Stored as part of Qdrant’s JSON payload

Vector type mapping to Qdrant

Qdrant only accepts vectors with fixed dimension. CocoIndex automatically detects vector shapes and maps unsupported or dynamic-dimension vectors into payloads instead.

In CocoIndex, data is structured as rows and fields:

• A row corresponds to a Qdrant point
• A field can either:
  • Be a named vector, if it fits Qdrant’s vector constraints
  • Be part of the payload, for non-conforming types

This hybrid approach gives you the flexibility of structured metadata with the power of vector search.

Finally

With the ability to support deeply typed vectors and multi-dimensional embeddings, CocoIndex brings structure and semantic clarity to vector data pipelines. Whether you're indexing images, text, audio, or abstract graph representations — our typing system ensures compatibility, debuggability, and correctness at scale.

We’re constantly adding more examples and improving our runtime. If you found this helpful, please ⭐ star CocoIndex on GitHub and share it with others.