Multi-vector image search with ColPali and Qdrant

Index images with ColPali multi-vector patch embeddings and a Qdrant MaxSim collection using CocoIndex v1: incremental, live, in plain async Python.

TutorialUpdated 10 min read
Search images patch by patch. An image of a giraffe is cut into a 4×4 grid of patches, and ColPali gives each patch its own vector, 16 vectors in all. The query "long neck" finds its best patch, the one on the neck, and that patch's vector.
Enlarge cover

Multi-vector image search with ColPali and Qdrant

Zoom in to see details.

Search images patch by patch. An image of a giraffe is cut into a 4×4 grid of patches, and ColPali gives each patch its own vector, 16 vectors in all. The query "long neck" finds its best patch, the one on the neck, and that patch's vector.

Most image search squeezes each image into a single vector. That works until the image is dense: a page with a chart and a table, a form with many fields, a photo with several objects. One vector blurs all of that together, so a query about one small region has to compete with everything else in the image.

ColPali keeps the regions apart. It splits each image into a grid of patches and embeds every patch into its own vector, so an image becomes a set of vectors instead of one. A text query is matched patch by patch. The approach is aimed at dense, text-heavy images, where a single embedding loses detail.

This tutorial builds an image index with ColPali and a Qdrant multivector collection, using CocoIndex v1. CocoIndex handles the incremental indexing, the managed Qdrant collection and live updates; ColPali does the embedding in plain Python. The full example is in the CocoIndex v1 repo.

What you build

  • A CocoIndex app, ImageSearchColpaliV1, in pipeline.py. It walks the img/ folder in live mode, embeds each .jpg, .jpeg or .png file with ColPali, and stores one Qdrant point per image.
  • A Qdrant collection, ImageSearchColpali, whose vector is a multi-vector with the max_sim comparator, so Qdrant scores queries with MaxSim.
  • A FastAPI server, api.py, that runs the app in live mode and serves GET /search. The example also ships a small React frontend.

You need Python 3.11+, Docker to run Qdrant, and Node.js with npm for the optional frontend. The example’s img/ folder holds 4 photos: cat1.jpeg, dog1.jpeg, elephant1.jpg and giraffe.jpg. The ColPali model, vidore/colpali-v1.2 by default, downloads from Hugging Face on the first run. Run it has the commands.

What the example builds. Indexing, in live mode: img/ holds 4 photos, watched. One process_file component per image, with memo=True, calls embed_image_bytes, which returns 1 vector per patch, and declares a point in the Qdrant collection ImageSearchColpali: multi-vector with max_sim, 1 point per image, with id uuid5(path) and payload filename. Search: GET /search calls embed_query, which returns about 1 vector per query token; query_points runs MaxSim and returns the top 5 images; the endpoint returns each filename and score.
What the example buildsEnlarge

What the example builds

Zoom in to see details.

What the example builds. Indexing, in live mode: img/ holds 4 photos, watched. One process_file component per image, with memo=True, calls embed_image_bytes, which returns 1 vector per patch, and declares a point in the Qdrant collection ImageSearchColpali: multi-vector with max_sim, 1 point per image, with id uuid5(path) and payload filename. Search: GET /search calls embed_query, which returns about 1 vector per query token; query_points runs MaxSim and returns the top 5 images; the endpoint returns each filename and score.

One vector per image, or one per patch

A few terms first:

  • Patch: a small square of the image. ColPali, built on the PaliGemma-3B vision-language model, resizes each image and cuts it into 1,024 patches (a 32×32 grid).
  • Patch embedding: the vector ColPali computes for one patch. Each has 128 numbers and also carries context from the rest of the image.
  • Multi-vector: a list of vectors stored as one value. A ColPali image embedding is a multi-vector with one vector per patch; a query embedding has about one vector per query token.
  • Late interaction: the query and the image are embedded separately, and the two sets of vectors are compared only at search time. ColPali follows the ColBERT text-retrieval model here.
  • MaxSim: the late-interaction score. Each query vector takes its best match among the image’s patch vectors, and the score is the sum of those best matches.
Two ways to embed the same image. A single-vector model encodes the whole image into 1 vector, so regions and layout mix into one vector. ColPali splits the image into a grid of patches and encodes each patch into its own vector; the outlined patch maps to the highlighted vector. Each image has 1,024 patches, with 128 numbers per vector.
One vector per image, or one per patchEnlarge

One vector per image, or one per patch

Zoom in to see details.

Two ways to embed the same image. A single-vector model encodes the whole image into 1 vector, so regions and layout mix into one vector. ColPali splits the image into a grid of patches and encodes each patch into its own vector; the outlined patch maps to the highlighted vector. Each image has 1,024 patches, with 128 numbers per vector.

The patch approach has these properties:

  • Fine-grained matching. Each patch is embedded with awareness of both visual and textual cues. A query becomes about one vector per token, and each query vector is matched against the most relevant patches, which is finer than one vector for the whole image.
  • Spatial structure is kept. A single global vector loses layout. Patch vectors keep the regions apart, so a match can come from one region: a diagram in a manual, a table on a form.
  • No OCR step. The pipeline embeds the image itself, so OCR errors cannot affect it, and visual elements that OCR skips (charts, drawings, logos) still count.
  • Little work per query. The image vectors are computed once, at index time. At query time only the query is embedded, and the comparison is a set of dot products. A cross-encoder, by contrast, runs the model on every query and image pair.

The cost is storage: one point holds about 1,030 vectors instead of one (see Step 1).

How MaxSim scores an image

The ColPali paper defines the score of a query against an image as a sum over the query vectors. For each query vector, take its maximum dot product with any of the image’s patch vectors, then add up those maxima.

Step through the query below. Each query vector is compared with every patch of the image; its best patch is outlined, and only that value is added to the score. Then switch to the other image and compare the two scores.

How MaxSim scores an image

Toy values on a 4×4 grid, not ColPali output
0.09
0.07
0.55
0.18
0.06
0.10
0.86
0.09
0.08
0.30
0.47
0.14
0.05
0.21
0.18
0.06

giraffe.jpg: 16 patch vectors (toy 4×4 grid)

query "long neck", 2 query vectors

MaxSim score1.54

0.68 + 0.86 = 1.54

Ranking for "long neck"

  1. giraffe.jpg1.54
  2. cat1.jpeg0.73

Every query vector scored. Press Reset to step through.

A real image has 1,024 patch vectors of 128 numbers, and a real query has more vectors than its words, so real scores are larger: the search results below give the giraffe 9.093 for this query.

An image scores high when every part of the query finds some region that matches it well. A patch that matches no query vector costs nothing, so a busy image is not penalized for its other content.

Quick check: an image has 1,024 patch vectors, and a query has 20 vectors. How many best matches does MaxSim add up for that image?

20, one per query vector. Each query vector is compared with all 1,024 patch vectors, but only its best match counts toward the score. So the sum has 20 terms, however many patches the image has.

Step 1: Embed images and queries with ColPali

The embedding is plain Python with the colpali_engine library. get_colpali() loads the model and processor once and caches them. embed_image_bytes() turns image bytes into a list of patch vectors, and embed_query() does the same for a text query. Both return list[list[float]]:

pipeline.py
QDRANT_COLLECTION = "ImageSearchColpali"
COLPALI_MODEL_NAME = os.getenv("COLPALI_MODEL", "vidore/colpali-v1.2")


@functools.cache
def get_colpali() -> tuple[ColPali, ColPaliProcessor, str]:
    model = ColPali.from_pretrained(COLPALI_MODEL_NAME)
    processor = ColPaliProcessor.from_pretrained(COLPALI_MODEL_NAME)
    device = get_torch_device("auto")
    model = model.to(device)
    model.eval()
    return model, processor, device


def _postprocess_embeddings(
    embeddings: torch.Tensor, processor: ColPaliProcessor
) -> list[list[float]]:
    padding_side = getattr(processor.tokenizer, "padding_side", "right")
    unpadded = unbind_padded_multivector_embeddings(
        embeddings, padding_side=padding_side
    )
    return unpadded[0].cpu().tolist()


def embed_query(text: str) -> list[list[float]]:
    model, processor, device = get_colpali()
    batch = processor.process_queries(texts=[text])
    batch = batch.to(device)
    with torch.no_grad():
        embeddings = model(**batch)
    return _postprocess_embeddings(embeddings, processor)


def embed_image_bytes(img_bytes: bytes) -> list[list[float]]:
    model, processor, device = get_colpali()
    image = Image.open(io.BytesIO(img_bytes)).convert("RGB")
    batch = processor.process_images([image])
    batch = batch.to(device)
    with torch.no_grad():
        embeddings = model(**batch)
    return _postprocess_embeddings(embeddings, processor)

_postprocess_embeddings removes the padding the model adds to fill a batch, so each result holds only real vectors. Because the query and the image go through the same model, their vectors live in the same space and can be compared one by one.

What you see. Each function returns a Python list of vectors, each a list of 128 floats. In the example on the model card, a page image gives 1,030 vectors and a one-sentence query gives 23. The 1,030 are the 1,024 patch vectors plus 6 vectors for the text tokens that ColPali adds to every image, a short prompt (“Describe the image.”) with its special tokens, as the ColPali paper explains. The count for a query depends on its length.

Step 2: Index each image as one Qdrant point

A lifespan function creates the Qdrant client once, when the CocoIndex runtime starts, and provides it under the context key QDRANT_DB. app_main passes that key to mount_collection_target in Step 3. The example also provides the same client under a second key, QDRANT_CLIENT. Nothing in pipeline.py or api.py reads that key, so you can drop both of its lines:

pipeline.py
def qdrant_url() -> str:
    return os.getenv("QDRANT_URL", "http://localhost:6334/")


QDRANT_DB = coco.ContextKey[QdrantClient]("image_search_colpali")
QDRANT_CLIENT = coco.ContextKey[QdrantClient]("qdrant_client")


@coco.lifespan
async def coco_lifespan(
    builder: coco.EnvironmentBuilder,
) -> AsyncIterator[None]:
    client = qdrant.create_client(qdrant_url(), prefer_grpc=True)
    builder.provide(QDRANT_DB, client)
    builder.provide(QDRANT_CLIENT, client)
    yield

One function processes one image: read the bytes, embed them, and declare a Qdrant point keyed by a stable uuid5 of the path. Declaring a point states that it should exist in Qdrant; CocoIndex works out the insert, update or delete. With memo=True, the result is cached, and an image whose content has not changed is not embedded again:

pipeline.py
@coco.fn(memo=True)
async def process_file(
    file: FileLike,
    target: qdrant.CollectionTarget,
) -> None:
    content = await file.read()
    embedding = embed_image_bytes(content)
    row_id = _image_id(file.file_path.path)
    point = qdrant.PointStruct(
        id=row_id,
        vector=embedding,
        payload={"filename": str(file.file_path.path)},
    )
    target.declare_point(point)


def _image_id(path: pathlib.PurePath) -> str:
    return str(uuid.uuid5(uuid.NAMESPACE_URL, str(path)))

process_file calls embed_image_bytes directly, so the model runs on the event loop thread. While one image is embedded, the other components wait, and so do requests to the API server, which shares that loop in live mode. For the 4 sample images this costs little. For a larger folder, run the model on CocoIndex’s GPU runner as multi-format indexing does: @coco.fn.as_async(runner=coco.GPU) turns a synchronous function into one that you await, and runs it on a dedicated thread pool, off the event loop, one call at a time per GPU:

pipeline.py
@coco.fn.as_async(runner=coco.GPU)
def embed_image_bytes(img_bytes: bytes) -> list[list[float]]:
    ...  # same body as above


# in process_file:
    embedding = await embed_image_bytes(content)

Decorating the function also makes CocoIndex track its code, so editing it invalidates the cached process_file results. /search still calls embed_query on the event loop; wrap it with await asyncio.to_thread(pipeline.embed_query, q) if queries should not wait behind indexing.

Unlike a typical image pipeline that stores one vector per image, each point here stores a multi-vector: the outer dimension is the number of patches, and the inner dimension is the model’s output size (128). Qdrant’s MaxSim scoring needs this shape. See how CocoIndex models a vector index schema for details, and multi-vector support in CocoIndex for how VectorSchema and MultiVectorSchema map to columns in Postgres, LanceDB and Qdrant.

Step 3: Configure the Qdrant multivector collection

A MultiVectorSchema with multivector_comparator="max_sim" tells Qdrant to do the MaxSim scoring, so the query side only hands over the query’s list of vectors. app_main reads the vector size from the model (128 if the model has no dim), mounts the collection, walks the image folder in live mode, and mounts one process_file per image:

pipeline.py
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    model, _, _ = get_colpali()
    dim = int(getattr(model, "dim", 128))

    target_collection = await qdrant.mount_collection_target(
        QDRANT_DB,
        collection_name=QDRANT_COLLECTION,
        schema=await qdrant.CollectionSchema.create(
            vectors=qdrant.QdrantVectorDef(
                schema=MultiVectorSchema(
                    vector_schema=VectorSchema(dtype=np.dtype(np.float32), size=dim)
                ),
                distance="cosine",
                multivector_comparator="max_sim",
            )
        ),
    )

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(
            included_patterns=["**/*.jpg", "**/*.jpeg", "**/*.png"]
        ),
        live=True,  # source supports live watch; api.py runs the app with live=True
    )
    await coco.mount_each(process_file, files.items(), target_collection)


app = coco.App(
    coco.AppConfig(name="ImageSearchColpaliV1"),
    app_main,
    sourcedir=pathlib.Path("./img"),
)

Each image is its own processing component: a unit of work that CocoIndex runs, tracks and cleans up on its own. Change detection is per image, and the component owns the point it declared.

With distance="cosine", each patch comparison is a cosine similarity, and the image score is the sum of one best match per query vector. So scores are not limited to 0–1, and they are comparable only between images for the same query.

Quick check: why is the point ID a uuid5 of the file path, and not a random UUID?

uuid5 is deterministic: the same path always gives the same ID. When an image changes, the new point has the same ID as the old one, so it replaces it in Qdrant. A random UUID would give a new ID on every run, and the old point would stay behind.

Step 4: Serve and query the index

The example wraps the app in a FastAPI server. Its lifespan starts the app in live mode, blocks startup until the first pass over the folder is READY, then keeps watching the folder in the background while it serves search:

api.py
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:  # type: ignore[override]
    global _client
    async with coco.runtime():
        _client = qdrant.create_client(pipeline.qdrant_url(), prefer_grpc=True)

        # Start a live update; block startup until the initial sweep finishes so
        # the collection is queryable, then keep it running in the background.
        update_handle = pipeline.app.update(live=True)
        async for snap in update_handle.watch():
            if snap.status is coco.UpdateStatus.READY:
                break
        update_task = asyncio.create_task(update_handle.result())

        try:
            yield
        finally:
            update_task.cancel()
            with contextlib.suppress(asyncio.CancelledError):
                await update_task
            _client = None

Search embeds the query text into its query vectors and lets Qdrant do the MaxSim scoring against every image’s patches:

api.py
@app.get("/search")
async def search(
    q: str = Query(..., description="Search query"),
    limit: int = Query(5, description="Number of results"),
) -> dict[str, Any]:
    query_embedding = pipeline.embed_query(q)

    if _client is None:
        raise RuntimeError("Qdrant client is not initialized.")

    results = pipeline._qdrant_search(
        _client,
        pipeline.QDRANT_COLLECTION,
        query_embedding,
        limit,
    )

    return {
        "results": [
            {
                "filename": (r.payload or {}).get("filename"),
                "score": r.score,
            }
            for r in results
        ]
    }

_qdrant_search in pipeline.py passes the list of query vectors to query_points. The collection’s max_sim comparator does the rest:

pipeline.py
def _qdrant_search(
    client: QdrantClient,
    collection_name: str,
    query_vector: list[list[float]],
    limit: int,
) -> list[qdrant_models.ScoredPoint]:
    if not hasattr(client, "query_points"):
        raise RuntimeError("qdrant-client must support query_points for ColPali.")
    response = client.query_points(
        collection_name=collection_name,
        query=query_vector,
        limit=limit,
        with_payload=True,
    )
    return response.points

Run it

Start Qdrant, then configure and install the example. From examples/image_search_colpali in the CocoIndex repo:

sh
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant
cp .env.example .env     # QDRANT_URL (defaults to the local container above)
pip install -e .

pip install -e . installs cocoindex[colpali,qdrant] with torch, transformers and pillow. Then run the server. It indexes img/ in live mode on startup, so there is no separate indexing command:

sh
python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000

To use the frontend, start it in a second terminal:

sh
cd frontend && npm install && npm run dev   # http://localhost:5173

What you see. The server starts to accept requests after the first pass over img/ is done. The frontend sends your query to /search and shows the images by score. A query such as “long neck” returns the giraffe first. The screenshot shows 4 queries over the 4 sample images, from a run of the example:

The example's search page, Image Search, with 4 queries over the 4 sample images. elephant: elephant 9.743, cat 6.641, giraffe 6.403, dogs 5.493. long neck: giraffe 9.093, elephant 7.566, cat 5.923, dogs 5.151. pets on the grass: dogs 10.306, elephant 8.856, giraffe 8.794, cat 8.013. tongue: dogs 8.100, elephant 6.761, giraffe 6.664, cat 6.065.
Search resultsEnlarge

Search results

Zoom in to see details.

The example's search page, Image Search, with 4 queries over the 4 sample images. elephant: elephant 9.743, cat 6.641, giraffe 6.403, dogs 5.493. long neck: giraffe 9.093, elephant 7.566, cat 5.923, dogs 5.151. pets on the grass: dogs 10.306, elephant 8.856, giraffe 8.794, cat 8.013. tongue: dogs 8.100, elephant 6.761, giraffe 6.664, cat 6.065.

Every query returns all 4 images, because the collection holds only 4 and limit defaults to 5. The scores are MaxSim sums, so values above 1, such as 9.093, are expected. For dense, text-heavy or busy images, compare with the single-vector CLIP image search example, which embeds each image into one vector.

What happens when img/ changes

The server keeps the source in live mode, so changes to img/ reach Qdrant while it runs, with no rebuild step. The example’s README puts the delay for a new photo at about a second. What happens depends on the change:

What live mode does for each change to img/. Photo added, in green: a new component, 1 ColPali call, 1 point added. Photo edited, in periwinkle: memo miss, 1 ColPali call, 1 point updated with the same ID. Photo unchanged, in dashed gray: memo hit, no ColPali call, point unchanged. Photo deleted, in red: the component is removed, no ColPali call, 1 point deleted. Touching a file without changing its content does not embed it again.
What live mode does when img/ changesEnlarge

What live mode does when img/ changes

Zoom in to see details.

What live mode does for each change to img/. Photo added, in green: a new component, 1 ColPali call, 1 point added. Photo edited, in periwinkle: memo miss, 1 ColPali call, 1 point updated with the same ID. Photo unchanged, in dashed gray: memo hit, no ColPali call, point unchanged. Photo deleted, in red: the component is removed, no ColPali call, 1 point deleted. Touching a file without changing its content does not embed it again.

memo=True compares the file’s modification time and, when it changed, a fingerprint of its content. So touching a file without changing its content does not embed it again.

Quick check: you delete dog1.jpeg from img/ while the server runs. What happens in Qdrant?

Its point is deleted. The live walk no longer yields that file, so app_main stops mounting its process_file component, and CocoIndex deletes the point that the component declared. No ColPali call runs.

What to try next

  • Add your own images. Copy .jpg, .jpeg or .png files into img/ while the server runs, then search for something in them.
  • Query from the command line. The API takes the query and an optional limit: curl "http://localhost:8000/search?q=long+neck&limit=2".
  • Try another ColPali checkpoint. Set COLPALI_MODEL in .env. app_main reads the vector size from the model when it creates the collection.
  • Index PDFs as well as images. Multi-format indexing renders each PDF page to an image and uses the same ColPali and Qdrant setup.
  • Read from another source. Swapping the source leaves the rest of the flow unchanged, and CocoIndex keeps the index current as files are added, updated or deleted. Source connectors include Google Drive, Amazon S3 and Azure Blob Storage.

Further reading:

Frequently asked questions

What is ColPali and why use it for image indexing?

ColPali is a multimodal retrieval model built on PaliGemma-3B that produces ColBERT-style multi-vector embeddings. Instead of reducing each image to a single dense vector, it breaks an image into many smaller patches and gives each patch its own embedding, together forming a multi-vector representation. This keeps local spatial and semantic structure, supports fine-grained visual search, and needs no OCR step.

See One vector per image, or one per patch.

How do I embed an image with ColPali in CocoIndex v1?

The embedding is plain Python with the colpali_engine library, called inside a CocoIndex function. Load the model once, then embed_image_bytes turns image bytes into a list[list[float]] (one vector per patch). Wrap the per-image function in @coco.fn(memo=True) so unchanged images are skipped, and declare a Qdrant point for each image.

See Step 1: Embed images and queries with ColPali.

What does ColPali actually store as an embedding?

Unlike typical image search pipelines that store one global vector per image, ColPali stores a multi-vector: the outer dimension is the number of patches and the inner dimension is the model's output size (128). This makes the collection multi-vector ready and compatible with late-interaction scoring such as MaxSim.

See Step 2: Index each image as one Qdrant point.

How does ColPali compare to single-vector CLIP image search?

Each page image is split into 1,024 patches (a 32×32 grid) and every patch is embedded with context from the rest of the image, so ColPali can match a query to one region of an image rather than collapsing the image into one global vector. It is aimed at dense, text-heavy or busy images, where a single embedding loses detail; the cost is about a thousand vectors per image instead of one.

See One vector per image, or one per patch.

How do you keep a ColPali image index up to date automatically?

Run the app in live mode. The example's FastAPI lifespan calls pipeline.app.update(live=True), blocks startup until the first pass is READY, then keeps watching the folder in the background. The example's README puts the delay for a new image at about a second, with no rebuild step, and deleting an image removes its Qdrant point automatically.

See What happens when img/ changes.

How are ColPali embeddings stored in Qdrant for MaxSim retrieval?

Mount a Qdrant collection target with qdrant.mount_collection_target and a MultiVectorSchema configured with multivector_comparator="max_sim". That makes Qdrant do the late-interaction scoring: at query time you embed the query into ColPali's per-token vectors, and Qdrant runs MaxSim against every image's patch vectors.

See Step 3: Configure the Qdrant multivector collection and Step 4: Serve and query the index.

Build fresh context for your agents.

CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.