Search PDFs and Images Without OCR Using ColPali and Qdrant

Index PDFs and images into one Qdrant collection with CocoIndex v1 and ColPali multi-vector embeddings. Each page is embedded as an image, so no OCR is needed.

TutorialUpdated 7 min read
A paper PDF and a company-report image become page images; ColPali splits a page into patches with one vector each; in Qdrant the text query "revenue growth" matches the chart patches with MaxSim and returns pages ranked best first.
Enlarge cover

Search PDFs and Images Without OCR Using ColPali and Qdrant

Zoom in to see details.

A paper PDF and a company-report image become page images; ColPali splits a page into patches with one vector each; in Qdrant the text query "revenue growth" matches the chart patches with MaxSim and returns pages ranked best first.
Updated for CocoIndex v1

This post uses the CocoIndex v1 API (@coco.fn, coco.App, coco.mount_each). New to v1? Start with the CocoIndex v1 launch post.

This post builds a search index over a folder of PDFs and images. You ask a question in plain words, and the index returns the pages that answer it: a slide, a chart in a report, page 7 of a paper. Nothing is converted to text first. Each page is embedded as an image with ColPali, so charts, tables and layout count in the search.

The full code is in examples/multi_format_indexing. This figure shows what it does on the example’s own source_files/ folder: 1 PDF of 16 pages and 3 JPEG pages from company reports. Each page becomes one Qdrant point: an ID, the page’s vectors, and a payload with the filename and page number.

Indexing: the example's source_files folder holds 1810.04805v2.pdf and 3 .jpg files. file_to_pages turns the PDF into 16 pages and each image into 1 page; ColPali embeds each page; Qdrant stores 19 points, one per page. Querying: the text query "revenue growth" is embedded with ColPali and searched with MaxSim over the 19 points; the top results are the pages sweetgreen_2023.jpg and restaurant_brands_international_2023.jpg.
What the example buildsEnlarge

What the example builds

Zoom in to see details.

Indexing: the example's source_files folder holds 1810.04805v2.pdf and 3 .jpg files. file_to_pages turns the PDF into 16 pages and each image into 1 page; ColPali embeds each page; Qdrant stores 19 points, one per page. Querying: the text query "revenue growth" is embedded with ColPali and searched with MaxSim over the 19 points; the top results are the pages sweetgreen_2023.jpg and restaurant_brands_international_2023.jpg.

What you can build with it

The unit of search is a page, and the query is plain text. That fits any collection where the answer is often in a figure, a table or the page layout:

  • Search a folder of reports: on the sample set, “revenue growth” ranks 2 company-report pages first, as in the sample run.
  • Find figures in research papers: a query such as “ablation table” can match a page by how its table looks as well as by its words.
  • Search scanned documents: a scanned page may have no text layer. This pipeline embeds the rendered image, so it does not need one.
  • Search slide decks: export the deck to PDF, and each slide becomes a searchable page.
  • Give an agent page-level context: each result carries the filename and page number, so an agent can cite the page, or re-render it from the file for a vision model.

The example accepts .pdf, .jpg, .jpeg and .png files. To add a format, extend the path matcher and file_to_pages.

Why embed the page image instead of its text

The usual pipeline runs OCR, splits the text into chunks and embeds each chunk. That works for prose. For a chart or a table, the bars, axes and cell positions do not survive as text, so the chunk embeddings miss them.

Two ways to index a report page that holds a bar chart. With OCR, the page becomes text chunks and one vector per chunk; the chart and the layout are lost. With ColPali, the page image is split into patches and each patch gets its own vector, so the chart and the layout stay in the embedding.
OCR versus page imagesEnlarge

OCR versus page images

Zoom in to see details.

Two ways to index a report page that holds a bar chart. With OCR, the page becomes text chunks and one vector per chunk; the chart and the layout are lost. With ColPali, the page image is split into patches and each patch gets its own vector, so the chart and the layout stay in the embedding.

ColPali splits the page image into patches and gives each patch its own embedding, so one page becomes a multi-vector: a set of vectors instead of one. A text query becomes about one vector per query token. MaxSim scores a page by adding up, for each query vector, its best match among the page’s patch vectors. Qdrant ranks pages this way when the vector is declared as a multi-vector with the max_sim comparator. Multi-vector image search with ColPali explains MaxSim step by step with an interactive demo, and multi-vector support in CocoIndex covers the MultiVectorSchema that declares the column.

Why CocoIndex

A one-off script can embed a folder once. Keeping the index correct as the folder changes is harder. A script that re-runs everything spends GPU time on pages that did not change, and it leaves points in Qdrant for files you deleted. To fix that by hand, you track which file produced which points, detect changes, and delete what no longer exists.

With CocoIndex you write the per-file logic once, in plain Python, and declare the points that each file should produce. CocoIndex compares that with the last run and applies the difference:

  • Unchanged files are skipped. process_file is memoized: CocoIndex caches its result, and a file with the same content is not rendered or embedded again.
  • Deleted files take their points with them. Each file is a processing component that owns the points it declared. Remove the file, and CocoIndex deletes those points.
  • Live mode keeps the index current: cocoindex update -L main watches the folder and applies changes as they happen.
  • GPU calls are scheduled for you. Functions on the coco.GPU runner get a whole GPU per call, so concurrent pages queue and run one at a time on it.

The section What happens when the folder changes shows each case.

How the code fits together

The pipeline has 3 main functions. app_main mounts one process_file component per file. process_file splits its file into pages and runs process_page on each one. process_page embeds the page and declares one Qdrant point.

app_main calls mount_each, which mounts one process_file component per file: 1810.04805v2.pdf, sweetgreen_2023.jpg, restaurant_brands_international_2023.jpg and healthcare_industry_test_p9.jpg. Inside each component, coco.map runs process_page once per page (16 for the PDF, 1 for each image), and each process_page declares one Qdrant point. The component owns those points.
Components and the points they ownEnlarge

Components and the points they own

Zoom in to see details.

app_main calls mount_each, which mounts one process_file component per file: 1810.04805v2.pdf, sweetgreen_2023.jpg, restaurant_brands_international_2023.jpg and healthcare_industry_test_p9.jpg. Inside each component, coco.map runs process_page once per page (16 for the PDF, 1 for each image), and each process_page declares one Qdrant point. The component owns those points.

The steps below go through the code from the inside out.

Step 1: turn every file into pages

A Page dataclass gives every format the same shape:

python
@dataclass
class Page:
    page_number: int | None  # 1-based for PDFs, None for standalone images
    image: bytes

file_to_pages takes a filename and its raw bytes, and branches on the MIME type:

python
@coco.fn.as_async(runner=coco.GPU)
def file_to_pages(filename: str, content: bytes) -> list[Page]:
    """PDF -> one image per page; image -> a single page; anything else -> []."""
    mime_type, _ = mimetypes.guess_type(filename)
    if mime_type == "application/pdf":
        pages = []
        for i, image in enumerate(convert_from_bytes(content, dpi=PDF_RENDER_DPI)):
            buf = io.BytesIO()
            image.save(buf, format="PNG")
            pages.append(Page(page_number=i + 1, image=buf.getvalue()))
        return pages
    if mime_type and mime_type.startswith("image/"):
        return [Page(page_number=None, image=content)]
    return []
file_to_pages turns every file into a list of Page objects. 1810.04805v2.pdf, MIME type application/pdf, becomes 16 pages, page_number 1 to 16, each rendered to PNG at 200 DPI. sweetgreen_2023.jpg, MIME type image/jpeg, becomes 1 page with page_number None and the original bytes. A file such as notes.txt never reaches file_to_pages, because the walk_dir path matcher only includes .pdf, .jpg, .jpeg and .png.
file_to_pagesEnlarge

file_to_pages

Zoom in to see details.

file_to_pages turns every file into a list of Page objects. 1810.04805v2.pdf, MIME type application/pdf, becomes 16 pages, page_number 1 to 16, each rendered to PNG at 200 DPI. sweetgreen_2023.jpg, MIME type image/jpeg, becomes 1 page with page_number None and the original bytes. A file such as notes.txt never reaches file_to_pages, because the walk_dir path matcher only includes .pdf, .jpg, .jpeg and .png.

A PDF renders to one PNG per page with pdf2image at 200 DPI (PDF_RENDER_DPI). After this step the pipeline only sees pages, so PDFs and images follow the same path.

@coco.fn.as_async(runner=coco.GPU) wraps the synchronous function so that the pipeline can await it, and runs it on the GPU runner. The runner gives each call a whole GPU, so PDF rendering, which only uses the CPU, waits in the same queue as the ColPali calls, one call per GPU at a time. That keeps rendering off the event loop with the same decorator the ColPali calls use. For a folder of many large PDFs, drop the runner: @coco.fn.as_async alone runs the function in a worker thread, so rendering can overlap with embedding.

Step 2: embed a page with ColPali

The model loads once and is cached. The example uses the colpali_engine package directly, with one helper for page images and one for text queries:

python
@functools.cache
def get_colpali() -> tuple[ColPali, ColPaliProcessor, str]:
    model = ColPali.from_pretrained(COLPALI_MODEL_NAME)
    processor = ColPaliProcessor.from_pretrained(COLPALI_MODEL_NAME)
    device = get_torch_device("auto")
    model = model.to(device).eval()
    return model, processor, device


def embed_image(img: Image.Image) -> list[list[float]]:
    model, processor, device = get_colpali()
    batch = processor.process_images([img]).to(device)
    with torch.no_grad():
        return _postprocess(model(**batch), processor)


def embed_query(text: str) -> list[list[float]]:
    model, processor, device = get_colpali()
    batch = processor.process_queries(texts=[text]).to(device)
    with torch.no_grad():
        return _postprocess(model(**batch), processor)

Both helpers return a list of vectors: the multi-vector that MaxSim compares. _postprocess removes the padding vectors from the batch output. embed_page decodes a page and calls embed_image on the GPU runner:

python
@coco.fn.as_async(runner=coco.GPU)
def embed_page(page_png: bytes) -> list[list[float]]:
    return embed_image(Image.open(io.BytesIO(page_png)).convert("RGB"))

Step 3: declare one point per page

process_page embeds one page and declares its Qdrant point. The point ID is a UUID derived from the filename and page number, so the same page always maps to the same point:

python
def _page_id(filename: str, page_number: int | None) -> str:
    return str(uuid.uuid5(uuid.NAMESPACE_URL, f"{filename}|{page_number}"))


@coco.fn
async def process_page(
    page: Page, filename: str, target: qdrant.CollectionTarget
) -> None:
    embedding = await embed_page(page.image)
    target.declare_point(
        qdrant.PointStruct(
            id=_page_id(filename, page.page_number),
            vector=embedding,
            payload={"filename": filename, "page": page.page_number},
        )
    )

declare_point states that this point should exist. It does not write it directly: CocoIndex compares the declared points with the points declared on the last run, and applies only the difference.

process_file reads a file, splits it into pages, and runs process_page on each page with coco.map:

python
@coco.fn(memo=True)
async def process_file(file: FileLike, target: qdrant.CollectionTarget) -> None:
    filename = str(file.file_path.path)
    pages = await file_to_pages(filename, await file.read())
    await coco.map(process_page, pages, filename, target)

memo=True caches the result per file. If the file is unchanged on the next run, CocoIndex skips it.

Step 4: create the collection and watch the folder

The Qdrant client is created once, in the app’s lifespan, and shared through a context key:

python
QDRANT_DB = coco.ContextKey[QdrantClient]("multi_format_qdrant")


@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    builder.provide(QDRANT_DB, qdrant.create_client(qdrant_url(), prefer_grpc=True))
    yield

app_main declares the collection, walks the folder, and mounts one component per file:

python
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    model, _, _ = get_colpali()
    dim = int(getattr(model, "dim", 128))

    target_collection = await qdrant.mount_collection_target(
        QDRANT_DB,
        collection_name=QDRANT_COLLECTION,
        schema=await qdrant.CollectionSchema.create(
            vectors=qdrant.QdrantVectorDef(
                schema=MultiVectorSchema(
                    vector_schema=VectorSchema(dtype=np.dtype(np.float32), size=dim)
                ),
                distance="cosine",
                multivector_comparator="max_sim",
            )
        ),
    )

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(
            included_patterns=["**/*.pdf", "**/*.jpg", "**/*.jpeg", "**/*.png"]
        ),
        live=True,
    )
    await coco.mount_each(process_file, files.items(), target_collection)


app = coco.App(
    coco.AppConfig(name="MultiFormatIndexing"),
    app_main,
    sourcedir=pathlib.Path("./source_files"),
)
  • MultiVectorSchema with multivector_comparator="max_sim" tells Qdrant to store each point as a multi-vector and rank with MaxSim.
  • walk_dir lists the matching files. live=True lets it watch the folder when the app runs in live mode.
  • coco.mount_each mounts one process_file component per file, keyed by the file path. Each component owns the points that its file declares.

What happens when the folder changes

Run cocoindex update main again after you change the folder, and CocoIndex compares the new state with the last run. Suppose that between runs you edit the PDF, add an image and delete another:

The next cocoindex update after 3 changes. 1810.04805v2.pdf was edited: its 16 pages are rendered and embedded again and its points are updated. sweetgreen_2023.jpg and healthcare_industry_test_p9.jpg are unchanged: memo hits, no ColPali call. new_report.png was added: 1 page is embedded and 1 point is added. restaurant_brands_international_2023.jpg was deleted: its component is removed and its 1 point is deleted.
The next runEnlarge

The next run

Zoom in to see details.

The next cocoindex update after 3 changes. 1810.04805v2.pdf was edited: its 16 pages are rendered and embedded again and its points are updated. sweetgreen_2023.jpg and healthcare_industry_test_p9.jpg are unchanged: memo hits, no ColPali call. new_report.png was added: 1 page is embedded and 1 point is added. restaurant_brands_international_2023.jpg was deleted: its component is removed and its 1 point is deleted.
  • Edited file: the content changed, so the cached result does not apply. The file is split again and all its pages are re-embedded. Points with the same page ID are overwritten. If the edit removes pages, the points for those pages are no longer declared, and CocoIndex deletes them.
  • Unchanged files: the cached result applies, so nothing is rendered or embedded.
  • New file: a new component. Its pages are embedded and their points added.
  • Deleted file: its component goes away, and CocoIndex deletes the points it owned.

Query the index

ColPali embeds text into the same space as page images, so the query path embeds the text with embed_query and asks Qdrant for the top pages:

python
def query(text: str, *, top_k: int = TOP_K) -> None:
    client = qdrant.create_client(qdrant_url(), prefer_grpc=True)
    for r in client.query_points(
        collection_name=QDRANT_COLLECTION,
        query=embed_query(text),
        limit=top_k,
        with_payload=True,
    ).points:
        payload = r.payload or {}
        page = f" page {payload['page']}" if payload.get("page") else ""
        print(f"[{r.score:.3f}] {payload.get('filename')}{page}")

Each result prints the score, the filename, and the page number when the page came from a PDF.

Run it

You need Qdrant and the ColPali dependencies (torch, transformers, pdf2image). pdf2image needs poppler to render PDFs (brew install poppler or apt install poppler-utils).

Start Qdrant:

sh
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant

Configure and install. .env sets QDRANT_URL, which defaults to the local container above:

sh
cp .env.example .env
pip install -e .

Build the index from the sample source_files/ folder (1 PDF of 16 pages and 3 report pages as images, so 19 points):

sh
cocoindex update main

To keep the index in sync while you add, edit or delete files, run in live mode instead:

sh
cocoindex update -L main

Search across formats:

sh
python main.py "revenue growth"

According to the example’s README, on the sample set this query ranks the 2 company-report pages first (Sweetgreen, then Restaurant Brands), above an unrelated healthcare page. TOP_K is 5, so pages of the PDF fill the rest of the list.

Source

The full example is in examples/multi_format_indexing in the CocoIndex repository.

Frequently asked questions

How do I index PDFs and images together in one vector index?

Walk a folder of PDFs and images, split each file into a list of page images, embed every page with ColPali, and store one point per page in a Qdrant collection. Because every page lands in the same vector space, one natural-language query searches the whole mixed collection. See What you can build with it.

How do I search documents without OCR?

Use ColPali, which embeds the page image directly and uses its layout, tables and figures. PDF pages are rendered to images and embedded as images, so the pipeline needs no OCR or text extraction. See Why embed the page image instead of its text.

How do I normalize PDFs and images into a uniform format for embedding?

Write a file_to_pages function that classifies each file by MIME type and returns a list of Page objects. For a PDF it renders each page to a 200 DPI PNG; for an image it returns a single page with the original bytes; other types return an empty list. See Step 1: turn every file into pages.

How does ColPali represent a document for retrieval?

A traditional bi-encoder reduces each page to a single dense vector. ColPali splits the image into many small patches and gives each patch its own embedding. Together these form a multi-vector representation of the page. Qdrant scores a query against it with MaxSim (multivector_comparator="max_sim"). See Why embed the page image instead of its text.

How do I embed a text query to search a ColPali visual index?

An image and a query are different types of data, so ColPali embeds them with different processors. The example's embed_query runs ColPaliProcessor.process_queries on the text and passes the result to Qdrant's query_points. See Query the index.

What happens when I add, change or delete a file?

process_file uses @coco.fn(memo=True), so CocoIndex skips unchanged files on the next run. A new or edited file has its own pages rendered and embedded again. coco.mount_each mounts one component per file, so deleting a file removes its points from Qdrant, and pages that an edit removes lose their points too. Run cocoindex update -L main to keep the index in sync while the folder changes. See What happens when the folder changes.

What can I build with OCR-free visual document search?

Search over a folder of reports, finding figures and tables in research papers, search over scanned pages that have no text layer, search over slide decks exported to PDF, and page-level context for an agent, since each result carries the filename and page number. See What you can build with it.

Build fresh context for your agents.

CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.