Extract Patient Intake Forms with DSPy and CocoIndex

Extract Pydantic-typed patient records from PDF intake forms with DSPy and Gemini vision into Postgres with pgvector, and see which edits re-run extraction.

TutorialUpdated 6 min read

Intake form extraction with DSPy

DSPy & CocoIndex

  1. PDF intake formsdata/patient_forms/*.pdf
  2. Page imagesPyMuPDF · one PNG per page
  3. DSPy ChainOfThoughtGemini 2.5 Flash → Pydantic Patient
  4. Postgres + pgvectorone row per form, kept in sync

This post extracts typed patient records from PDF intake forms with DSPy and CocoIndex. DSPy declares the extraction as a typed signature over page images, and Gemini 2.5 Flash reads the pages. CocoIndex runs the extraction once per PDF and keeps one Postgres row per form in step with the folder, with a pgvector embedding of the visit for search.

DSPy replaces hand-written prompts with a programming model. You declare what an LLM step takes and returns (a signature) and pick a strategy (a module such as ChainOfThought); DSPy writes the prompt and parses the answer into your types. The same signature can later be tuned with DSPy’s optimizers.

The post covers the DSPy side: the Pydantic models, the signature and module, the page images the model sees, and which edits make CocoIndex re-run the extraction. The CocoIndex app is the same as in the BAML version of this post, so it is shown once, in a collapsed block. For a no-framework baseline and a comparison of plain prompts, BAML, DSPy and a local Ollama model, see Extract patient intake forms with an LLM.

The Pydantic models, the DSPy signature and module, and the PDF-to-image step come from the DSPy patient intake example in the CocoIndex repo (Apache 2.0), which runs on CocoIndex v1 and writes one JSON file per form. This post writes to Postgres instead and adds the embedding.

DSPy and BAML on this task

The two versions share the CocoIndex app and differ in the extraction layer:

AspectDSPy (this post)BAML
ExtractionA dspy.Signature and a ChainOfThought module. DSPy writes the prompt from the signature.A BAML function with a prompt you write, compiled to a Python client.
PDF inputPyMuPDF renders each page to a PNG at 2x; Gemini reads the images.baml_py.Pdf passes the PDF itself; Gemini reads it directly.
SchemaPydantic models in models.py. dob is a datetime.date.BAML classes in patient.baml, compiled to Pydantic. dob is a string.
Missing fieldsNo rule in the signature.The prompt asks for "N/A".
Not seen by the cachemodels.py, the signature and module, dspy.LM(...).patient.baml.
CocoIndex appSame code: one memoized component per PDF, a Postgres table with pgvector.Same code.

DSPy is built around Python modules and optimizer-driven prompt tuning. BAML is built around a typed prompt language, schema checks, and generated clients for several languages.

What you build

PatientIntakeExtractionDSPy

main.py + models.py

localfs.walk_dirdata/patient_forms/**/*.pdf
process_patient_form
extract_patientDSPy extractPyMuPDF pages → PatientExtractor · Gemini
Patient
EMBEDDERembed visit_summaryall-MiniLM-L6-v2
vector(384)
postgres · table.declare_rowpatients_dspykey: filename
namereason_for_visitpatient_info jsonbembedding vector
  • Read the PDF intake forms from a local directory.
  • For each PDF, render the pages to images, run the DSPy extractor to get a Patient, and embed a short summary of the visit.
  • Write one row per form to a Postgres table: a few columns for the key fields, the whole Patient as jsonb, and the embedding as a pgvector vector.

Prerequisites

  1. Run Postgres with the pgvector extension. With Docker:

    sh
    docker run -d --name pgvector -p 5432:5432 \
      -e POSTGRES_USER=cocoindex -e POSTGRES_PASSWORD=cocoindex -e POSTGRES_DB=cocoindex \
      pgvector/pgvector:pg17
  2. Install CocoIndex with its Postgres and sentence-transformers extras, and the other dependencies:

    sh
    pip install -U "cocoindex[postgres,sentence_transformers]" dspy-ai pydantic pymupdf pillow pgvector python-dotenv
  3. Create a .env file in the project directory. COCOINDEX_DB is the local file where CocoIndex keeps its state between runs, and POSTGRES_URL is the database that holds the patients_dspy table:

    sh
    COCOINDEX_DB=./cocoindex.db
    POSTGRES_URL=postgres://cocoindex:cocoindex@localhost/cocoindex
    GEMINI_API_KEY=your_api_key_here
  4. Copy a few intake forms to data/patient_forms/. The example ships 4 artificial PDFs in that folder.

Define Patient in models.py

The Pydantic classes follow a FHIR-inspired patient schema. Patient nests the others: an Address, lists of Condition, Medication, Allergy and Surgery, and an optional Insurance and Pharmacy, where Pharmacy has an Address of its own.

models.py · Patient

nested · repeated · optional

class Patient:
    name: str
    …
    address: Address
    …
    past_conditions: list[Condition]
    …
    pharmacy: Pharmacy | None
    …
address · nested
class Address:
    street: str
    city: str
    state: str
    zip_code: str
past_conditions · repeated
class Condition:
    name: str
    diagnosed: bool
pharmacy · optional
class Pharmacy:
    name: str
    phone: str
    address: Address  # nested again
models.py
"""Pydantic models for patient intake form extraction."""

from __future__ import annotations

import datetime

from pydantic import BaseModel, Field


class Contact(BaseModel):
    """Emergency contact information."""

    name: str
    phone: str
    relationship: str


class Address(BaseModel):
    """Physical address."""

    street: str
    city: str
    state: str
    zip_code: str


class Pharmacy(BaseModel):
    """Pharmacy information."""

    name: str
    phone: str
    address: Address


class Insurance(BaseModel):
    """Insurance information."""

    provider: str
    policy_number: str
    group_number: str | None = None
    policyholder_name: str
    relationship_to_patient: str


class Condition(BaseModel):
    """Medical condition."""

    name: str
    diagnosed: bool


class Medication(BaseModel):
    """Current medication."""

    name: str
    dosage: str


class Allergy(BaseModel):
    """Known allergy."""

    name: str


class Surgery(BaseModel):
    """Past surgery."""

    name: str
    date: str


class Patient(BaseModel):
    """Complete patient information extracted from intake form."""

    name: str
    dob: datetime.date
    gender: str
    address: Address
    phone: str
    email: str
    preferred_contact_method: str
    emergency_contact: Contact
    insurance: Insurance | None = None
    reason_for_visit: str
    symptoms_duration: str
    past_conditions: list[Condition] = Field(default_factory=list)
    current_medications: list[Medication] = Field(default_factory=list)
    allergies: list[Allergy] = Field(default_factory=list)
    surgeries: list[Surgery] = Field(default_factory=list)
    occupation: str | None = None
    pharmacy: Pharmacy | None = None
    consent_given: bool
    consent_date: str | None = None

The models do more than describe the shape:

  • Validation. Pydantic checks every field’s type, so dob has to parse as a datetime.date, and it raises if the model returns data that does not fit.
  • Defaults. Field(default_factory=list) gives an empty list when a form lists no conditions or medications.
  • Serialization. model_dump(mode="json") turns the record into JSON values for the database, with dob as an ISO date string such as "1992-07-22".

The DSPy signature and module

PatientExtractionSignature is a dspy.Signature: it declares the inputs and outputs of the LLM step and leaves the prompt to DSPy.

main.py
# DSPy Signature for patient information extraction from images
class PatientExtractionSignature(dspy.Signature):
    """Extract structured patient information from a medical intake form image."""

    form_images: list[dspy.Image] = dspy.InputField(
        desc="Images of the patient intake form pages"
    )
    patient: Patient = dspy.OutputField(
        desc="Extracted patient information with all available fields filled"
    )
  • The input form_images is a list of page images.
  • The output patient is typed as the Pydantic Patient, so DSPy parses the answer into that class.
  • The docstring and the desc strings go into the prompt DSPy writes.

PatientExtractor is a dspy.Module, the unit DSPy composes and optimizes. It implements the signature with ChainOfThought:

main.py
class PatientExtractor(dspy.Module):
    """DSPy module for extracting patient information from intake form images."""

    def __init__(self) -> None:
        super().__init__()
        self.extract = dspy.ChainOfThought(PatientExtractionSignature)

    def forward(self, form_images: list[dspy.Image]) -> Patient:
        """Extract patient information from form images and return as a Pydantic model."""
        result = self.extract(form_images=form_images)
        return result.patient

ChainOfThought asks the model to reason about the page images before it fills in patient. forward is DSPy’s standard entry point: calling the module runs it, and it returns the validated Patient.

Render the pages and run the module

extract_patient is the only place the CocoIndex app touches DSPy: PDF bytes in, a Patient out. It opens the PDF with PyMuPDF, renders each page at 2x (pymupdf.Matrix(2, 2)) so small print stays legible, wraps each PNG in a dspy.Image, and passes the list to the module in one call.

main.py
@coco.fn
def extract_patient(pdf_content: bytes) -> Patient:
    """Extract patient information from PDF content."""
    pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")

    form_images = []
    for page in pdf_doc:
        pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
        img_bytes = pix.tobytes("png")
        form_images.append(dspy.Image(img_bytes))

    pdf_doc.close()

    extractor = PatientExtractor()
    patient = extractor(form_images=form_images)
    return patient

For the 4 sample forms, these are the pages the model gets:

extract_patient · PDF → list[dspy.Image]

Matrix(2, 2) · 1 ChainOfThought call per form

Patient_Intake_Form_Emily_Artificial.pdf3 pages · A4 · 1191 × 1684 px each
  1. p1
  2. p2
  3. p3
Patient_Intake_Form_David_Artificial.pdf2 pages · US Letter · 1224 × 1584 px each
  1. p1
  2. p2
Patient_Intake_Form_Joe_Artificial.pdf1 page · US Letter · 1224 × 1584 px each
  1. p1
Patient_Intake_From_Jane_Artificial.pdf1 page · US Letter · 1224 × 1584 px each
  1. p1

4 forms → 4 model calls, 7 PNG page images

@coco.fn puts the function’s source under CocoIndex’s change tracking. models.py, the DSPy classes, and the dspy.LM(...) setting are not in that source, which matters for caching.

The CocoIndex app

The rest of main.py is the same app as in the BAML post, apart from the imports, the extraction call, and the table name. Three parts carry it:

  • process_patient_form, decorated with @coco.fn(memo=True), runs once per PDF: it calls extract_patient, embeds a short visit summary, and declares one row with table.declare_row. If the PDF’s bytes and the code of the functions it calls are unchanged, the next run skips it, with no rendering, no Gemini call, and no embedding.
  • PatientRow is the table schema. patient_info: dict[str, Any] becomes a jsonb column, and the embedding field, annotated with the embedder, becomes a vector(384) column.
  • app_main mounts the patients_dspy table with postgres.mount_table_target, walks data/patient_forms/ with localfs.walk_dir, and mounts one processing component per file with coco.mount_each. A row is a target state: CocoIndex inserts, updates, or deletes it to match what the components declare.

dspy.LM("gemini/gemini-2.5-flash") is configured once, when main.py loads, and reads GEMINI_API_KEY from the environment.

The full main.py (imports, row schema, DSPy classes, per-form component, app, search)
main.py
from __future__ import annotations

import asyncio
import os
import pathlib
import sys
from dataclasses import dataclass
from typing import Annotated, Any, AsyncIterator

import asyncpg
import dspy
import pymupdf
from dotenv import load_dotenv
from numpy.typing import NDArray
from pgvector.asyncpg import register_vector

import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

from models import Patient

DATABASE_URL = os.getenv(
    "POSTGRES_URL", "postgres://cocoindex:cocoindex@localhost/cocoindex"
)
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
TABLE_NAME = "patients_dspy"

PG_DB = coco.ContextKey[asyncpg.Pool]("patient_intake_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)


@dataclass
class PatientRow:
    filename: str
    name: str
    reason_for_visit: str
    patient_info: dict[str, Any]
    embedding: Annotated[NDArray, EMBEDDER]


@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    async with asyncpg.create_pool(DATABASE_URL) as pool:
        builder.provide(PG_DB, pool)
        builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
        yield


# DSPy Signature for patient information extraction from images
class PatientExtractionSignature(dspy.Signature):
    """Extract structured patient information from a medical intake form image."""

    form_images: list[dspy.Image] = dspy.InputField(
        desc="Images of the patient intake form pages"
    )
    patient: Patient = dspy.OutputField(
        desc="Extracted patient information with all available fields filled"
    )


class PatientExtractor(dspy.Module):
    """DSPy module for extracting patient information from intake form images."""

    def __init__(self) -> None:
        super().__init__()
        self.extract = dspy.ChainOfThought(PatientExtractionSignature)

    def forward(self, form_images: list[dspy.Image]) -> Patient:
        """Extract patient information from form images and return as a Pydantic model."""
        result = self.extract(form_images=form_images)
        return result.patient


@coco.fn
def extract_patient(pdf_content: bytes) -> Patient:
    """Extract patient information from PDF content."""
    pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")

    form_images = []
    for page in pdf_doc:
        pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
        img_bytes = pix.tobytes("png")
        form_images.append(dspy.Image(img_bytes))

    pdf_doc.close()

    extractor = PatientExtractor()
    patient = extractor(form_images=form_images)
    return patient


@coco.fn
def visit_summary(patient: Patient) -> str:
    """The text to embed: why the patient came in, and their history."""
    conditions = ", ".join(c.name for c in patient.past_conditions)
    medications = ", ".join(m.name for m in patient.current_medications)
    return (
        f"Reason for visit: {patient.reason_for_visit}\n"
        f"Symptoms duration: {patient.symptoms_duration}\n"
        f"Past conditions: {conditions}\n"
        f"Current medications: {medications}"
    )


@coco.fn(memo=True)
async def process_patient_form(
    file: FileLike, table: postgres.TableTarget[PatientRow]
) -> None:
    """Extract one intake form, embed its visit summary, and declare its row."""
    content = await file.read()
    patient_info = extract_patient(content)
    embedding = await coco.use_context(EMBEDDER).embed(visit_summary(patient_info))
    table.declare_row(
        row=PatientRow(
            filename=file.file_path.path.name,
            name=patient_info.name,
            reason_for_visit=patient_info.reason_for_visit,
            patient_info=patient_info.model_dump(mode="json"),
            embedding=embedding,
        ),
    )


@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    """Main application function that processes patient intake forms."""
    table = await postgres.mount_table_target(
        PG_DB,
        table_name=TABLE_NAME,
        table_schema=await postgres.TableSchema.from_class(
            PatientRow, primary_key=["filename"]
        ),
    )
    files = localfs.walk_dir(
        sourcedir,
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
    )
    await coco.mount_each(process_patient_form, files.items(), table)


lm = dspy.LM("gemini/gemini-2.5-flash")
dspy.configure(lm=lm)

app = coco.App(
    coco.AppConfig(name="PatientIntakeExtractionDSPy"),
    app_main,
    sourcedir=pathlib.Path("./data/patient_forms"),
)


async def search(query: str, top_k: int = 3) -> None:
    embedder = SentenceTransformerEmbedder(EMBED_MODEL)
    query_vec = await embedder.embed(query)
    async with asyncpg.create_pool(DATABASE_URL, init=register_vector) as pool:
        rows = await pool.fetch(
            f"""
            SELECT name, reason_for_visit, embedding <=> $1 AS distance
            FROM {TABLE_NAME}
            ORDER BY distance
            LIMIT $2
            """,
            query_vec,
            top_k,
        )
    for r in rows:
        print(f"[{1.0 - r['distance']:.3f}] {r['name']}: {r['reason_for_visit']}")


if __name__ == "__main__":
    load_dotenv()
    asyncio.run(search(" ".join(sys.argv[1:])))

You can read from other sources too, for example Amazon S3 or Google Drive. To keep the table updated as files change, see live mode.

Run it

Run the app. The update scans the forms, extracts, embeds, writes the rows, and exits:

sh
cocoindex update main.py

The patients_dspy table now has one row per form, and patient_info holds the full record, so Postgres can query any field in it, for example patient_info->>'dob'.

python main.py "<query>" runs the search at the end of main.py. It embeds the query with the same model and prints the 3 nearest forms, ordered by pgvector’s cosine distance (<=>), with 1 - distance as the score:

sh
python main.py "headache"

The DSPy signature has no rule for missing fields (the BAML prompt asks for "N/A"), so what a sparse form such as Jane Doe’s returns for reason_for_visit depends on the model. The BAML post shows sample scores for the same search.

What invalidates the cache

Run cocoindex update main.py again after a change, and CocoIndex re-runs only the forms the change affects. Which edits count depends on what CocoIndex can see. Pick one:

One edit, then cocoindex update main.py: what re-runs?

4 sample forms, all cached before the edit

data/patient_forms/
- Patient_Intake_Form_David_Artificial.pdf   (old bytes)
+ Patient_Intake_Form_David_Artificial.pdf   (new bytes)
1 of 4 forms re-run · 1 Gemini call
formprocess_patient_formrow
Emilycachedunchanged
Davidre-runsupdated
Joecachedunchanged
Janecachedunchanged

The file content is an input of process_patient_form. Only the component for the changed file misses the cache.

We checked the PDF, code, models.py and dspy.LM cases above against the same Postgres table, and these three as well:

  • Nothing changed: every form is a cache hit, and the run makes no LLM call.
  • A form is added: only that PDF is rendered, extracted, and embedded, and its row is inserted.
  • A form is deleted: its component is not mounted on the next run, so CocoIndex deletes the row it declared.

CocoIndex tracks the source of @coco.fn functions and the context values read with detect_change=True; the Pydantic models, the DSPy classes, and the module-level dspy.LM are plain Python outside them. To re-extract after such a change, do one of these:

  • Run cocoindex update main.py --full-reprocess once, which bypasses every cache for that run.
  • Add version= to @coco.fn on extract_patient and increase it with each change. With version set, CocoIndex tracks that number instead of the function’s source code.
  • For the model, pass its name as deps=: set LM_NAME = "gemini/gemini-2.5-flash", configure dspy.LM(LM_NAME), and decorate extract_patient with @coco.fn(deps=LM_NAME). A new name then re-runs every form.

Frequently asked questions

What is DSPy and how is it different from prompt engineering?

Traditional LLM apps rely on prompt engineering: you write a prompt with instructions, few-shot examples, and formatting, then call the model and parse raw text. That is fragile because small changes can break the output, and logic is buried in strings that are hard to test, compose, or version. DSPy replaces this with a programming model: you declare what each LLM step should do (inputs, outputs, constraints) and the framework figures out how to prompt the model to satisfy that spec.

See DSPy and BAML on this task.

How do you extract structured data from PDFs using DSPy vision and CocoIndex?

A @coco.fn function receives PDF bytes, renders each page to a PNG with PyMuPDF at 2x resolution, wraps each PNG in a DSPy Image, and passes the list to a DSPy module that returns a Pydantic Patient. There is no Markdown or OCR-plus-regex step. CocoIndex runs this once per PDF and writes one Postgres row per form.

See Render the pages and run the module.

What are DSPy Signatures and Modules?

A Signature declares the inputs and outputs of an LLM step and leaves the prompt to DSPy. The post's PatientExtractionSignature declares a list of form_images as input and a Patient as output. A Module is a composable, potentially trainable building block that implements a Signature. PatientExtractor uses dspy.ChainOfThought, a primitive module that calls the LLM with reasoning-style prompting to satisfy the Signature.

See The DSPy signature and module.

Why use Pydantic models for patient intake extraction?

Pydantic gives validation (fields have the correct type, e.g. dob is a date), nested models (Patient nests Address, Contact, Insurance), defaults and optional fields (Field(default_factory=list) yields empty lists), serialization to JSON with model_dump(mode="json"), and error checking on data that does not fit. The schema is FHIR-inspired.

See Define Patient in models.py.

How does CocoIndex make the DSPy extraction pipeline incremental?

process_patient_form is decorated with @coco.fn(memo=True). If a PDF's content and the code of the functions it calls are unchanged, the next cocoindex update skips that form: no rendering, no Gemini call, and no embedding. An added or replaced form is processed, and a deleted form's row is removed. Changes to models.py, the DSPy signature or module, or the LM are not tracked, so after those run cocoindex update main.py --full-reprocess bump version= on extract_patient, or pass the model name as deps=.

See What invalidates the cache.

Should I use DSPy or BAML for structured extraction with CocoIndex?

Both build LLM applications with structured outputs but emphasize different things. DSPy focuses on modular Python workflows and optimizer-driven prompt tuning; in this example it renders PDF pages to images for Gemini vision and uses Pydantic models directly as output types. BAML focuses on a typed DSL, schema guarantees, and multi-language client generation; its example passes the PDF to Gemini directly and compiles BAML classes to Pydantic models. Both run on the same CocoIndex code.

See DSPy and BAML on this task.

Which model and database does this DSPy example use?

It uses Gemini 2.5 Flash for vision extraction, configured when main.py loads with dspy.LM("gemini/gemini-2.5-flash") and dspy.configure(lm=lm). The records go to a Postgres table, patients_dspy, keyed by file name: the whole Patient as jsonb plus a pgvector vector(384) embedding from all-MiniLM-L6-v2. CocoIndex inserts, updates, and deletes rows as the PDFs change.

See The CocoIndex app.

How do you search the extracted patient records with pgvector?

Each row stores an embedding of a short visit summary: the reason for the visit, its duration, past conditions, and current medications. To search, embed the query with the same model and order the rows by pgvector's cosine distance, embedding <=> $1; python main.py "headache" prints the 3 nearest forms.

See Run it.

Build fresh context for your agents.

CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.