This post extracts typed patient records from PDF intake forms with DSPy and CocoIndex. DSPy declares the extraction as a typed signature over page images, and Gemini 2.5 Flash reads the pages. CocoIndex runs the extraction once per PDF and keeps one Postgres row per form in step with the folder, with a pgvector embedding of the visit for search.
DSPy replaces hand-written prompts with a programming model. You declare what an LLM step takes and returns (a signature) and pick a strategy (a module such as ChainOfThought); DSPy writes the prompt and parses the answer into your types. The same signature can later be tuned with DSPy’s optimizers.
The post covers the DSPy side: the Pydantic models, the signature and module, the page images the model sees, and which edits make CocoIndex re-run the extraction. The CocoIndex app is the same as in the BAML version of this post, so it is shown once, in a collapsed block. For a no-framework baseline and a comparison of plain prompts, BAML, DSPy and a local Ollama model, see Extract patient intake forms with an LLM.
The Pydantic models, the DSPy signature and module, and the PDF-to-image step come from the DSPy patient intake example in the CocoIndex repo (Apache 2.0), which runs on CocoIndex v1 and writes one JSON file per form. This post writes to Postgres instead and adds the embedding.
DSPy and BAML on this task
The two versions share the CocoIndex app and differ in the extraction layer:
dspy.Signature and a ChainOfThought module. DSPy writes the prompt from the signature.A BAML function with a prompt you write, compiled to a Python client.baml_py.Pdf passes the PDF itself; Gemini reads it directly.models.py. dob is a datetime.date.BAML classes in patient.baml, compiled to Pydantic. dob is a string."N/A".models.py, the signature and module, dspy.LM(...).patient.baml.DSPy is built around Python modules and optimizer-driven prompt tuning. BAML is built around a typed prompt language, schema checks, and generated clients for several languages.
What you build
- Read the PDF intake forms from a local directory.
- For each PDF, render the pages to images, run the DSPy extractor to get a
Patient, and embed a short summary of the visit. - Write one row per form to a Postgres table: a few columns for the key fields, the whole
Patientasjsonb, and the embedding as a pgvectorvector.
Prerequisites
-
Run Postgres with the pgvector extension. With Docker:
docker run -d --name pgvector -p 5432:5432 \ -e POSTGRES_USER=cocoindex -e POSTGRES_PASSWORD=cocoindex -e POSTGRES_DB=cocoindex \ pgvector/pgvector:pg17 -
Install CocoIndex with its Postgres and sentence-transformers extras, and the other dependencies:
pip install -U "cocoindex[postgres,sentence_transformers]" dspy-ai pydantic pymupdf pillow pgvector python-dotenv -
Create a
.envfile in the project directory.COCOINDEX_DBis the local file where CocoIndex keeps its state between runs, andPOSTGRES_URLis the database that holds thepatients_dspytable:COCOINDEX_DB=./cocoindex.db POSTGRES_URL=postgres://cocoindex:cocoindex@localhost/cocoindex GEMINI_API_KEY=your_api_key_here -
Copy a few intake forms to
data/patient_forms/. The example ships 4 artificial PDFs in that folder.
Define Patient in models.py
The Pydantic classes follow a FHIR-inspired patient schema. Patient nests the others: an Address, lists of Condition, Medication, Allergy and Surgery, and an optional Insurance and Pharmacy, where Pharmacy has an Address of its own.
class Patient: name: str … address: Address … past_conditions: list[Condition] … pharmacy: Pharmacy | None …
class Address: street: str city: str state: str zip_code: str
class Condition: name: str diagnosed: bool
class Pharmacy: name: str phone: str address: Address # nested again
"""Pydantic models for patient intake form extraction."""
from __future__ import annotations
import datetime
from pydantic import BaseModel, Field
class Contact(BaseModel):
"""Emergency contact information."""
name: str
phone: str
relationship: str
class Address(BaseModel):
"""Physical address."""
street: str
city: str
state: str
zip_code: str
class Pharmacy(BaseModel):
"""Pharmacy information."""
name: str
phone: str
address: Address
class Insurance(BaseModel):
"""Insurance information."""
provider: str
policy_number: str
group_number: str | None = None
policyholder_name: str
relationship_to_patient: str
class Condition(BaseModel):
"""Medical condition."""
name: str
diagnosed: bool
class Medication(BaseModel):
"""Current medication."""
name: str
dosage: str
class Allergy(BaseModel):
"""Known allergy."""
name: str
class Surgery(BaseModel):
"""Past surgery."""
name: str
date: str
class Patient(BaseModel):
"""Complete patient information extracted from intake form."""
name: str
dob: datetime.date
gender: str
address: Address
phone: str
email: str
preferred_contact_method: str
emergency_contact: Contact
insurance: Insurance | None = None
reason_for_visit: str
symptoms_duration: str
past_conditions: list[Condition] = Field(default_factory=list)
current_medications: list[Medication] = Field(default_factory=list)
allergies: list[Allergy] = Field(default_factory=list)
surgeries: list[Surgery] = Field(default_factory=list)
occupation: str | None = None
pharmacy: Pharmacy | None = None
consent_given: bool
consent_date: str | None = None
The models do more than describe the shape:
- Validation. Pydantic checks every field’s type, so
dobhas to parse as adatetime.date, and it raises if the model returns data that does not fit. - Defaults.
Field(default_factory=list)gives an empty list when a form lists no conditions or medications. - Serialization.
model_dump(mode="json")turns the record into JSON values for the database, withdobas an ISO date string such as"1992-07-22".
The DSPy signature and module
PatientExtractionSignature is a dspy.Signature: it declares the inputs and outputs of the LLM step and leaves the prompt to DSPy.
# DSPy Signature for patient information extraction from images
class PatientExtractionSignature(dspy.Signature):
"""Extract structured patient information from a medical intake form image."""
form_images: list[dspy.Image] = dspy.InputField(
desc="Images of the patient intake form pages"
)
patient: Patient = dspy.OutputField(
desc="Extracted patient information with all available fields filled"
)
- The input
form_imagesis a list of page images. - The output
patientis typed as the PydanticPatient, so DSPy parses the answer into that class. - The docstring and the
descstrings go into the prompt DSPy writes.
PatientExtractor is a dspy.Module, the unit DSPy composes and optimizes. It implements the signature with ChainOfThought:
class PatientExtractor(dspy.Module):
"""DSPy module for extracting patient information from intake form images."""
def __init__(self) -> None:
super().__init__()
self.extract = dspy.ChainOfThought(PatientExtractionSignature)
def forward(self, form_images: list[dspy.Image]) -> Patient:
"""Extract patient information from form images and return as a Pydantic model."""
result = self.extract(form_images=form_images)
return result.patient
ChainOfThought asks the model to reason about the page images before it fills in patient. forward is DSPy’s standard entry point: calling the module runs it, and it returns the validated Patient.
Render the pages and run the module
extract_patient is the only place the CocoIndex app touches DSPy: PDF bytes in, a Patient out. It opens the PDF with PyMuPDF, renders each page at 2x (pymupdf.Matrix(2, 2)) so small print stays legible, wraps each PNG in a dspy.Image, and passes the list to the module in one call.
@coco.fn
def extract_patient(pdf_content: bytes) -> Patient:
"""Extract patient information from PDF content."""
pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")
form_images = []
for page in pdf_doc:
pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
img_bytes = pix.tobytes("png")
form_images.append(dspy.Image(img_bytes))
pdf_doc.close()
extractor = PatientExtractor()
patient = extractor(form_images=form_images)
return patient
For the 4 sample forms, these are the pages the model gets:
p1
p2
p3
p1
p2
p1
p1
4 forms → 4 model calls, 7 PNG page images
@coco.fn puts the function’s source under CocoIndex’s change tracking. models.py, the DSPy classes, and the dspy.LM(...) setting are not in that source, which matters for caching.
The CocoIndex app
The rest of main.py is the same app as in the BAML post, apart from the imports, the extraction call, and the table name. Three parts carry it:
process_patient_form, decorated with@coco.fn(memo=True), runs once per PDF: it callsextract_patient, embeds a short visit summary, and declares one row withtable.declare_row. If the PDF’s bytes and the code of the functions it calls are unchanged, the next run skips it, with no rendering, no Gemini call, and no embedding.PatientRowis the table schema.patient_info: dict[str, Any]becomes ajsonbcolumn, and theembeddingfield, annotated with the embedder, becomes avector(384)column.app_mainmounts thepatients_dspytable withpostgres.mount_table_target, walksdata/patient_forms/withlocalfs.walk_dir, and mounts one processing component per file withcoco.mount_each. A row is a target state: CocoIndex inserts, updates, or deletes it to match what the components declare.
dspy.LM("gemini/gemini-2.5-flash") is configured once, when main.py loads, and reads GEMINI_API_KEY from the environment.
The full main.py (imports, row schema, DSPy classes, per-form component, app, search)
from __future__ import annotations
import asyncio
import os
import pathlib
import sys
from dataclasses import dataclass
from typing import Annotated, Any, AsyncIterator
import asyncpg
import dspy
import pymupdf
from dotenv import load_dotenv
from numpy.typing import NDArray
from pgvector.asyncpg import register_vector
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
from models import Patient
DATABASE_URL = os.getenv(
"POSTGRES_URL", "postgres://cocoindex:cocoindex@localhost/cocoindex"
)
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
TABLE_NAME = "patients_dspy"
PG_DB = coco.ContextKey[asyncpg.Pool]("patient_intake_db")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@dataclass
class PatientRow:
filename: str
name: str
reason_for_visit: str
patient_info: dict[str, Any]
embedding: Annotated[NDArray, EMBEDDER]
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
async with asyncpg.create_pool(DATABASE_URL) as pool:
builder.provide(PG_DB, pool)
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
# DSPy Signature for patient information extraction from images
class PatientExtractionSignature(dspy.Signature):
"""Extract structured patient information from a medical intake form image."""
form_images: list[dspy.Image] = dspy.InputField(
desc="Images of the patient intake form pages"
)
patient: Patient = dspy.OutputField(
desc="Extracted patient information with all available fields filled"
)
class PatientExtractor(dspy.Module):
"""DSPy module for extracting patient information from intake form images."""
def __init__(self) -> None:
super().__init__()
self.extract = dspy.ChainOfThought(PatientExtractionSignature)
def forward(self, form_images: list[dspy.Image]) -> Patient:
"""Extract patient information from form images and return as a Pydantic model."""
result = self.extract(form_images=form_images)
return result.patient
@coco.fn
def extract_patient(pdf_content: bytes) -> Patient:
"""Extract patient information from PDF content."""
pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")
form_images = []
for page in pdf_doc:
pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))
img_bytes = pix.tobytes("png")
form_images.append(dspy.Image(img_bytes))
pdf_doc.close()
extractor = PatientExtractor()
patient = extractor(form_images=form_images)
return patient
@coco.fn
def visit_summary(patient: Patient) -> str:
"""The text to embed: why the patient came in, and their history."""
conditions = ", ".join(c.name for c in patient.past_conditions)
medications = ", ".join(m.name for m in patient.current_medications)
return (
f"Reason for visit: {patient.reason_for_visit}\n"
f"Symptoms duration: {patient.symptoms_duration}\n"
f"Past conditions: {conditions}\n"
f"Current medications: {medications}"
)
@coco.fn(memo=True)
async def process_patient_form(
file: FileLike, table: postgres.TableTarget[PatientRow]
) -> None:
"""Extract one intake form, embed its visit summary, and declare its row."""
content = await file.read()
patient_info = extract_patient(content)
embedding = await coco.use_context(EMBEDDER).embed(visit_summary(patient_info))
table.declare_row(
row=PatientRow(
filename=file.file_path.path.name,
name=patient_info.name,
reason_for_visit=patient_info.reason_for_visit,
patient_info=patient_info.model_dump(mode="json"),
embedding=embedding,
),
)
@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
"""Main application function that processes patient intake forms."""
table = await postgres.mount_table_target(
PG_DB,
table_name=TABLE_NAME,
table_schema=await postgres.TableSchema.from_class(
PatientRow, primary_key=["filename"]
),
)
files = localfs.walk_dir(
sourcedir,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.pdf"]),
)
await coco.mount_each(process_patient_form, files.items(), table)
lm = dspy.LM("gemini/gemini-2.5-flash")
dspy.configure(lm=lm)
app = coco.App(
coco.AppConfig(name="PatientIntakeExtractionDSPy"),
app_main,
sourcedir=pathlib.Path("./data/patient_forms"),
)
async def search(query: str, top_k: int = 3) -> None:
embedder = SentenceTransformerEmbedder(EMBED_MODEL)
query_vec = await embedder.embed(query)
async with asyncpg.create_pool(DATABASE_URL, init=register_vector) as pool:
rows = await pool.fetch(
f"""
SELECT name, reason_for_visit, embedding <=> $1 AS distance
FROM {TABLE_NAME}
ORDER BY distance
LIMIT $2
""",
query_vec,
top_k,
)
for r in rows:
print(f"[{1.0 - r['distance']:.3f}] {r['name']}: {r['reason_for_visit']}")
if __name__ == "__main__":
load_dotenv()
asyncio.run(search(" ".join(sys.argv[1:])))You can read from other sources too, for example Amazon S3 or Google Drive. To keep the table updated as files change, see live mode.
Run it
Run the app. The update scans the forms, extracts, embeds, writes the rows, and exits:
cocoindex update main.py
The patients_dspy table now has one row per form, and patient_info holds the full record, so Postgres can query any field in it, for example patient_info->>'dob'.
python main.py "<query>" runs the search at the end of main.py. It embeds the query with the same model and prints the 3 nearest forms, ordered by pgvector’s cosine distance (<=>), with 1 - distance as the score:
python main.py "headache"
The DSPy signature has no rule for missing fields (the BAML prompt asks for "N/A"), so what a sparse form such as Jane Doe’s returns for reason_for_visit depends on the model. The BAML post shows sample scores for the same search.
What invalidates the cache
Run cocoindex update main.py again after a change, and CocoIndex re-runs only the forms the change affects. Which edits count depends on what CocoIndex can see. Pick one:
One edit, then cocoindex update main.py: what re-runs?
- Patient_Intake_Form_David_Artificial.pdf (old bytes) + Patient_Intake_Form_David_Artificial.pdf (new bytes)
| form | process_patient_form | row |
|---|---|---|
| Emily | cached | unchanged |
| David | re-runs | updated |
| Joe | cached | unchanged |
| Jane | cached | unchanged |
The file content is an input of process_patient_form. Only the component for the changed file misses the cache.
def extract_patient(pdf_content: bytes) -> Patient: - pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2)) + pix = page.get_pixmap(matrix=pymupdf.Matrix(3, 3))
| form | process_patient_form | row |
|---|---|---|
| Emily | re-runs | updated |
| David | re-runs | updated |
| Joe | re-runs | updated |
| Jane | re-runs | updated |
extract_patient is a @coco.fn, so its source is part of the memo key of every component that called it.
- EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2" + EMBED_MODEL = "sentence-transformers/all-MiniLM-L12-v2"
| form | process_patient_form | row |
|---|---|---|
| Emily | re-runs | updated |
| David | re-runs | updated |
| Joe | re-runs | updated |
| Jane | re-runs | updated |
EMBEDDER is a context key with detect_change=True, so the embedder is part of the key. The Gemini call repeats too: the memo boundary is the whole component, and extract_patient has no cache of its own.
class Patient(BaseModel): ... + blood_type: str | None = None
| form | process_patient_form | row |
|---|---|---|
| Emily | cached | unchanged, old shape |
| David | cached | unchanged, old shape |
| Joe | cached | unchanged, old shape |
| Jane | cached | unchanged, old shape |
Pydantic models are plain Python, outside any @coco.fn. CocoIndex does not see the change, and the rows keep the old shape, without blood_type.
To re-extract: run cocoindex update main.py --full-reprocess once, or bump version= on extract_patient.
- lm = dspy.LM("gemini/gemini-2.5-flash") + lm = dspy.LM("gemini/gemini-2.5-pro")
| form | process_patient_form | row |
|---|---|---|
| Emily | cached | unchanged, old model |
| David | cached | unchanged, old model |
| Joe | cached | unchanged, old model |
| Jane | cached | unchanged, old model |
The LM is configured at module level, outside any @coco.fn. Every row keeps the values the previous model extracted.
To re-extract: run --full-reprocess once, or pass the model name to the function, @coco.fn(deps=LM_NAME), so a new name re-runs every form.
- @coco.fn(version=1) + @coco.fn(version=2) def extract_patient(pdf_content: bytes) -> Patient:
| form | process_patient_form | row |
|---|---|---|
| Emily | re-runs | updated |
| David | re-runs | updated |
| Joe | re-runs | updated |
| Jane | re-runs | updated |
With version= set, the number replaces the source as the function's logic. Bumping it re-runs every form; editing the body alone no longer does.
We checked the PDF, code, models.py and dspy.LM cases above against the same Postgres table, and these three as well:
- Nothing changed: every form is a cache hit, and the run makes no LLM call.
- A form is added: only that PDF is rendered, extracted, and embedded, and its row is inserted.
- A form is deleted: its component is not mounted on the next run, so CocoIndex deletes the row it declared.
CocoIndex tracks the source of @coco.fn functions and the context values read with detect_change=True; the Pydantic models, the DSPy classes, and the module-level dspy.LM are plain Python outside them. To re-extract after such a change, do one of these:
- Run
cocoindex update main.py --full-reprocessonce, which bypasses every cache for that run. - Add
version=to@coco.fnonextract_patientand increase it with each change. Withversionset, CocoIndex tracks that number instead of the function’s source code. - For the model, pass its name as
deps=: setLM_NAME = "gemini/gemini-2.5-flash", configuredspy.LM(LM_NAME), and decorateextract_patientwith@coco.fn(deps=LM_NAME). A new name then re-runs every form.
Related
- Extract patient intake forms with an LLM: the no-framework baseline, and how BAML, DSPy and Ollama compare on this task.
- The BAML version: the same app with a BAML schema and prompt, and the
role("user")gotcha for PDF input on OpenAI. - The example in the CocoIndex repo, with a walkthrough in the docs that writes JSON files instead of a table.
Frequently asked questions
What is DSPy and how is it different from prompt engineering?
Traditional LLM apps rely on prompt engineering: you write a prompt with instructions, few-shot examples, and formatting, then call the model and parse raw text. That is fragile because small changes can break the output, and logic is buried in strings that are hard to test, compose, or version. DSPy replaces this with a programming model: you declare what each LLM step should do (inputs, outputs, constraints) and the framework figures out how to prompt the model to satisfy that spec.
How do you extract structured data from PDFs using DSPy vision and CocoIndex?
A @coco.fn function receives PDF bytes, renders each page to a PNG with PyMuPDF at 2x resolution, wraps each PNG in a DSPy Image, and passes the list to a DSPy module that returns a Pydantic Patient. There is no Markdown or OCR-plus-regex step. CocoIndex runs this once per PDF and writes one Postgres row per form.
What are DSPy Signatures and Modules?
A Signature declares the inputs and outputs of an LLM step and leaves the prompt to DSPy. The post's PatientExtractionSignature declares a list of form_images as input and a Patient as output. A Module is a composable, potentially trainable building block that implements a Signature. PatientExtractor uses dspy.ChainOfThought, a primitive module that calls the LLM with reasoning-style prompting to satisfy the Signature.
Why use Pydantic models for patient intake extraction?
Pydantic gives validation (fields have the correct type, e.g. dob is a date), nested models (Patient nests Address, Contact, Insurance), defaults and optional fields (Field(default_factory=list) yields empty lists), serialization to JSON with model_dump(mode="json"), and error checking on data that does not fit. The schema is FHIR-inspired.
How does CocoIndex make the DSPy extraction pipeline incremental?
process_patient_form is decorated with @coco.fn(memo=True). If a PDF's content and the code of the functions it calls are unchanged, the next cocoindex update skips that form: no rendering, no Gemini call, and no embedding. An added or replaced form is processed, and a deleted form's row is removed. Changes to models.py, the DSPy signature or module, or the LM are not tracked, so after those run cocoindex update main.py --full-reprocess bump version= on extract_patient, or pass the model name as deps=.
Should I use DSPy or BAML for structured extraction with CocoIndex?
Both build LLM applications with structured outputs but emphasize different things. DSPy focuses on modular Python workflows and optimizer-driven prompt tuning; in this example it renders PDF pages to images for Gemini vision and uses Pydantic models directly as output types. BAML focuses on a typed DSL, schema guarantees, and multi-language client generation; its example passes the PDF to Gemini directly and compiles BAML classes to Pydantic models. Both run on the same CocoIndex code.
Which model and database does this DSPy example use?
It uses Gemini 2.5 Flash for vision extraction, configured when main.py loads with dspy.LM("gemini/gemini-2.5-flash") and dspy.configure(lm=lm). The records go to a Postgres table, patients_dspy, keyed by file name: the whole Patient as jsonb plus a pgvector vector(384) embedding from all-MiniLM-L6-v2. CocoIndex inserts, updates, and deletes rows as the PDFs change.
See The CocoIndex app.
How do you search the extracted patient records with pgvector?
Each row stores an embedding of a short visit summary: the reason for the visit, its duration, past conditions, and current medications. To search, embed the query with the same model and order the rows by pgvector's cosine distance, embedding <=> $1; python main.py "headache" prints the 3 nearest forms.
See Run it.
Build fresh context for your agents.
CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.

