Schema inference for Qdrant collections

CocoIndex has a native Qdrant connector. It runs on CocoIndex’s Rust engine with incremental processing end to end, so an edited file re-embeds that file only.

The connector also sets up the Qdrant collection. You declare the collection in Python, next to the code that writes points into it, and CocoIndex creates it on the first cocoindex update. The vector size is read from the embedding model, so the collection and the model stay in agreement. When the declaration changes, for example after a switch to a model with a different dimension, CocoIndex drops the collection, creates it again and writes the points back.

The same mechanism sets up vector columns in the Postgres, Neo4j and LanceDB connectors. This post walks through it for Qdrant, with CocoIndex v1 and the text_embedding_qdrant example.

No more manual setup

Without schema inference, you create the collection by hand before indexing:

curl -X PUT 'http://localhost:6333/collections/image_search' \
  -H 'Content-Type: application/json' \
  -d '{
    "vectors": {
      "embedding": {
        "size": 768,
        "distance": "Cosine"
      }
    }
  }'

The size (768 here) has to match the embedding model, and the vector name (embedding) has to match the code that writes points. Both live outside that code. Switch to a model with another dimension and Qdrant rejects the writes until someone recreates the collection with the new size.

With CocoIndex, the collection is declared in the indexing code, and you don’t manage it by hand.

How it works

The example: Markdown files into Qdrant

The text_embedding_qdrant example walks a folder of Markdown files, splits each file into chunks (chunk_size=2000, chunk_overlap=500), embeds every chunk with all-MiniLM-L6-v2 and stores one Qdrant point per chunk in the TextEmbedding collection. The docs walkthrough covers setup and querying; this post follows the collection schema.

{
  "result": {
    …
    "config": {
      "params": {
        "vectors": {
          "size": 384,
          "distance": "Cosine"
        },
        …

mount_collection_target: the collection, declared in code

app_main declares the collection before it walks the files:

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    target_collection = await qdrant.mount_collection_target(
        QDRANT_DB,
        collection_name=QDRANT_COLLECTION,
        schema=await qdrant.CollectionSchema.create(
            vectors=qdrant.QdrantVectorDef(schema=EMBEDDER)
        ),
    )
    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
        live=True,  # source supports live watch; pass -L to `cocoindex update` to actually run live
    )
    await coco.mount_each(process_file, files.items(), target_collection)

mount_collection_target takes the Qdrant client’s context key, the collection name and a CollectionSchema. This schema has one unnamed dense vector, and its definition, QdrantVectorDef(schema=EMBEDDER), gives no size. The distance defaults to cosine.

The collection is a target state: CocoIndex records what was declared, compares it with the previous run, and creates, replaces or deletes the collection to match.

QdrantVectorDef(schema=EMBEDDER): vector size from the embedding model

EMBEDDER is a context key for the embedder, provided once in the lifespan:

QDRANT_URL = "http://localhost:6334"
QDRANT_COLLECTION = "TextEmbedding"
TOP_K = 5


EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
QDRANT_DB = coco.ContextKey[QdrantClient]("text_embedding_qdrant")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)

_splitter = RecursiveSplitter()


@coco.lifespan
async def coco_lifespan(
    builder: coco.EnvironmentBuilder,
) -> AsyncIterator[None]:
    client = qdrant.create_client(QDRANT_URL, prefer_grpc=True)
    builder.provide(QDRANT_DB, client)
    builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
    yield

When CollectionSchema.create resolves QdrantVectorDef(schema=EMBEDDER), it looks up the embedder behind the key and calls its __coco_vector_schema__() method. SentenceTransformerEmbedder answers with the model’s own dimension: VectorSchema(dtype=np.dtype(np.float32), size=384) for all-MiniLM-L6-v2. CocoIndex then creates the collection with a vector of size 384 and cosine distance.

The number 384 appears nowhere in main.py. Set EMBED_MODEL to a model with 768 dimensions and the collection follows on the next update (see below). For a model without __coco_vector_schema__(), pass an explicit VectorSchema(dtype=np.dtype(np.float32), size=768) instead of the key.

Qdrant needs a schema only for vectors: their size and distance. The payload is schemaless, so the payload fields (filename, chunk_start, chunk_end, text) need no declaration.

declare_point: the points that fill the collection

Each chunk becomes one point:

@coco.fn
async def process_chunk(
    chunk: Chunk,
    filename: pathlib.PurePath,
    id_gen: IdGenerator,
    target: qdrant.CollectionTarget,
) -> None:
    embedding_vec = await coco.use_context(EMBEDDER).embed(chunk.text)

    point = qdrant.PointStruct(
        id=await id_gen.next_id(chunk.text),
        vector=embedding_vec.tolist(),
        payload={
            "filename": str(filename),
            "chunk_start": chunk.start.char_offset,
            "chunk_end": chunk.end.char_offset,
            "text": chunk.text,
        },
    )
    target.declare_point(point)

The point’s vector is a plain list, which fits the single unnamed vector the schema declared. The vector comes from the same EMBEDDER that sized the collection, so its length is 384. declare_point declares the point as a target state: CocoIndex upserts new and changed points, and deletes points whose chunk is gone. id_gen.next_id(chunk.text) gives each point an ID derived from its chunk text that stays the same across runs.

Named vectors: one size per model

A collection that holds more than one vector takes a dict of names to definitions. Each entry resolves its own size, so 2 embedding models give 2 named vectors in Qdrant, each with its own size and distance. From the connector docs:

collection = await qdrant.mount_collection_target(
    QDRANT_DB,
    "multimodal_docs",
    await qdrant.CollectionSchema.create(
        vectors={
            "text": qdrant.QdrantVectorDef(
                schema=text_embedder,
                distance="cosine",
            ),
            "image": qdrant.QdrantVectorDef(
                schema=VectorSchema(dtype=np.dtype(np.float32), size=512),
                distance="dot",
            ),
        }
    ),
)

Points then pass vector={"text": ..., "image": ...}. Sparse vectors (QdrantSparseVectorDef) go in the same dict. However many vectors the collection has, CocoIndex derives the Qdrant schema from these definitions.

cocoindex update: setup and update in one command

CocoIndex v1 has no separate setup step. The cocoindex CLI builds and updates the index in one command:

cocoindex update main          # catch-up: scan, sync, exit
cocoindex update -L main       # live: keep watching for file changes

On the first run, cocoindex update main creates the TextEmbedding collection, then writes the points. Each later run compares the declared collection schema with the one recorded on the previous run:

• Same schema: the collection stays, and only changed points are written.
• Different schema, such as a new vector size or distance: CocoIndex deletes the collection and creates it with the new schema. Qdrant cannot change a vector’s size in place, and the new collection starts empty, so CocoIndex reprocesses every component that wrote into it and the points are written again.

A model change shows both cases. EMBEDDER is declared with detect_change=True, so a new model invalidates the memoized process_file calls and every chunk is embedded again. If the new model has the same dimension, the collection stays and the points are upserted with the new vectors. If the dimension differs, the collection is recreated first.

Schema changes are applied by cocoindex update itself. To see what an update will do before it runs, use --preview, which computes the target actions and prints them without applying them:

cocoindex update --preview main

cocoindex drop: remove the app and its collection

To drop the app, run:

cocoindex drop main

cocoindex drop reverts every target state the app created and clears the app’s internal state. The Qdrant collection is derived data owned by the app, so it is deleted too, the same way a Postgres table would be. A collection declared with managed_by="user" is the exception: CocoIndex assumes the collection already exists, and leaves it in place on drop.

Why the target schema should follow the code

An indexing pipeline usually spans several systems, and each holds part of the schema:

• Vector databases, such as Qdrant or Postgres with pgvector
• Relational databases, such as PostgreSQL
• Graph databases, such as Neo4j or FalkorDB
• The pipeline’s own state, which tracks what was processed for incremental updates

The data types the code outputs and the target schema must match. Keeping them aligned by hand means:

• Setting up and syncing schemas manually.
• Coordinating developers, DevOps and data engineers, because the people who write the pipeline are often not the people who deploy and run it.
• Debugging mismatches between the pipeline logic and the storage layer.
• Planning each production rollout around the schema change.

A mismatch fails in one of 2 ways. Some failures are visible: if nobody created the collection, the first write errors out, and you still have to work out the exact size and distance it needs. Others are quiet: the pipeline’s state and the target fall out of step. If someone drops and recreates the collection but not the pipeline’s state, or the other way round, the pipeline’s record of what was written no longer matches the store, and the bugs that follow are hard to trace. A pipeline usually needs several backends, and keeping them in sync manually is error-prone.

The work also repeats. New fields need indexing, old ones get deprecated, models get replaced, and each change to the pipeline can change the target schema with it.

In CocoIndex, every step produces derived data, down to the target. Deriving the target schema from the code keeps data consistent between the pipeline and the target stores, with less manual configuration.

Declarative indexing: CocoIndex does the rest

When we started CocoIndex, the aim was for developers to declare the transformation and indexing logic and leave the rest to CocoIndex. Automatic schema setup is one part of that.

For Qdrant, CocoIndex derives the collection setup from the code: the vector names, sizes and distances. The payload stays schemaless.

If you keep indexing logic and storage setup in sync by hand today, tell us on Discord what you would like CocoIndex to handle next.

About the author

Linghua JinCofounder & CEO of CocoIndex · @linghuaj

CEO and cofounder of CocoIndex. Writes about incremental data infrastructure, agents, and the engineering decisions behind the engine.