Hybrid search over SEC EDGAR filings with Apache Doris

Index SEC 10-K text and XBRL facts into one Apache Doris table with CocoIndex v1, then rank each chunk by meaning and by keywords, fused with RRF.

TutorialUpdated 15 min read
Hybrid search over SEC EDGAR filings. The question cloud computing and AI risk is ranked by meaning and by keywords over one Apache Doris table: Microsoft 10-K, distance 0.795, keyword match, the top hit; Apple 10-K, 1.031, keyword match; Microsoft facts, 1.329, no match.
Enlarge cover

Hybrid search over SEC EDGAR filings with Apache Doris

Zoom in to see details.

Hybrid search over SEC EDGAR filings. The question cloud computing and AI risk is ranked by meaning and by keywords over one Apache Doris table: Microsoft 10-K, distance 0.795, keyword match, the top hit; Apple 10-K, 1.031, keyword match; Microsoft facts, 1.329, no match.

Every public company in the United States files reports with the SEC (the U.S. Securities and Exchange Commission). The reports are public on EDGAR, the SEC’s filing system. Say you want to ask questions across them: which companies name cybersecurity as a risk, what a company reported as revenue, or how often a risk comes up by filing year.

Two things make this harder than a text search. First, one company’s data comes in different formats: its risks are prose in an annual report, and its financial numbers are structured data. Second, a question such as “cloud computing and AI risk” needs both kinds of search. Keyword search finds the exact words. Search by meaning finds passages that say the same thing in other words.

An agent or a RAG pipeline that retrieves from filings needs both. A question that names a company, a CIK or a metric such as “net income” must hit those exact words; a vector places them near similar words but does not guarantee a match. A question about “risks of hosted services” must also find a 10-K that says “a data breach in our cloud services”. The same company’s numbers are in JSON, not prose, so both formats have to land in one searchable form.

One company, 2 formats on EDGAR, from the example's sample data for Apple. Left, the 10-K annual report as text, 0000320193_2025-11-01_10-K.txt: APPLE INC. FORM 10-K ANNUAL REPORT, Item 1A. Risk Factors, Cybersecurity threats are a material risk. A data breach or ransomware attack on our systems could disrupt operations and harm our reputation, and at the end a contact line with an email address and a phone number; 778 characters of prose. Right, the company facts as XBRL JSON, 0000320193.json: cik 320193, entityName Apple Inc., us-gaap Revenues with val 391035000000 for the period ending 2024-09-28 and 383285000000 for the one before, then NetIncomeLoss and ResearchAndDevelopmentExpense; 557 characters of JSON, 3 metrics. The sample is a short synthetic excerpt; the pipeline turns both files into text and indexes them in one table.
One company, 2 formatsEnlarge

One company, 2 formats

Zoom in to see details.

One company, 2 formats on EDGAR, from the example's sample data for Apple. Left, the 10-K annual report as text, 0000320193_2025-11-01_10-K.txt: APPLE INC. FORM 10-K ANNUAL REPORT, Item 1A. Risk Factors, Cybersecurity threats are a material risk. A data breach or ransomware attack on our systems could disrupt operations and harm our reputation, and at the end a contact line with an email address and a phone number; 778 characters of prose. Right, the company facts as XBRL JSON, 0000320193.json: cik 320193, entityName Apple Inc., us-gaap Revenues with val 391035000000 for the period ending 2024-09-28 and 383285000000 for the one before, then NetIncomeLoss and ResearchAndDevelopmentExpense; 557 characters of JSON, 3 metrics. The sample is a short synthetic excerpt; the pipeline turns both files into text and indexes them in one table.

This tutorial walks through the SEC EDGAR Financial Analytics example, a CocoIndex v1 pipeline that handles both. It reads the 2 formats, removes personal data and splits the text into chunks. It tags each chunk with topics and turns it into a vector (a list of numbers that captures its meaning). Everything lands in one Apache Doris table that has 2 indexes: an inverted index for keywords and a vector index for meaning. A search then ranks every chunk both ways and combines the 2 rankings into one score.

The numbers in the figures come from the example’s own code and sample data: download.py writes 2 short 10-K excerpts and 2 company-facts files, for Apple and Microsoft. We ran the example’s functions on them and made the vectors with the same model, all-MiniLM-L6-v2. We did not run Doris; the search ranks follow the rules in search.py. The code is in the CocoIndex v1 repo.

A few terms used below:

  • 10-K: a company’s annual report to the SEC. Its “Item 1A. Risk Factors” section lists what could go wrong. The 10-Q is the quarterly report.
  • CIK (Central Index Key): the number the SEC assigns to each filer, such as 0000320193 for Apple.
  • XBRL (eXtensible Business Reporting Language): the tagged format for the numbers in a filing. A company-facts JSON file lists the values a company reported, such as revenue and net income, by period.

The pipeline: 4 files in, 4 rows out

The example is one CocoIndex app, SECFilingAnalytics, in one main.py. Each format has its own function that reads a file and turns it into text with metadata. Both functions then call one shared function, _index_text, that does the rest.

SECFilingAnalytics: 2 source formats, 1 Doris table. Counts are for the sample data: 4 files in, 4 chunks, 4 rows. process_filing reads data/filings, *.txt 10-K, 2 files. process_facts reads data/company_facts, *.json XBRL, 2 files. Both call _index_text, the shared path for both formats: _scrub_pii removes SSN, phone and email; RecursiveSplitter with 1000 / 200; embedder.embed makes 384 numbers; _extract_topics adds topic tags. declare_row writes to filing_chunks, an Apache Doris table with 4 rows, a vector index on embedding and an inverted index on text. One SQL query in search.py fuses a meaning rank and a keyword rank with RRF, k = 60.
Pipeline architectureEnlarge

Pipeline architecture

Zoom in to see details.

SECFilingAnalytics: 2 source formats, 1 Doris table. Counts are for the sample data: 4 files in, 4 chunks, 4 rows. process_filing reads data/filings, *.txt 10-K, 2 files. process_facts reads data/company_facts, *.json XBRL, 2 files. Both call _index_text, the shared path for both formats: _scrub_pii removes SSN, phone and email; RecursiveSplitter with 1000 / 200; embedder.embed makes 384 numbers; _extract_topics adds topic tags. declare_row writes to filing_chunks, an Apache Doris table with 4 rows, a vector index on embedding and an inverted index on text. One SQL query in search.py fuses a meaning rank and a keyword rank with RRF, k = 60.

CocoIndex is a data transformation framework with a Rust engine. You describe what the target (here, the Doris table) should contain as a function of the source, target_state = transformation(source_state), in plain Python. CocoIndex keeps the target in sync with incremental processing: when a file or the code changes, it redoes only the work that the change affects.

Apache Doris is an open-source analytical database. It is the target here because one Doris table can hold an inverted (full-text) index and a vector index side by side, so one table serves both kinds of search.

The app starts with these imports and the app object:

main.py
import json
import os
import re
import uuid
from collections.abc import AsyncIterator
from dataclasses import dataclass
from typing import Annotated

from numpy.typing import NDArray

import cocoindex as coco
from cocoindex.connectors import doris, localfs
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

app = coco.App(coco.AppConfig(name="SECFilingAnalytics"), app_main)

Step 1: read each format

app_main: one table, then one processor per file

app_main sets up (mounts) the Doris table, then walks each source folder and runs a processor once for every file that matches:

main.py
@coco.fn
async def app_main() -> None:
    table = await doris.mount_table_target(
        DORIS_DB,
        TABLE,
        await doris.TableSchema.from_class(FilingChunk, primary_key=["chunk_id"]),
        vector_indexes=[
            doris.VectorIndexDef(field_name="embedding", metric_type="l2_distance")
        ],
        inverted_indexes=[doris.InvertedIndexDef(field_name="text", parser="unicode")],
    )

    txt = localfs.walk_dir(
        localfs.FilePath(path="./data/filings"),
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.txt"]),
    )
    await coco.mount_each(process_filing, txt.items(), table)

    facts = localfs.walk_dir(
        localfs.FilePath(path="./data/company_facts"),
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.json"]),
    )
    await coco.mount_each(process_facts, facts.items(), table)
Source folderFile patternProcessorsource_type
./data/filings**/*.txtprocess_filing"filing"
./data/company_facts**/*.jsonprocess_facts"facts"

localfs.walk_dir lists the files, and coco.mount_each gives each file its own processing component. A component is a unit of work that CocoIndex runs, tracks and cleans up on its own. That is what lets it update one file without touching the others. The 2 indexes on the table come up again in Step 3.

process_filing: metadata from the file name

A 10-K file is named {CIK}_{date}_{form}.txt. process_filing splits the name to get the company, the filing date and the form type, then passes the text and the metadata to _index_text:

main.py
@coco.fn(memo=True)
async def process_filing(
    file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
    """10-K text filing: metadata from the {CIK}_{date}_{form}.txt filename."""
    name = file.file_path.path.name
    parts = name.rsplit(".", 1)[0].split("_")
    cik = parts[0] if parts else "unknown"
    filing_date = parts[1] if len(parts) > 1 else "2024-01-01"
    form_type = parts[2] if len(parts) > 2 else "10-K"
    await _index_text(
        await file.read_text(), "filing", name, cik, filing_date, form_type, table
    )
From file name to metadata in process_filing. The sample file name 0000320193_2025-11-01_10-K.txt goes through name.rsplit('.', 1)[0].split('_'). parts[0] becomes cik 0000320193, parts[1] becomes filing_date 2025-11-01, and parts[2] becomes form_type 10-K. A missing part falls back to a default: unknown, 2024-01-01 or 10-K. A string split, no LLM call.
File name to metadataEnlarge

File name to metadata

Zoom in to see details.

From file name to metadata in process_filing. The sample file name 0000320193_2025-11-01_10-K.txt goes through name.rsplit('.', 1)[0].split('_'). parts[0] becomes cik 0000320193, parts[1] becomes filing_date 2025-11-01, and parts[2] becomes form_type 10-K. A missing part falls back to a default: unknown, 2024-01-01 or 10-K. A string split, no LLM call.

The metadata comes from a string split. When you know the file-name format in advance, a parser like this skips an LLM call per document and returns the same result every time. For formats you do not know in advance, CocoIndex can extract fields with an LLM.

memo=True turns on memoization: CocoIndex stores the result, and on the next run it skips a file whose content and function code are unchanged.

process_facts: XBRL numbers as text

A company-facts file is JSON, not prose. To make it searchable, process_facts renders it as text with _company_facts_to_text and passes that text to the same _index_text:

main.py
@coco.fn(memo=True)
async def process_facts(
    file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
    """XBRL company-facts JSON: render to text, then index."""
    name = file.file_path.path.name
    content = await file.read_text()
    cik = name.replace("CIK", "").replace(".json", "")
    filing_date = (
        json.loads(content).get("filingDate", "2024-01-01") if content else "2024-01-01"
    )
    await _index_text(
        _company_facts_to_text(content), "facts", name, cik, filing_date, "FACTS", table
    )

This is what _company_facts_to_text makes of Apple’s sample file:

XBRL facts become text a search can read: _company_facts_to_text on Apple's sample file, real output. Left, 0000320193.json, us-gaap tag and val: Revenues 391035000000 for the period ending 2024-09-28 and 383285000000 ending 2023-09-30; NetIncomeLoss 93736000000 ending 2024-09-28; ResearchAndDevelopmentExpense 31370000000 ending 2024-09-28. Right, the text passed to _index_text: # Company Financial Facts: Apple Inc., CIK: 320193, ## US-GAAP Financial Metrics, ### Revenues, Recent values: 2024-09-28: $391.0B, 2023-09-30: $383.3B, ### Net Income, Recent values: 2024-09-28: $93.7B, ### R&D Expense, Recent values: 2024-09-28: $31.4B. Each metric gets its label and its newest values, at most 3, scaled: 391035000000 becomes $391.0B. The function looks for 6 metrics; Apple's sample file has 3 of them.
XBRL facts become textEnlarge

XBRL facts become text

Zoom in to see details.

XBRL facts become text a search can read: _company_facts_to_text on Apple's sample file, real output. Left, 0000320193.json, us-gaap tag and val: Revenues 391035000000 for the period ending 2024-09-28 and 383285000000 ending 2023-09-30; NetIncomeLoss 93736000000 ending 2024-09-28; ResearchAndDevelopmentExpense 31370000000 ending 2024-09-28. Right, the text passed to _index_text: # Company Financial Facts: Apple Inc., CIK: 320193, ## US-GAAP Financial Metrics, ### Revenues, Recent values: 2024-09-28: $391.0B, 2023-09-30: $383.3B, ### Net Income, Recent values: 2024-09-28: $93.7B, ### R&D Expense, Recent values: 2024-09-28: $31.4B. Each metric gets its label and its newest values, at most 3, scaled: 391035000000 becomes $391.0B. The function looks for 6 metrics; Apple's sample file has 3 of them.

To add a third format such as PDF, you write one more process_* function that turns the file into text and calls _index_text. Manuals to Structured Data shows a PDF path with docling.

Step 2: clean, chunk, tag and embed

Both formats end in _index_text. It removes personal data, splits the text into chunks, and declares one row per chunk:

main.py
_splitter = RecursiveSplitter()

async def _index_text(text, source_type, filename, cik, filing_date, form_type, table):
    embedder = coco.use_context(EMBEDDER)
    for chunk in _splitter.split(
        _scrub_pii(text), chunk_size=1000, chunk_overlap=200, language="markdown"
    ):
        table.declare_row(row=FilingChunk(
            chunk_id=_chunk_id(filename, chunk.start.char_offset, chunk.end.char_offset),
            source_type=source_type,
            doc_filename=filename,
            cik=cik,
            filing_date=filing_date,
            form_type=form_type,
            text=chunk.text,
            topics=_extract_topics(chunk.text),
            embedding=await embedder.embed(chunk.text),
        ))

_scrub_pii: remove personal data first

PII (personally identifiable information) includes Social Security numbers (SSNs), phone numbers and email addresses. A filing can contain them by accident, and a search index should not. _scrub_pii replaces each match of 3 regular expressions with a placeholder:

Personal data is removed before the text is split. The last line of Apple's sample 10-K. Before: Contact for investor relations: ir.contact@example.com, (555) 010-2003, with the email address and the phone number marked in red. After: Contact for investor relations: [EMAIL REDACTED], [PHONE REDACTED]. The file goes from 778 to 774 characters. 3 regular expressions run in this order: SSN, backslash b, 3 digits, dash, 2 digits, dash, 4 digits, to [SSN REDACTED]; phone, an optional parenthesized 3-digit area code, 3 digits and 4 digits with optional separators, to [PHONE REDACTED]; email, to [EMAIL REDACTED]. Why before splitting: a phone number cut in half by a chunk boundary no longer matches the pattern; scrubbing the whole document first means the splitter, the embedder and the index only see the redacted text.
PII scrubbingEnlarge

PII scrubbing

Zoom in to see details.

Personal data is removed before the text is split. The last line of Apple's sample 10-K. Before: Contact for investor relations: ir.contact@example.com, (555) 010-2003, with the email address and the phone number marked in red. After: Contact for investor relations: [EMAIL REDACTED], [PHONE REDACTED]. The file goes from 778 to 774 characters. 3 regular expressions run in this order: SSN, backslash b, 3 digits, dash, 2 digits, dash, 4 digits, to [SSN REDACTED]; phone, an optional parenthesized 3-digit area code, 3 digits and 4 digits with optional separators, to [PHONE REDACTED]; email, to [EMAIL REDACTED]. Why before splitting: a phone number cut in half by a chunk boundary no longer matches the pattern; scrubbing the whole document first means the splitter, the embedder and the index only see the redacted text.

The order matters. _scrub_pii runs on the whole document, before the split. If it ran on each chunk instead, a phone number cut in half by a chunk boundary would no longer match the pattern. Both halves would then reach the index.

Regular expressions catch only the patterns they describe. A name or a home address passes through, so treat this as a first filter. It does not catch every kind of personal data.

RecursiveSplitter: chunks of up to about 1,000 characters

A search should return the passage that answers the question, not a whole report. So the text is split into chunks, and each chunk becomes its own row with its own vector. RecursiveSplitter with language="markdown" picks split points from the structure: headings first, then paragraphs, then sentences. chunk_size=1000 and chunk_overlap=200 are in bytes; for English text that is about the same number of characters.

Chunks: up to about 1,000 characters, neighbors may share up to 200. RecursiveSplitter with chunk_size=1000 and chunk_overlap=200, both in bytes, 1 byte per character here. The 4 sample chunks, scrubbed and trimmed, on an axis from 0 to 1,000: Apple 10-K 773, Microsoft 10-K 504, Apple facts 250, Microsoft facts 189; each fits in 1 chunk. Below, a longer filing becomes several chunks of at most 1,000 each, with split points that follow the structure; neighboring chunks share at most 200 characters, outlined.
ChunksEnlarge

Chunks

Zoom in to see details.

Chunks: up to about 1,000 characters, neighbors may share up to 200. RecursiveSplitter with chunk_size=1000 and chunk_overlap=200, both in bytes, 1 byte per character here. The 4 sample chunks, scrubbed and trimmed, on an axis from 0 to 1,000: Apple 10-K 773, Microsoft 10-K 504, Apple facts 250, Microsoft facts 189; each fits in 1 chunk. Below, a longer filing becomes several chunks of at most 1,000 each, with split points that follow the structure; neighboring chunks share at most 200 characters, outlined.

The sample documents are short, so each one is a single chunk: 4 files give 4 chunks. A full 10-K is far longer and gives many chunks. Chunk size is a tradeoff: a large chunk matches many questions loosely, and a small one may lack the context to make sense on its own.

_extract_topics: tags from keyword lists

_extract_topics gives each chunk a list of topic tags, such as RISK:CYBER or TOPIC:AI. Each tag has a keyword list in _TOPIC_KEYWORDS, and a chunk gets the tag when its lower-cased text contains any keyword from the list:

main.py
def _extract_topics(text: str) -> list[str]:
    low = text.lower()
    return [t for t, kws in _TOPIC_KEYWORDS.items() if any(k in low for k in kws)]
Topic tags, a keyword list per label: _extract_topics on the 4 sample chunks. Each cell shows a matched keyword and how many more matched. Apple 10-K: RISK:CYBER ransomware +3, RISK:CLIMATE carbon +3, RISK:SUPPLY shortage +3, RISK:REGULATORY sec +1, TOPIC:AI machine learning +1. Microsoft 10-K: RISK:CYBER data breach +1, RISK:REGULATORY sec +3, TOPIC:AI artificial intelligence, TOPIC:CLOUD azure +1. Apple facts and Microsoft facts: TOPIC:FINANCIAL revenue +1. The 2 sec cells are outlined: matching is by substring on lower-cased text, so the keyword sec also matches inside cybersecurity, so a chunk that only mentions cybersecurity still gets RISK:REGULATORY. Here both 10-Ks have other regulatory words.
Topic tagsEnlarge

Topic tags

Zoom in to see details.

Topic tags, a keyword list per label: _extract_topics on the 4 sample chunks. Each cell shows a matched keyword and how many more matched. Apple 10-K: RISK:CYBER ransomware +3, RISK:CLIMATE carbon +3, RISK:SUPPLY shortage +3, RISK:REGULATORY sec +1, TOPIC:AI machine learning +1. Microsoft 10-K: RISK:CYBER data breach +1, RISK:REGULATORY sec +3, TOPIC:AI artificial intelligence, TOPIC:CLOUD azure +1. Apple facts and Microsoft facts: TOPIC:FINANCIAL revenue +1. The 2 sec cells are outlined: matching is by substring on lower-cased text, so the keyword sec also matches inside cybersecurity, so a chunk that only mentions cybersecurity still gets RISK:REGULATORY. Here both 10-Ks have other regulatory words.

The tags are stored as a JSON array, so you can filter on them in SQL, for example json_contains(topics, '"RISK:CYBER"'). Substring matching is simple and fast, but crude: the keyword sec also matches inside “cybersecurity”. On the sample data that does not change any tag, because both 10-Ks also contain “regulation”. For production tags, match whole words or use an LLM.

embedder.embed: 384 numbers per chunk

embedder.embed(chunk.text) turns the chunk into a vector of 384 numbers with the all-MiniLM-L6-v2 model. Texts with similar meaning get vectors that are close together, which is what the meaning side of the search uses. The embedder is created once when the app starts and shared through context, together with the Doris connection:

main.py
TABLE = "filing_chunks"
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"

DORIS_DB = coco.ContextKey[doris.ManagedConnection]("sec_doris")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)

@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
    builder.provide(DORIS_DB, doris.connect(doris.DorisConnectionConfig(...)))
    builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
    yield

DorisConnectionConfig(...) reads the Doris host, ports, user and database from environment variables (see .env.example in the example). detect_change=True tells CocoIndex to watch the embedder itself: if you change EMBED_MODEL, the next run embeds every chunk again.

FilingChunk: one typed row per chunk

The row is a plain dataclass. CocoIndex creates the Doris table from it, so you do not write CREATE TABLE. The embedding field is annotated with the embedder, so CocoIndex gets the vector size (384) from the model:

main.py
@dataclass
class FilingChunk:
    chunk_id: str            # primary key: uuid5 of (filename, chunk offsets)
    source_type: str         # "filing" | "facts"
    doc_filename: str
    cik: str
    filing_date: str
    form_type: str
    text: str
    topics: list[str]
    embedding: Annotated[NDArray, EMBEDDER]

This is the real row for Apple’s sample 10-K:

One chunk becomes one row. The FilingChunk dataclass is the table schema; this is the row for Apple's sample 10-K. Field, Doris column, value: chunk_id, VARCHAR(512), cc92a5a5-b7d1-5695-9c56-deeec5f91bad, the primary key; source_type, TEXT, filing; doc_filename, TEXT, 0000320193_2025-11-01_10-K.txt; cik, TEXT, 0000320193; filing_date, TEXT, 2025-11-01; form_type, TEXT, 10-K; text, TEXT, APPLE INC. FORM 10-K ANNUAL REPORT and so on, 773 characters; topics, JSON, RISK:CYBER, RISK:CLIMATE, RISK:SUPPLY and more, 5 tags; embedding, ARRAY<FLOAT>, -0.0073, 0.0154, 0.0623, 0.0320 and more, 384 numbers. chunk_id is a uuid5 of the file name and the chunk's start and end offsets, so the same chunk gets the same id on every run.
One rowEnlarge

One row

Zoom in to see details.

One chunk becomes one row. The FilingChunk dataclass is the table schema; this is the row for Apple's sample 10-K. Field, Doris column, value: chunk_id, VARCHAR(512), cc92a5a5-b7d1-5695-9c56-deeec5f91bad, the primary key; source_type, TEXT, filing; doc_filename, TEXT, 0000320193_2025-11-01_10-K.txt; cik, TEXT, 0000320193; filing_date, TEXT, 2025-11-01; form_type, TEXT, 10-K; text, TEXT, APPLE INC. FORM 10-K ANNUAL REPORT and so on, 773 characters; topics, JSON, RISK:CYBER, RISK:CLIMATE, RISK:SUPPLY and more, 5 tags; embedding, ARRAY<FLOAT>, -0.0073, 0.0154, 0.0623, 0.0320 and more, 384 numbers. chunk_id is a uuid5 of the file name and the chunk's start and end offsets, so the same chunk gets the same id on every run.

declare_row does not write to Doris at once. It declares a target state, a row that should exist. CocoIndex compares the declared rows with the ones from the last run and writes only the difference. Because chunk_id is computed from the file name and the chunk’s offsets, a re-run updates rows in place instead of adding copies.

Quick check: a filing has a phone number that starts at character 990, across the boundary of the first 1,000-character chunk. Can part of the number end up in the index?

No. _scrub_pii replaces the number in the whole document before RecursiveSplitter runs, so the splitter only sees [PHONE REDACTED]. If scrubbing ran per chunk, the chunk that ends mid-number would hold a fragment that the regular expression cannot match, and that fragment would be indexed.

Step 3: one table, two indexes

mount_table_target in app_main asks for 2 indexes on the same table:

  • An inverted index on text. It maps each word to the rows that contain it, so a keyword search does not read every row. parser="unicode" sets how the text is split into words. SQL queries use it with MATCH_ANY, which is true when the text contains any of the given words.
  • A vector index on embedding, of the ANN (approximate nearest neighbor) kind. It finds the vectors closest to the question’s vector without comparing it with every row. Closeness is l2_distance, the straight-line distance between 2 vectors: smaller means closer.
One table, 2 indexes, 2 ways to find a row, both on the same 4 rows. Left, the inverted index on text maps a word to the rows that contain it: cloud, Microsoft 10-K; risk, Apple 10-K and Microsoft 10-K; cybersecurity, Apple 10-K and Microsoft 10-K; revenues, Apple facts and Microsoft facts; azure, Microsoft 10-K. It finds exact words and is used by MATCH_ANY. Left, the words are tokenized in Python. Right, the vector (ANN) index on embedding orders rows by distance to the question's vector. For cloud computing and AI risk: Microsoft 10-K 0.795, Apple 10-K 1.031, Microsoft facts 1.329, Apple facts 1.361. l2_distance, smaller is closer; it finds meaning even with different words.
Two indexesEnlarge

Two indexes

Zoom in to see details.

One table, 2 indexes, 2 ways to find a row, both on the same 4 rows. Left, the inverted index on text maps a word to the rows that contain it: cloud, Microsoft 10-K; risk, Apple 10-K and Microsoft 10-K; cybersecurity, Apple 10-K and Microsoft 10-K; revenues, Apple facts and Microsoft facts; azure, Microsoft 10-K. It finds exact words and is used by MATCH_ANY. Left, the words are tokenized in Python. Right, the vector (ANN) index on embedding orders rows by distance to the question's vector. For cloud computing and AI risk: Microsoft 10-K 0.795, Apple 10-K 1.031, Microsoft facts 1.329, Apple facts 1.361. l2_distance, smaller is closer; it finds meaning even with different words.

The 2 indexes answer different questions. The inverted index finds exact words, such as the product name “azure”. A vector does not guarantee an exact word match; it only places “azure” near similar words. The vector index finds passages that say the same thing in other words. Hybrid search uses both kinds of match.

search.py keeps its SQL simple: it scores every row exactly, with l2_distance and with MATCH_ANY inside a CASE. On 4 rows that is instant. On a large table you would take the top k from each side instead: ORDER BY l2_distance_approximate(...) LIMIT k uses the vector index, and MATCH_ANY in a WHERE clause uses the inverted index.

Step 4: hybrid search with RRF

search.py ranks the chunks twice in one SQL query and combines the 2 ranks with Reciprocal Rank Fusion (RRF):

search.py
def search(query: str, source_type: str | None = None, limit: int = 5) -> None:
    vec = asyncio.run(_embedder.embed(query))
    vstr = "[" + ",".join(f"{x:.6f}" for x in vec) + "]"
    keywords = " ".join(re.findall(r"[A-Za-z]{3,}", query.lower()))
    where = f"source_type = '{source_type}'" if source_type else "1 = 1"

    sql = f"""
    WITH semantic AS (
        SELECT chunk_id, doc_filename, source_type, topics, text,
               ROW_NUMBER() OVER (ORDER BY l2_distance(embedding, {vstr})) AS rk
        FROM filing_chunks WHERE {where}
    ),
    lexical AS (
        SELECT chunk_id,
               ROW_NUMBER() OVER (ORDER BY CASE WHEN text MATCH_ANY '{keywords}'
                                  THEN 0 ELSE 1 END) AS rk
        FROM filing_chunks WHERE {where}
    )
    SELECT s.doc_filename, s.source_type, s.topics, s.text,
           1.0/(60 + s.rk) + 1.0/(60 + l.rk) AS rrf
    FROM semantic s JOIN lexical l USING (chunk_id)
    ORDER BY rrf DESC LIMIT {limit}
    """

The query has 3 parts:

  1. semantic gives each chunk its meaning rank. search.py first embeds the question in Python with the same model; semantic then numbers the chunks by l2_distance, closest first.
  2. lexical gives each chunk its keyword rank: it numbers the chunks again, the ones that match any keyword first.
  3. The last SELECT gives each chunk the score 1/(60 + meaning rank) + 1/(60 + keyword rank) and returns the highest scores.

RRF uses only the ranks, not the raw scores, so a distance and a keyword match never need to be on the same scale. The constant 60 (often written k) keeps the gap between rank 1 and rank 2 small, so a top rank in one list alone rarely decides the result. A chunk ranked 1st in both lists scores 1/61 + 1/61 = 0.0328; a chunk ranked 1st by meaning but 100th by keywords scores 1/61 + 1/160 = 0.0226.

This is the ranking for “cloud computing and AI risk” on the sample data:

Hybrid search: 2 rankings, 1 score, for cloud computing and AI risk on the sample data, with ranks computed with the same model and rules as search.py. Keywords: cloud, computing, and, risk; AI is dropped because the regex keeps words of 3 or more letters. Microsoft 10-K: distance 0.795, meaning rank 1, keywords matched cloud, computing, risk, keyword rank 1, RRF score 0.0328, the top hit. Apple 10-K: distance 1.031, meaning rank 2, matched risk, keyword rank 2, score 0.0323. Microsoft facts: 1.329, meaning rank 3, no keyword matched, keyword rank 3, score 0.0317. Apple facts: 1.361, meaning rank 4, none, keyword rank 4, score 0.0312. RRF score = 1 / (60 + meaning rank) + 1 / (60 + keyword rank); Microsoft gets 1/61 + 1/61 = 0.0328. The keyword side is only match or no match, so the 2 10-Ks tie; Doris numbers tied rows in any order, and if Apple gets keyword rank 1, both 10-Ks score 0.0325; the 2 facts rows tie too, and swapped they both score 0.0315.
Hybrid search with RRFEnlarge

Hybrid search with RRF

Zoom in to see details.

Hybrid search: 2 rankings, 1 score, for cloud computing and AI risk on the sample data, with ranks computed with the same model and rules as search.py. Keywords: cloud, computing, and, risk; AI is dropped because the regex keeps words of 3 or more letters. Microsoft 10-K: distance 0.795, meaning rank 1, keywords matched cloud, computing, risk, keyword rank 1, RRF score 0.0328, the top hit. Apple 10-K: distance 1.031, meaning rank 2, matched risk, keyword rank 2, score 0.0323. Microsoft facts: 1.329, meaning rank 3, no keyword matched, keyword rank 3, score 0.0317. Apple facts: 1.361, meaning rank 4, none, keyword rank 4, score 0.0312. RRF score = 1 / (60 + meaning rank) + 1 / (60 + keyword rank); Microsoft gets 1/61 + 1/61 = 0.0328. The keyword side is only match or no match, so the 2 10-Ks tie; Doris numbers tied rows in any order, and if Apple gets keyword rank 1, both 10-Ks score 0.0325; the 2 facts rows tie too, and swapped they both score 0.0315.

Two details of search.py show up here. The keyword regex keeps only words of 3 or more letters, so “AI” is not a keyword; the meaning side still reads the whole question. And the keyword side has only 2 values, match or no match, so all matching chunks tie. ROW_NUMBER() numbers tied rows in an order that Doris picks, so the 2 10-Ks can swap their keyword ranks between runs. The 2 facts rows tie the same way; swapped, both score 0.0315. Doris may also ignore “and” as a stopword, which does not change the ranks here. With a full set of filings, a keyword relevance score (or RANK(), which gives tied rows the same rank) would make the keyword side more useful.

The playground below runs the same rules on the 4 sample chunks. The distances are fixed at the values in the figure; the keyword side is computed from the chunk texts. Turn keywords on and off, move k, and swap the tie order:

RRF playground: “cloud computing and AI risk”

Sample data; distances from all-MiniLM-L6-v2
Keywords sent to MATCH_ANY
andDoris stopwordAIdropped by the regex
Try words that are not in the question
search.py uses 60
ROW_NUMBER() numbers tied rows in an order Doris picks
chunkdistancemeaning rankkeywords matchedkeyword rankRRF score
Microsoft 10-K0000789019_2025-10-15_10-K.txt0.7951–––
Apple 10-K0000320193_2025-11-01_10-K.txt1.0312–––
Microsoft facts0000789019.json1.3293–––
Apple facts0000320193.json1.3614–––

With the question’s own keywords, k changes the scores but not the order: the Microsoft 10-K matches every word the Apple 10-K matches, so the 2 lists agree. To see k change the order, keep only apple on and swap the tie order. The Apple 10-K is 2nd in both lists. The Microsoft 10-K is 1st by meaning and 4th by keywords, and the Apple facts row is the reverse. At k = 60 the Apple 10-K comes first (0.0323 against 0.0320). At k = 1 the step from rank 1 to rank 2 is large, and both rows with a 1st place pass it (0.7000 against 0.6667).

Keep only revenues on, and the facts rows take keyword ranks 1 and 2. Each one then ties with a 10-K: RRF sees only ranks, and 1st plus 3rd scores the same as 3rd plus 1st.

To search one format only, call search(question, source_type="facts") (or "filing").

Quick check: chunk A ranks 1st by meaning and 100th by keywords. Chunk B ranks 2nd in both. Which one does RRF put first?

Chunk B. A scores 1/61 + 1/160 = 0.0226. B scores 1/62 + 1/62 = 0.0323. Here, A’s 1st place by meaning does not make up for its 100th place by keywords.

Run it

You need Python 3.11+ and Docker with Docker Compose. Compose runs Apache Doris, which must be 4.0 or later for the vector index. Clone the CocoIndex repo and work in examples/sec_edgar_analytics:

bash
cd examples/sec_edgar_analytics

# 1. Start Doris: one frontend (FE) and one backend (BE)
docker compose up -d fe be

# 2. Write the sample data, configure, and install
python download.py           # writes data/filings/*.txt + data/company_facts/*.json
cp .env.example .env         # Doris host and ports
pip install -e .

# 3. Build the index
cocoindex update main

In a Doris cluster, the FE (frontend) receives queries and keeps metadata, and the BE (backend) stores data and runs queries. Wait until both containers have started before step 3.

The first cocoindex update main loads 4 rows into filing_chunks, 1 per sample file. It also creates the vector index idx_vec_embedding and the inverted index idx_inv_text. The tags follow each filing’s wording: Apple gets RISK:CYBER, RISK:CLIMATE, RISK:SUPPLY, RISK:REGULATORY, TOPIC:AI, and Microsoft RISK:CYBER, RISK:REGULATORY, TOPIC:AI, TOPIC:CLOUD.

Then search:

bash
python search.py "cloud computing and AI risk"

search.py prints each hit with its RRF score, its file and the first 160 characters of the chunk. On the sample data the 10-Ks come first:

text
Hybrid search: "cloud computing and AI risk"

[0.0328] 0000789019_2025-10-15_10-K.txt (filing)  topics=["RISK:CYBER","RISK:REGULATORY","TOPIC:AI","TOPIC:CLOUD"]
    MICROSOFT CORPORATION FORM 10-K ANNUAL REPORT

Item 1A. Risk Factors

Our cloud computing business (Azure) faces intense competition. Outages or a
data breach i

[0.0323] 0000320193_2025-11-01_10-K.txt (filing)  topics=["RISK:CYBER","RISK:CLIMATE","RISK:SUPPLY","RISK:REGULATORY","TOPIC:AI"]
    APPLE INC. FORM 10-K ANNUAL REPORT

Item 1A. Risk Factors

Cybersecurity threats are a material risk. A data breach or ransomware attack
on our systems could di

We computed these scores without Doris, using the tie order in the figure above. If Doris breaks the keyword tie the other way, both lines show 0.0325. Doris computes 1.0/(60 + rk) in decimal arithmetic, so the last digit can differ. The 2 company-facts rows follow below the 10-Ks.

A re-run redoes only what changed

Run cocoindex update main again, and CocoIndex does only the work that a change needs:

What a re-run does, in 6 cases, with what CocoIndex does, the chunks embedded and the rows in Doris. Nothing changed, dashed gray: both processors reuse the last run's results, 0 embedded, rows unchanged. Microsoft 10-K edited, periwinkle: process_filing re-runs for that file, its new text is embedded, its rows replaced. A new filing added, green: a new component runs for it, its chunks embedded, its rows added. A file deleted, red: its component is removed, 0 embedded, its rows deleted. EMBED_MODEL changed, periwinkle: detect_change=True invalidates the memos, all 4 embedded, every vector updated. _TOPIC_KEYWORDS edited, dashed red: nothing re-runs, because plain helpers are not tracked, 0 embedded, the tags stay old (stale). After editing a helper such as _TOPIC_KEYWORDS, _scrub_pii or the chunk size, run cocoindex update main --full-reprocess.
What a re-run doesEnlarge

What a re-run does

Zoom in to see details.

What a re-run does, in 6 cases, with what CocoIndex does, the chunks embedded and the rows in Doris. Nothing changed, dashed gray: both processors reuse the last run's results, 0 embedded, rows unchanged. Microsoft 10-K edited, periwinkle: process_filing re-runs for that file, its new text is embedded, its rows replaced. A new filing added, green: a new component runs for it, its chunks embedded, its rows added. A file deleted, red: its component is removed, 0 embedded, its rows deleted. EMBED_MODEL changed, periwinkle: detect_change=True invalidates the memos, all 4 embedded, every vector updated. _TOPIC_KEYWORDS edited, dashed red: nothing re-runs, because plain helpers are not tracked, 0 embedded, the tags stay old (stale). After editing a helper such as _TOPIC_KEYWORDS, _scrub_pii or the chunk size, run cocoindex update main --full-reprocess.

memo=True on the 2 processors skips files that did not change, and embed() is memoized too, so only new chunk text goes to the model (unless EMBED_MODEL changes). A deleted file’s component goes away, and its rows are deleted with it.

The last row in the figure needs a manual step. CocoIndex tracks the code of functions decorated with @coco.fn. _index_text, _scrub_pii, _extract_topics and _TOPIC_KEYWORDS are plain Python, so an edit to them does not invalidate the stored results. After you change one of them, run cocoindex update main --full-reprocess, which skips all memoized results and processes every file again (including the embeddings).

Quick check: you edit the Microsoft 10-K file and run cocoindex update main again. Which files does the pipeline process?

Only the edited file. process_filing and process_facts use memo=True, so files whose content and code are unchanged are skipped. chunk_id comes from the file name and chunk offsets, so the re-run replaces that file’s rows instead of adding copies.

Query Doris directly

The table is plain SQL, so you can also query it over the MySQL protocol:

bash
mysql -h localhost -P 9030 -u root

Filter by topic tag:

sql
-- Chunks tagged with cybersecurity risk
SELECT doc_filename, text, topics
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"');

-- Chunks with either of 2 tags
SELECT doc_filename, text
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"')
   OR json_contains(topics, '"RISK:CLIMATE"');

The 3 closest chunks per company, for a question vector:

sql
WITH scored AS (
    SELECT cik, doc_filename, text,
           l2_distance(embedding, [...]) AS score   -- [...] is the question's 384-number vector
    FROM sec_analytics.filing_chunks
    WHERE cik IN ('0000320193', '0000789019')
),
ranked AS (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY cik ORDER BY score) AS rk
    FROM scored
)
SELECT * FROM ranked WHERE rk <= 3;

Keyword matches by filing year:

sql
SELECT LEFT(filing_date, 4) AS filing_year,
       COUNT(DISTINCT cik) AS companies,
       COUNT(*) AS chunks
FROM sec_analytics.filing_chunks
WHERE text MATCH_ANY 'cybersecurity risk'
GROUP BY filing_year
ORDER BY filing_year DESC;

What to try next

  • Index your own filing. Save a 10-K as {CIK}_{date}_{form}.txt in data/filings, for example 0000320193_2025-11-01_10-K.txt, and run cocoindex update main. Only the new file is processed.
  • Add a topic tag. Add an entry to _TOPIC_KEYWORDS in main.py, then run cocoindex update main --full-reprocess so every chunk is tagged again.
  • Preview before you write. cocoindex update main --preview computes the changes to the target and prints them without applying them.
  • Add a third format. Write a process_pdf that turns a PDF into text and calls _index_text, and mount it with coco.mount_each like the other 2.
  • Filter, then search. Add a json_contains(topics, ...) condition to the where in search.py, so the hybrid search only ranks chunks with a given tag.

The same pattern fits other document collections, such as contracts or support tickets: several formats into one table, personal data removed first, tags for filters, and hybrid search. The full source code is in the CocoIndex v1 repo, with the sample-data generator and the search script.

Frequently asked questions

How do I build a search index over SEC EDGAR filings?

This example builds a CocoIndex v1 pipeline that reads 2 SEC formats (10-K filings as text and XBRL company-facts JSON), removes personal data, splits the text into chunks, tags and embeds each chunk, and loads one row per chunk into an Apache Doris table with a full-text index and a vector index. Both formats go through one shared function, _index_text.

See The pipeline: 4 files in, 4 rows out

How do you combine several document formats into one search index?

Each format gets its own processor that turns a file into text plus metadata: process_filing parses the 10-K file name, and process_facts renders the XBRL JSON as text. Both call the same _index_text, which declares rows into one Doris table with one schema. A third format, such as PDF, is one more processor that calls _index_text.

See Step 1: read each format

Why scrub PII before chunking instead of after?

_scrub_pii runs on the whole document before the split. If it ran on each chunk, a phone number or SSN cut in half by a chunk boundary would no longer match the regular expression, and both halves would reach the index.

See Remove personal data first

What chunk size does the SEC EDGAR pipeline use?

RecursiveSplitter with chunk_size=1000 and chunk_overlap=200, in bytes (about the same number of characters for English text). With language="markdown" it splits at headings first, then paragraphs, then sentences. Each of the 4 short sample documents fits in 1 chunk.

See Chunks of up to about 1,000 characters

How does hybrid search with Reciprocal Rank Fusion work here?

search.py ranks every chunk by vector distance (l2_distance) and by keyword match (MATCH_ANY), then gives each chunk the score 1/(60 + rank) summed over both lists. A chunk ranked 1st in both scores 1/61 + 1/61 = 0.0328; 1st by meaning but 100th by keywords scores 1/61 + 1/160 = 0.0226.

See Step 4: hybrid search with RRF

When should I use a deterministic parser instead of an LLM for extraction?

When you know the file format in advance. Here the 10-K metadata (CIK, filing date, form type) comes from splitting the file name {CIK}_{date}_{form}.txt, which skips an LLM call per document and returns the same result every time. For formats you do not know in advance, CocoIndex can extract fields with an LLM.

See Metadata from the file name

Why use Apache Doris as the target for hybrid search?

One Doris table can hold an inverted index on text for keyword search and an ANN vector index on embedding for search by meaning, so a single SQL query can rank by both. Doris 4.0 or later is needed for the vector index.

See Step 3: one table, two indexes

Build fresh context for your agents.

CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.