Every public company in the United States files reports with the SEC (the U.S. Securities and Exchange Commission). The reports are public on EDGAR, the SEC’s filing system. Say you want to ask questions across them: which companies name cybersecurity as a risk, what a company reported as revenue, or how often a risk comes up by filing year.
Two things make this harder than a text search. First, one company’s data comes in different formats: its risks are prose in an annual report, and its financial numbers are structured data. Second, a question such as “cloud computing and AI risk” needs both kinds of search. Keyword search finds the exact words. Search by meaning finds passages that say the same thing in other words.
An agent or a RAG pipeline that retrieves from filings needs both. A question that names a company, a CIK or a metric such as “net income” must hit those exact words; a vector places them near similar words but does not guarantee a match. A question about “risks of hosted services” must also find a 10-K that says “a data breach in our cloud services”. The same company’s numbers are in JSON, not prose, so both formats have to land in one searchable form.
This tutorial walks through the SEC EDGAR Financial Analytics example, a CocoIndex v1 pipeline that handles both. It reads the 2 formats, removes personal data and splits the text into chunks. It tags each chunk with topics and turns it into a vector (a list of numbers that captures its meaning). Everything lands in one Apache Doris table that has 2 indexes: an inverted index for keywords and a vector index for meaning. A search then ranks every chunk both ways and combines the 2 rankings into one score.
The numbers in the figures come from the example’s own code and sample data: download.py writes 2 short 10-K excerpts and 2 company-facts files, for Apple and Microsoft. We ran the example’s functions on them and made the vectors with the same model, all-MiniLM-L6-v2. We did not run Doris; the search ranks follow the rules in search.py. The code is in the CocoIndex v1 repo.
A few terms used below:
- 10-K: a company’s annual report to the SEC. Its “Item 1A. Risk Factors” section lists what could go wrong. The 10-Q is the quarterly report.
- CIK (Central Index Key): the number the SEC assigns to each filer, such as
0000320193for Apple. - XBRL (eXtensible Business Reporting Language): the tagged format for the numbers in a filing. A company-facts JSON file lists the values a company reported, such as revenue and net income, by period.
The pipeline: 4 files in, 4 rows out
The example is one CocoIndex app, SECFilingAnalytics, in one main.py. Each format has its own function that reads a file and turns it into text with metadata. Both functions then call one shared function, _index_text, that does the rest.
CocoIndex is a data transformation framework with a Rust engine. You describe what the target (here, the Doris table) should contain as a function of the source, target_state = transformation(source_state), in plain Python. CocoIndex keeps the target in sync with incremental processing: when a file or the code changes, it redoes only the work that the change affects.
Apache Doris is an open-source analytical database. It is the target here because one Doris table can hold an inverted (full-text) index and a vector index side by side, so one table serves both kinds of search.
The app starts with these imports and the app object:
import json
import os
import re
import uuid
from collections.abc import AsyncIterator
from dataclasses import dataclass
from typing import Annotated
from numpy.typing import NDArray
import cocoindex as coco
from cocoindex.connectors import doris, localfs
from cocoindex.ops.text import RecursiveSplitter
from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
app = coco.App(coco.AppConfig(name="SECFilingAnalytics"), app_main)
Step 1: read each format
app_main: one table, then one processor per file
app_main sets up (mounts) the Doris table, then walks each source folder and runs a processor once for every file that matches:
@coco.fn
async def app_main() -> None:
table = await doris.mount_table_target(
DORIS_DB,
TABLE,
await doris.TableSchema.from_class(FilingChunk, primary_key=["chunk_id"]),
vector_indexes=[
doris.VectorIndexDef(field_name="embedding", metric_type="l2_distance")
],
inverted_indexes=[doris.InvertedIndexDef(field_name="text", parser="unicode")],
)
txt = localfs.walk_dir(
localfs.FilePath(path="./data/filings"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.txt"]),
)
await coco.mount_each(process_filing, txt.items(), table)
facts = localfs.walk_dir(
localfs.FilePath(path="./data/company_facts"),
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.json"]),
)
await coco.mount_each(process_facts, facts.items(), table)
| Source folder | File pattern | Processor | source_type |
|---|---|---|---|
./data/filings | **/*.txt | process_filing | "filing" |
./data/company_facts | **/*.json | process_facts | "facts" |
localfs.walk_dir lists the files, and coco.mount_each gives each file its own processing component. A component is a unit of work that CocoIndex runs, tracks and cleans up on its own. That is what lets it update one file without touching the others. The 2 indexes on the table come up again in Step 3.
process_filing: metadata from the file name
A 10-K file is named {CIK}_{date}_{form}.txt. process_filing splits the name to get the company, the filing date and the form type, then passes the text and the metadata to _index_text:
@coco.fn(memo=True)
async def process_filing(
file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
"""10-K text filing: metadata from the {CIK}_{date}_{form}.txt filename."""
name = file.file_path.path.name
parts = name.rsplit(".", 1)[0].split("_")
cik = parts[0] if parts else "unknown"
filing_date = parts[1] if len(parts) > 1 else "2024-01-01"
form_type = parts[2] if len(parts) > 2 else "10-K"
await _index_text(
await file.read_text(), "filing", name, cik, filing_date, form_type, table
)
The metadata comes from a string split. When you know the file-name format in advance, a parser like this skips an LLM call per document and returns the same result every time. For formats you do not know in advance, CocoIndex can extract fields with an LLM.
memo=True turns on memoization: CocoIndex stores the result, and on the next run it skips a file whose content and function code are unchanged.
process_facts: XBRL numbers as text
A company-facts file is JSON, not prose. To make it searchable, process_facts renders it as text with _company_facts_to_text and passes that text to the same _index_text:
@coco.fn(memo=True)
async def process_facts(
file: FileLike, table: doris.DorisTableTarget[FilingChunk]
) -> None:
"""XBRL company-facts JSON: render to text, then index."""
name = file.file_path.path.name
content = await file.read_text()
cik = name.replace("CIK", "").replace(".json", "")
filing_date = (
json.loads(content).get("filingDate", "2024-01-01") if content else "2024-01-01"
)
await _index_text(
_company_facts_to_text(content), "facts", name, cik, filing_date, "FACTS", table
)
This is what _company_facts_to_text makes of Apple’s sample file:
To add a third format such as PDF, you write one more process_* function that turns the file into text and calls _index_text. Manuals to Structured Data shows a PDF path with docling.
Step 2: clean, chunk, tag and embed
Both formats end in _index_text. It removes personal data, splits the text into chunks, and declares one row per chunk:
_splitter = RecursiveSplitter()
async def _index_text(text, source_type, filename, cik, filing_date, form_type, table):
embedder = coco.use_context(EMBEDDER)
for chunk in _splitter.split(
_scrub_pii(text), chunk_size=1000, chunk_overlap=200, language="markdown"
):
table.declare_row(row=FilingChunk(
chunk_id=_chunk_id(filename, chunk.start.char_offset, chunk.end.char_offset),
source_type=source_type,
doc_filename=filename,
cik=cik,
filing_date=filing_date,
form_type=form_type,
text=chunk.text,
topics=_extract_topics(chunk.text),
embedding=await embedder.embed(chunk.text),
))
_scrub_pii: remove personal data first
PII (personally identifiable information) includes Social Security numbers (SSNs), phone numbers and email addresses. A filing can contain them by accident, and a search index should not. _scrub_pii replaces each match of 3 regular expressions with a placeholder:
The order matters. _scrub_pii runs on the whole document, before the split. If it ran on each chunk instead, a phone number cut in half by a chunk boundary would no longer match the pattern. Both halves would then reach the index.
Regular expressions catch only the patterns they describe. A name or a home address passes through, so treat this as a first filter. It does not catch every kind of personal data.
RecursiveSplitter: chunks of up to about 1,000 characters
A search should return the passage that answers the question, not a whole report. So the text is split into chunks, and each chunk becomes its own row with its own vector. RecursiveSplitter with language="markdown" picks split points from the structure: headings first, then paragraphs, then sentences. chunk_size=1000 and chunk_overlap=200 are in bytes; for English text that is about the same number of characters.
The sample documents are short, so each one is a single chunk: 4 files give 4 chunks. A full 10-K is far longer and gives many chunks. Chunk size is a tradeoff: a large chunk matches many questions loosely, and a small one may lack the context to make sense on its own.
_extract_topics: tags from keyword lists
_extract_topics gives each chunk a list of topic tags, such as RISK:CYBER or TOPIC:AI. Each tag has a keyword list in _TOPIC_KEYWORDS, and a chunk gets the tag when its lower-cased text contains any keyword from the list:
def _extract_topics(text: str) -> list[str]:
low = text.lower()
return [t for t, kws in _TOPIC_KEYWORDS.items() if any(k in low for k in kws)]
The tags are stored as a JSON array, so you can filter on them in SQL, for example json_contains(topics, '"RISK:CYBER"'). Substring matching is simple and fast, but crude: the keyword sec also matches inside “cybersecurity”. On the sample data that does not change any tag, because both 10-Ks also contain “regulation”. For production tags, match whole words or use an LLM.
embedder.embed: 384 numbers per chunk
embedder.embed(chunk.text) turns the chunk into a vector of 384 numbers with the all-MiniLM-L6-v2 model. Texts with similar meaning get vectors that are close together, which is what the meaning side of the search uses. The embedder is created once when the app starts and shared through context, together with the Doris connection:
TABLE = "filing_chunks"
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
DORIS_DB = coco.ContextKey[doris.ManagedConnection]("sec_doris")
EMBEDDER = coco.ContextKey[SentenceTransformerEmbedder]("embedder", detect_change=True)
@coco.lifespan
async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:
builder.provide(DORIS_DB, doris.connect(doris.DorisConnectionConfig(...)))
builder.provide(EMBEDDER, SentenceTransformerEmbedder(EMBED_MODEL))
yield
DorisConnectionConfig(...) reads the Doris host, ports, user and database from environment variables (see .env.example in the example). detect_change=True tells CocoIndex to watch the embedder itself: if you change EMBED_MODEL, the next run embeds every chunk again.
FilingChunk: one typed row per chunk
The row is a plain dataclass. CocoIndex creates the Doris table from it, so you do not write CREATE TABLE. The embedding field is annotated with the embedder, so CocoIndex gets the vector size (384) from the model:
@dataclass
class FilingChunk:
chunk_id: str # primary key: uuid5 of (filename, chunk offsets)
source_type: str # "filing" | "facts"
doc_filename: str
cik: str
filing_date: str
form_type: str
text: str
topics: list[str]
embedding: Annotated[NDArray, EMBEDDER]
This is the real row for Apple’s sample 10-K:
declare_row does not write to Doris at once. It declares a target state, a row that should exist. CocoIndex compares the declared rows with the ones from the last run and writes only the difference. Because chunk_id is computed from the file name and the chunk’s offsets, a re-run updates rows in place instead of adding copies.
Quick check: a filing has a phone number that starts at character 990, across the boundary of the first 1,000-character chunk. Can part of the number end up in the index?
No. _scrub_pii replaces the number in the whole document before RecursiveSplitter runs, so the splitter only sees [PHONE REDACTED]. If scrubbing ran per chunk, the chunk that ends mid-number would hold a fragment that the regular expression cannot match, and that fragment would be indexed.
Step 3: one table, two indexes
mount_table_target in app_main asks for 2 indexes on the same table:
- An inverted index on
text. It maps each word to the rows that contain it, so a keyword search does not read every row.parser="unicode"sets how the text is split into words. SQL queries use it withMATCH_ANY, which is true when the text contains any of the given words. - A vector index on
embedding, of the ANN (approximate nearest neighbor) kind. It finds the vectors closest to the question’s vector without comparing it with every row. Closeness isl2_distance, the straight-line distance between 2 vectors: smaller means closer.
The 2 indexes answer different questions. The inverted index finds exact words, such as the product name “azure”. A vector does not guarantee an exact word match; it only places “azure” near similar words. The vector index finds passages that say the same thing in other words. Hybrid search uses both kinds of match.
search.py keeps its SQL simple: it scores every row exactly, with l2_distance and with MATCH_ANY inside a CASE. On 4 rows that is instant. On a large table you would take the top k from each side instead: ORDER BY l2_distance_approximate(...) LIMIT k uses the vector index, and MATCH_ANY in a WHERE clause uses the inverted index.
Step 4: hybrid search with RRF
search.py ranks the chunks twice in one SQL query and combines the 2 ranks with Reciprocal Rank Fusion (RRF):
def search(query: str, source_type: str | None = None, limit: int = 5) -> None:
vec = asyncio.run(_embedder.embed(query))
vstr = "[" + ",".join(f"{x:.6f}" for x in vec) + "]"
keywords = " ".join(re.findall(r"[A-Za-z]{3,}", query.lower()))
where = f"source_type = '{source_type}'" if source_type else "1 = 1"
sql = f"""
WITH semantic AS (
SELECT chunk_id, doc_filename, source_type, topics, text,
ROW_NUMBER() OVER (ORDER BY l2_distance(embedding, {vstr})) AS rk
FROM filing_chunks WHERE {where}
),
lexical AS (
SELECT chunk_id,
ROW_NUMBER() OVER (ORDER BY CASE WHEN text MATCH_ANY '{keywords}'
THEN 0 ELSE 1 END) AS rk
FROM filing_chunks WHERE {where}
)
SELECT s.doc_filename, s.source_type, s.topics, s.text,
1.0/(60 + s.rk) + 1.0/(60 + l.rk) AS rrf
FROM semantic s JOIN lexical l USING (chunk_id)
ORDER BY rrf DESC LIMIT {limit}
"""
The query has 3 parts:
semanticgives each chunk its meaning rank.search.pyfirst embeds the question in Python with the same model;semanticthen numbers the chunks byl2_distance, closest first.lexicalgives each chunk its keyword rank: it numbers the chunks again, the ones that match any keyword first.- The last
SELECTgives each chunk the score1/(60 + meaning rank) + 1/(60 + keyword rank)and returns the highest scores.
RRF uses only the ranks, not the raw scores, so a distance and a keyword match never need to be on the same scale. The constant 60 (often written k) keeps the gap between rank 1 and rank 2 small, so a top rank in one list alone rarely decides the result. A chunk ranked 1st in both lists scores 1/61 + 1/61 = 0.0328; a chunk ranked 1st by meaning but 100th by keywords scores 1/61 + 1/160 = 0.0226.
This is the ranking for “cloud computing and AI risk” on the sample data:
Two details of search.py show up here. The keyword regex keeps only words of 3 or more letters, so “AI” is not a keyword; the meaning side still reads the whole question. And the keyword side has only 2 values, match or no match, so all matching chunks tie. ROW_NUMBER() numbers tied rows in an order that Doris picks, so the 2 10-Ks can swap their keyword ranks between runs. The 2 facts rows tie the same way; swapped, both score 0.0315. Doris may also ignore “and” as a stopword, which does not change the ranks here. With a full set of filings, a keyword relevance score (or RANK(), which gives tied rows the same rank) would make the keyword side more useful.
The playground below runs the same rules on the 4 sample chunks. The distances are fixed at the values in the figure; the keyword side is computed from the chunk texts. Turn keywords on and off, move k, and swap the tie order:
RRF playground: “cloud computing and AI risk”
Sample data; distances from all-MiniLM-L6-v2MATCH_ANYsearch.py uses 60 ROW_NUMBER() numbers tied rows in an order Doris picksWith the question’s own keywords, k changes the scores but not the order: the Microsoft 10-K matches every word the Apple 10-K matches, so the 2 lists agree. To see k change the order, keep only apple on and swap the tie order. The Apple 10-K is 2nd in both lists. The Microsoft 10-K is 1st by meaning and 4th by keywords, and the Apple facts row is the reverse. At k = 60 the Apple 10-K comes first (0.0323 against 0.0320). At k = 1 the step from rank 1 to rank 2 is large, and both rows with a 1st place pass it (0.7000 against 0.6667).
Keep only revenues on, and the facts rows take keyword ranks 1 and 2. Each one then ties with a 10-K: RRF sees only ranks, and 1st plus 3rd scores the same as 3rd plus 1st.
To search one format only, call search(question, source_type="facts") (or "filing").
Quick check: chunk A ranks 1st by meaning and 100th by keywords. Chunk B ranks 2nd in both. Which one does RRF put first?
Chunk B. A scores 1/61 + 1/160 = 0.0226. B scores 1/62 + 1/62 = 0.0323. Here, A’s 1st place by meaning does not make up for its 100th place by keywords.
Run it
You need Python 3.11+ and Docker with Docker Compose. Compose runs Apache Doris, which must be 4.0 or later for the vector index. Clone the CocoIndex repo and work in examples/sec_edgar_analytics:
cd examples/sec_edgar_analytics
# 1. Start Doris: one frontend (FE) and one backend (BE)
docker compose up -d fe be
# 2. Write the sample data, configure, and install
python download.py # writes data/filings/*.txt + data/company_facts/*.json
cp .env.example .env # Doris host and ports
pip install -e .
# 3. Build the index
cocoindex update main
In a Doris cluster, the FE (frontend) receives queries and keeps metadata, and the BE (backend) stores data and runs queries. Wait until both containers have started before step 3.
The first cocoindex update main loads 4 rows into filing_chunks, 1 per sample file. It also creates the vector index idx_vec_embedding and the inverted index idx_inv_text. The tags follow each filing’s wording: Apple gets RISK:CYBER, RISK:CLIMATE, RISK:SUPPLY, RISK:REGULATORY, TOPIC:AI, and Microsoft RISK:CYBER, RISK:REGULATORY, TOPIC:AI, TOPIC:CLOUD.
Then search:
python search.py "cloud computing and AI risk"
search.py prints each hit with its RRF score, its file and the first 160 characters of the chunk. On the sample data the 10-Ks come first:
Hybrid search: "cloud computing and AI risk"
[0.0328] 0000789019_2025-10-15_10-K.txt (filing) topics=["RISK:CYBER","RISK:REGULATORY","TOPIC:AI","TOPIC:CLOUD"]
MICROSOFT CORPORATION FORM 10-K ANNUAL REPORT
Item 1A. Risk Factors
Our cloud computing business (Azure) faces intense competition. Outages or a
data breach i
[0.0323] 0000320193_2025-11-01_10-K.txt (filing) topics=["RISK:CYBER","RISK:CLIMATE","RISK:SUPPLY","RISK:REGULATORY","TOPIC:AI"]
APPLE INC. FORM 10-K ANNUAL REPORT
Item 1A. Risk Factors
Cybersecurity threats are a material risk. A data breach or ransomware attack
on our systems could di
We computed these scores without Doris, using the tie order in the figure above. If Doris breaks the keyword tie the other way, both lines show 0.0325. Doris computes 1.0/(60 + rk) in decimal arithmetic, so the last digit can differ. The 2 company-facts rows follow below the 10-Ks.
A re-run redoes only what changed
Run cocoindex update main again, and CocoIndex does only the work that a change needs:
memo=True on the 2 processors skips files that did not change, and embed() is memoized too, so only new chunk text goes to the model (unless EMBED_MODEL changes). A deleted file’s component goes away, and its rows are deleted with it.
The last row in the figure needs a manual step. CocoIndex tracks the code of functions decorated with @coco.fn. _index_text, _scrub_pii, _extract_topics and _TOPIC_KEYWORDS are plain Python, so an edit to them does not invalidate the stored results. After you change one of them, run cocoindex update main --full-reprocess, which skips all memoized results and processes every file again (including the embeddings).
Quick check: you edit the Microsoft 10-K file and run cocoindex update main again. Which files does the pipeline process?
Only the edited file. process_filing and process_facts use memo=True, so files whose content and code are unchanged are skipped. chunk_id comes from the file name and chunk offsets, so the re-run replaces that file’s rows instead of adding copies.
Query Doris directly
The table is plain SQL, so you can also query it over the MySQL protocol:
mysql -h localhost -P 9030 -u root
Filter by topic tag:
-- Chunks tagged with cybersecurity risk
SELECT doc_filename, text, topics
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"');
-- Chunks with either of 2 tags
SELECT doc_filename, text
FROM sec_analytics.filing_chunks
WHERE json_contains(topics, '"RISK:CYBER"')
OR json_contains(topics, '"RISK:CLIMATE"');
The 3 closest chunks per company, for a question vector:
WITH scored AS (
SELECT cik, doc_filename, text,
l2_distance(embedding, [...]) AS score -- [...] is the question's 384-number vector
FROM sec_analytics.filing_chunks
WHERE cik IN ('0000320193', '0000789019')
),
ranked AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY cik ORDER BY score) AS rk
FROM scored
)
SELECT * FROM ranked WHERE rk <= 3;
Keyword matches by filing year:
SELECT LEFT(filing_date, 4) AS filing_year,
COUNT(DISTINCT cik) AS companies,
COUNT(*) AS chunks
FROM sec_analytics.filing_chunks
WHERE text MATCH_ANY 'cybersecurity risk'
GROUP BY filing_year
ORDER BY filing_year DESC;
What to try next
- Index your own filing. Save a 10-K as
{CIK}_{date}_{form}.txtindata/filings, for example0000320193_2025-11-01_10-K.txt, and runcocoindex update main. Only the new file is processed. - Add a topic tag. Add an entry to
_TOPIC_KEYWORDSinmain.py, then runcocoindex update main --full-reprocessso every chunk is tagged again. - Preview before you write.
cocoindex update main --previewcomputes the changes to the target and prints them without applying them. - Add a third format. Write a
process_pdfthat turns a PDF into text and calls_index_text, and mount it withcoco.mount_eachlike the other 2. - Filter, then search. Add a
json_contains(topics, ...)condition to thewhereinsearch.py, so the hybrid search only ranks chunks with a given tag.
The same pattern fits other document collections, such as contracts or support tickets: several formats into one table, personal data removed first, tags for filters, and hybrid search. The full source code is in the CocoIndex v1 repo, with the sample-data generator and the search script.
Frequently asked questions
How do I build a search index over SEC EDGAR filings?
This example builds a CocoIndex v1 pipeline that reads 2 SEC formats (10-K filings as text and XBRL company-facts JSON), removes personal data, splits the text into chunks, tags and embeds each chunk, and loads one row per chunk into an Apache Doris table with a full-text index and a vector index. Both formats go through one shared function, _index_text.
How do you combine several document formats into one search index?
Each format gets its own processor that turns a file into text plus metadata: process_filing parses the 10-K file name, and process_facts renders the XBRL JSON as text. Both call the same _index_text, which declares rows into one Doris table with one schema. A third format, such as PDF, is one more processor that calls _index_text.
Why scrub PII before chunking instead of after?
_scrub_pii runs on the whole document before the split. If it ran on each chunk, a phone number or SSN cut in half by a chunk boundary would no longer match the regular expression, and both halves would reach the index.
What chunk size does the SEC EDGAR pipeline use?
RecursiveSplitter with chunk_size=1000 and chunk_overlap=200, in bytes (about the same number of characters for English text). With language="markdown" it splits at headings first, then paragraphs, then sentences. Each of the 4 short sample documents fits in 1 chunk.
How does hybrid search with Reciprocal Rank Fusion work here?
search.py ranks every chunk by vector distance (l2_distance) and by keyword match (MATCH_ANY), then gives each chunk the score 1/(60 + rank) summed over both lists. A chunk ranked 1st in both scores 1/61 + 1/61 = 0.0328; 1st by meaning but 100th by keywords scores 1/61 + 1/160 = 0.0226.
When should I use a deterministic parser instead of an LLM for extraction?
When you know the file format in advance. Here the 10-K metadata (CIK, filing date, form type) comes from splitting the file name {CIK}_{date}_{form}.txt, which skips an LLM call per document and returns the same result every time. For formats you do not know in advance, CocoIndex can extract fields with an LLM.
Why use Apache Doris as the target for hybrid search?
One Doris table can hold an inverted index on text for keyword search and an ANN vector index on embedding for search by meaning, so a single SQL query can rank by both. Doris 4.0 or later is needed for the vector index.
Build fresh context for your agents.
CocoIndex keeps derived data in sync with its sources. Only what changed re-runs.




