AI-Native Data Pipeline - Why We Made It
Why the next wave of AI needs open source, scalable, and AI-native data infrastructure, and how CocoIndex is building the foundation for the future of intelligent data pipelines.
45 Posts
Index PDF elements - text, images with mixed embedding models and metadata
Extracts, embeds, and stores multimodal PDF elements — text with SentenceTransformers and images with CLIP — in vector database for unified semantic search. Includes full metadata traceability, thumbnail generation, and flow definitions for building multimodal retrieval systems.
Bring your own data: Index any data with Custom Sources
CocoIndex now officially supports custom sources — giving you the power to read data from any system you want. You can use CocoIndex for anything, and enjoy the robust incremental computing to build fresh knowledge for AI.
CocoIndex Changelog 2025-10-19
Featuring significant optimizations for production ready infrastructure: durable execution, efficient incremental processing optimizations over large datasets, GPU isolation, etc. and better support over native building blocks.
Automated invoice processing with AI, Snowflake and CocoIndex - with incremental processing
Discover how to rapidly optimize your data indexing strategy with CocoIndex and CocoInsight. Learn to define query handlers, trace search results back to source data, and seamlessly integrate AI-powered transformations for efficient, end-to-end retrieval and analytics.
Fast iterate your indexing strategy - trace back from query to data
Discover how to rapidly optimize your data indexing strategy with CocoIndex and CocoInsight. Learn to define query handlers, trace search results back to source data, and seamlessly integrate AI-powered transformations for efficient, end-to-end retrieval and analytics.
Incrementally Transform Structured + Unstructured Data from Postgres with AI
Comprehensive walkthrough on using CocoIndex to build unified, incrementally updated search and analytics pipelines across both structured and unstructured data sources, using PostgreSQL tables as the main origin and target for indexed data.
Build a Visual Document Index from multiple formats all at once - PDFs, Images, Slides - with ColPali
Build a unified visual document index from multiple file formats—including PDFs, images, and slides—using CocoIndex and ColPali, No OCR needed.
CocoIndex Changelog 2025-08-18
Featuring production readiness, scalability, and reliability. More flexibility with customization and native integrations. Extended features for multi-modalities pipelines and more.
Control Processing Concurrency in CocoIndex
Learn how CocoIndex's layered concurrency control features help you optimize data processing performance, prevent system overload, and ensure stable, efficient pipelines at scale.
Multi-Dimensional Vector Support in CocoIndex
CocoIndex natively handles typed multi-dimensional vectors — from simple arrays to multi-vector embeddings, unlocks multimodal AI pipelines at scale.
Indexing Faces for Scalable Visual Search - Build your own Google Photo Search
Build a scalable face detection and recognition pipeline using CocoIndex. This tutorial covers extracting and embedding faces from images, structuring data for visual search, and exporting to a vector database for face similarity queries.
Index academic papers and extract metadata for AI agents
How to index academic research papers by extracting metadata (e.g., title, authors, abstract) for AI agents and AI workflows using LLMs and CocoIndex
CocoIndex Changelog 2025-07-07
CocoIndex updates: in-process API, CLI improvements, EmbedText support, codebase indexing enhancements, and more.
CocoIndex Changelog 2025-05-31
CocoIndex updates: Amazon S3 as a data source, updates on query handling, standalone, and more.
Real-time data transformation pipeline with Amazon S3 bucket, SQS and CocoIndex
Build real-time data transformation pipeline with S3 and CocoIndex.
Build image search and query with natural language with vision model CLIP
Indexing images with CocoIndex and Vision Model in real-time: multi-modal embedding, and build vector index for efficient retrieval.
How to build index with text embeddings
Indexing text with CocoIndex and text embeddings, and query it with natural language.
Story of CocoIndex, at 1k stars 🎉
CocoIndex is the world's first open-source engine that supports both custom transformation logic and incremental processing specialized for data indexing. We just crossed 1k stars, thank you so much!
Build Real-Time Product Recommendation Engine with LLM and Graph Database
Build a real-time product recommendation engine with LLM and graph database, from the aspect of product category (taxonomy) understanding.
CocoIndex Changelog 2025-04-30
CocoIndex updates: Knowledge Graphs, Qdrant, Supabase, KTable/LTable, and more LLM providers.
Build Real-Time Knowledge Graph For Documents with LLM
CocoIndex now supports knowledge graph with incremental processing. Build live knowledge for agents is super easy with CocoIndex!
CocoIndex Changelog 2025-04-07
CocoIndex updates: Incremental processing with live update mode, evaluation utilities, support for date/time types, Google Drive, and assorted core/performance improvements
Continuous update derived data on source updates, automatically
CocoIndex continuously watches source changes and keeps derived data in sync, with low latency and minimal performance overhead.
Incremental Processing with CocoIndex
CocoIndex helps to keep index up to date with source changes, super efficient and low latency - with the support of incremental processing.
Structured Extraction from Patient Intake Form with LLM
Extract structured data from patient intake forms in PDF/Word with LLM by CocoIndex.
Build text embeddings from Google Drive for RAG
Tutorial to create text embeddings from docs on Google Drive, save in vector stores for semantics search / RAG, using CocoIndex.
CocoIndex Changelog 2025-03-20
First release of CocoIndex Changelog: LLM support, codebase indexing, custom functions, and assorted core/performance improvements
Build Real-Time Codebase Indexing for AI Code Generation
Indexing codebase for RAG with CocoIndex and Tree-sitter in real-time: chunking, embedding, semantic search, and build vector index for efficient retrieval.
On-premise structured extraction with LLM using Ollama
Learn to use CocoIndex extracting structured data from PDF/Markdown with Ollama's local LLM models. All running on premise without sending data to external APIs.
We are officially open sourced! 🎉
CocoIndex is the world's first open-source engine that supports both custom transformation logic and incremental processing specialized for data indexing. We are now officially open sourced!
Customizable Data Indexing Pipelines
Explain what customizable data indexing pipelines are through comparisons and examples.
What Makes Indexing Pipelines Different?
Understanding the unique characteristics of indexing pipelines compared to other data processing systems. Learn why indexing requires special handling for incremental processing and persistence.
Handling System Updates and CocoIndex Automatic Schema Inference
Explore how CocoIndex handles system updates in indexing flows and our approach to automatic schema inference. Learn about the challenges of managing data and logic evolution, infrastructure setup, and how CocoIndex simplifies these processes through smart automation.
Processing Large Files in Data Indexing Systems
Learn best practices for handling large files in data indexing systems. Understand processing granularity, fan-in/fan-out scenarios, and strategies for efficient processing of large datasets like patent XML files. Discover how CocoIndex helps manage memory pressure and ensures reliable processing.
Data Consistency in Indexing Pipelines
Explore the challenges and solutions for maintaining data consistency in indexing pipelines. Learn about concurrent updates, data exposure risks, and best practices for ensuring reliable, up-to-date indexes using CocoIndex's data-driven approach.
Data Indexing and Common Challenges
Explore the fundamentals of data indexing pipelines for RAG applications. Learn about key characteristics of effective indexing systems, common challenges in production, and how CocoIndex solves them. Discover best practices for building maintainable, cost-effective, and scalable data indexing infrastructure.
CocoIndex - A Data Indexing Platform for AI Applications
Discover how CocoIndex simplifies data indexing for AI applications. Learn about our comprehensive platform that handles data ingestion, processing, and management for RAG, semantic search, and other AI use cases.
Welcome to CocoIndex
Welcome to the official CocoIndex blog! We're excited to share our journey in building high-performance indexing infrastructure for AI applications.