LiteLLM embeddings & speech-to-text

Embed text and transcribe audio through LiteLLM's unified API across 100+ providers (OpenAI, Azure, Vertex AI, Bedrock, Cohere, and more) — with VectorSchemaProvider for automatic vector column configuration.

Version

v 1.0.14

Last reviewed

Jul 4, 2026

The cocoindex.ops.litellm module provides integration with the LiteLLM library for text embeddings (LiteLLMEmbedder) and speech-to-text transcription (LiteLLMTranscriber).

from cocoindex.ops.litellm import LiteLLMEmbedder, LiteLLMTranscriber

pip install cocoindex[litellm]

Overview

The LiteLLMEmbedder class is a wrapper around LiteLLM’s embedding API that:

• Implements VectorSchemaProvider for seamless integration with CocoIndex connectors
• Supports 100+ embedding providers (OpenAI, Azure, Vertex AI, Cohere, Bedrock, etc.) through a unified API
• Provides a simple async embed() method
• Passes through all additional arguments to the LiteLLM embedding API, except timeout and max_inflight_requests, which CocoIndex owns
• Returns properly typed numpy arrays

Basic usage

Creating an embedder

All extra keyword arguments are passed through to every litellm.aembedding call (api_key, api_base, dimensions, …) — except timeout and max_inflight_requests, which are CocoIndex’s own knobs (see Timeout and Limiting concurrent requests). See Supported providers for provider-specific model strings and configuration.

from datetime import timedelta

from cocoindex.ops.litellm import LiteLLMEmbedder

embedder = LiteLLMEmbedder("text-embedding-3-small")

# With explicit API key and base URL
embedder = LiteLLMEmbedder("text-embedding-3-small", api_key="sk-...", api_base="https://my-proxy.example.com")

# With custom dimensions (OpenAI text-embedding-3 models)
embedder = LiteLLMEmbedder("text-embedding-3-small", dimensions=512)

# With a cap on concurrent requests to the backend
embedder = LiteLLMEmbedder("text-embedding-3-small", max_inflight_requests=8)

# With a longer timeout, for a slow self-hosted endpoint
embedder = LiteLLMEmbedder("openai/bge-m3", api_base="http://localhost:7997", timeout=timedelta(minutes=30))

Embedding text

The embed() method converts text into a numpy.ndarray of float32. It’s an async method — use await when calling it:

# In a CocoIndex function
embedding = await embedder.embed("Hello, world!")

# Use the embedding in a dataclass row, store in a vector database, etc.
table.declare_row(row=DocEmbedding(text="Hello, world!", embedding=embedding))

Limiting concurrent requests

Concurrent embed() calls are batched: CocoIndex groups them into requests of up to 64 texts each, and sends as many such requests in parallel as there is work queued. max_inflight_requests caps how many requests this embedder keeps open against the backend at once — useful for a self-hosted endpoint (Infinity, TEI, Ollama, vLLM, …) that rejects bursts with 429s:

embedder = LiteLLMEmbedder(
    "openai/bge-m3",
    api_base="http://localhost:7997",
    max_inflight_requests=8,
)

The default is None (no cap). The cap is per embedder instance: two embedders pointed at the same endpoint have independent limits, so share one instance when you want one limit.

Using as a type annotation

The LiteLLMEmbedder implements VectorSchemaProvider, which means it can be used directly as metadata in Annotated type annotations. This is the recommended way to declare vector columns — CocoIndex connectors automatically extract the vector dimension and dtype from the annotation when creating tables.

from dataclasses import dataclass
from typing import Annotated
from numpy.typing import NDArray

embedder = LiteLLMEmbedder("text-embedding-3-small")

@dataclass
class DocEmbedding:
    id: int
    filename: str
    text: str
    embedding: Annotated[NDArray, embedder]

When you pass this dataclass to a connector’s TableSchema.from_class(), the connector automatically reads the embedder annotation to determine the vector column’s dimension and dtype. For example, with Postgres:

from cocoindex.connectors import postgres

table_schema = await postgres.TableSchema.from_class(
    DocEmbedding,
    primary_key=["id"],
)
target_table = await postgres.mount_table_target(
    PG_DB,
    table_name="doc_embeddings",
    table_schema=table_schema,
    pg_schema_name="my_schema",
)

The connector automatically creates the appropriate vector(N) column. See the Connectors docs for other supported backends (LanceDB, Qdrant, SQLite).

Speech-to-text

The LiteLLMTranscriber class is a wrapper around LiteLLM’s transcription API that turns audio into text through any LiteLLM-supported speech-to-text provider (OpenAI Whisper, ElevenLabs, Groq, and more) using a unified interface.

Creating a transcriber

All extra keyword arguments are passed through to every litellm.atranscription call (api_key, api_base, language, extra_body, …) — except timeout, which is CocoIndex’s own bound on the whole request (see Timeout).

from datetime import timedelta

from cocoindex.ops.litellm import LiteLLMTranscriber

transcriber = LiteLLMTranscriber("whisper-1")

# With an explicit API key
transcriber = LiteLLMTranscriber("whisper-1", api_key="sk-...")

# With a default language hint applied to every call
transcriber = LiteLLMTranscriber("whisper-1", language="en")

# With a longer timeout, for long audio or a slow endpoint
transcriber = LiteLLMTranscriber("whisper-1", timeout=timedelta(minutes=30))

Transcribing audio

The transcribe() method takes a FileLike object (such as a localfs.File) containing audio data and returns the transcribed text as a str. It’s an async method — use await when calling it. Per-call keyword arguments override the defaults provided at construction time — except timeout, which is settable only on the constructor (see Timeout):

# In a CocoIndex function, `file` is a FileLike (e.g. localfs.File)
transcript = await transcriber.transcribe(file)

# Override or add per-call options
transcript = await transcriber.transcribe(file, response_format="verbose_json")

A complete pipeline that walks local audio files, transcribes each one, and stores the transcripts in Postgres is available in the audio_to_text example.

Supported transcription providers

LiteLLMTranscriber accepts any model string supported by LiteLLM’s transcription API. A few common examples:

See the LiteLLM audio transcription docs for the full list of providers and model strings.

Shared behavior

LiteLLMEmbedder and LiteLLMTranscriber reach their providers the same way: one retry loop, one clock, one memoization contract. Everything in this section applies to both.

Timeout

timeout bounds each request end to end; the default is 10 minutes. Transient failures — HTTP 429, 5xx, a dropped connection, and others below — are retried with backoff inside that window; a request that times out fails with DeadlineExceededError and is not retried. It takes a timedelta, or seconds as a number.

embedder = LiteLLMEmbedder("text-embedding-3-small", timeout=timedelta(minutes=30))
transcriber = LiteLLMTranscriber("whisper-1", timeout=timedelta(minutes=30))

timeout is CocoIndex’s own clock and the only one on the request — litellm’s per-request timeout is not forwarded. What to size it against is where the two differ:

• Embedding — one request carries up to 64 texts, so raise it for an endpoint that needs longer than the default to embed a full batch.
• Transcription — one request carries one whole audio file, so size it against the longest file you expect. It is settable only on the constructor: passing timeout to transcribe(), alongside the per-call provider arguments, raises TypeError.

Retries

Within the timeout window a failed request is retried with exponential backoff — 1s, doubling, capped at 30s. There is no attempt cap; time is the only brake.

The 408 row and the timeout row look contradictory and aren’t: an HTTP 408 response arrives quickly, so nothing has spent the budget, while a timeout we observed ourselves has by definition already spent its duration. Retrying that inside the request’s own bound just re-spends what is left, and against an overloaded backend it amplifies the load that caused it. The remedy for a genuine timeout is a longer timeout, not more requests.

A surrounding coco.timeout() merges with this one by taking whichever is shorter, so it can only cut retries short — raising timeout is the only way to give a request longer.

Memoization

embed() and transcribe() are both memoized (memo=True): inside a processing component, each distinct input’s result is cached across that component’s runs, so a caller that re-executes doesn’t re-pay the API call for unchanged input. timeout and max_inflight_requests are deliberately absent from the memo key — they bound and pace requests but cannot change a result, so retuning them invalidates nothing.

Mind the memoization placement rule; it lands differently on the two:

• Embedding — the memo holds one full vector per distinct text. That is what makes the common per-file-component pattern incremental, but it is redundant when your components are already keyed by the embedded content.
• Transcription — the memo holds one transcript per file: small entries standing in for an expensive call, which is the opposite trade from embedding. It hits whenever the enclosing component re-runs for a reason other than the audio changing — its code edited, its version bumped, another of its inputs changed — so it is usually worth keeping even in the per-file-component pattern, where an unchanged file skips the component entirely.

Callsite control over an op’s memoization is tracked in #2275.

Supported providers

Below are common providers with their model strings and configuration. The litellm module is re-exported from cocoindex.ops.litellm for setting provider-specific variables. See the LiteLLM embedding docs for the full list.

Ollama

No API key required. Ollama must be running locally (default http://localhost:11434). Pull the model first with ollama pull <model-name>.

embedder = LiteLLMEmbedder("ollama/nomic-embed-text", api_base="http://localhost:11434")

OpenAI

Environment variables: OPENAI_API_KEY

embedder = LiteLLMEmbedder("text-embedding-3-small")

Azure OpenAI

The model string uses your Azure deployment name, not the OpenAI model name.

Environment variables: AZURE_API_KEY, AZURE_API_BASE, AZURE_API_VERSION

embedder = LiteLLMEmbedder(
    "azure/my-deployment-name",
    api_key="your-azure-api-key",
    api_base="https://my-resource.openai.azure.com",
    api_version="2024-02-01",
)

Gemini (Google AI Studio)

Environment variables: GEMINI_API_KEY

embedder = LiteLLMEmbedder("gemini/text-embedding-004")

Vertex AI

Environment variables: GOOGLE_APPLICATION_CREDENTIALS (path to service account JSON)

Additional configuration: Set project and location via the litellm module or environment variables VERTEXAI_PROJECT and VERTEXAI_LOCATION:

from cocoindex.ops.litellm import LiteLLMEmbedder, litellm

litellm.vertex_project = "my-gcp-project"
litellm.vertex_location = "us-central1"

embedder = LiteLLMEmbedder("vertex_ai/text-embedding-004")

AWS Bedrock

Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION_NAME

embedder = LiteLLMEmbedder("bedrock/amazon.titan-embed-text-v2:0")

Mistral AI

Environment variables: MISTRAL_API_KEY

embedder = LiteLLMEmbedder("mistral/mistral-embed")

Voyage AI

Environment variables: VOYAGE_API_KEY

embedder = LiteLLMEmbedder("voyage/voyage-3.5")

Cohere

Environment variables: COHERE_API_KEY

Additional configuration: V3 models require an input_type parameter (defaults to "search_document"; use "search_query" for queries):

embedder = LiteLLMEmbedder("cohere/embed-english-v3.0", input_type="search_document")

Nebius AI

Environment variables: NEBIUS_API_KEY

embedder = LiteLLMEmbedder("nebius/BAAI/bge-en-icl")