Azure Blob Storage connector

Read blobs from Azure Blob Storage containers with prefix filters, path matchers, size limits, and stable AzureBlobFilePath values for memoization.

Version

v 1.0.14

The azure_blob connector provides utilities for reading blobs from Azure Blob Storage containers.

use cocoindex::connectors::azure_blob::{
    self, AzureBlobFile, ContainerClient, Credential, ListOptions,
};

cocoindex = { git = "https://github.com/cocoindex-io/cocoindex", features = ["azure_blob"] }

As source

The connector provides three read helpers:

• list_blobs() lists blobs in a container, with optional prefix and filtering.
• get_blob() fetches a single blob by name.
• read() reads blob content directly by name.

All helpers take the connector’s own azure_blob::ContainerClient, a handle to one container built with ContainerClient::new(account_url, container_name, credential).

Connection setup

az login
az account set --subscription "<subscription-name-or-id>"

use cocoindex::connectors::azure_blob::{ContainerClient, Credential, TokenCredential};

let client = ContainerClient::new(
    "https://myaccount.blob.core.windows.net",
    "docs",
    Credential::TokenCredential(TokenCredential::Default),
)?;

TokenCredential::Default takes the sources of the Azure SDK’s DefaultAzureCredential that need no interactive sign-in. It uses the first source that is configured where the process runs, a service principal from AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET, the managed identity of the Azure resource the process runs on, or the account signed in with az login. TokenCredential::ClientSecret { tenant_id, client_id, client_secret }, TokenCredential::ManagedIdentity { client_id } (None for the system-assigned identity) and TokenCredential::AzureCli name one source. The client gets the token on the first request and renews it shortly before it expires. Credential::BearerToken takes an access token for the https://storage.azure.com/ resource that you obtained yourself, such as the one az account get-access-token --resource https://storage.azure.com/ prints, and does not renew it.

To use a SAS token:

let client = ContainerClient::new(
    "https://myaccount.blob.core.windows.net",
    "docs",
    Credential::SasToken(sas_token),
)?;

The SAS token is the query string Azure generates, with or without the leading ?.

To use a storage account access key:

let client = ContainerClient::new(
    "https://myaccount.blob.core.windows.net",
    "docs",
    Credential::AccountKey(account_key),
)?;

A ContainerClient is Clone and has no close(); declare a key with cocoindex::context_key!(static CONTAINER: ContainerClient) and provide the client with Environment::builder().provide_key(&CONTAINER, client). An account URL with a path, such as the Azurite emulator’s http://127.0.0.1:10000/devstoreaccount1, names the account by the first path segment.

The identity must have permission to list and read blobs. With Azure role-based access control, Storage Blob Data Reader is enough for read-only indexing.

list_blobs

List blobs in an Azure Blob Storage container. Returns an AzureBlobWalker.

pub fn list_blobs(client: &ContainerClient, options: ListOptions) -> AzureBlobWalker

pub struct ListOptions {
    /// Only list blobs in this folder.
    pub prefix: String,
    /// Filter by relative path (after prefix stripping). Defaults to match-all.
    pub path_matcher: Option<Arc<dyn FilePathMatcher>>,
    /// Skip blobs larger than this many bytes.
    pub max_file_size: Option<u64>,
}

pub async fn AzureBlobWalker::list(&self) -> Result<Vec<AzureBlobFile>>
pub async fn AzureBlobWalker::items(&self) -> Result<Vec<(String, AzureBlobFile)>>

ListOptions::default() lists the whole container. The walker is not an iterator: list() returns every matching blob, and items() the (key, file) pairs for mount_each!.

Parameters:

• container_client - The ContainerClient (the argument is named client).
• prefix - Only list blobs in this folder. A prefix names a folder: "docs" and "docs/" both list the blobs under docs/, and neither lists docsa.md. The folder is stripped from relative paths in returned files. To select blobs by the start of their name (for example 2024-*.csv), use path_matcher.
• path_matcher - Optional filter for files. Patterns are matched against the relative path after prefix stripping. See PatternFilePathMatcher.
• max_file_size - Skip blobs larger than this size in bytes.

Returns: An AzureBlobWalker whose list() and items() list the container once each time you call them.

Iterating files

list_blobs() lists AzureBlobFile objects implementing FileLike:

let files = azure_blob::list_blobs(
    client,
    ListOptions {
        prefix: "docs/".to_string(),
        ..Default::default()
    },
)
.list()
.await?;
for file in files {
    let text = file.read_text().await?;
    // ...
}

Keyed iteration with items()

AzureBlobWalker::items() returns (String, AzureBlobFile) pairs. The key is the blob name relative to the prefix, exactly as stored. Keys are not normalized, so a//b.txt and a/b.txt are two items:

let files = azure_blob::list_blobs(
    &client,
    ListOptions { prefix: "docs".to_string(), ..Default::default() },
)
.items()
.await?;
mount_each!(files, |file| index_blob(ctx, file)).await?;

file.key() is the same relative blob name.

Filtering files

Use PatternFilePathMatcher to include or exclude blobs by glob pattern. Patterns are matched against the relative path after prefix stripping:

use cocoindex::resources::file::PatternFilePathMatcher;

let files = azure_blob::list_blobs(
    client,
    ListOptions {
        prefix,
        path_matcher: Some(Arc::new(
            PatternFilePathMatcher::include(["**/*.md"])
                .map_err(|e| Error::engine(format!("matcher: {e}")))?,
        )),
        ..Default::default()
    },
)
.list()
.await?;

PatternFilePathMatcher::new(included, excluded) takes exclude patterns too.

Limiting file size

Use max_file_size to skip blobs above a size threshold:

let small = azure_blob::list_blobs(
    client,
    ListOptions {
        max_file_size: Some(10 * 1024 * 1024),
        ..Default::default()
    },
)
.list()
.await?;

get_blob

Fetch a single blob by name.

pub async fn get_blob(client: &ContainerClient, blob_name: &str) -> Result<AzureBlobFile>

Parameters:

• container_client - The ContainerClient (the argument is named client).
• blob_name - The full blob name in the container.

Returns: An AzureBlobFile for the specified blob.

Example:

let file = azure_blob::get_blob(client, "docs/readme.md").await?;
let text = file.read_text().await?;

read

Read blob content directly by name, without fetching metadata first.

pub async fn read(
    client: &ContainerClient,
    blob_name: &str,
    size: Option<usize>,
) -> Result<Vec<u8>>

Parameters:

• container_client - The ContainerClient (the argument is named client).
• blob_name - The full blob name in the container.
• size - Number of bytes to read. None reads the entire blob.

Returns: The blob content as bytes.

Example:

let data = azure_blob::read(client, "docs/readme.md", None).await?;

AzureBlobFilePath

Each file returned by the connector has an AzureBlobFilePath, a FilePath specialized for Azure Blob Storage:

• Relative path (file.file_path().path(), a &str) - The blob name relative to the walker prefix, or the full blob name if no prefix was used. file.key() returns it as a String, the key for mount_each!.
• Resolved path (file.file_path().resolve(), also blob_name()) - The full blob name in the container.
• Account and container (account_name(), container_name()).
• Identity - The memo key is (account_name, container_name, relative_path).

For example, with prefix: "docs/" and blob name "docs/readme.md":

• file.file_path().path() returns "readme.md"
• file.file_path().resolve() returns "docs/readme.md"

AzureBlobFile has the public fields size, modified_secs and etag, plus key(), read(), read_size(n) and read_text(); it implements FileLike and MemoInput. Its memo key holds only stable metadata, never the client.

Example

use std::sync::Arc;

use cocoindex::connectors::azure_blob::{self, AzureBlobFile, ContainerClient, ListOptions};
use cocoindex::prelude::*;
use cocoindex::resources::file::PatternFilePathMatcher;

cocoindex::context_key!(static CONTAINER: ContainerClient);

#[cocoindex::function(memo)]
async fn process_file(_ctx: &Ctx, file: AzureBlobFile) -> Result<()> {
    let text = file.read_text().await?;
    // Process the file content.
    Ok(())
}

async fn app_main(ctx: Ctx, prefix: String) -> Result<()> {
    let client = ctx.get_key(&CONTAINER)?;
    let files = azure_blob::list_blobs(
        client,
        ListOptions {
            prefix,
            path_matcher: Some(Arc::new(
                PatternFilePathMatcher::include(["**/*.md"])
                    .map_err(|e| Error::engine(format!("matcher: {e}")))?,
            )),
            ..Default::default()
        },
    )
    .list()
    .await?;

    mount_each!(
        files.into_iter().map(|f| (f.key(), f)),
        |file| process_file(ctx, file)
    )
    .await?;
    Ok(())
}

See examples/rust/azure_blob_embedding for a runnable project.