Amazon S3 connector

Read objects from Amazon S3 buckets and S3-compatible services (e.g. MinIO, Tigris) — with prefix filters, path matchers, size limits, and stable S3FilePath for memoization.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The amazon_s3 connector provides utilities for reading objects from Amazon S3 buckets and S3-compatible services (e.g. MinIO, Tigris).

use cocoindex::connectors::amazon_s3::{self, ListOptions, S3Client, S3File};

cocoindex = { git = "https://github.com/cocoindex-io/cocoindex", features = ["amazon_s3"] }

As source

The connector provides two ways to read from S3:

• list_objects() — List the objects in a bucket (with optional prefix and filtering)
• S3Client::get_object() / S3Client::get_object_uri() — Fetch a single object by its key or s3:// URI
• S3Client::read() — Read an object’s content

All of them go through an S3Client, which you create and provide to the app as a context key:

cocoindex::context_key!(static S3: S3Client);

let s3 = S3Client::connect().await?;
let app = Environment::builder()
    .provide_key(&S3, s3)
    .build()
    .await?
    .app("AmazonS3EmbeddingRust")
    .await?;

S3Client::connect() reads the standard AWS environment: region, credentials and AWS_ENDPOINT_URL. When AWS_ENDPOINT_URL is set (for MinIO or Tigris), path-style addressing is enabled. For a client configured by hand, build an aws_sdk_s3::Client and wrap it with S3Client::from_client(state_id, client), where state_id is the stable identity used in memo keys. An S3Client is Clone and has no close().

list_objects

List objects in an S3 bucket. Returns an S3Walker.

pub fn list_objects(
    client: &S3Client,
    bucket: impl Into<String>,
    options: ListOptions,
) -> S3Walker

pub struct ListOptions {
    /// Only list objects whose key starts with this prefix.
    pub prefix: String,
    /// Filter by relative path (after prefix stripping). Defaults to match-all.
    pub path_matcher: Option<Arc<dyn FilePathMatcher>>,
    /// Skip objects larger than this many bytes.
    pub max_file_size: Option<u64>,
}

pub async fn S3Walker::list(&self) -> Result<Vec<S3File>>
pub async fn S3Walker::items(&self) -> Result<Vec<(String, S3File)>>

The walker is not an iterator: list() returns every matching object, and items() the (key, file) pairs for mount_each!.

Parameters:

• client — The S3Client.
• bucket — The S3 bucket name.
• options.prefix — Only list objects whose key starts with this prefix. The prefix and a / after it are stripped from the relative keys of the returned files.
• options.path_matcher — Optional filter for files. Patterns are matched against the relative path (after prefix stripping). See PatternFilePathMatcher.
• options.max_file_size — Skip objects larger than this size in bytes.

Returns: An S3Walker; list() and items() list the bucket once each time you call them.

Iterating files

list_objects() returns an S3Walker that lists S3File objects (implementing the FileLike base class):

let files = amazon_s3::list_objects(
    &client,
    "my-bucket",
    ListOptions {
        prefix: "data/".to_string(),
        ..Default::default()
    },
)
.list()
.await?;
for file in files {
    let text = file.read_text().await?;
    // ...
}

A listed file carries the client, so file.read_text() works; client.read_text(&file) reads through a client you hold.

See FileLike for details on the file objects.

Keyed iteration with items()

S3Walker::items() returns (String, S3File) pairs, useful for associating each file with a stable string key: the object key relative to the prefix, exactly as stored. Keys are not normalized, so a//b.txt and a/b.txt are two items:

let items = amazon_s3::list_objects(&client, "my-bucket", ListOptions::default())
    .items()
    .await?;
for (key, file) in items {
    let content = file.read().await?;
    // ...
}

Filtering files

Use PatternFilePathMatcher to filter which objects are included. Patterns are matched against the relative path (after prefix stripping):

use cocoindex::resources::file::PatternFilePathMatcher;

let files = amazon_s3::list_objects(
    ctx.get_key(&S3)?,
    &bucket,
    ListOptions {
        prefix,
        path_matcher: Some(Arc::new(PatternFilePathMatcher::include(["**/*.md"])?)),
        ..Default::default()
    },
)
.items()
.await?;

PatternFilePathMatcher::new(included, excluded) takes exclude patterns too.

Limiting file size

Use max_file_size to skip objects that exceed a size threshold:

// Skip objects larger than 10 MB
let small = amazon_s3::list_objects(
    &client,
    "my-bucket",
    ListOptions {
        max_file_size: Some(10 * 1024 * 1024),
        ..Default::default()
    },
)
.list()
.await?;

get_object

Fetch a single object from an S3 bucket by its key.

pub async fn S3Client::get_object(&self, bucket: &str, key: &str) -> Result<S3File>
pub async fn S3Client::get_object_uri(&self, uri: &str) -> Result<S3File>

Parameters:

• bucket, key — The bucket name and the full S3 object key, for get_object.
• uri — A full S3 URI (s3://bucket/key), for get_object_uri.

Returns: An S3File (FileLike) for the specified object. Its relative path is the full key. It is an error if the object does not exist.

Example:

// Via S3 URI:
let f = client.get_object_uri("s3://my-bucket/data/config.json").await?;
let data = client.read(&f).await?;

// Via bucket name + key:
let f = client.get_object("my-bucket", "data/config.json").await?;
let data = f.read().await?;

read

pub async fn S3Client::read(&self, file: &S3File) -> Result<Vec<u8>>
pub async fn S3Client::read_range(&self, file: &S3File, len: u64) -> Result<Vec<u8>>
pub async fn S3Client::read_text(&self, file: &S3File) -> Result<String>

read_range reads the first len bytes with a ranged request. read_text decodes the content as UTF-8 (BOM-aware, lossy on invalid UTF-8).

Example:

let f = client.get_object_uri("s3://my-bucket/data/config.json").await?;
let data = client.read(&f).await?;
let head = client.read_range(&f, 3).await?;

S3FilePath

Each file returned by the connector has an S3FilePath — a FilePath specialized for S3:

• Relative path (file.file_path().path(), a &str) — The object key relative to the walker prefix (or the full key if no prefix was used). file.key() returns it as a String, the key for mount_each!.
• Resolved path (file.file_path().resolve()) — The full S3 object key.
• Bucket (file.file_path().bucket()).
• Identity — The memo key is the bucket and the relative key.

For example, with prefix: "data/" and an object key "data/docs/readme.md":

• file.file_path().path() → "docs/readme.md"
• file.file_path().resolve() → "data/docs/readme.md"

S3File has the public fields size, modified_secs and etag, plus key(), read(), read_size(n) and read_text(); it implements FileLike and MemoInput. Its memo key holds only stable metadata (the path, size, modified time and ETag), never the client.

Example

use std::sync::Arc;

use cocoindex::connectors::amazon_s3::{self, ListOptions, S3Client, S3File};
use cocoindex::prelude::*;
use cocoindex::resources::file::PatternFilePathMatcher;

cocoindex::context_key!(static S3: S3Client);

#[cocoindex::function(memo)]
async fn process_file(_ctx: &Ctx, file: S3File) -> Result<()> {
    let text = file.read_text().await?;
    // ... process the file content ...
    Ok(())
}

async fn app_main(ctx: Ctx, bucket: String, prefix: String) -> Result<()> {
    let files = amazon_s3::list_objects(
        ctx.get_key(&S3)?,
        &bucket,
        ListOptions {
            prefix,
            path_matcher: Some(Arc::new(PatternFilePathMatcher::include(["**/*.md"])?)),
            ..Default::default()
        },
    )
    .items()
    .await?;

    mount_each!(files, |file| process_file(ctx, file)).await?;
    Ok(())
}

The full program, which chunks and embeds each object into Postgres, is examples/rust/amazon_s3_embedding.