# Amazon S3 connector

> **CocoIndex v1, Rust SDK (experimental).** This page documents the CocoIndex **v1** Rust SDK (the `cocoindex` crate). The Rust SDK is experimental: any release may change or remove its API, and it is not yet published on crates.io. The Python SDK pages are at https://cocoindex.io/docs/.
>
> Source: https://cocoindex.io/docs/rust/connectors/amazon_s3/ · Rust docs index: https://cocoindex.io/docs/rust/llms.txt · Rust SDK tour: https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/SHOWCASE.md · Rust examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust

The `amazon_s3` connector provides utilities for reading objects from Amazon S3 buckets and S3-compatible services (e.g. MinIO, Tigris).

```rust
use cocoindex::connectors::amazon_s3::{self, ListOptions, S3Client, S3File};
```

**Note — Installation**
Enable the `amazon_s3` feature of the `cocoindex` crate:

```toml
cocoindex = { git = "https://github.com/cocoindex-io/cocoindex", features = ["amazon_s3"] }
```

The connector uses the official `aws-sdk-s3` crate, re-exported as `cocoindex::connectors::amazon_s3::aws_sdk_s3`, so you can create buckets and upload objects without a direct dependency.

## As source

The connector provides two ways to read from S3:

- `list_objects()` — List the objects in a bucket (with optional prefix and filtering)
- `S3Client::get_object()` / `S3Client::get_object_uri()` — Fetch a single object by its key or `s3://` URI
- `S3Client::read()` — Read an object's content

All of them go through an `S3Client`, which you create and provide to the app as a context key:

```rust
cocoindex::context_key!(static S3: S3Client);

let s3 = S3Client::connect().await?;
let app = Environment::builder()
    .provide_key(&S3, s3)
    .build()
    .await?
    .app("AmazonS3EmbeddingRust")
    .await?;
```

`S3Client::connect()` reads the standard AWS environment: region, credentials and `AWS_ENDPOINT_URL`. When `AWS_ENDPOINT_URL` is set (for MinIO or Tigris), path-style addressing is enabled. For a client configured by hand, build an `aws_sdk_s3::Client` and wrap it with `S3Client::from_client(state_id, client)`, where `state_id` is the stable identity used in memo keys. An `S3Client` is `Clone` and has no `close()`.

### list_objects

List objects in an S3 bucket. Returns an `S3Walker`.

```rust
pub fn list_objects(
    client: &S3Client,
    bucket: impl Into<String>,
    options: ListOptions,
) -> S3Walker

pub struct ListOptions {
    /// Only list objects whose key starts with this prefix.
    pub prefix: String,
    /// Filter by relative path (after prefix stripping). Defaults to match-all.
    pub path_matcher: Option<Arc<dyn FilePathMatcher>>,
    /// Skip objects larger than this many bytes.
    pub max_file_size: Option<u64>,
}

pub async fn S3Walker::list(&self) -> Result<Vec<S3File>>
pub async fn S3Walker::items(&self) -> Result<Vec<(String, S3File)>>
```

The walker is not an iterator: `list()` returns every matching object, and `items()` the `(key, file)` pairs for `mount_each!`.

**Parameters:**

- `client` — The `S3Client`.
- `bucket` — The S3 bucket name.
- `options.prefix` — Only list objects whose key starts with this prefix. The prefix and a `/` after it are stripped from the relative keys of the returned files.
- `options.path_matcher` — Optional filter for files. Patterns are matched against the relative path (after prefix stripping). See [PatternFilePathMatcher](/docs/rust/common_resources/data_types/#patternfilepathmatcher).
- `options.max_file_size` — Skip objects larger than this size in bytes.

**Returns:** An `S3Walker`; `list()` and `items()` list the bucket once each time you call them.

The prefix is passed to S3 as given, not read as a folder: `"data"` also matches `database.csv`. Name a folder with its trailing slash, `"data/"`, and select objects by the start of their name with `path_matcher`.

### Iterating files

`list_objects()` returns an `S3Walker` that lists `S3File` objects (implementing the [`FileLike`](/docs/rust/common_resources/data_types/#filelike) base class):

```rust
let files = amazon_s3::list_objects(
    &client,
    "my-bucket",
    ListOptions {
        prefix: "data/".to_string(),
        ..Default::default()
    },
)
.list()
.await?;
for file in files {
    let text = file.read_text().await?;
    // ...
}
```

A listed file carries the client, so `file.read_text()` works; `client.read_text(&file)` reads through a client you hold.

See [`FileLike`](/docs/rust/common_resources/data_types/#filelike) for details on the file objects.

### Keyed iteration with `items()`

`S3Walker::items()` returns `(String, S3File)` pairs, useful for associating each file with a stable string key: the object key relative to the prefix, exactly as stored. Keys are not normalized, so `a//b.txt` and `a/b.txt` are two items:

```rust
let items = amazon_s3::list_objects(&client, "my-bucket", ListOptions::default())
    .items()
    .await?;
for (key, file) in items {
    let content = file.read().await?;
    // ...
}
```

### Filtering files

Use `PatternFilePathMatcher` to filter which objects are included. Patterns are matched against the relative path (after prefix stripping):

```rust
use cocoindex::resources::file::PatternFilePathMatcher;

let files = amazon_s3::list_objects(
    ctx.get_key(&S3)?,
    &bucket,
    ListOptions {
        prefix,
        path_matcher: Some(Arc::new(PatternFilePathMatcher::include(["**/*.md"])?)),
        ..Default::default()
    },
)
.items()
.await?;
```

`PatternFilePathMatcher::new(included, excluded)` takes exclude patterns too.

### Limiting file size

Use `max_file_size` to skip objects that exceed a size threshold:

```rust
// Skip objects larger than 10 MB
let small = amazon_s3::list_objects(
    &client,
    "my-bucket",
    ListOptions {
        max_file_size: Some(10 * 1024 * 1024),
        ..Default::default()
    },
)
.list()
.await?;
```

### get_object

Fetch a single object from an S3 bucket by its key.

```rust
pub async fn S3Client::get_object(&self, bucket: &str, key: &str) -> Result<S3File>
pub async fn S3Client::get_object_uri(&self, uri: &str) -> Result<S3File>
```

**Parameters:**

- `bucket`, `key` — The bucket name and the full S3 object key, for `get_object`.
- `uri` — A full S3 URI (`s3://bucket/key`), for `get_object_uri`.

**Returns:** An `S3File` (FileLike) for the specified object. Its relative path is the full key. It is an error if the object does not exist.

**Example:**

```rust
// Via S3 URI:
let f = client.get_object_uri("s3://my-bucket/data/config.json").await?;
let data = client.read(&f).await?;

// Via bucket name + key:
let f = client.get_object("my-bucket", "data/config.json").await?;
let data = f.read().await?;
```

### read

Read an object's content through the client.

```rust
pub async fn S3Client::read(&self, file: &S3File) -> Result<Vec<u8>>
pub async fn S3Client::read_range(&self, file: &S3File, len: u64) -> Result<Vec<u8>>
pub async fn S3Client::read_text(&self, file: &S3File) -> Result<String>
```

`read_range` reads the first `len` bytes with a ranged request. `read_text` decodes the content as UTF-8 (BOM-aware, lossy on invalid UTF-8).

The client reads an `S3File`, not a URI: get the file with `get_object_uri` first, which costs one metadata request.

**Example:**

```rust
let f = client.get_object_uri("s3://my-bucket/data/config.json").await?;
let data = client.read(&f).await?;
let head = client.read_range(&f, 3).await?;
```

### S3FilePath

Each file returned by the connector has an `S3FilePath` — a [`FilePath`](/docs/rust/common_resources/data_types/#filepath) specialized for S3:

- **Relative path** (`file.file_path().path()`, a `&str`) — The object key relative to the walker prefix (or the full key if no prefix was used). `file.key()` returns it as a `String`, the key for `mount_each!`.
- **Resolved path** (`file.file_path().resolve()`) — The full S3 object key.
- **Bucket** (`file.file_path().bucket()`).
- **Identity** — The memo key is the bucket and the relative key.

For example, with `prefix: "data/"` and an object key `"data/docs/readme.md"`:
- `file.file_path().path()` → `"docs/readme.md"`
- `file.file_path().resolve()` → `"data/docs/readme.md"`

`S3File` has the public fields `size`, `modified_secs` and `etag`, plus `key()`, `read()`, `read_size(n)` and `read_text()`; it implements `FileLike` and `MemoInput`. Its memo key holds only stable metadata (the path, size, modified time and ETag), never the client.

`S3FilePath` has no derived-path methods (`parent`, `/`, `with_name()`, `with_suffix()`).

### Example

```rust
use std::sync::Arc;

use cocoindex::connectors::amazon_s3::{self, ListOptions, S3Client, S3File};
use cocoindex::prelude::*;
use cocoindex::resources::file::PatternFilePathMatcher;

cocoindex::context_key!(static S3: S3Client);

#[cocoindex::function(memo)]
async fn process_file(_ctx: &Ctx, file: S3File) -> Result<()> {
    let text = file.read_text().await?;
    // ... process the file content ...
    Ok(())
}

async fn app_main(ctx: Ctx, bucket: String, prefix: String) -> Result<()> {
    let files = amazon_s3::list_objects(
        ctx.get_key(&S3)?,
        &bucket,
        ListOptions {
            prefix,
            path_matcher: Some(Arc::new(PatternFilePathMatcher::include(["**/*.md"])?)),
            ..Default::default()
        },
    )
    .items()
    .await?;

    mount_each!(files, |file| process_file(ctx, file)).await?;
    Ok(())
}
```

The full program, which chunks and embeds each object into Postgres, is `examples/rust/amazon_s3_embedding`.