Google Drive connector

Read files recursively from Google Drive folders, via a service account or a personal account signed in with OAuth. Supports MIME-type filtering and auto-exports Google Docs, Sheets, and Slides to text or CSV.

Version

v 1.0.14

Last reviewed

Nov 4, 2026

The google_drive connector provides utilities for reading files from Google Drive, authenticated with a service account or with a personal Google account signed in through OAuth.

use cocoindex::connectors::gdrive::{DriveFile, GoogleDriveClient, GoogleDriveSource};

cocoindex = { git = "https://github.com/cocoindex-io/cocoindex", features = ["google_drive"] }

As source

The connector provides two ways to read from Google Drive:

• GoogleDriveSource — the source, with list_files() and items()
• list_files() — a function over a GoogleDriveSourceSpec

Both read through a GoogleDriveClient, which authenticates with one of:

• a service account with access to the target Drive folders (GoogleDriveClient::from_service_account_file(path)?), the fit for a shared or automated setup;
• a personal Google account signed in once through OAuth (GoogleDriveClient::from_user_account(&account)?, experimental), the fit for your own Drive: see Signing in with a personal account.

The client is Clone; provide it under a context key so that functions can read files through it.

Setting up a service account

1. Create a service account in the Google Cloud Console
2. Download the JSON credential file
3. Share the target Drive folders with the service account’s email address

gws auth setup
gws auth login
gws drive files list

GoogleDriveSource

The primary source class for iterating over Google Drive files.

pub fn GoogleDriveSource::new(client: GoogleDriveClient, root_folder_ids: Vec<String>) -> Self
pub fn GoogleDriveSource::mime_types(self, mime_types: Vec<String>) -> Self

For example, with the client provided under the GDRIVE key:

cocoindex::context_key!(static GDRIVE: GoogleDriveClient);

let source = GoogleDriveSource::new(ctx.get_key(&GDRIVE)?.clone(), root_folder_ids);

Parameters:

• client — The GoogleDriveClient, signed in with a service account or a UserAccount.
• root_folder_ids — Google Drive folder IDs to scan. Subfolders are traversed recursively.
• mime_types — Optional MIME types to include, set with .mime_types(...). Without it, all file types are included.

Iterating files

GoogleDriveSource::list_files() lists DriveFile objects (implementing the FileLike base class):

let client = GoogleDriveClient::from_service_account_file("./credentials.json")?;
let source = GoogleDriveSource::new(client, vec!["1abc...xyz".to_string()]);

for file in source.list_files().await? {
    let text = file.read_text().await?;
    // ...
}

list_files() returns the whole listing as a Vec<DriveFile>; there is no async iterator. A listed file carries the client, so file.read_text() works; client.read_text(&file) reads through a client you hold.

Keyed iteration with items()

items() returns (String, DriveFile) pairs keyed by the file’s Google Drive ID. IDs remain stable across renames and avoid collisions when files share a name, including within the same folder. This is useful with mount_each!:

let files = source.items().await?;
mount_each!(files, |file| process_file(ctx, file, table)).await?;

file.key() is the same Google Drive ID.

Filtering by MIME type

Use mime_types to restrict which files are returned:

let source = GoogleDriveSource::new(client, vec!["1abc...xyz".to_string()])
    .mime_types(vec!["application/pdf".to_string(), "text/plain".to_string()]);

Google Workspace files (Docs, Sheets, Slides) are automatically exported:

list_files

A lower-level sync iterator for listing files:

pub struct GoogleDriveSourceSpec {
    pub client: GoogleDriveClient,
    pub root_folder_ids: Vec<String>,
    pub mime_types: Option<Vec<String>>,
}

pub async fn list_files(spec: GoogleDriveSourceSpec) -> Result<Vec<DriveFile>>

Parameters:

• spec — A GoogleDriveSourceSpec with the same fields as GoogleDriveSource: the client, the root folder IDs and the optional MIME types.

Returns: Every listed DriveFile, as a Vec.

DriveFile

DriveFile implements FileLike with Google Drive-specific behavior:

• Public fields file_id, name, path, mime_type, size, modified_time (RFC 3339, as Drive returns it) and md5_checksum (binary files only); info() returns them as a DriveFileInfo, and key() returns the file ID.
• file_path() — A DriveFilePath with path(), the path relative to the configured root folder, and file_id(); resolve() returns the Google Drive file ID. If roots overlap or a file is reachable more than once, it is emitted once using the first path encountered in breadth-first traversal. Each folder’s children are visited in order of name, then ID, so the same path is kept on every run.
• read() / read_text() — Downloads file content via the Google Drive API. A listed file carries the client; client.read(&file) and client.read_text(&file) read through a client you hold.

• Every Drive API request, listing and download alike, is retried with backoff (up to five more attempts) when Google answers with a server error, HTTP 429, or 403 with a rate-limit reason, or when the connection fails.
• Change detection uses the Drive MD5 checksum of binary files, so an unchanged file is not downloaded again. Google Docs, Sheets and Slides have no checksum; their exported content is hashed instead. DriveFile implements MemoInput: a memoized function that takes it runs again only for a file whose content changed. Its memo key holds only the file’s metadata, never the client.

Signing in with a personal account

A service account only sees folders shared with it. To read your own Drive, sign in with your Google account instead. The sign-in is a one-time interactive step, run from the shell, outside any CocoIndex run; it uses the OAuth flow Google recommends for command-line applications (an authorization code with PKCE, redirected to a listener on this machine’s loopback interface), from a Desktop app OAuth client. CocoIndex asks for read-only access to Drive (drive.readonly) and nothing else. CocoIndex’s use and transfer of information received from Google APIs adheres to the Google API Services User Data Policy, including the Limited Use requirements; see how CocoIndex handles Google user data.

CocoIndex’s own OAuth client is not in a release yet: Google has to verify it first. Until a release includes it, sign in with an OAuth client you create in your own Google Cloud project, as below. A release that includes it needs nothing but your Google account: cocoindex auth sign-in google-drive without a client file.

The built-in client is compiled in only when COCOINDEX_GOOGLE_OAUTH_CLIENT_ID and COCOINDEX_GOOGLE_OAUTH_CLIENT_SECRET are set at build time, as a release build sets them. A cocoindex crate or cocoindex CLI built from the git repository has none, so its sign-in takes your own client file.

Setting up the OAuth client

Once per Google Cloud project:

1. In the Google Cloud Console, create a project (or pick one) and enable the Google Drive API under APIs & Services > Library.
2. Under APIs & Services > OAuth consent screen (called Google Auth Platform in newer consoles), configure the consent screen: external user type, any app name, your email as the contact. Leave the publishing status at Testing and add your own Google account under Test users.
3. Under APIs & Services > Credentials, create an OAuth client ID of type Desktop app and download its JSON file.

What to expect from a client you own, as Google documents it:

• drive.readonly is a restricted scope, so Google would require app verification (and a security assessment) before the public could use the client. Neither applies to a client in Testing status used by its listed test users, which is the setup for reading your own Drive.
• In Testing status, Google expires the consent seven days after it was given: a run then fails with SignInRequiredError and you sign in again. Publishing the client to In production without verification removes the seven-day limit; the consent screen then shows an “unverified app” warning you pass through Advanced, and the client is limited to 100 users over its lifetime.
• The downloaded JSON contains a client secret. For a Desktop app Google treats it as not confidential, but keep the file out of git all the same; CocoIndex copies what it needs into the token file and never writes either into the state database, a memo key or a log.

Signing in

cocoindex auth sign-in google-drive ~/Downloads/client_secret_XXXX.json

Without the file argument, the command uses CocoIndex’s own OAuth client, and a release that doesn’t include one says so.

The command opens the consent page in your browser. Approve the request, including the Google Drive permission; the browser lands on a page that says the sign-in completed, and the terminal prints the signed-in address.

On a machine with no browser (over SSH, or with no display), the command prints the link instead. Open it in a browser on another machine and sign in; the browser then lands on http://127.0.0.1:<port>/?code=...&state=... and fails to load, since that address is the other machine. Copy the whole address from the address bar and paste it into the terminal. (If you forward the port with ssh -L <port>:127.0.0.1:<port> <host> in a second session, the page loads instead and nothing is left to paste.)

The tokens are stored at ~/.config/cocoindex/google_drive_token.json on Linux and macOS (under $XDG_CONFIG_HOME when set) and under %APPDATA%\cocoindex on Windows; --token-path picks another location.

cocoindex auth sign-out google-drive      # revokes the sign-in at Google and deletes the token file

These are commands of the cocoindex CLI for Rust projects. The Python CLI has the same two commands over the same token file, at the same default location, so one sign-in serves both SDKs. From code, UserAccount::new()?.sign_in(Some(client_secrets_path)).await? signs in with your own client and sign_in(None) with the client built into the release, when it has one; sign_out().await? revokes the sign-in and deletes the file. See Signing in from your own program.

Signing in from your own program

A program can sign in by itself instead of asking you to run the command first: check is_signed_in(), and call sign_in() when it is false, before the app runs.

• sign_in waits for a person, up to five minutes, to finish the consent in the browser. So call it before the run starts, never from inside the run.
• On a host with no browser, it prints the link and reads the address the browser lands on, pasted back into the terminal, as the command does.
• The command and the library use the same token file, at the same default location, so a sign-in with one serves the other.
• is_signed_in() reads only the token file. A stored sign-in that Google has since refused (the consent expired or was revoked) still counts; the run then fails with SignInError::SignInRequired, and you sign in again.

use std::path::{Path, PathBuf};

use cocoindex::connectors::gdrive::{GoogleDriveClient, UserAccount};
use cocoindex::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
    let account = UserAccount::new()?;
    if !account.is_signed_in() {
        // A client file of your own; `None` for CocoIndex's own client.
        let email = account.sign_in(Some(Path::new("client_secret.json"))).await?;
        println!("Signed in as {email}.");
    }
    let gdrive = GoogleDriveClient::from_user_account(&account)?;
    let root_folder_ids = vec!["1abc...xyz".to_string()];
    let app = Environment::builder()
        .db_path(PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(".cocoindex_db"))
        .provide_key(&GDRIVE, gdrive)
        .build()
        .await?
        .app("GoogleDriveTextEmbeddingRust")
        .await?;
    let stats = app.run(move |ctx| app_main(ctx, root_folder_ids)).await?;
    println!("{stats}");
    Ok(())
}

GDRIVE and app_main are those of the example below.

UserAccount

pub fn UserAccount::new() -> Result<Self>
pub fn UserAccount::with_token_path(token_path: impl Into<PathBuf>) -> Self
pub fn UserAccount::token_path(&self) -> &Path
pub async fn UserAccount::sign_in(&self, client_secrets_path: Option<&Path>) -> Result<String>
pub async fn UserAccount::sign_out(&self) -> Result<bool>
pub fn UserAccount::email(&self) -> Result<String>
pub fn UserAccount::is_signed_in(&self) -> bool

new() uses the default token path; with_token_path picks another location. sign_in returns the signed-in address. sign_out deletes the token file even when Google can’t be reached, and returns whether the file existed. email() fails with SignInError::SignInRequired when no sign-in is stored. is_signed_in() is false as well when the file exists but can’t be used, for example because other users can access it.

The handle a GoogleDriveClient signs with. GoogleDriveClient::from_user_account(&account) reads the token file at token_path (default: the path sign-in writes to) when it builds the client:

• the access token is refreshed with the stored refresh token as it expires, and the refreshed token is written back to the file;
• email() is the signed-in account’s address, and the client’s identity is that address: a refreshed token does not invalidate memoized work, a different account does;
• when no sign-in is stored, or Google refuses the stored one (the consent expired or was revoked, or the OAuth client was deleted, disabled or given a new secret), the call fails with SignInError::SignInRequired, whose message says which and to sign in again. Other sign-in failures are SignInError::Failed. SignInError::of(&error) recovers the SignInError from the SDK’s Error. The message never contains a token.

let account = UserAccount::new()?;
let client = GoogleDriveClient::from_user_account(&account)?;
let files = GoogleDriveSource::new(client, folder_ids).list_files().await?;
mount_each!(files.into_iter().map(|file| (file.key(), file)), |file| {
    process_file(ctx, file)
})
.await?;

Example

use std::path::PathBuf;

use cocoindex::connectors::gdrive::{DriveFile, GoogleDriveClient, GoogleDriveSource};
use cocoindex::prelude::*;

cocoindex::context_key!(static GDRIVE: GoogleDriveClient);

#[cocoindex::function(memo)]
async fn process_file(_ctx: &Ctx, file: DriveFile) -> Result<()> {
    let text = file.read_text().await?;
    // ... process the file content ...
    Ok(())
}

async fn app_main(ctx: Ctx, root_folder_ids: Vec<String>) -> Result<()> {
    let source = GoogleDriveSource::new(ctx.get_key(&GDRIVE)?.clone(), root_folder_ids);
    let files = source.items().await?;
    println!("indexing {} Google Drive files", files.len());

    mount_each!(files, |file| process_file(ctx, file)).await?;
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    let gdrive = GoogleDriveClient::from_service_account_file("./credentials.json")?;
    let root_folder_ids = vec!["1abc...xyz".to_string()];
    let app = Environment::builder()
        .db_path(PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(".cocoindex_db"))
        .provide_key(&GDRIVE, gdrive)
        .build()
        .await?
        .app("GoogleDriveTextEmbeddingRust")
        .await?;
    let stats = app.run(move |ctx| app_main(ctx, root_folder_ids)).await?;
    println!("{stats}");
    Ok(())
}

The full program, which chunks and embeds each file into Postgres, is examples/rust/gdrive_text_embedding.