Working with list operations

Some services return potentially large lists of items, such as rows or resource descriptions. To keep CPU and memory usage under control, services return these resources in pages: smaller subsets of the items with a continuation token to request the next subset.

Iterating over items by page can be tedious. The client libraries provide adapters to convert the pages into asynchronous iterators. This guide shows you how to work with these adapters.

Prerequisites

This guide uses the Secret Manager service to demonstrate list operations, but the concepts apply to other services as well.

You may want to follow the service quickstart, which shows you how to enable the service and ensure that you've logged in and that your account has the necessary permissions.

For complete setup instructions for the Rust libraries, see Setting up your development environment.

Dependencies

Add the Secret Manager library to your Cargo.toml file:

cargo add google-cloud-secretmanager-v1

Iterating list methods

To help iterate the items in a list method, the APIs return an implementation of the ItemPaginator trait. Introduce it into scope via a use declaration:

    use google_cloud_gax::paginator::ItemPaginator as _;

To iterate the items, use the by_item function.

    let mut list = client
        .list_secrets()
        .set_parent(format!("projects/{project_id}"))
        .by_item();
    while let Some(secret) = list.next().await {
        let secret = secret?;
        println!("  secret={}", secret.name)
    }

In rare cases, pages might contain extra information that you need access to. Or you may need to checkpoint your progress across processes. In these cases, you can iterate over full pages instead of individual items.

First introduce Paginator into scope via a use declaration:

    use google_cloud_gax::paginator::Paginator as _;

Then iterate over the pages using by_page:

    let mut list = client
        .list_secrets()
        .set_parent(format!("projects/{project_id}"))
        .by_page();
    while let Some(page) = list.next().await {
        let page = page?;
        println!("  next_page_token={}", page.next_page_token);
        page.secrets
            .into_iter()
            .for_each(|secret| println!("    secret={}", secret.name));
    }

Working with `futures::Stream`

You may want to use these APIs in the larger Rust ecosystem of asynchronous streams, such as tokio::Stream. This is readily done, but you must first enable the unstable-streams feature in the google_cloud_gax crate:

cargo add google-cloud-gax --features unstable-stream

The name of this feature is intended to convey that we consider these APIs unstable, because they are! You should only use them if you are prepared to deal with any breaks that result from incompatible changes to the futures::Stream trait.

The following examples also use the futures::stream::StreamExt trait, which you enable by adding the futures crate.

cargo add futures

Add the required use declarations:

    use futures::stream::StreamExt;
    use google_cloud_gax::paginator::ItemPaginator as _;

Then use the into_stream function to convert ItemPaginator into a futures::Stream of items.

    let list = client
        .list_secrets()
        .set_parent(format!("projects/{project_id}"))
        .by_item()
        .into_stream();
    list.map(|secret| -> gax::Result<()> {
        println!("  secret={}", secret?.name);
        Ok(())
    })
    .fold(Ok(()), async |acc, result| -> gax::Result<()> {
        acc.and(result)
    })
    .await?;

Similarly, you can use the into_stream function to convert Paginator into a futures::Stream of pages.

    let list = client
        .list_secrets()
        .set_parent(format!("projects/{project_id}"))
        .by_page()
        .into_stream();
    list.enumerate()
        .map(|(index, page)| -> gax::Result<()> {
            println!("page={}, next_page_token={}", index, page?.next_page_token);
            Ok(())
        })
        .fold(Ok(()), async |acc, result| -> gax::Result<()> {
            acc.and(result)
        })
        .await?;

Resuming list methods by setting next page token

In some cases, such as an interrupted list operation, you can set the next page token to resume paginating from a specific page.

    let page = client
        .list_secrets()
        .set_parent(format!("projects/{project_id}"))
        .send()
        .await;
    let page = page?;
    let mut next_page_token = page.next_page_token.clone();
    page.secrets
        .into_iter()
        .for_each(|secret| println!("    secret={}", secret.name));

    while !next_page_token.is_empty() {
        println!("  next_page_token={next_page_token}");

        let page = client
            .list_secrets()
            .set_parent(format!("projects/{project_id}"))
            .set_page_token(next_page_token)
            .send()
            .await;
        let page = page?;
        next_page_token = page.next_page_token.clone();

        page.secrets
            .into_iter()
            .for_each(|secret| println!("    secret={}", secret.name));
    }

Additional paginator technical details

The standard Google API List method follows the pagination guideline defined by AIP-158. Each call to a List method for a resource returns a page of resource items (e.g. secrets) along with a next-page token that can be passed to the List method to retrieve the next page.

The Google Cloud Client Libraries for Rust provide an adapter to convert the list RPCs as defined by AIP-4233 into a streams that can be iterated over in an async fashion.

What's next

Learn more about working with the Cloud Client Libraries for Rust:

Keyboard shortcuts

Google Cloud Client Libraries for Rust