google-cloud-cpp

How-to Guide: Set Up your Development Environment

This guide is intended for contributors to the google-cloud-cpp libraries. This will walk you through the steps necessary to set up your development workstation to compile the code, run the unit and integration tests, and send contributions to the project.

• Packaging maintainers or developers who prefer to install the library in a fixed directory (such as /usr/local or /opt) should consult the packaging guide.
• Developers who prefer using a package manager such as vcpkg, or Conda, should follow the instructions for their package manager.
• Developers wanting to use the libraries as part of a larger CMake or Bazel project should consult the quickstart guides for the library or libraries they want to use.
• Developers wanting to compile the library just to run some examples or tests should consult the building and installing section of the top-level README file.
• Contributors and developers to google-cloud-cpp, this is the right document.

Table of Contents

• Linux
• Windows
• macOS
• Appendix: Linux VM on Google Compute Engine
• Appendix: Windows VM on Google Compute Engine

Linux

Install the dependencies needed for your distribution. The top-level README file lists the minimal development tools necessary to compile google-cloud-cpp. But for active development you may want to install additional tools to run the unit and integration tests.

These instructions will describe how to install these tools for Ubuntu 18.04 (Bionic Beaver). For other distributions you may consult the Dockerfiles in ci/cloudbuild/dockerfiles/ If you use a different distribution, you will need to use the corresponding package manager (dnf, zypper, apk, etc.) and find the corresponding package names.

Install the basic development tools:

sudo curl -o /usr/bin/bazelisk -sSL "https://github.com/bazelbuild/bazelisk/releases/download/v1.18.0/bazelisk-linux-amd64"
sudo chmod +x /usr/bin/bazelisk
sudo ln -s /usr/bin/bazelisk /usr/bin/bazel

sudo apt-get update
sudo apt-get --no-install-recommends install -y apt-transport-https apt-utils \
        automake build-essential cmake ca-certificates curl git \
        gcc g++ libc-ares-dev libc-ares2 libcurl4-openssl-dev libre2-dev \
        libssl-dev m4 make pkg-config tar wget zlib1g-dev
sudo apt install -y ninja-build
sudo apt-get install curl zip unzip tar

Install docker and its buildx component:

sudo apt install -y docker.io docker-buildx

Configure your user to launch docker without needing sudo:

sudo usermod -aG docker $USER

Update all packages and reboot:

sudo apt upgrade
sudo shutdown -r now

Verify docker is working:

docker run hello-world

Clone the repository and run a test build:

git clone git@github.com:${GITHUB_USER}/google-cloud-cpp.git

time env -C google-cloud-cpp bazel build //google/cloud/storage/...
#
# real    6m36.547s
# user    0m0.825s
# sys     0m0.630s

You need to install the Google Cloud SDK. These instructions work for a GCE VM, but you may need to adapt them for your environment. Check the instructions on the Google Cloud SDK website for alternatives.

type gcloud || sudo google-cloud-cpp/ci/install-cloud-sdk.sh

By default, gcloud will use the GCE instance credentials. That may be enough in most cases, but you may need to use your own credentials:

/usr/local/google-cloud-sdk/bin/gcloud auth login

Don’t forget the second ~breakfast~ login:

/usr/local/google-cloud-sdk/bin/gcloud auth application-default login

Test checkers:

time env -C google-cloud-cpp ci/cloudbuild/build.sh -t checkers-pr
# The first time is slow, as it needs to create the docker image. The second
# time it should run
# real    0m30.997s
# user    0m0.282s
# sys     0m0.232s

(
echo ENABLED_FEATURES=storage,experimental-storage_grpc,opentelemetry
echo RUN_CLANG_TIDY_FIX=NO
) >$HOME/.cloudcxxrc
time env -C google-cloud-cpp ci/cloudbuild/build.sh --distro fedora-latest-cmake --build development
# The second run should get times such as
# real    1m48.347s
# user    0m0.259s
# sys     0m0.247s

Optional Tools

Most of the time we use these tools in docker images, so there is no need to install them in your development environment.

Install the buildifier tool, which we use to format BUILD.bazel files:

sudo curl -fsSL -o /usr/bin/buildifier https://github.com/bazelbuild/buildtools/releases/download/6.0.0/buildifier-linux-amd64
sudo chmod 755 /usr/bin/buildifier

Install shfmt tool, which we use to format our shell scripts:

sudo curl -L -o /usr/local/bin/shfmt \
    "https://github.com/mvdan/sh/releases/download/v3.4.3/shfmt_v3.4.3_linux_amd64"
sudo chmod 755 /usr/local/bin/shfmt

Install cmake_format to automatically format the CMake list files. We pin this tool to a specific version because the formatting changes when the “latest” version is updated, and we do not want the builds to break just because some third party changed something.

sudo apt install -y python3 python3-pip
pip3 install --upgrade pip
pip3 install cmake_format==0.6.13

Install `black``, which we use to format our Python scripts:

pip3 install black==22.3.0

Install mdformat, which we use to format our .md documents:

pip3 install mdformat-gfm==0.3.5 \
                 mdformat-frontmatter==0.4.1 \
                 mdformat-footnote==0.1.1

Install typos-cli for spell and typo checking:

sudo apt install cargo
cargo install typos-cli --version 1.16.1

Install the Python modules used in the integration tests:

pip3 install setuptools wheel
pip3 install git+https://github.com/googleapis/storage-testbench

Add the pip directory to your PATH:

export PATH=$PATH:$HOME/.local/bin

(Optional) Enable clang-based tooling in your IDE

Many IDEs use clang for smart code completion, refactoring, etc. To do this, clang needs to know how to compile each file. This is commonly done by creating a compile_commands.json file at the top of the repo, which CMake knows how to create for us.

For the purposes of clang-tooling, it’s best to build all the code and dependencies from source so that clangd can find all the symbols and jump you to the right spot in the source files. We prefer using vcpkg for this, so clone the vcpkg repo if you don’t already have it checked out somewhere.

VCPKG_ROOT=~/vcpkg  # We'll use this later
git clone https://github.com/microsoft/vcpkg "${VCPKG_ROOT}"

Next, use CMake with the CMAKE_EXPORT_COMPILE_COMMANDS option to compile the google-cloud-cpp code, and tell it to use vcpkg to build all the dependencies.

cmake -S . -B .build -G Ninja \
  --toolchain "${VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake" \
  -DGOOGLE_CLOUD_CPP_ENABLE_CCACHE=ON \
  -DGOOGLE_CLOUD_CPP_ENABLE_WERROR=ON \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cmake --build .build

Finally, symlink the .build/compile_commands.json file into the top of the repo.

ln -s .build/compile_commands.json .

Note: It’s also possible to create compile_commands.json using bazel, but it’s not quite as easy. If you want to do that, take a look at https://github.com/grailbio/bazel-compilation-database.

Clone and compile google-cloud-cpp

You may need to clone and compile the code as described here.

Run the tests using:

env -C cmake-out/home ctest --output-on-failure -LE integration-test

Run the Google Cloud Storage integration tests:

env -C cmake-out/home \
    $PWD/google/cloud/storage/ci/run_integration_tests_emulator_cmake.sh .

Run the Google Cloud Bigtable integration tests:

env -C cmake-out/home \
    $PWD/google/cloud/bigtable/ci/run_integration_tests_emulator_cmake.sh .

Installing Docker

You may want to install Docker. This will allow you to use the build scripts to test on multiple platforms.

Once Docker is installed, to avoid needing to prepend sudo to Docker invocations, add yourself to the Docker group:

sudo usermod -aG docker $USER

Windows

If you mainly use Windows as your development environment, you need to install a number of tools. We use Chocolatey to drive the installation, so you would need to install it first. This needs to be executed in a cmd.exe shell, running as the Administrator:

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -ExecutionPolicy Bypass -Command "iex (
(New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

Then you can install the dependencies in the same shell:

choco install -y visualstudio2019community visualstudio2019-workload-nativedesktop visualstudio2019buildtools
choco install -y git cmake ninja bazelisk

Clone google-cloud-cpp

You may need to create a new key pair to connect to GitHub. Search the web for how to do this. Then you can clone the code:

> cd \Users\%USERNAME%
> git clone git@github.com:<GITHUB-USERNAME_HERE>/google-cloud-cpp.git
> cd google-cloud-cpp

Use the CI scripts to compile google-cloud-cpp

You can use the CI driver scripts to compile the code. You need to load the MSVC environment variables:

> set MSVC_VERSION=2022
> call "c:\Program Files (x86)\Microsoft Visual Studio\%MSVC_VERSION%\Community\VC\Auxiliary\Build\vcvars64.bat"

Or to setup for 32-bit builds:

> set MSVC_VERSION=2022
> call "c:\Program Files (x86)\Microsoft Visual Studio\%MSVC_VERSION%\Community\VC\Auxiliary\Build\vcvars32.bat"

Then run the CI scripts, for example, to compile and run the tests with CMake in debug mode:

> cd google-cloud-cpp
> powershell -exec bypass ci/kokoro/windows/build.ps1 cmake-debug

While to compile and run the tests with Bazel in debug mode you would use:

> cd google-cloud-cpp
> powershell -exec bypass ci/kokoro/windows/build.ps1 bazel-debug

We used to have instructions to setup manual builds with CMake and Bazel on Windows, but they quickly get out of date.

macOS

:warning: This is a work in progress. These instructions might be incomplete because we don’t know how to create a “fresh” macOS install to verify we did not miss a step.

To build on macOS you will need the command-line development tools, and some packages from homebrew. Start by installing the command-line development tools:

sudo xcode-select --install

Verify that worked by checking your compiler:

c++ --version
# Expected output: Apple clang .*

Install homebrew, their home page should have up to date instructions. Once you have installed homebrew, use it to install some development tools:

brew update
brew install cmake bazel openssl git ninja

You may also want to install ccache to improve the rebuild times, and google-cloud-sdk if you are planning to run the integration tests.

Then clone the repository:

cd $HOME
git clone git@github.com:<GITHUB-USERNAME_HERE>/google-cloud-cpp.git
cd google-cloud-cpp

Manual builds with CMake

The guide generally works for macOS too. We recommend using Ninja and vcpkg for development. When using vcpkg you should disable the OpenSSL checks:

git clone -C $HOME https://github.com/microsoft/vcpkg.git
env VCPKG_ROOT=$HOME/vcpkg $HOME/vcpkg/bootstrap-vcpkg.sh
cmake -GNinja -S. -Bcmake-out/ \
  -DCMAKE_TOOLCHAIN_FILE=$HOME/vcpkg/scripts/buildsystems/vcpkg.cmake \
  -DGOOGLE_CLOUD_CPP_ENABLE_MACOS_OPENSSL_CHECK=OFF
cmake --build cmake-out

Running the CI scripts

The CI scripts follow a similar pattern to the scripts for Linux and Windows:

./ci/kokoro/macos/build.sh bazel        # <-- Run the `bazel` CI build
./ci/kokoro/macos/build.sh cmake-vcpkg  # <-- Build with CMake

Appendix: Linux VM on Google Compute Engine

From time to time you may want to setup a Linux VM in Google Compute Engine. This might be useful to run performance tests in isolation, but “close” to the service you are doing development for. The following instructions assume you have already created a project:

PROJECT_ID=$(gcloud config get-value project)
# Or manually set it if you have not configured your default project:

Select a zone to run your VM:

gcloud compute zones list
$ ZONE=... # Pick a zone.

Select the name of the VM:

VM=... # e.g. VM=my-windows-devbox

Then create the virtual machine using:

IMAGE_PROJECT=ubuntu-os-cloud

IMAGE=$(gcloud compute images list \
    --project=${IMAGE_PROJECT} \
    --filter="family ~ ubuntu-2204" \
    --sort-by=~name \
    --format="value(name)" \
    --limit=1)

PROJECT_NUMBER=$(gcloud projects list \
    --filter="project_id=${PROJECT_ID}" \
    --format="value(project_number)" \
    --limit=1)

gcloud compute instances create ${VM} \
    --project=${PROJECT_ID} \
    --service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
    --create-disk=auto-delete=yes,boot=yes,device-name=${VM},image=projects/${IMAGE_PROJECT}/global/images/${IMAGE},mode=rw,size=2048,type=projects/${PROJECT_ID}/zones/${ZONE}/diskTypes/pd-balanced \
    --zone=${ZONE} \
    --machine-type=n2d-standard-128 \
    --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --labels=goog-ec-src=vm_add-gcloud \
    --reservation-affinity=any

Update the ssh configuration to include this new VM:

# You need to do this every time you add GCE instances.
gcloud compute config-ssh

# Googlers should do this once. It configures SSH to use a tunnel and to forward
# the authentication agent on all VMs in your project.
cat >>.ssh/config <__EOF__
Host *.${PROJECT_ID}
    ProxyCommand=corp-ssh-helper %h %p
    ForwardAgent yes
__EOF__

Then login using:

ssh -A ${VM}.${ZONE}.${PROJECT_ID}

Appendix: Windows VM on Google Compute Engine

If you do not have a Windows workstation, but need a Windows development environment to troubleshoot a test or build problem, it might be convenient to create a Windows VM. The following commands assume you have already created a project:

$ PROJECT_ID=... # Set to your project id

Select a zone to run your VM:

$ gcloud compute zones list
$ ZONE=... # Pick a zone close to where you are...

Select the name of the VM:

$ VM=... # e.g. VM=my-windows-devbox

Then create the virtual machine using:

$ IMAGE=$(gcloud compute images list \
    --sort-by=~name \
    --format="value(name)" \
    --limit=1)
$ PROJECT_NUMBER=$(gcloud projects list \
    --filter="project_id=${PROJECT_ID}" \
    --format="value(project_number)" \
    --limit=1)
$ gcloud compute --project "${PROJECT_ID}" instances create "${VM}" \
  --zone "${ZONE}" \
  --image "${IMAGE}" --image-project "windows-cloud" \
  --boot-disk-size "1024" --boot-disk-type "pd-standard" \
  --boot-disk-device-name "${VM}" \
  --service-account "${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --machine-type "n1-standard-8" \
  --subnet "default" \
  --maintenance-policy "MIGRATE" \
  --scopes "https://www.googleapis.com/auth/bigtable.admin","https://www.googleapis.com/auth/bigtable.data","https://www.googleapis.com/auth/cloud-platform"

Reset the password for your account:

$ gcloud compute --project "${PROJECT_ID}" reset-windows-password --zone "${ZONE}" "${VM}"

Save that password in some kind of password manager. Then connect to the VM using your favorite RDP client. The Google Cloud Compute Engine documentation suggests some third-party clients that may be useful.

This site is open source. Improve this page.