feat(ci): Implement CI/CD pipeline for parallel Ubuntu Docker builds (#4952)

This PR establishes a robust CI/CD pipeline for building and publishing
Docker images, addressing the requirements of the Ubuntu 24.04
migration. The new system introduces separate, parallel build workflows
for Pull Requests and merges to the master branch.

Part of: [b/441792502](b/441792502)

***

### Key Changes

- **New CI/CD Workflow:** Implemented a two-stage pipeline for Docker
image builds:
1. **Pull Request Verification:** A new trigger, `Build-Base-Images-PR`,
automatically runs on PRs that modify files in the `docker/` directory.
It performs a parallel, **build-only** check for all Ubuntu versions to
validate the Dockerfiles. The success of this build is required for
merging.
2. **Master Branch Deployment:** A second new trigger,
`Build-And-Push-Base-Images-Master`, runs upon merging to `master`. It
executes the same parallel build process but also **pushes** the tagged
images to the Artifact Registry.

- **Parallel Execution:** The `docker/cloudbuild.yaml` configuration was
updated to run the builds for `latest`, `ubuntu-20-04`, and
`ubuntu-24-04` simultaneously, significantly reducing pipeline execution
time.

- **Conditional Push Logic:** The `docker/build.sh` script was
refactored to accept a `--no-push` flag, allowing the CI pipeline to
control whether images are only built (for PRs) or built and pushed (for
master).

***

### How to Verify

1. **Pull Request:** Observe that the `Build-Base-Images-PR` check has
run and passed on this PR, as it contains changes in the `docker/`
directory.
2. **Master Branch:** After this PR is merged, the
`Build-And-Push-Base-Images-Master` trigger will be active. To verify
its functionality, a subsequent PR that modifies a file in `docker/` can
be merged. The Cloud Build dashboard for the `clusterfuzz-images`
project should then show this trigger running and successfully pushing
the images.

Author: hunsche

.gitignore

@@ -55,3 +55,7 @@ bazel-*
 *.tfstate*
 *.lock.hcl
 **/.terraform/*
+
+# Ignore temporary build files.
+docker/base/Pipfile
+docker/base/Pipfile.lock

Pipfile.lock

@@ -1,33 +1,61 @@
-# Docker Build Instructions
+# Docker Images
 
-## Local testing
+This directory contains the Dockerfiles for various images used by ClusterFuzz.
 
-You can build a specific image locally using the `docker build` command from the root of the **clusterfuzz** repository.
+## Building Images Locally
 
-For example, to build the `base` image, run the following command from the root of the **clusterfuzz** repository:
+The `build.sh` script is the primary way to build images locally. It handles the complexities of build contexts and tagging for all images.
+
+### Prerequisites
+
+Ensure you are in the `docker/` directory before running the script:
+```bash
+cd docker
+```
+
+### Usage
+
+The script builds all images defined in its configuration. You must specify the Ubuntu version to build for.
 
 ```bash
-docker build -f docker/base/Dockerfile docker/base
+# Usage: ./build.sh <ubuntu-version> [git-hash] [--no-push]
 ```
 
-### Command Explanation
+-   `<ubuntu-version>`: **(Required)** The Ubuntu version to build (e.g., `20.04`, `24.04`). The script will look for a corresponding `.Dockerfile` for each image.
+-   `git-hash`: (Optional) A git hash to use for tagging the image. Defaults to the current `HEAD`.
+-   `--no-push`: (Optional) A flag to prevent the script from pushing the built images to the container registry. This is useful for local testing.
+
+### Examples
+
+**1. Build all `ubuntu-24.04` images for local testing:**
+```bash
+./build.sh ubuntu-24-04 --no-push
+```
 
-*   `docker build`: The standard command to build a Docker image.
-*   `-f docker/base/Dockerfile`: This flag specifies the path to the `Dockerfile` to be used, relative to the repository root.
-*   `docker/base`: This is the "build context". The Docker daemon will have access to all files and folders within this path. This is necessary for the `COPY` instructions in the `Dockerfile`.
+**2. Build and push all `ubuntu-20.04` images:**
+```bash
+./build.sh ubuntu-20.04
+```
 
-You can adapt this command to build other images by changing the path to the `Dockerfile` and the build context accordingly.
+## Manual Build (for a single image)
 
-## Production
+While using `build.sh` is recommended, you can build a specific image manually. You must run the `docker build` command from the **root of the clusterfuzz repository**.
 
-To build all images on container builder, run:
+For example, to build the `base` image for `ubuntu-24.04`:
 
 ```bash
-./build_on_container_builder.sh
+# Note: This command requires manually handling the build context.
+# The build.sh script handles this automatically.
+docker build -f docker/base/ubuntu-24-04.Dockerfile docker/base
 ```
 
-Note that your checkout needs to be on the latest deployed commit.
-You also need to have access to the `clusterfuzz-images` project.
+## Production Builds
+
+Production images are built using Google Cloud Build. The configuration is defined in `cloudbuild.yaml`, which uses the `build.sh` script. The build process is automated with the following triggers:
+
+-   **Pull Requests:** A trigger runs on every pull request that modifies files under the `docker/` directory. It builds all images with the `--no-push` flag to validate the changes without deploying them.
+-   **Push to `master`:** A trigger runs on every push to the `master` branch that modifies files under the `docker/` directory. It builds and pushes all images to the container registry, making them available for production use.
+
 ## Docker Image Dependency Tree
 
 ```mermaid
@@ -79,4 +107,4 @@ graph TD
     H --> O;
 
     O --> P;
-```
+```

docker/README.md

@@ -128,7 +128,7 @@ RUN locale-gen en_US.UTF-8
 ENV LANG en_US.UTF-8
 ENV PYTHONIOENCODING UTF-8
 
-COPY setup_common.sh setup_clusterfuzz.sh start_clusterfuzz.sh setup_mock_metadata.sh Pipfile Pipfile.lock start.sh /data/
+COPY Pipfile Pipfile.lock setup_common.sh setup_clusterfuzz.sh start_clusterfuzz.sh setup_mock_metadata.sh start.sh /data/
 RUN cd /data && \
     # Make pip3.11 the default so that pipenv install --system works.
     mv /usr/local/bin/pip3.11 /usr/local/bin/pip && \

docker/base/ubuntu-24-04.Dockerfile

@@ -31,14 +31,48 @@ IMAGES=(
   gcr.io/clusterfuzz-images/fuchsia
 )
 
-# The first argument is the version tag, e.g., 'latest', 'ubuntu-20-04'.
-VERSION_TAG=${1:-latest}
-# The second argument is the git hash.
-GIT_HASH_ARG=${2}
+# Default values
+VERSION_TAG="latest"
+GIT_HASH_ARG=""
+PUSH="true"
+NEEDS_ROOT_PIPFILE=false
+
+# Set up a trap to clean up Pipfiles on exit.
+function cleanup() {
+  if [[ "$NEEDS_ROOT_PIPFILE" == "true" ]]; then
+    rm -f base/Pipfile base/Pipfile.lock
+  fi
+}
+trap cleanup EXIT
+
+# Parse command-line arguments
+# The first two arguments are positional for backwards compatibility.
+if [ -n "$1" ] && ! [[ "$1" =~ ^-- ]]; then
+    VERSION_TAG="$1"
+    shift
+fi
+if [ -n "$1" ] && ! [[ "$1" =~ ^-- ]]; then
+    GIT_HASH_ARG="$1"
+    shift
+fi
+
+# Parse optional flags
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --no-push) PUSH="false";;
+        "") ;; # Ignore empty arguments, which can be passed by Cloud Build.
+        *) echo "Unknown parameter passed: $1"; exit 1 ;;
+    esac
+    shift
+done
 
 function docker_push {
-  docker push "$image_with_version_tag"
-  docker push "$image_with_stamp"
+  if [ "$PUSH" == "true" ]; then
+    docker push "$image_with_version_tag"
+    docker push "$image_with_stamp"
+  else
+    echo "Skipping push for $image_with_version_tag."
+  fi
 }
 
 if [ -z "$GIT_HASH_ARG" ]; then
@@ -60,7 +94,7 @@ for image_name in "${IMAGES[@]}"; do
   if [ "$VERSION_TAG" == "latest" ]; then
     dockerfile="$image_dir/Dockerfile"
   else
-    dockerfile="$image_dir/$VERSION_TAG.Dockerfile"
+    dockerfile="$image_dir/${VERSION_TAG}.Dockerfile"
   fi
 
   if [ ! -f "$dockerfile" ]; then
@@ -71,9 +105,27 @@ for image_name in "${IMAGES[@]}"; do
   image_with_version_tag="$image_name:$VERSION_TAG"
   image_with_stamp="$image_name:$stamp"
 
+  # Copy Pipfile to base for the ubuntu-24.04 build, as it's
+  # needed but not in the build context.
+  if [[ "$image_dir" == "base" && "$dockerfile" == *"ubuntu-24-04"* ]]; then
+    NEEDS_ROOT_PIPFILE=true
+    cp ../Pipfile ../Pipfile.lock base/
+  fi
+
   docker build -t "$image_with_version_tag" -f "$dockerfile" "$image_dir"
+
+  # Clean up the copied files.
+  if [[ "$NEEDS_ROOT_PIPFILE" == "true" ]]; then
+    rm base/Pipfile base/Pipfile.lock
+    NEEDS_ROOT_PIPFILE=false
+  fi
+
   docker tag "$image_with_version_tag" "$image_with_stamp"
   docker_push
 done
 
-echo "Built and pushed images successfully for version $VERSION_TAG with stamp $stamp"
+if [ "$PUSH" == "true" ]; then
+  echo "Built and pushed images successfully for version $VERSION_TAG with stamp $stamp"
+else
+  echo "Built images successfully (without push) for version $VERSION_TAG with stamp $stamp"
+fi

docker/build.sh

@@ -11,6 +11,7 @@ FROM gcr.io/clusterfuzz-images/base:ubuntu-24-04
 ENV UPDATE_WEB_TESTS True
 
 # Note: snapcraft installation seems to always fail.
+RUN ln -s /usr/local/bin/python3.11 /usr/local/bin/python3
 RUN echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | debconf-set-selections &&     curl 'https://chromium.googlesource.com/chromium/src/+/main/build/install-build-deps.py?format=TEXT' | base64 -d > /tmp/install-build-deps.py &&     sed -i s/snapcraft/doesnotexist/ /tmp/install-build-deps.py &&     sed -i "s/if requires_pinned_linux_libc():/if False:/" /tmp/install-build-deps.py &&     chmod u+x /tmp/install-build-deps.py &&     /tmp/install-build-deps.py --backwards-compatible --no-prompt --no-chromeos-fonts --syms --lib32
 
 

docker/chromium/base/ubuntu-24-04.Dockerfile

@@ -15,29 +15,38 @@
 steps:
 - name: 'gcr.io/cloud-builders/docker'
   id: 'Build latest'
+  waitFor: ['-']
   entrypoint: bash
+  dir: 'docker'
   args:
   - -ex
   - build.sh
   - latest
   - ${_GIT_HASH}
+  - ${_PUSH_FLAG}
 - name: 'gcr.io/cloud-builders/docker'
   id: 'Build Ubuntu 20.04'
+  waitFor: ['-']
   entrypoint: bash
+  dir: 'docker'
   args:
   - -ex
   - build.sh
   - ubuntu-20-04
   - ${_GIT_HASH}
+  - ${_PUSH_FLAG}
 - name: 'gcr.io/cloud-builders/docker'
   id: 'Build Ubuntu 24.04'
+  waitFor: ['-']
   entrypoint: bash
+  dir: 'docker'
   args:
   - -ex
   - build.sh
   - ubuntu-24-04
   - ${_GIT_HASH}
+  - ${_PUSH_FLAG}
 timeout: 14400s
 options:
-  machineType: N1_HIGHCPU_32
+  machineType: E2_HIGHCPU_32
   diskSizeGb: 500