Skip to content

CI: consolidate space freeing steps #18722

@Jefffrey

Description

@Jefffrey

Is your feature request related to a problem or challenge?

We've faced issues with storage space on our CI runners, so we introduced steps to free up space for them across various actions:

# Run extended tests (with feature 'extended_tests')
linux-test-extended:
name: cargo test 'extended_tests' (amd64)
needs: [linux-build-lib]
runs-on: ubuntu-latest
# note: do not use amd/rust container to preserve disk space
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
ref: ${{ github.event.inputs.pr_head_sha }} # will be empty if triggered by push
submodules: true
fetch-depth: 1
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
- name: Install Rust
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env
rustup toolchain install
- name: Install Protobuf Compiler
run: sudo apt-get install -y protobuf-compiler
# For debugging, test binaries can be large.
- name: Show available disk space
run: |
df -h

# Library and integration tests
linux-test:
name: cargo test (amd64)
needs: linux-build-lib
runs-on: ubuntu-latest
container:
image: amd64/rust
volumes:
- /usr/local:/host/usr/local
steps:
- name: Remove unnecessary preinstalled software
run: |
echo "Disk space before cleanup:"
df -h
# remove tool cache: about 8.5GB (github has host /opt/hostedtoolcache mounted as /__t)
rm -rf /__t/* || true
# remove Haskell runtime: about 6.3GB (host /usr/local/.ghcup)
rm -rf /host/usr/local/.ghcup || true
# remove Android library: about 7.8GB (host /usr/local/lib/android)
rm -rf /host/usr/local/lib/android || true
echo "Disk space after cleanup:"
df -h

linux-test-example:
name: cargo examples (amd64)
needs: linux-build-lib
runs-on: ubuntu-latest
container:
image: amd64/rust
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
submodules: true
fetch-depth: 1
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Rust Dependency Cache
uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
save-if: ${{ github.ref_name == 'main' }}
shared-key: "amd-ci-linux-test-example"
- name: Remove unnecessary preinstalled software
run: |
echo "Disk space before cleanup:"
df -h
apt-get clean
rm -rf /__t/CodeQL
rm -rf /__t/PyPy
rm -rf /__t/Java_Temurin-Hotspot_jdk
rm -rf /__t/Python
rm -rf /__t/go
rm -rf /__t/Ruby
echo "Disk space after cleanup:"
df -h

Describe the solution you'd like

We should look into consolidating them together if possible so can apply to all our runners/actions.

Need to consider that some actions are on bare machine runners and others are on containers.

Describe alternatives you've considered

No response

Additional context

See #18709

Metadata

Metadata

Assignees

No one assigned

    Labels

    development-processRelated to development process of DataFusionenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions