Computing digest fails with `No space left on device` despite having space on disk

[PhysicsIsAwesome]

Computing digest fails with No space left on device despite having space on disk. Empty space on disk is about 10x the image size, nevertheless it fails.

Log:

admin@archlinux PIAbootc]$ sudo podman image ls
REPOSITORY                   TAG         IMAGE ID      CREATED       SIZE
ghcr.io/bootcrew/arch-bootc  latest      8e0a3630a6c8  4 hours ago   1.54 GB
localhost/bootc-bin          latest      39d5bc27d5a2  19 hours ago  5.23 GB
[admin@archlinux PIAbootc]$ sudo ./hack/build-sealed composefs-sealeduki-sdboot localhost/bootc-bin localhost/bootc --secret=id=secureboot_key,src=target/test-secureboot/db.key --secret=id=secureboot_cert,src=target/test-secureboot/db.crt
DEBUG argv0="bootc"
DEBUG Computing digest of 39d5bc27d5a245034df72c68074810eecc9526ee1b5de3fa50d44782a8934ead
DEBUG new_with_config: Spawned skopeo pid=2 config=ImageProxyConfig { authfile: None, auth_data: None, auth_anonymous: false, certificate_directory: None, decryption_keys: None, insecure_skip_tls_verification: None, debug: false, skopeo_cmd: Some(STORAGE_OPTS="additionalimagestore=/run/host-container-storage" "skopeo") }
DEBUG new_with_config: Remote protocol version: 0.2.8 config=ImageProxyConfig { authfile: None, auth_data: None, auth_anonymous: false, certificate_directory: None, decryption_keys: None, insecure_skip_tls_verification: None, debug: false, skopeo_cmd: Some(STORAGE_OPTS="additionalimagestore=/run/host-container-storage" "skopeo") }
DEBUG open_image: opening image self=ImageProxy imgref="containers-storage:39d5bc27d5a245034df72c68074810eecc9526ee1b5de3fa50d44782a8934ead"
DEBUG finish_pipe: closing pipe self=ImageProxy pipeid=PipeId(6)
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:39d5bc27d5a245034df72c68074810eecc9526ee1b5de3fa50d44782a8934ead", split: 6 }, size: 10922, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:39d5bc27d5a245034df72c68074810eecc9526ee1b5de3fa50d44782a8934ead", split: 6 } size=10922
DEBUG finish_pipe: closing pipe self=ImageProxy pipeid=PipeId(6)
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:242e6849d1986b715e02b4a9025f6457529e3f4470e8bff71388972fa714d8b8", split: 6 }, size: 3684185088, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:242e6849d1986b715e02b4a9025f6457529e3f4470e8bff71388972fa714d8b8", split: 6 } size=3684185088
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:37a50fe28757cfe63f9b025d38d51bf2ecd10710bc862b19e8cffa9731e503c1", split: 6 }, size: 1005191680, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:37a50fe28757cfe63f9b025d38d51bf2ecd10710bc862b19e8cffa9731e503c1", split: 6 } size=1005191680
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:aebb4d4a6be21dce981822641ca56fd82d4d58977403e17040a4e71ca91d0867", split: 6 }, size: 535168512, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:aebb4d4a6be21dce981822641ca56fd82d4d58977403e17040a4e71ca91d0867", split: 6 } size=535168512
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:aa505f34dbdb110b908f01ae6562e5f27a27176d9334d113572530bb419b1a85", split: 6 }, size: 418304, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:aa505f34dbdb110b908f01ae6562e5f27a27176d9334d113572530bb419b1a85", split: 6 } size=418304
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:d4057825841b6856b09b7feadae0a74f98cb7ed209b4c6e017897d50b876232e", split: 6 }, size: 41472, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d4057825841b6856b09b7feadae0a74f98cb7ed209b4c6e017897d50b876232e", split: 6 } size=41472
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:18d534537d356819a30d2f834c2bfe3b83563081978415efc63ad8f669a9646d", split: 6 }, size: 8704, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:18d534537d356819a30d2f834c2bfe3b83563081978415efc63ad8f669a9646d", split: 6 } size=8704
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:b8d8d24f98190ed3b9a854a8f9d1260893558133f5e16bc67087bd371defb811", split: 6 }, size: 4608, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:b8d8d24f98190ed3b9a854a8f9d1260893558133f5e16bc67087bd371defb811", split: 6 } size=4608
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:8efbe6f58af9febbf6c32f2c35c88cea2eeda70dabed5da56d0668c28e0d84f1", split: 6 }, size: 4096, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:8efbe6f58af9febbf6c32f2c35c88cea2eeda70dabed5da56d0668c28e0d84f1", split: 6 } size=4096
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:918b71a34a40328548dd68ae82c539650673207ba0874ddb5da10bfc50bfe3fe", split: 6 }, size: 1536, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:918b71a34a40328548dd68ae82c539650673207ba0874ddb5da10bfc50bfe3fe", split: 6 } size=1536
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:d65a1579178c68ea1a3cffffa91333528718a838f821ce217c909ccbaabc5ccf", split: 6 }, size: 1536, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d65a1579178c68ea1a3cffffa91333528718a838f821ce217c909ccbaabc5ccf", split: 6 } size=1536
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayer, digest: Digest { algorithm: Sha256, value: "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", split: 6 }, size: 1024, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", split: 6 } size=1024
error: Pulling image: Unable to pull container image containers-storage:39d5bc27d5a245034df72c68074810eecc9526ee1b5de3fa50d44782a8934ead: Failed to pull config Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:39d5bc27d5a245034df72c68074810eecc9526ee1b5de3fa50d44782a8934ead", split: 6 }, size: 10922, urls: None, annotations: None, platform: None, artifact_type: None, data: None }: No space left on device (os error 28)
[admin@archlinux PIAbootc]$ sudo ./hack/build-sealed composefs-sealeduki-sdboot localhost/bootc-bin localhost/bootc --secret=id=secureboot_key,src=target/test-secureboot/db.key --secret=id=secureboot_cert,src=target/test-secureboot/db.crt^C
[admin@archlinux PIAbootc]$ sudo df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
dev             7,7G       0  7,7G    0% /dev
run              20G    1,5M   20G    1% /run
efivarfs        256K    159K   93K   64% /sys/firmware/efi/efivars
/dev/vda2        99G     36G   57G   39% /
tmpfs           7,7G     84K  7,7G    1% /dev/shm
tmpfs           1,0M       0  1,0M    0% /run/credentials/systemd-journald.service
tmpfs           7,7G    4,0K  7,7G    1% /tmp
/dev/vda1      1022M    711M  312M   70% /boot
/dev/vda2        99G     36G   57G   39% /home
/dev/vda2        99G     36G   57G   39% /var/cache/pacman/pkg
/dev/vda2        99G     36G   57G   39% /var/log
tmpfs           1,6G    100K  1,6G    1% /run/user/1000
/dev/vdc3        39G    4,9G   32G   14% /mnt/archbootc
/dev/vdc2      1022M     64M  959M    7% /mnt/archbootc/boot

Dockerfile:

FROM ghcr.io/bootcrew/arch-bootc:latest

RUN rm -rf /opt
RUN mkdir -p /opt

RUN --mount=type=tmpfs,dst=/tmp --mount=type=tmpfs,dst=/var/cache/pacman --mount=type=tmpfs,dst=/usr/lib/sysimage/cache/pacman pacman -Syu --noconfirm && pacman -Scc --noconfirm

RUN --mount=type=tmpfs,dst=/tmp --mount=type=tmpfs,dst=/var/cache/pacman --mount=type=tmpfs,dst=/usr/lib/sysimage/cache/pacman pacman -Sy --noconfirm --needed nano fsverity-utils sddm plasma-meta konsole kate && pacman -Scc --noconfirm

RUN --mount=type=tmpfs,dst=/tmp --mount=type=tmpfs,dst=/var/cache/pacman --mount=type=tmpfs,dst=/usr/lib/sysimage/cache/pacman pacman -Sy whois --noconfirm --needed && pacman -Scc --noconfirm
# RUN usermod -p "$(echo "changeme" | mkpasswd -s)" root

RUN rm -rf /boot /var/cache && \
    mkdir /boot /var/cache

RUN bootc container lint

If I install less packages, resulting in a smaller image, it works as expected, including further steps like writing to filesystem and booting into it.

Base image Dockerfile: https://github.com/bootcrew/arch-bootc/blob/b2f9a83e2e9c8d7386dbc21db934cc5346771cb6/Containerfile

[chaserhkj]

IIRC skopeo uses some staging area (/var/tmp under most installations) to extract layers and if you get a No space left on device it is probably the staging area running out of space not your underlying physical disk.

If you are running bootc from within a container as it is instructed and that container image contains symlinks /var/tmp -> /tmp, then the inner tmpfs mount is used for the staging area and that could run out in your case since your default tmpfs size seems to be 7.7G and skopeo will try to uncompress the layers bloating tmpfs large enough to create an error.

This is very hard to track if you do podman run --rm for your bootc invocation since 1. the innner tmpfs mount is invisible from the root mount ns 2. container gets removed whatsoever and no traces is left.

I would suggest do podman run <flags> image bash and call bootc from there. After it drops out you back to the container shell you can check if inner /tmp or /var/tmp is full.

[PhysicsIsAwesome]

This was very helpful. Thank you.

/var/tmp was not a symlink to /tmp, but nevertheless a tmpfs mount with only 7.7G size. I fixed the error by adding --tmpfs /var/tmp:size=90% in the following podman run command:

https://github.com/bootc-dev/bootc/blob/bd9026997bfbc155f0b1efef6ed75523a5be40b8/hack/compute-composefs-digest#L10C1-L11C125

In the end my solution is still limited by the RAM size. It might be worth fixing this command in the compute-composefs-digest for other users as well.

[cgwalters]

This relates to containers/container-libs#144 (comment) which is a general issue about inefficiency of reading from containers-storage.