Use zstd compression for docker images

[jchorl]

Is your feature request related to a problem? Please describe.

The megalinter images are big. They take a while to pull over the wire and extract. This hits cold CI workers particularly hard. It is great that there are variants but even those are large, and sometimes you want the whole gamut of linters.

Describe the solution you'd like

Docker/OCI have supported zstd for a while now. You can get better compression ratios with more efficient extraction.

I ran a quick test locally:

docker buildx build \       
    --platform linux/amd64 \
    --output type=oci,dest=image-zstd.tar.zstd,compression=zstd,compression-level=9,force-compression=true \
    .

docker buildx build \
    --platform linux/amd64 \
    --output type=oci,dest=image-gzip.tar.gz,compression=gzip,compression-level=9,force-compression=true \  
    .

du -sh image-*
4.5G    image-gzip.tar.gz
4.5G    image-raw.tar
3.8G    image-zstd.tar.zstd

I believe all that's required is to swap

megalinter/.github/workflows/deploy-RELEASE.yml

Line 110 in 9e9d746

push: true

with something like:

outputs: type=registry,compression=zstd,compression-level=9,force-compression=true

See docs on the github action: https://github.com/docker/build-push-action/tree/789f68658055d3ca993799b232b5c46dfe3f114d

Of note, I would expect this to slow down the build. It's a pay-once-every-pull-benefits type of change.

Describe alternatives you've considered

N/A

Additional context

N/A

[echoix]

If you’re set up for that, could you try to also note the time to run one or two workloads, then test by setting the exporter to estargz? https://github.com/containerd/stargz-snapshotter

The goal, if you are able to use containerd (that is now more closer to reality than it was a couple years ago), in theory you should be able to run the image while it is getting downloaded, and if some chunks are not used at all (because you don’t use that language), they might not even get downloaded.

It’s always been one of my wishes, but it also means that the github actions might need to be changed to be based on the npm runner, as when based on docker images, the GitHub runner downloads the full image before we can run any steps to configure the runner.

[jchorl]

I was able to do some benchmarking.

Push configurations:

• zstd: type=registry,compression=zstd,compression-level=9,force-compression=true
• esgz: type=registry,compression=estargz,compression-level=9,force-compression=true
• upstream: just docker pull/docker push to ECR

I was using EC2 and ECR with a VPC endpoint to S3, so network should be reasonably fast. Disk, maybe not so much. I ran on a c7i.4xl (16 cores).

I ran two benchmarks:

• No linters enabled, just to benchmark pull/extract
• All linters enabled, to stress the esgz online fs access

I ran this on a moderate sized codebase - not tiny, certainly not enterprise company size.

Results:

Not running any linters:

• zstd: 2:32.20
• esgz: 0:36.71
• esgz with no local esgz support (fallback behaviour): 2:39.42
• upstream: 2:46.37

Running all linters:

• zstd: 3m6.680s
• esgz: 1m28.601s
• esgz with no local esgz support: 3m23.820s
• normal: 3m25.648s

Note that configuring docker for esgz support is somewhat painful (it's not the default). As best I can tell, you need to grab the binaries, put them in PATH, and configure docker daemon/containerd to use that snapshotter. Which is to say, I suspect most folks are not going to have this configured by default.

My conclusion here is that zstd gives a modest improvement with no client changes needed. esgz gives a large improvement but client changes are required.

I'm not sure there's a hybrid approach available. I don't think docker has any way to say "use zstd if esgz is unavailable, else use esgz". My biased view is to just make the minor change to zstd to get a free simple improvement. But I understand if you'd rather go straight to esgz.

[echoix]

Wow! Many thanks for that benchmark! Thanks for sharing your experience on what it takes to make it use estargz on the client side. I knew it would fall back to be just a normal image if there isn’t any support. It’s been a while since we got introduced to it by a user.

I agree with your conclusion that zstd is easier. Just have to note if there is a specific docker engine version required if it ends up being our default. Most of the usage is done in CI, so should be fine.

For estargz, the numbers let me think that it could be a nice substitute to flavors, as you end up downloading only what you need. I’ll keep that in mind.