[RFC] [ci] move management of CI images into this repo?

[jameslamb]

Description

Recently, we faced rate-limiting issues from DockerHub on Azure DevOps jobs (#6866 (review)).

@shiyu1994 worked around those by manually pushing the relevant images to Azure Container Registry repos: #6866 (comment)

No one else was given access to those repos and no plan was communicated for how we might get updates from https://github.com/guolinke/lightgbm-ci-docker into those repos.

This inspired an idea.... what if we build those CI images from right here in the LightGBM repo and publish them to the GitHub Container Registry (ghcr.io)?

Opening this to discuss that.

Benefits of this work

• reduces the risk of of rate-limiting issues
  • ref: https://github.com/orgs/community/discussions/49671#discussioncomment-8795596
• removes a dependency on @guolinke 's personal GitHub account and personal DockerHub account
  • where no one else can be an admin: ef: [ci] allow pushes from any branch guolinke/lightgbm-ci-docker#30 (comment)
• consolidates permissions / access management... if you have write access to this repo, you have access to update the CI images
  • less risk of being blocked because one person is unavailable
  • removes reliance on @guolinke 's personal GitHub account and personal DockerHub credentials
• allows for PRs that both update the images AND update other code in the repo

Acceptance criteria

• LightGBM CI does not rely on https://github.com/guolinke/lightgbm-ci-docker or https://hub.docker.com/repository/docker/lightgbm/vsts-agent/tags

Approach

I built a proof-of-concept in another repo: jameslamb/lightgbm-dask-testing#75

I'm proposing the following:

• Dockerfiles and other context for the images are checked into source control here in LightGBM
• GitHub Actions workflow(s) for publishing those images, with the following characteristics:
  • only triggered by workflow_dispatch (a maintainer clicking a button in the GItHub UI)
  • can be triggered from any branch
• images are only for CI... not proposing (for now) using this mechanism to publish user-facing images that include LightGBM (ref: WIP: [docker] [ci] fix dockerfiles, test docker builds in CI #6638 (comment))

Notes

If folks are supportive of trying this, I'll open a draft PR showing what I'm thinking of. I think this could really help here, and make it easier to do updates like #5596.

cc @StrikerRUS @jmoralez @guolinke @shiyu1994 @borchero

[guolinke]

I am +1

[StrikerRUS]

Great idea!
Much better than my previous proposal to use Google mirror to overcome Docker rate limits.

[jameslamb]

Thanks! Sorry, I forgot to mention that Google mirror idea here too.

I think this GHCR thing could be really nice not ONLY for rate limits but also just for general development. I'll put up something to review as time allows 😁

[borchero]

Little late but definitely +1 from my side :)