Using nat_router since 2.18.x indirectly leads to ImagePullBackOff when deploying multiple applications subsquently

[ToshY]

Problem

Since 2.18.x I've started using the nat_router options in combination with use_control_plane_lb.

  nat_router = {
    server_type = "cax21"
    location    = "nbg1"
    enable_sudo = false
    labels      = {}
  }

  # Control plane load balancer
  use_control_plane_lb = true
  control_plane_lb_type = "lb11"
  control_plane_lb_enable_public_interface = true

After the cluster has been created succesfully, I start deploying multiple applications (with Helm and/or kustomize) in quick succesion. Here's the list of following applications: longhorn, percona mysql, percona postgres, redis (with haproxy), rabbitmq, velero, cert-manager, error-pages, traefik (patched), whoami, uptime-kuma, meilisearch, crowdsec, redis-insight, argocd, it-tools, stirling-pdf. Between 15-20 applications are deployed within <5 minutes.

In 2.17.x I had no issues deploying all these applications, but in 2.18.x I started seeing ImagePullBackOff warnings for quite a lot of pods, sometimes on just 1 or 2 of the 4 agent nodes.

Failed to pull image "percona/percona-xtradb-cluster-operator:1.17.0": failed to pull and unpack image "docker.io/percona/percona-xtradb-cluster-operator:1.17.0": failed to resolve reference "docker.io/percona/percona-xtradb-cluster-operator:1.17.0": failed to do request: Head "https://registry-1.docker.io/v2/percona/percona-xtradb-cluster-operator/manifests/1.17.0": dial tcp: lookup registry-1.docker.io: Try again

While I queried Google/AI for this and it concluded this must be DNS networking issue (either on the pod or node), I'm confused on why/how this happened. My initial thought was that maybe because outgoing traffic is going through the NAT router, and it might get ratelimited by the docker registry, but the error does not seem to be that way. I recreated the pods and rebooted the specific nodes that had these issues, but it did not have any effect. At that point I destroyed the cluster (created with 2.18.1), recreated it with 2.17.4, and had no image pull issues when deploying all the above-mentioned applications.

Has anyone experienced a similiar issues as well since upgrading to 2.18.x ?

[dustinmoris]

I am experiencing the exact same issue. I've enabled image mirroring for all registries as well and it hasn't helped. It's definitely not rate limiting, because when I deploy new pods which pull new images which definitely wouldn't have been cached yet they succeed on all my nodes except one. All ImagePullBackOff are coming from the same single node. Given that I have also enabled the NAT router feature all traffic is going out via the same IP, so if it works on other nodes then it's not an IP issue. I'm using 2.18.3.

[ToshY]

Hey @dustinmoris

Are you also still experiencing this problem?

I tried again with 2.18.4 (new cluster), with and without nat_router (or explicitely setting use_control_plane_lb = false), and still I'm having ImagePullBackOff issues.

If this issue still persist for you as well, I will create an issue for this instead.

[dustinmoris]

I've not have that issue since upgrading to 2.18.4, but I also additionally enabled the Spiegel feature in k3s:

  k3s_registries = <<-EOT
    mirrors:
      "*":
  EOT

[ToshY]

Interesting, I willl try that as well. Thanks for the info 🙂

[ToshY]

I think I found what my problem was, which is that Bitnami made changes to their registries: bitnami/containers#83267. So apparently now it was just not pulling the images because bitnami has been changed to bitnamilegacy (which are no longer maintained/updated).

p.s.

I changed to using https://github.com/open-bitnami/containers instead, using oci://ghcr.io/open-bitnami/charts and manually setting the global.imageRegistry: ghcr.io/open-bitnami, image.repository: open-bitnami/<app> and image.tag in values.yaml for all Bitnami charts, and now it pulls images again without problems, even with nat_router and use_control_plane_lb = true. I kept the k3s_registries in my kube.tf like you suggested, seems like a good addition to have.

Thanks for the help.