Image Update Automation - Can't clone git repo

[dzirkler]

So, starting here because I feel like I may be missing something simple, but can move to an Issue if needed.

Basically, my image updater cannot clone the same git repo that created the image updater in the first place! It always has some sort of connection issue reaching github.com.

You can see below that my ImageRepository and ImagePolicy seem to be processing correctly, and I can provide those definitions if it's helpful.

I tried adding the EXPERIMENTAL_GIT_TRANSPORT flag, and it had no effect.

The most puzzling part is the same repo is what's running Flux and it's obviously working because that's what's applying these CRDs, so I'm hoping this is just something dumb on my part. Is there something obvious I'm missing?

TIA for any assistance!

---

Additional Details:

NAME                    LAST SCAN                       SUSPENDED       READY   MESSAGE
imagerepository/esphome 2023-11-02T05:30:26-07:00       False           True    successful scan: found 1683 tags

NAME                    LATEST IMAGE                    READY   MESSAGE
imagepolicy/esphome     esphome/esphome:2023.10.5       True    Latest image tag for 'esphome/esphome' resolved to 2023.10.5

NAME                                            LAST RUN                        SUSPENDED       READY   MESSAGE

imageupdateautomation/esphome-imageupdater      2023-11-02T09:06:08-07:00       False           False   unable to clone 'ssh://[email protected]/my/fluxcd': ssh: handshake failed: read tcp 10.10.120.11:58530->20.29.134.23:22: read: connection timed out

Describing the image updater has these lines in the Events:

Events:
  Type     Reason  Age                     From                         Message
  ----     ------  ----                    ----                         -------
  Warning  error   36m                     image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 140.82.113.4:22: connect: connection timed out
  Warning  error   5m49s (x18 over 7h43m)  image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 20.29.134.23:22: connect: connection timed out
  Warning  error   3m16s                   image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': ssh: handshake failed: read tcp 10.10.120.11:58530->20.29.134.23:22: read: connection timed out
  Warning  error   62s (x2 over 38m)       image-automation-controller  unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 140.82.112.3:22: connect: connection timed out

Logs from the image-automation-controller pod look similar:

{"level":"error","ts":"2023-11-02T16:23:44.989Z","msg":"Reconcilererror","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"e4bc2dad-238a-4cd3-b865-38c309646ab0","error":"unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 20.29.134.23:22: connect: connection timed out"}
{"level":"info","ts":"2023-11-02T16:23:46.842Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"d741bd6a-0352-4a40-943a-5beca476de82"}
{"level":"info","ts":"2023-11-02T16:26:02.161Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"4dacd010-cffa-448e-8934-f95e175036fa"}
{"level":"info","ts":"2023-11-02T16:26:03.653Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"b0547cfd-f8a7-4c6b-a832-a612ab3e0a55"}
{"level":"info","ts":"2023-11-02T16:26:22.472Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"esphome-imageupdater","namespace":"esphome"},"namespace":"esphome","name":"esphome-imageupdater","reconcileID":"72f031fe-2a18-4d62-96ec-c3969680818b"}
{"level":"error","ts":"2023-11-02T16:28:56.340Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"83782a82-603e-4b09-854f-073e805eae4e","error":"unable to clone 'ssh://[email protected]/my/fluxcd': dial tcp 140.82.113.4:22: connect: connection timed out"}
{"level":"info","ts":"2023-11-02T16:28:58.330Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"a7f4f03 ││ 1-82d2-4ac2-a25f-34824b0bac95"}
{"level":"info","ts":"2023-11-02T16:29:15.990Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"cc249f9 ││ a-9d4a-477c-a13c-949574083ac9"}
{"level":"info","ts":"2023-11-02T16:29:17.444Z","msg":"no changes made in working directory; no commit","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io" ││ ,"controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"055cb01 ││ 9-4637-41e1-bcbf-769f616f45f5"}

ImageUpdateAutomation.yml:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: esphome-imageupdater
  namespace: esphome
spec:
  interval: 3m # 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: [email protected]
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: main
  update:
    path: clusters/home-k3s
    strategy: Setters
  suspend: false

[makkes]

Can you run the following command and paste its output here, please? I have a suspicion that the image-automation-controller Pod is running on a node from where it can't reach github.com.

kubectl get po -n flux-system -o wide

[dzirkler]

Looks like it's able to open the connection and get some text back:

<<K9s-Shell>> Pod: flux-system/image-automation-controller-754ff45ff8-hr6wg | Container: manager
~ $ nc github.com 22
SSH-2.0-babeld-f8b1fc6c
login
^Cpunt!

[GIG0i]

@dzirkler I'm aware this is a relatively old post however I wonder if you found a solution at the end?
I'm seeing the same exact issue with the added variable that when I kill the pod sometimes it starts working! 🤔

[dchien234]

Pump it up... Encountering the same issue where my ImageUpdateAutomation cannot clone the git repo with the same exception. Any ideas? really appreciate it.

[matheuscscp]

It's obviously a network issue. Check everything is right in the network path from the controller to your Git server.