How to recover from a "install retries exhausted"

[cer]

One of the consequences of missing HelmRepository secret is because those HelmReleases were not installed, a HelmRelease from a different HelmRepository was not able to become ready.

As a result, that HelmRelease ended up in the state "install retries exhausted".

To fix the problem I had to use flux suspend/resume. But why? Is this documented somewhere?

[stefanprodan]

Bumping the retries would fix it spec.install.remediation.retries. You could also set the retries to -1 and this will make Flux retry forever.

Docs: https://fluxcd.io/flux/components/helm/helmreleases/#configuring-failure-remediation

[cer]

Thanks for the tip.

But let's imagine that it "install retries exhausted" occurs.
Is flux suspend/resume the correct way to tell FluxCD to try again?
And, is this (state machine model + how to recover from failure) documented somewhere?

[stefanprodan]

Any changes to .spec of a HelmRelease resets the retry count, toggling .spec.suspend or changing something in the .spec.values has the same effect. This behavior is alluded to in the docs, maybe we need to call it out under failure remediation section cc @hiddeco

[cer]

In my scenario, HelmRelease X wasn't installed successfully - pod couldn't become ready - because it was depended upon HelmReleases Y and Z being installed successfully.

As a result, the solution didn't involve changing X. Instead, I had to tell Flux to try again with X.

[hiddeco]

In a future release there will be specific annotations to be able to control the reset of the failure count, or a one off "forced" retry.

This is however pending some bigger architectural changes, for which I am trying (very hard) to accumulate time for at the moment.

[youcefguichi]

hey @stefanprodan, i was reproducing the install retries exhausted and upgrade retries exhausted problem.

however when i fix the helmrelease values on my gitrepo, flux sync automatically and it doesn't stuck on the install retries exhausted state, i'm just curious if this would be always the case ? or it's not consistent and it could stuck sometimes ?

Or it used to stuck but you guys improved the helm controller lately ?

[hiddeco]

It gets stuck when it's out of retries according to the defined .spec.*.remediation.retries setting. Changing values will reset the counter, which may result in it then succeeding in e.g. one try, as described by Stefan in #3851 (reply in thread).

[youcefguichi]

Thanks @hiddeco for your reply, it helps us a lot on the investigation that we were doing about flux's helm controller and how it works and deals with errors, we tried to put together in a simple article the lessons we gained from our experience of the upgrade retries exhausted issue.

the article: https://gimlet.io/blog/error-handling-in-helm-controller-how-to-solve-the-infamous-upgrade-retries-exhausted-error

[smeeklai]

Great article

[pavel-mikhalchuk]

Hey @youcefguichi ,
The link is unfortunately broken. I'll add the archive version here - https://web.archive.org/web/20250213232116/https://gimlet.io/blog/error-handling-in-helm-controller-how-to-solve-the-infamous-upgrade-retries-exhausted-error