Rollout Doesn't Rollback During A syncHandler/Reconciliation Error. Gets Stuck in a Infinite Progressing State

[satyamsareen007]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

I tried updating an argument for my analysis template using the fieldRef/ valueFrom way. Argo Rollouts controller was not able to find the field under the provided path and the Rollout is stuck in an infinte Progressing state.
The controller has alreay retired close to 5000 times.
This looks like a bug, if the controller is not able find the field due to missing a value or a incorrectly typed path, controller should retry only upto a configurable finite number of times before rolling back and entering a degraded state.
Also, there are not statuses being emited abouth this by the Rollout, only way to check this is through logs.
If this happens in an actual deployment, there should also be a quick way to check it for eg. a status field gets updated which we can look out for/get notifications on.

Also, a separate question; does fieldRef only work for labels on the Rollout resource itself, does it not work for the labels under the template section which get added to the pod.

  args:
  - name: rollout-version
    valueFrom:
      fieldRef:
        fieldPath: spec.template.metadata.labels['version']

To Reproduce

Use the below Rollout config:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: usage-events-ingester
  namespace: usage-events-ingester
  labels:
    app: usage-events-ingester
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5}
      - analysis:
          templates:
            - templateName: example-template
          args:
          - name: rollout-version
            valueFrom:
              fieldRef:
                fieldPath: spec.template.metadata.labels['version']
      - setWeight: 100
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: usage-events-ingester
  template:
    metadata:
      labels:
        app: usage-events-ingester
        version: "3"
    spec:
      containers:
      - name: rollouts-demo-analysis
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m

Expected behavior

Rollout should have rolled back after a configurable number of retries and entered into a "Degraded" state

Screenshots

Version

1.8.3

Logs

time="2025-10-22T07:28:22Z" level=info msg="rollout syncHandler queue retries: 3968 : key \"usage-events-ingester/usage-events-ingester\"" namespace=usage-events-ingester rollout=usage-events-ingester
time="2025-10-22T07:28:22Z" level=error msg="invalid path spec.template.metadata.labels['version'] in rollout" error="<nil>"
time="2025-10-22T07:28:32Z" level=info msg="Started syncing rollout" generation=18 namespace=usage-events-ingester resourceVersion=1119184795 rollout=usage-events-ingester
time="2025-10-22T07:28:32Z" level=info msg="No TrafficRouting Reconcilers found" namespace=usage-events-ingester rollout=usage-events-ingester
time="2025-10-22T07:28:32Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=usage-events-ingester rollout=usage-events-ingester
time="2025-10-22T07:28:32Z" level=error msg="roCtx.reconcile err invalid path spec.template.metadata.labels['version'] in rollout" generation=18 namespace=usage-events-ingester resourceVersion=1119184795 rollout=usage-events-ingester
time="2025-10-22T07:28:32Z" level=info msg="Reconciliation completed" generation=18 namespace=usage-events-ingester resourceVersion=1119184795 rollout=usage-events-ingester time_ms=3.248486
time="2025-10-22T07:28:32Z" level=error msg="rollout syncHandler error: invalid path spec.template.metadata.labels['version'] in rollout" namespace=usage-events-ingester rollout=usage-events-ingester
time="2025-10-22T07:28:32Z" level=info msg="rollout syncHandler queue retries: 3969 : key \"usage-events-ingester/usage-events-ingester\"" namespace=usage-events-ingester rollout=usage-events-ingester
time="2025-10-22T07:28:32Z" level=error msg="invalid path spec.template.metadata.labels['version'] in rollout" error="<nil>"

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[krapie]

I'm not sure whether we have maximum retry mechanism in the reconciler, but I've seen similar issue where the reconciler keeps retrying rollouts indefinitely and gets stuck. We might need a way to prevent this kind of problem, perhaps by adding a configurable maximum retry limit in the reconciler for repeated failures, as you mentioned, or by exploring other approaches.

Also, a separate question; does fieldRef only work for labels on the Rollout resource itself, does it not work for the labels under the template section which get added to the pod.

@satyamsareen007 Could you try using the dot notation format spec.template.metadata.labels.version? I noticed that the extractValueFromRollout() function is used for other valueFrom fields other than metadata prefix, and it seems to successfully extract values from the spec template section when using dot notation.

argo-rollouts/utils/analysis/factory.go

Line 267 in 92f4801

func extractValueFromRollout(r *v1alpha1.Rollout, path string) (string, error) {

[satyamsareen007]

Thanks @krapie for the workaround, spec.template.metadata.labels.version worked. Any specific reason why there are 2 formats for fieldRef?

Do you know if the updates to the retry mechanism are something that is already in the roadmap for future releases, or can it be prioritised?

[krapie]

Any specific reason why there are 2 formats for fieldRef?

There are two format types (dot notation and accessor notation) for reading object fields. It seems that extractValueFromRollout(), which is used for paths like $.spec.template.metadata, does not support the accessor notation format, whereas ExtractFieldPathAsString(), used for $.metadata, does support it.

Do you know if the updates to the retry mechanism are something that is already in the roadmap for future releases, or can it be prioritised?

Not sure about the plans for this kind of feature, but I think we can push to prioritize it. @zachaller might know more about the roadmap.

BTW, I think there are a few ways (or a combination of ways) to mitigate this issue:

• Add detection mechanism to monitor the reconciler for specific retry patterns or recurring errors during the rollout process for logging for further actions in reconcile progress
• Add notification mechanism to alert users when the reconciler encounters repeated retries or errors during rollout for manual abort or rollback
• Add maximum retry mechanism with configurable retries to rollback rollout on threshold breach (automatic, but may cause unexpected rollbacks during the rollout)

Hence, my first suggestion is to implement a detection/notification mechanism to notify users via the CLI or dashboard when syncHandler queue retries occur, so that manual rollback can be performed with human confirmation.

[krapie]

Also, if dot notation format spec.template.metadata.labels.version worked, we could update the Analysis documentation to mention that valueFrom supports retrieving values from anywhere in the object when using dot notation in fieldPath.

[zachaller]

Kubernetes controller in general should try forever to reconcile, ideally with an exponential backoff so hopefully rollouts did backoff and start reconciling less frequently. There are also probably improvements we could make to inform the users of erros weather it's k8s events or through conditions on the status.