Skip to content

EKS Pod Identity support for Metrics provider with Amazon Managed Prometheus #4536

@punkwalker

Description

@punkwalker

Checklist:

  • I've included steps to reproduce the bug.
  • I've included the version of argo rollouts.

Describe the bug
Pod Identity is not supported, especially for metrics provider using Amazon Managed Prometheus. It uses SigV4, and the SIGV4 signer does not support Pod Identity Association as it is using aws-go-sdk=v1.38.35 whereas minimum required SDK version is v1.47.11.

We first need to bump up the version for aws-go-sdk in prometheus Sigv4 signer and then bump it's version here when the new release is cut.

To Reproduce

  • Create Pod Identity Association for argo-rollouts
  • Create AnalysisRun/AnalysisTemplate with Metrics Provider using SigV4
  • The AnalysisRun will fail with error:
    2025/11/11 19:50:33 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed.
    11-11T19:39:08Z\",\"lastUpdateTime\":\"2025-11-11T19:39:08Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2025-11-11T19:51:13Z\",\"lastUpdateTime\":\"2025-11-11T19:51:13Z\",\"message\":\"Rollout aborted update to revision 2: Metric \\\"metric[0]-rust-backend: rocket_http_requests_total\\\" assessed Error due to consecutiveErrors (5) \\u003e consecutiveErrorLimit (4): \\\"Error Message: could not get SigV4 credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see
    

Expected behavior
EKS Pod Identity should work with Metrics provider and SigV4

Screenshots

Version

v1.8.2

Logs

time="2025-11-11T19:50:33Z" level=info msg="Enqueueing parent of team-rust/rust-backend-67b6f6d884-2-8.2: Rollout team-rust/rust-backend"
2025/11/11 19:50:33 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed. <nil>
2025/11/11 19:50:33 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed. <nil>
time="2025-11-11T19:50:33Z" level=error msg="Error creating SigV4 RoundTripper: could not get SigV4 credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2025-11-11T19:50:33Z" level=error msg="Error in getting metric provider :could not get SigV4 credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2025-11-11T19:50:33Z" level=info msg="Measurement Completed. Result: Error" analysisrun=rust-backend-67b6f6d884-2-8.2 metric="metric[1]-rust-backend: rocket_http_requests_duration_seconds_sum" namespace=team-rust

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions