Add tests for EKS Pod Identity #489

xdu31 · 2025-02-26T21:49:03Z

Issue #, if available:

Description of changes:

Add tests for EKS Pod Identity

Create a pipeline for testing EKS Pod Identity, after managed node group creation, create a Pod Identity Association in the default namespace on default service account, and then run load tests with Pod Identity.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

hakuna-matatah · 2025-02-27T00:45:33Z

Please let me know once this has been tested.

hakuna-matatah · 2025-02-28T18:19:50Z

tests/tasks/setup/eks/awscli-pod-identity-association.yaml

+  - name: endpoint
+    default: ""
+  - name: namespace-prefix
+    default: "default"


why do we need this ? can we add description to this ?

hakuna-matatah · 2025-02-28T18:22:07Z

tests/tasks/setup/eks/awscli-pod-identity-association.yaml

+    default: ""
+  - name: namespace-prefix
+    default: "default"
+  - name: namespaces


How about namespace-count ? or something that tells its the count of namespaces

hakuna-matatah · 2025-02-28T18:25:28Z

tests/tasks/setup/eks/awscli-pod-identity-association.yaml

+      MANAGED_POLICY_ARN="arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
+      TRUST_POLICY_FILE="pia-trust-policy.json"
+      # create a trust policy json file
+      cat > $TRUST_POLICY_FILE <<EOF


let's put trust policy under assets here https://github.com/awslabs/kubernetes-iteration-toolkit/tree/main/tests/assets and provide a url to pull it from there ?

we want to decouple this from the task def.

hakuna-matatah · 2025-02-28T18:26:41Z

tests/tasks/setup/eks/awscli-pod-identity-association.yaml

+      aws eks $ENDPOINT_FLAG --region $(params.region) list-pod-identity-associations --cluster-name $(params.cluster-name) --query 'associations'
+
+      echo "waiting for 30 seconds..."
+      sleep 30


is there a better way to know pod-identity-association went through than sleeping before we proceed ?

We have synced up offline on this. The answer is no, currently there is not an interface on this

hakuna-matatah · 2025-02-28T18:28:17Z

tests/tasks/setup/eks/awscli-cp-with-vpc.yaml

+      if [ -n "$(params.endpoint)" ]; then
+        ENDPOINT_FLAG="--endpoint $(params.endpoint)"
+      fi
+      aws eks $ENDPOINT_FLAG create-addon --cluster-name $(params.cluster-name) --addon-name eks-pod-identity-agent --addon-version v1.3.5-eksbuild.2


let's not hardcode the addon version as its going to change. Let's take that as a param and have it defaulted to this.

hakuna-matatah · 2025-02-28T18:30:36Z

tests/tasks/generators/clusterloader/load-pod-identity.yaml

+    default: "5000"
+  - name: cl2-default-qps
+    description: "default qps"
+    default: "500"


let's use the regular defaults, so there is no queueing for this if the server side has lower value than this. We can pass more aggressive ones in Pipeline params depending on type of cluster we test it on ?

hakuna-matatah · 2025-02-28T18:41:52Z

tests/tasks/generators/clusterloader/load-pod-identity.yaml

+  params:
+  - name: giturl
+    description: "git url to clone the package"
+    default: https://github.com/xdu31/perf-tests.git


let's not have forked branches of this. Its hard to maintain. Is there a plan to upstream whatever change you have ?

You can have this file under assets- https://github.com/kubernetes/perf-tests/blob/1df9f0af35a123c12a3324295774379411e1efb0/clusterloader2/testing/eks-pod-identity/pod-default.yaml

and copy it into CL2 before building it, so you have access.

And also, this config here is just bringing up the pods not measuring any pod start up latencies.

I think you can just use this for your test purposes - https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/scheduler-throughput/config.yaml and supply your pod-default.

I think you can just use this for your test purposes - https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/scheduler-throughput/config.yaml and supply your pod-default.

We can't use the same pod spec, we need to know the namespaces before hand so we can create the pod identity associations before this load tests, I definitely evaluated the option to re-use this

I'm not sure if upstream will agree to merge this, since this is eks specific test, and I can try to push for the merge and let's not make this the blocker, since we already use forked repo in some other tests, e.g https://github.com/awslabs/kubernetes-iteration-toolkit/blob/main/tests/tasks/generators/clusterloader/load.yaml#L12

hakuna-matatah · 2025-03-04T19:38:15Z

Summary of what @xdu31 and I synced offline:

Next steps for this PR

put eks-pod-identity config and pod-spec under assets folder, avoid the fork for maintenance overhead
Copy pod-spec into cl2 before building cl2
download this test config and refer from local dir and passing to cl2 as param as test config
add a new pipeline copying the existing pipeline and add this new pod identity task as a step so we can run periodics on that pipeline.

hakuna-matatah · 2025-03-10T16:17:48Z

tests/pipelines/eks/awscli-cl2-load-with-addons-slos.yaml

+  - name: cl2-default-qps
+    default: "500"
+  - name: cl2-default-burst
+    default: "1000"


lets not use this as default :D , it way above normal values.

just changed the values

hakuna-matatah · 2025-03-10T16:48:25Z

tests/pipelines/eks/awscli-cl2-load-with-addons-slos.yaml

+  - name: cl2-default-burst
+    default: "400"
+  - name: cl2-uniform-qps
+    default: "500"


let's change this to 200 too to begin with. This is what your config is using https://github.com/awslabs/kubernetes-iteration-toolkit/pull/489/files#diff-d2d660edac904aa96e330bfae7bf67ef6885190877c5ab7668f0f157057da03fR45

hakuna-matatah · 2025-03-10T21:43:26Z

LGTM!

Thank you.
Once we have a successful run on experimental infra, we can merge this.

hakuna-matatah · 2025-03-11T02:31:38Z

/lgtm
/approve
/hold - to verify if trigger templates are up to date.

xdu31 force-pushed the eks-pod-identity branch from 2b7c24e to 310f106 Compare February 27, 2025 17:22

xdu31 requested a review from hakuna-matatah February 27, 2025 18:42

hakuna-matatah reviewed Feb 28, 2025

View reviewed changes

xdu31 requested a review from hakuna-matatah March 1, 2025 04:06

xdu31 force-pushed the eks-pod-identity branch from e019b6d to 428d50c Compare March 4, 2025 18:13

xdu31 force-pushed the eks-pod-identity branch from fb6e202 to cb7290f Compare March 5, 2025 22:22

hakuna-matatah reviewed Mar 10, 2025

View reviewed changes

xdu31 requested a review from hakuna-matatah March 10, 2025 16:34

hakuna-matatah reviewed Mar 10, 2025

View reviewed changes

xdu31 force-pushed the eks-pod-identity branch 2 times, most recently from 581fbed to 22c334d Compare March 10, 2025 20:46

xdu31 force-pushed the eks-pod-identity branch from a241c54 to 7526cf6 Compare March 11, 2025 00:44

xdu31 requested a review from hakuna-matatah March 11, 2025 02:36

hakuna-matatah previously approved these changes Mar 11, 2025

View reviewed changes

xdu31 added 4 commits March 10, 2025 19:40

Add tests for EKS Pod Identity

7a3c700

Review comments

5b9d697

Refactoring to use awscli-cl2-load-with-addons-slos pipeline

88e1632

Use normal bundle QPS

4a4f220

xdu31 dismissed hakuna-matatah’s stale review via 0b0ed88 March 11, 2025 02:43

xdu31 force-pushed the eks-pod-identity branch from 7526cf6 to 0b0ed88 Compare March 11, 2025 02:43

Move pod identity tests before cl2

b3ece89

xdu31 force-pushed the eks-pod-identity branch from 0b0ed88 to b3ece89 Compare March 11, 2025 02:45

xdu31 requested a review from hakuna-matatah March 11, 2025 02:47

hakuna-matatah approved these changes Mar 11, 2025

View reviewed changes

xdu31 merged commit 2689526 into awslabs:main Mar 11, 2025
7 checks passed

Add tests for EKS Pod Identity #489

Add tests for EKS Pod Identity #489

Uh oh!

Conversation

xdu31 commented Feb 26, 2025

Uh oh!

hakuna-matatah commented Feb 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hakuna-matatah commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hakuna-matatah commented Mar 10, 2025

Uh oh!

hakuna-matatah commented Mar 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hakuna-matatah commented Mar 4, 2025 •

edited

Loading