Skip to content

Listener pod names conflict when using the same runnerScaleSetName in multiple orgs. #3587

@akhilp6

Description

@akhilp6

Checks

Controller Version

0.9.0

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

gha-scale-set-controller installed with helm chart defaults in action-runner-system namespace.
gha-runner-scale-set chart installed with for org1 in runners namespace with values.yaml:

githubConfigUrl: 'https://github.com/org1'
runnerGroup: staging
runnerScaleSetName: staging-centos7

gha-runner-scale-set chart installed with for org2 in runners namespace with values.yaml:

githubConfigUrl: 'https://github.com/org2'
runnerGroup: staging
runnerScaleSetName: staging-centos7

Describe the bug

The controller creates AutoscalingListener/staging-centos7-7c5bdbdc-listener in the local action-runner-system namespace for org1 but this gets stuck in a error/crash loop for the org2 listener as it creates it with the same name. I believe due to a continuous conflict in which org its trying to register the listener.

listener name is computed here. it basically takes in hash of the namespace, since both the scalesets are deployed in runner namespace, it basically gets the same name.
Listener pod logs
https://gist.github.com/akhilp6/9f22d2da2da9a52ed4b02e9aeb8ae0d1

This happens even when we have multiple cluster which different controllers and we try to deploy scalesets with same name, it basically creates on one cluster and then throws an error on the other cluster.
Application returned an error: createSession failed: failed to create session: 409 - had issue communicating with Actions backend: The runner scale set staging-centos7 already has an active session for owner

We probably can workaround with having these scalesets creating in different namespace, but we are having some issue with this approach, one issue is we need to copy the secrets in all the namespace which we create(which we would like to avoid)

Describe the expected behavior

We would like to have "Adding org to the AutoscalingListener object name" or "placing the listener in the runner namespace instead of the local namespace"

Additional Context

org1 values.yaml

githubConfigUrl: 'https://github.com/org1'
runnerGroup: staging
runnerScaleSetName: staging-centos7

org2 values.yaml

githubConfigUrl: 'https://github.com/org2'
runnerGroup: staging
runnerScaleSetName: staging-centos7

Controller Logs

https://gist.github.com/akhilp6/6d759fa1f1bd26b427e8c3c225dc0e73

Runner Pod Logs

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggha-runner-scale-setRelated to the gha-runner-scale-set mode

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions