Skip to content

Conversation

@bradfordwagner
Copy link

@bradfordwagner bradfordwagner commented Oct 24, 2025

Fixes #7198

Motivation

Enable sso connections to identity provider to use a custom cacert to enable chain of trust verification.

Modifications

  • Added chain of trust certificate options for rootCA (pem contents). Allowed for use of SSL_CERT_DIR, and SSL_CERT_FILE by using SystemCertPool()

Documentation

  • Available in sso configuration page.

Verification

  1. make start UI_SECURE=true PROFILE=sso
Screenshot 2025-11-05 at 10 35 08 AM
  • successful handshake between dex (tls'd) and argo-server.
  • cacert present by using rootCA in sso configuration in workflow-controller-configmap.
  1. Removing rootCA in sso configuration in workflow-controller-configmap yields an expected tls verification error
Screenshot 2025-11-05 at 10 50 08 AM

Signed-off-by: bradfordwagner <[email protected]>
Signed-off-by: bradfordwagner <[email protected]>
Signed-off-by: bradfordwagner <[email protected]>
Signed-off-by: bradfordwagner <[email protected]>
Signed-off-by: bradfordwagner <[email protected]>
Signed-off-by: bradfordwagner <[email protected]>
@bradfordwagner
Copy link
Author

bradfordwagner commented Oct 24, 2025

Testing Evidence

insecureSkipVerify=true, rootCA=‘’, rootCAFile=‘’, cacerts_mount=unmounted

argo-server time=2025-10-24T19:36:55.950Z level=INFO msg="not enabling pprof debug endpoints"
argo-server time=2025-10-24T19:36:55.951Z level=INFO msg="Starting Argo Server" authModes="[client sso]" namespace=adpe-argo-workflows managedNamespace="" ssoNamespace=adpe-argo-workflows baseHRef=/ secure=false
argo-server time=2025-10-24T19:36:55.951Z level=WARN msg="You are running in insecure mode. Learn how to enable transport layer security: https://argo-workflows.readthedocs.io/en/latest/tls/"
argo-server time=2025-10-24T19:36:56.083Z level=INFO msg="SSO configuration" redirectUrl=http://adpe-argo-workflows-argoworkflows-server.adpe-argo-workflows.svc.cluster.local:2746/oauth2/callback issuer=https://adpe-vault-ui.adpe-vault.svc.cluster.local/v1/identity/oidc/provider/dev issuerAlias=DISABLED clientId="{LocalObjectReference:{Name:adpe-vault-oidc-client} Key:client_id Optional:<nil>}" scopes="[adpe_argo_workflows_default openid]" insecureSkipVerify=true filterGroupsRegex=[adpe_argo_workflows_oidc.*] rootCA=""
argo-server time=2025-10-24T19:36:56.185Z level=INFO msg="SSO enabled"
argo-server time=2025-10-24T19:36:56.188Z level=INFO msg="No artifact drivers configured, skipping validation"
argo-server time=2025-10-24T19:36:56.188Z level=INFO msg="No artifact drivers configured, skipping connection validation"
argo-server time=2025-10-24T19:36:56.188Z level=INFO msg="Starting Argo Server" goVersion=go1.24.4 version=untagged buildDate=2025-10-24T19:32:28Z gitCommit=f2b5581cce301ebe84514417ced58f61c9ca6cf1 gitTag=untagged gitTreeState=clean instanceID=""
argo-server time=2025-10-24T19:36:56.291Z level=DEBUG msg="Node status offloading config" ttl=5m0s
argo-server time=2025-10-24T19:36:56.291Z level=DEBUG msg=CanI resource=clusterworkflowtemplates namespace="" name="" verb=get
argo-server time=2025-10-24T19:36:56.293Z level=DEBUG msg=CanI verb=list resource=clusterworkflowtemplates namespace="" name=""
argo-server time=2025-10-24T19:36:56.294Z level=DEBUG msg=CanI verb=watch resource=clusterworkflowtemplates namespace="" name=""
argo-server time=2025-10-24T19:36:56.295Z level=INFO msg="Creating event controller" workerCount=4 operationQueueSize=16 asyncDispatch=false
argo-server time=2025-10-24T19:36:56.501Z level=INFO msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
argo-server time=2025-10-24T19:36:56.501Z level=INFO msg="Argo Server started successfully" url=http://localhost:2746

insecureSkipVerify=false, rootCA=‘’, rootCAFile=‘’, cacerts_mount=unmounted

  • missing cacert failure expected
argo-server time=2025-10-24T19:44:16.948Z level=INFO msg="not enabling pprof debug endpoints"
argo-server time=2025-10-24T19:44:16.948Z level=INFO msg="Starting Argo Server" secure=false authModes="[client sso]" namespace=adpe-argo-workflows managedNamespace="" ssoNamespace=adpe-argo-workflows baseHRef=/
argo-server time=2025-10-24T19:44:16.948Z level=WARN msg="You are running in insecure mode. Learn how to enable transport layer security: https://argo-workflows.readthedocs.io/en/latest/tls/"
argo-server Error: Get "https://adpe-vault-ui.adpe-vault.svc.cluster.local/v1/identity/oidc/provider/dev/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority
stream closed: EOF for adpe-argo-workflows/adpe-argo-workflows-argoworkflows-server-5d48645d98-b8xmg (argo-server)

insecureSkipVerify=false, rootCA=‘’, rootCAFile=‘/cacerts/ca.crt’, cacerts_mount=/cacerts

argo-server time=2025-10-24T19:45:01.094Z level=INFO msg="not enabling pprof debug endpoints"
argo-server time=2025-10-24T19:45:01.094Z level=INFO msg="Starting Argo Server" ssoNamespace=adpe-argo-workflows baseHRef=/ secure=false authModes="[client sso]" namespace=adpe-argo-workflows managedNamespace=""
argo-server time=2025-10-24T19:45:01.094Z level=WARN msg="You are running in insecure mode. Learn how to enable transport layer security: https://argo-workflows.readthedocs.io/en/latest/tls/"
argo-server time=2025-10-24T19:45:01.603Z level=INFO msg="SSO configuration" issuerAlias=DISABLED insecureSkipVerify=false issuer=https://adpe-vault-ui.adpe-vault.svc.cluster.local/v1/identity/oidc/provider/dev clientId="{LocalObjectReference:{Name:adpe-vault-oidc-client} Key:client_id Optional:<nil>}" filterGroupsRegex=[adpe_argo_workflows_oidc.*] rootCAFile=/cacerts/ca.crt scopes="[adpe_argo_workflows_default openid]" rootCA="" redirectUrl=http://adpe-argo-workflows-argoworkflows-server.adpe-argo-workflows.svc.cluster.local:2746/oauth2/callback
argo-server time=2025-10-24T19:45:01.704Z level=INFO msg="SSO enabled"
argo-server time=2025-10-24T19:45:01.706Z level=INFO msg="No artifact drivers configured, skipping validation"
argo-server time=2025-10-24T19:45:01.706Z level=INFO msg="No artifact drivers configured, skipping connection validation"
argo-server time=2025-10-24T19:45:01.706Z level=INFO msg="Starting Argo Server" goVersion=go1.24.4 version=untagged instanceID="" buildDate=2025-10-24T19:39:34Z gitCommit=485d3464070de4c07582c27864f24c4129202912 gitTag=untagged gitTreeState=clean
argo-server time=2025-10-24T19:45:01.796Z level=DEBUG msg="Node status offloading config" ttl=5m0s
argo-server time=2025-10-24T19:45:01.796Z level=DEBUG msg=CanI namespace="" name="" verb=get resource=clusterworkflowtemplates
argo-server time=2025-10-24T19:45:01.797Z level=DEBUG msg=CanI verb=list resource=clusterworkflowtemplates namespace="" name=""
argo-server time=2025-10-24T19:45:01.798Z level=DEBUG msg=CanI namespace="" name="" verb=watch resource=clusterworkflowtemplates
argo-server time=2025-10-24T19:45:01.799Z level=INFO msg="Creating event controller" operationQueueSize=16 asyncDispatch=false workerCount=4
argo-server time=2025-10-24T19:45:02.006Z level=INFO msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
argo-server time=2025-10-24T19:45:02.006Z level=INFO msg="Argo Server started successfully" url=http://localhost:2746
argo-server time=2025-10-24T19:45:17.936Z level=INFO msg="HTTP request" size=488 duration=2.93025ms path=/ method=GET status=200

insecureSkipVerify=false, rootCA=‘’, rootCAFile=‘’, cacerts_mount=/etc/ssl/certs

argo-server time=2025-10-24T19:47:23.501Z level=INFO msg="not enabling pprof debug endpoints"
argo-server time=2025-10-24T19:47:23.501Z level=INFO msg="Starting Argo Server" namespace=adpe-argo-workflows managedNamespace="" ssoNamespace=adpe-argo-workflows baseHRef=/ secure=false authModes="[client sso]"
argo-server time=2025-10-24T19:47:23.501Z level=WARN msg="You are running in insecure mode. Learn how to enable transport layer security: https://argo-workflows.readthedocs.io/en/latest/tls/"
argo-server time=2025-10-24T19:47:24.307Z level=INFO msg="SSO configuration" redirectUrl=http://adpe-argo-workflows-argoworkflows-server.adpe-argo-workflows.svc.cluster.local:2746/oauth2/callback clientId="{LocalObjectReference:{Name:adpe-vault-oidc-client} Key:client_id Optional:<nil>}" filterGroupsRegex=[adpe_argo_workflows_oidc.*] rootCAFile="" insecureSkipVerify=false issuer=https://adpe-vault-ui.adpe-vault.svc.cluster.local/v1/identity/oidc/provider/dev scopes="[adpe_argo_workflows_default openid]" rootCA="" issuerAlias=DISABLED
argo-server time=2025-10-24T19:47:24.407Z level=INFO msg="SSO enabled"
argo-server time=2025-10-24T19:47:24.410Z level=INFO msg="No artifact drivers configured, skipping validation"
argo-server time=2025-10-24T19:47:24.410Z level=INFO msg="No artifact drivers configured, skipping connection validation"
argo-server time=2025-10-24T19:47:24.410Z level=INFO msg="Starting Argo Server" gitCommit=485d3464070de4c07582c27864f24c4129202912 gitTag=untagged gitTreeState=clean goVersion=go1.24.4 version=untagged instanceID="" buildDate=2025-10-24T19:39:34Z
argo-server time=2025-10-24T19:47:24.485Z level=DEBUG msg="Node status offloading config" ttl=5m0s
argo-server time=2025-10-24T19:47:24.485Z level=DEBUG msg=CanI name="" verb=get resource=clusterworkflowtemplates namespace=""
argo-server time=2025-10-24T19:47:24.487Z level=DEBUG msg=CanI verb=list resource=clusterworkflowtemplates namespace="" name=""
argo-server time=2025-10-24T19:47:24.487Z level=DEBUG msg=CanI namespace="" name="" verb=watch resource=clusterworkflowtemplates
argo-server time=2025-10-24T19:47:24.488Z level=INFO msg="Creating event controller" workerCount=4 operationQueueSize=16 asyncDispatch=false
argo-server time=2025-10-24T19:47:24.694Z level=INFO msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
argo-server time=2025-10-24T19:47:24.694Z level=INFO msg="Argo Server started successfully" url=http://localhost:2746

insecureSkipVerify=false, rootCA=’MIIGFTCCA/2gAwIBAgIUKNdK/1S4x7Rm2WKDDYfGZYI76igwDQYJKoZIhvcNAQEL’, rootCAFile=‘’, cacerts_mount=unmounted

argo-server time=2025-10-24T19:46:15.212Z level=INFO msg="not enabling pprof debug endpoints"
argo-server time=2025-10-24T19:46:15.213Z level=INFO msg="Starting Argo Server" ssoNamespace=adpe-argo-workflows baseHRef=/ secure=false authModes="[client sso]" namespace=adpe-argo-workflows managedNamespace=""
argo-server time=2025-10-24T19:46:15.213Z level=WARN msg="You are running in insecure mode. Learn how to enable transport layer security: https://argo-workflows.readthedocs.io/en/latest/tls/"
argo-server time=2025-10-24T19:46:16.020Z level=INFO msg="SSO configuration" issuer=https://adpe-vault-ui.adpe-vault.svc.cluster.local/v1/identity/oidc/provider/dev clientId="{LocalObjectReference:{Name:adpe-vault-oidc-client} Key:client_id Optional:<nil>}" scopes="[adpe_argo_workflows_default openid]" filterGroupsRegex=[adpe_argo_workflows_oidc.*] rootCA="-----BEGIN CERTIFICATE-----\nMIIGFTCCA/2gAwIBAgIUKNdK/1S4x7Rm2WKDDYfGZYI76igwDQYJKoZIhvcNAQEL\nBQAwUDEkMCIGA1UEChMbR2xvYmFsIEluZm9ybWF0aW9uIFNlY3VyaXR5MQwwCgYD\nVQQLEwNHSVMxGjAYBgNVBAMTEUJsYWNrUm9jayBSb290IENBMB4XDTI1MTAxNDEy\nMjI1NVoXDTMwMTAxMzEyMjMyMlowUDEkMCIGA1UEChMbR2xvYmFsIEluZm9ybWF0\naW9uIFNlY3VyaXR5MQwwCgYDVQQLEwNHSVMxGjAYBgNVBAMTEUJsYWNrUm9jayBS\nb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9vdKyLE2keLo\nolLs+Yq9Ctw0aSKZSc7LRNzdOAzlA/rn0FhStlaXKRh4ck1a+N1ACIkKK9bazHiu\na3xuO4F6yk8QBgN2qiIYNA/5WLHkE6nSoTHICI9aQfOewMSllcgrDClwqFGgmRBW\nQJ8M5GBz7vkuBFC6UTi8lEPsOoEX+xWZMIOKUmSPAoB5gOrsBLHOGmRwFHtxACpb\n0KK9GWRkW18krO5QFL46+Y8rwpV5i3Hxj0Sleq2jBqyNGm5rxhtJbKOGfItXrHzW\ntOo25KkRZsnI75B3yTbKv8mrj4V4pC2IqkijNyk8GB7Ol07WDkbAHBmEVT8JqUyh\n5L72L4IOUfGXGeS6H6Ia16x0vc1CFUuL4+7p6hXI4BjQFpsIzCK2QVBRpDMOH2Ki\neziVv8nk0MKlYIUBPgCS/tyPc18PT6jw+FzvtvlRmR7iRX96B6Rl0jKN0gcAw/rP\ncaCA+OSptms5tzv4ZgmuP/YJ2W2KjBnyQLIsF08Gnjk0o7joQ6wKwPPhoNkDV4Gn\nkix9InZ14VQa006WbnsgAB+SJcd85368gUTLEMw6v+Gn0f7D+GHM8mBIgntQ8G8O\nXwjCUdAAIcTr1Dr2YLiOA0JL/cAXmDBHnyTkUjm/4B5Bxu1fvx+jiSUaTpBJclak\nEsBfWE//aRg+Z2YAMg/7D9DBS0ID8e8CAwEAAaOB5jCB4zAOBgNVHQ8BAf8EBAMC\nAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQURO0DElQX1MoEcrGYqSKbWYcr\n4uwwHwYDVR0jBBgwFoAURO0DElQX1MoEcrGYqSKbWYcr4uwwRAYIKwYBBQUHAQEE\nODA2MDQGCCsGAQUFBzAChihodHRwOi8vbG9jYWxob3N0OjgyMDAvdjEvcm9vdF9h\naXJfZ2FwL2NhMDoGA1UdHwQzMDEwL6AtoCuGKWh0dHA6Ly9sb2NhbGhvc3Q6ODIw\nMC92MS9yb290X2Fpcl9nYXAvY3JsMA0GCSqGSIb3DQEBCwUAA4ICAQDHPPzbd3IO\n6J1tFb/IhqDz9oPGPJKiGVsg7vvFsKOjA421Glohbl1ntJqlnj2j7eMNyMWplw5S\njYNR5au+vnkSFcWZFGqn8J9iy0iOyj7KK3sCjtwhZLcHIXnnimrq1KxXQ68gIzl3\nF5jEgkF+GF/v7grI3Qkhqwsa/+Dy8ii5mcxTltsLQyt7sUi9sgmgghcs1zdD83GO\noiX9dms/grPYY5KszcViKHBkBkInp8/p7JXjuM/EKNPHQNsuOn5kSJnARTTlYU6V\n3OVHsKAJgmY+as2u3XjMgSQ18P4cglSzagplAq7N1TV6DujPBM8ibLzKjRu4Bun0\nmDUDGxzrZGarctx3xjdKsOerML4bHPVf7h/AV9VgoSB5LfaCuZqpqo/ykAsYa6yp\nqhvHY4u+TOle9MrxnX1LC1RxJK8YdpaqKdK688/utcCGO9Cs92m9yheZQiihefCw\n9GhGDkTUUPArorkKJ+3UtDHlfb5+v8JvK5cEuIIV/BSE3zb3XYqUs93MiGP9sFXE\nrtSL1izYIeiEyc4/77O8o+1Vg9fwoKzFmk6eANTdulatsOi9OyNqr0MJau6nGy3S\nfJBilpPF8D4EEgS33ty7UhsxQqk6rZGfMJ2f5fx0+wLdEMlclQinbtx16j5dapX5\nhbuCuVn1nuS/v/0kiTqnMO8BYMyOdn1iXw==\n-----END CERTIFICATE-----" redirectUrl=http://adpe-argo-workflows-argoworkflows-server.adpe-argo-workflows.svc.cluster.local:2746/oauth2/callback issuerAlias=DISABLED insecureSkipVerify=false rootCAFile=""
argo-server time=2025-10-24T19:46:16.121Z level=INFO msg="SSO enabled"
argo-server time=2025-10-24T19:46:16.123Z level=INFO msg="No artifact drivers configured, skipping validation"
argo-server time=2025-10-24T19:46:16.123Z level=INFO msg="No artifact drivers configured, skipping connection validation"
argo-server time=2025-10-24T19:46:16.123Z level=INFO msg="Starting Argo Server" instanceID="" gitTreeState=clean goVersion=go1.24.4 version=untagged buildDate=2025-10-24T19:39:34Z gitCommit=485d3464070de4c07582c27864f24c4129202912 gitTag=untagged
argo-server time=2025-10-24T19:46:16.193Z level=DEBUG msg="Node status offloading config" ttl=5m0s
argo-server time=2025-10-24T19:46:16.193Z level=DEBUG msg=CanI namespace="" name="" verb=get resource=clusterworkflowtemplates
argo-server time=2025-10-24T19:46:16.194Z level=DEBUG msg=CanI verb=list resource=clusterworkflowtemplates namespace="" name=""
argo-server time=2025-10-24T19:46:16.195Z level=DEBUG msg=CanI name="" verb=watch resource=clusterworkflowtemplates namespace=""
argo-server time=2025-10-24T19:46:16.196Z level=INFO msg="Creating event controller" workerCount=4 operationQueueSize=16 asyncDispatch=false
argo-server time=2025-10-24T19:46:16.402Z level=INFO msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
argo-server time=2025-10-24T19:46:16.402Z level=INFO msg="Argo Server started successfully" url=http://localhost:2746

@bradfordwagner
Copy link
Author

@MasonM - I felt more comfortable signing off on changes I could test (adding custom ca certificate support to the idp connection) so I decided to raise a separate PR. Connections which require mTLS to IDP should be a simple enough PR from these changes. I also allowed for default cacertificate ie /etc/ssl/certs to be loaded which would not require setting either rootCA, or rootCAFile.

Copy link
Member

@MasonM MasonM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bradfordwagner! I think we should update the local development environment to support testing this, because we (the maintainers) need to be able to reproduce bug reports involving this feature. I created a quick POC for that here off this branch: MasonM@0633548

Use this link to create a CodeSpace from this commit: https://github.com/codespaces/new?hide_repo_select=true&ref=add-dev-environment&repo=856626547&skip_quickstart=true&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json&geo=UsWest

I did some brief testing and it appears to work using make start UI_SECURE=true PROFILE=sso. I'm not happy about the hacky sed command, but I can't think of a better approach. If that does work for you too, then it just needs to be documented in docs/running-locally.md, as was done for #14894

Copy link
Member

@MasonM MasonM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! Just a few minor comments

bradfordwagner and others added 3 commits November 9, 2025 08:38
Co-authored-by: Mason Malone <[email protected]>
Signed-off-by: Bradford Wagner <[email protected]>
Co-authored-by: Mason Malone <[email protected]>
Signed-off-by: Bradford Wagner <[email protected]>
…p.yaml

Co-authored-by: Mason Malone <[email protected]>
Signed-off-by: Bradford Wagner <[email protected]>
Copy link
Member

@MasonM MasonM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! I appreciate your persistence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

custom root CA for argo-server SSO

2 participants