Failed calling webhook "webhook.openpolicyagent.org" error

[marcusdroy]

While validating upgrading to the latest version of OPA and kube-mgmt on Kubernetes 1.26, we are seeing Failed calling webhook "webhook.openpolicyagent.org" error in the api server logs.

E0321 15:06:15.586026 dispatcher.go:183] failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server could not find the requested resource
W0321 15:06:17.357212 1 dispatcher.go:176] Failed calling webhook, failing open webhook.openpolicyagent.org: failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server could not find the requested resource
E0321 15:06:17.357275 1 dispatcher.go:183] failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server could not find the requested resource

Reproduction Steps
Starting with blank kind cluster kind create cluster --image=kindest/node:v1.26.0

1. Install latest chart via Helm
   helm upgrade -i -n opa --create-namespace opa opa/opa-kube-mgmt --set admissionController.enabled=true --version 8.1.0

2. validatingwebhookconfiguration is created as expected below

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    meta.helm.sh/release-name: opa
    meta.helm.sh/release-namespace: opa
  creationTimestamp: "2023-03-21T15:04:27Z"
  generation: 1
  labels:
    app: opa-opa-kube-mgmt
    app.kubernetes.io/managed-by: Helm
    chart: opa-kube-mgmt-8.1.0
    heritage: Helm
    release: opa
  name: opa-opa-kube-mgmt
  resourceVersion: "622"
  uid: c696feb5-c5dc-4fbd-a842-c02105027269
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    caBundle: "<redacted>"
    service:
      name: opa-opa-kube-mgmt
      namespace: opa
      port: 8181
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: webhook.openpolicyagent.org
  namespaceSelector:
    matchExpressions:
    - key: openpolicyagent.org/webhook
      operator: NotIn
      values:
      - ignore
  objectSelector: {}
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - '*'
    resources:
    - '*'
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

3. Service is available k get svc opa-opa-kube-mgmt -n opa

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE opa-opa-kube-mgmt ClusterIP 10.96.164.211 <none> 8181/TCP 5m38s

5. However seeing this error in the api server logs
   E0321 15:06:15.586026 dispatcher.go:183] failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server could not find the requested resource W0321 15:06:17.357212 1 dispatcher.go:176] Failed calling webhook, failing open webhook.openpolicyagent.org: failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server could not find the requested resource E0321 15:06:17.357275 1 dispatcher.go:183] failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server could not find the requested resource

[charlieegan3]

Thanks for opening this, just to confirm you aren't seeing any logs from OPA at all? I'm trying to work out if the server could not find the requested resource is related to a 404 or the k8s API not being able to find a healthy instance to call (I presume OPA pods have started ok?).

[marcusdroy]

Looks like both opa mgmt have started

opa-opa-kube-mgmt-7bcc55fbd5-9sw45 2/2 Running 0 54m

mgmt
time="2023-03-21T15:04:41Z" level=info msg="Policy/data ConfigMap processor connected to K8s: namespaces=[opa]"
opa
{"client_addr":"10.244.0.1:26501","level":"info","msg":"Received request.","req_id":4421,"req_method":"POST","req_path":"/","time":"2023-03-21T15:59:49Z"} {"client_addr":"10.244.0.1:26501","level":"info","msg":"Sent response.","req_id":4421,"req_method":"POST","req_path":"/","resp_bytes":86,"resp_duration":1.90967,"resp_status":404,"time":"2023-03-21T15:59:49Z"}

Interesting to note along with health checks its also returning 404 errors, prehaps the webhook service path needs to be updated?

[charlieegan3]

Yeah, that doesn't seem right. I think that the url is going to need to be set in the client config too, i.e. https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#url

I think that you want to set this to this path: https://www.openpolicyagent.org/docs/latest/rest-api/#get-a-document-webhook i.e. /v0/data/system/main if you have the same structure of policy here: https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/#policy-2-prohibit-hostname-conflicts

[marcusdroy]

I tried the configuration below but got a auth error:

    service:
      name: opa-opa-kube-mgmt
      namespace: opa
      path: /v0/data/system/main
      port: 8181

failed calling webhook "webhook.openpolicyagent.org": failed to call webhook: the server has asked for the client to provide credentials

In the opa pod logs it has:

{"client_addr":"10.244.0.1:48895","level":"info","msg":"Sent response.","req_id":11321,"req_method":"POST","req_path":"/v0/data/system/main","resp_bytes":87,"resp_duration":0.884191,"resp_status":401,"time":"2023-03-21T17:25:42Z"}

I havent loaded any bundles in yet besides what the chart is providing

[charlieegan3]

I think I got this wrong. The reason that no path is set in the web hook config is that there's an expectation that default_decision will be used. You can read about that here: https://www.openpolicyagent.org/docs/latest/configuration/#miscellaneous

I think that what you need is a policy to be loaded into OPA with the package system and a main rule, as we see here: https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/#combine-policies

I think that OPA is running with authz on, but it should allow anyone into the root path and the default decision:

            package system.authz
            default allow = false
            # Allow anonymous access to the default policy decision.
            allow { input.path = [""]; input.method = "POST" }
            allow { input.path = [""]; input.method = "GET" }