[proto]: bolt a DRA driver frontend on the topology-aware policy.

[klihub]

This prototype patch set bolts a DRA allocation frontend on top of the existing topology aware resource policy plugin. The main intention with of this patch set is

• provide something practical to play around with for the feasibility study of enabling DRA-based CPU allocation,
• allow (relatively) easy experimentation with how to expose CPU as DRA devices (IOW test various CPU DRA attributes)
• allow testing how DRA-based CPU allocation (using non-trivial CEL expressions) would scale with cluster and cluster node size

Notes:
This patched NRI plugin, especially in its current state and form, is not a proposal for a first real DRA-based CPU driver.

If you want to play around with this (for instance modify the exposed CPU abstraction), the easiest way is to

1. fork the main NRI Reference Plugins repo
2. enable github actions in your personal fork
3. make any changes you want (for instance, to alter the CPU abstraction, take a look at cpu.DRA()
4. Push your changes to ssh://git@github.com/$YOUR_FORK/nri-plugins/refs/heads/test/build/dra-driver.
5. Wait for the image and Helm chart publishing actions to succeed
6. Once done, you can pull the result in to your cluster with something like helm install --devel -n kube-system test oci://ghcr.io/$YOUR_GITHUB_USERID/nri-plugins/helm-charts/nri-resource-policy-topology-aware --version v0.9-dra-driver-unstable

You can then test if things work with something like

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: any-cores
spec:
  spec:
    devices:
      requests:
      - name: cpu
        exactly:
          deviceClassName: native.cpu
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: p-cores
spec:
  spec:
    devices:
      requests:
      - name: cpu
        exactly:
          deviceClassName: native.cpu
          selectors:
            - cel:
                expression: device.attributes["native.cpu"].coreType == "P-core"
          count: 1
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: e-cores
spec:
  spec:
    devices:
      requests:
      - name: cpu
        exactly:
          deviceClassName: native.cpu
          selectors:
            - cel:
                expression: device.attributes["native.cpu"].coreType == "E-core"
          count: 1
---
apiVersion: v1
kind: Pod
metadata:
  name: pcore-test
  labels:
    app: pod
spec:
  containers:
  - name: ctr0
    image: busybox
    imagePullPolicy: IfNotPresent
    args:
      - /bin/sh
      - -c
      - trap 'exit 0' TERM; sleep 3600 & wait
    resources:
      requests:
        cpu: 1
        memory: 100M
      limits:
        cpu: 1
        memory: 100M
      claims:
      - name: claim-pcores
  resourceClaims:
  - name: claim-pcores
    resourceClaimTemplateName: p-cores
  terminationGracePeriodSeconds: 1

[vigashinikesavan]

After a Pod with DRA claim is terminated, general workloads do not gain access to the released Unclaimed CPUs. Considering the scenario, where a DRA claimed pod access to core 1, and the other generic shared workloads has access to 2-16 cores. After terminating the claimed Pod, the internal shared pool gets updated, but the generic workloads still has access to only 2-16 cores.

During Pod termination, StopContainer is invoked first followed by NodeUnprepareResources function. This 'unclaimCPUs' function falls under 'NodeUnprepareResources' flow, it unclaims and release the claimed CPUs through updateSharedAllocation for remaining workloads. This update happens internally, but is not reflected in the end container.

Releasing the claimed CPUs back to shared/reserved pool in function 'ReleaseCPU' (part of the StopContainer flow) seems to work, the general workloads can access full shared pool of 0-16 upon a claimed pod termination. But, not sure if this would be a valid placement.

[klihub]

@vigashinikesavan It was a long while ago when we rolled this early prototype, so it took me a while to pick up where we left off. I tried a few simple cases and indeed once you release a claim, if you have burstable or a best-effort container which would be eligible to run on the freed up CPUs (if they run in a shared pool where the CPU was returned to), their effective cgroup cpuset might not get correctly updated.(.. immediately.)

But based on a quick look at the code, and my few simple tests, this does not happen as/because of what you state above. Instead what happens is that we do update the remaining containers correctly, but because the update did not happen in the context of a normal resource allocation or release (CreateContainer or StopContainer) the cpuset/resource updates we did for the containers do not get propagated back to the runtime as part of an NRI response but instead they stay cached in the plugin, until the next container creation or release. You can test this easily when you are in the incorrect state, by creating for instance a new best effort container, which will then flush all the collected pending updates in the response, including the ones collected during the release/unprepare of the claim. This will then cause the containers running in the shared pools to finally get correctly updated. What we should do in this situation is to update the affected containers using an unsolicited container update, but the current code does not do it.

Anyway, this was an early prototype to see how far we could get with what was available at the time we rolled this draft PR. This needs to be updated/reworked pretty much, because a lot has happened since on the DRA front, and the approach taken in this proto is now outdated.

[vigashinikesavan]

Got it. Thanks for the clarification.