Adjustment audit logging

[klihub]

We want to allow logging NRI OCI Spec container adjustments for improved observability. This is also one of the unmet requirements for tagging a first stable v1.0 NRI version. There are some related questions to discuss.

The questions I have in mind are mostly about how much details to log, should it be configurable, and should it be on/off configurable. Based on my discussions with @mikebrow a while ago, he was also pondering these (but correct me Mike if I misinterpreted you).

There are two rather obvious alternatives places to implement container adjustment logging. One is an where adjustment is collected, the other where the adjustment is applied to the OCI Spec. Both have some pros and cons.

The linked PR #268 rolls a possible implementation of the latter approach. IOW, it logs adjustments as they are applied to the OCI Spec. The PR in its current form only enables on/off configurability, leaving the decision and details of which to the runtime integration code. When logging adjustments, the PR logs plugins responsible for the adjustments and the adjustment details. Additionally (and thanks to the comment by @acurtiz) the PR also updates returned errors to include the plugin name if the error originated in a plugin.

/cc @samuelkarp @chrishenzie @mikebrow Please chime in.

[acurtiz]

A critical aspect of audit logging is telling which plugin did what, which is increasingly important as usage of NRI grows (given we'd expect more nodes to have multiple plugins running simultaneously). IMO we should reliably be able to answer at least the following two questions:

1. Which plugins made what modifications?
2. Which plugin resulted in a given error?

For #2, imagine a node with 3 NRI plugins running and one of them returns an error during a RunPodSandbox callback. containerd logs don't actually reveal which plugin caused that error (see below), but arguably should & could in a structured manner, which brings improved debuggability and the ability for higher level automation to react to failing plugins in specific ways.

"jsonPayload": {
  "_EXE": "/usr/bin/containerd",
  "MESSAGE": "time=\"2026-02-09T16:44:32.545908121Z\" level=error msg=\"RunPodSandbox for &PodSandboxMetadata{Name:pod-7c8m8,Uid:098341e4-1aed-47bc-86ab-fe9d22dcfbd1,Namespace:kube-system,Attempt:1,} failed, error\" error=\"NRI RunPodSandbox failed: rpc error: code = Unknown desc = internal error: reconciliation failed\"",
  "_SYSTEMD_SLICE": "system.slice",
  "SYSLOG_IDENTIFIER": "containerd",
  ...
}

[klihub]

1. Which plugins made what modifications?
2. Which plugin resulted in a given error?

@acurtiz Thank you. Spelling those out so clearly was very useful feedback for me.

So I reworked the linked PoC/PR and the related containerd working tree so that now we

1. pass the collected plugin ownership to the runtime integration code along with adjustments
2. pass this information and use it during OCI Spec adjustment, to log any plugin the changes originate from
3. for any error that originates/gets reported by a plugin, include the plugin in any final NRI error which is passed back to the runtime

I am not anywhere nearly happy with many of the details, but I think this is now closer to providing the relevant information we want in audit log messages.

[chrishenzie]

I agree with @acurtiz's comment that our focus should be on answering the "who" (which plugin?) and "what" (modification was made?) to be traceable.

Regarding the configuration, I think audit logging should probably be "always on" since plugins can transparently change the OCI spec.

I left some comments on the PR regarding the implementation details, but overall this approach looks good to me.

[samuelkarp]

Regarding the configuration, I think audit logging should probably be "always on" since plugins can transparently change the OCI spec.

I think this configurability should live in the runtime. NRI library should report audit information up to the runtime as part of responses, then runtime can decide about logging. In the runtime we could tie this to an always-on log, or emit it into the runtime standard log, or whatever we decide there (and that might be different depending on the runtime).

[klihub]

Regarding the configuration, I think audit logging should probably be "always on" since plugins can transparently change the OCI spec.

I think this configurability should live in the runtime. NRI library should report audit information up to the runtime as part of responses, then runtime can decide about logging. In the runtime we could tie this to an always-on log, or emit it into the runtime standard log, or whatever we decide there (and that might be different depending on the runtime).

I think the linked PR #268 also takes this approach. The runtime decides whether to log audit messages or not and at which logging level (by omitting or providing a function to log messages). Extra audit information is passed from NRI to the runtime, although not strictly as part of but rather along with responses.