[cocl-operator] Define the node identifier and workflow to generate the encryption password

[alicefr]

The Confidential Cluster Operator is responsible for configuring the trustee and ensuring that secrets are available following successful attestation.

On first boot, a node attests itself and, upon success, receives a password from Trustee. This password is used to encrypt the root disk. On subsequent boots, the node performs attestation again to retrieve the same password for disk decryption.

To ensure the trustee returns the correct secret, each node must present a unique identifier during attestation. This identifier is used to match the node to its corresponding password.

The password used for disk encryption can be generated at different stages:

Option 1: Password Generated at Machine Object Creation

In this approach, the password is created as soon as the operator detects a new Machine object. The operator extracts an identifier (e.g., IP address or hostname) from the object, generates the password, and associates it with that identifier. During attestation, the node must present the same identifier to retrieve the correct secret.

This option correspond to the RATS Background-Check model.

Pros:

• Simple design.
• Direct correlation between the Machine object and the identifier.

Cons:

• IP address or hostname is managed by the MCO (Machine Config Operator), not the Confidential Cluster Operator.
• This information may not be available early enough when needed.

Option 2: Password Generated After Successful First Attestation

In this approach, the password and identifier are generated and registered after a successful firstboot attestation. The node receives the identifier and password at that time, and reuses the identifier for future attestations to retrieve the same secret.

This model is described as a combination of the background-check model and the password model where we can use the attestation result to retrieve the resource from a separate rely party.

Pros:

• Decouples the operator from the MCO, reducing dependency on external timing or data.

Cons:

• More complex
• Slightly slower, since password generation occurs during attestation.
• If an external secret store is used, it may introduce latency.
• Requires a mechanism to associate the attesting node with its corresponding Kubernetes object (for deletion/update tracking and lifecycle management).
• we need a second url to contact the cluster operator service to generate the id and password

[niteshnlal]

@alicefr
for Option 2, how would the first boot attestation look like?
for Option 1, is their a way to pin point when IP address or hostname managed by the MCO (Machine Config Operator) are made available? Can we delay the invocation of the attestation until those details are available?

[alicefr]

@alicefr for Option 2, how would the first boot attestation look like?

It would exactly the same as we have used so far, but upon a successful attestation the node will receive the secret for the encryption AND the identifier which needs to be store somewhere and used in the following boots.

for Option 1, is their a way to pin point when IP address or hostname managed by the MCO (Machine Config Operator) are made available? Can we delay the invocation of the attestation until those details are available?

We are trying to understand this right now. When I have the information I will post it here.

Based on the findings one option might be possible or not. For example, also for option 2 I'm not quite sure where the identifier could fit in the attestation protocol. I hope we can return some extra information in the attestation token or something similar

[travier]

The idea behind the IP/hostname as identifier is that the attester server can verify that the connection performing the attestation/resource request actually comes from that IP/hostname. This means that you can not spawn a node with a random IP and query all secrets from the cluster. You actually have to replace the existing node with its IP to be able to do so.

This in turn is no longer discreet (having two systems with the same IP on the same network will likely not go well and I don't think cloud providers accept that).

Moreover, in the future, we will remove older PCR values for nodes that have been updated, so if you were exploiting a vulnerability in the boot image, you would not longer be able to do after the new node joins the cluster.

Overall, this scenario really needs a threat model otherwise we are not going to solve the right thing here.

[travier]

(Copy/pasting from internal discussion channels)

The goal that we are trying to achieve here is:

• a distinct luks root key for each node
• each node has the same initdata

Attack scenario is:

• if you get root access to a node for whatever reason after the node passes attestation, you should not be able to get the secret of another node, only the one for the current node
• for example, the attacker finds a vulnerability in the initrd that it can exploit after the remote attestation or finds a vulnerability in our boot images that we don't update often

If we specialize the initdata for each node then this is "mostly solved", but this would be a lot of work.
I want to find a way that does not involve that.

One option was checking the IP of the connection initiated by the node to the attestation server. This makes it easy to distinguish which secret to hand out but it is difficult to have multiple node using the same IP in a cluster/network so it makes the attack difficult. However node IPs can change and a node can have multiple IPs so this makes this a "not so great option".

Another option was to find an identifier that is both known from the cloud side of things (i.e. the cloud specific ID/name for the VM) and from the node itself, ideally measure as part of the Conf Comp quote, so that we can tie a connection to a secret looking at that.

If we "just" use an identifier that is not protected by something else (Conf Comp quote or Cloud provider constraints) then the attacker (which could be the cloud provider) could create another VM with the same identifier, exploit the vulnerability and get the key.

Is finding such a vulnerability realistic? Yes. See: https://nerds.xyz/2025/07/linux-initramfs-security-flaw-secure-boot-bypass/

We are not impacted on FCOS as we closed this gap earlier AFAIK but there could still be things we missed.

[travier]

If we reduce the scope here and only consider the case where the attacker gets root access on a node but does not have cloud user or cloud provider level access, then we can do option 2 from #11 (comment) or trusted-execution-clusters/operator#5, measuring the ID in a PCR before doing the remote attestation request and validating it as part of the ressource access policy.

[travier]

For a smaller scope above but still getting a secret per node without changing initdata:

First boot flow would be:

• Node initramfs generates an ID and makes a request to the operator to generate a LUKS root key for this ID
• Operator:
  • generates a secret
  • sets up the trustee ressource policy so that only systems with a PCR (8?) extended with this ID can access the path /conf-cluster/<ID> and secrets under
  • sets up the secret there: /conf-cluster/<ID>/luuks-root-key
• Node initramfs extends a PCR value with this ID
• Node initramfs binds LUKS device using Clevis PIN bind and config:
  • trustee URLs: ...
  • ressource path: /conf-cluster/<ID>/...

Subsequent boot does not require more interactions:

• Node initramfs opens LUKS device using Clevis PIN:
  • trustee URLs: ...
  • ressource path: /conf-cluster/<ID>/...

[alicefr]

For a smaller scope above but still getting a secret per node without changing initdata:

First boot flow would be:

• Node initramfs generates an ID and makes a request to the operator to generate a LUKS root key for this ID
• Operator:

It doesn't change too much the flow, but I would rather let the operator generating the ID and the initramfs fetch it. In any case, the clevis pin would require the URL for this service as well.

• generates a secret
• sets up the trustee ressource policy so that only systems with a PCR (8?) extended with this ID can access the path /conf-cluster/<ID> and secrets under
• sets up the secret there: /conf-cluster/<ID>/luuks-root-key

The format here is actually not so free, it needs to correspond to the secret name in the namespace, so I think the ID needs to be as third element. Maybe instead of be set by ignition we could return the path together with the ID by the operator since it is the one that generates the secret.

• Node initramfs extends a PCR value with this ID

• Node initramfs binds LUKS device using Clevis PIN bind and config:

  • trustee URLs: ...
  • ressource path: /conf-cluster/<ID>/...

Subsequent boot does not require more interactions:

• Node initramfs opens LUKS device using Clevis PIN:

  • trustee URLs: ...
  • ressource path: /conf-cluster/<ID>/...

[alicefr]

The registration service for the node is implemented by trusted-execution-clusters/operator#7

[travier]

It doesn't change too much the flow, but I would rather let the operator generating the ID and the initramfs fetch it.

Ah indeed, it makes more sense this way as we could make sure there are no conflicts.

In any case, the clevis pin would require the URL for this service as well.

I don't think the pin should do that. This is logic that is too specific to our use case. The code for that should live elsewhere.

• sets up the trustee ressource policy so that only systems with a PCR (8?) extended with this ID can access the path /conf-cluster/<ID> and secrets under
• sets up the secret there: /conf-cluster/<ID>/luuks-root-key

The format here is actually not so free, it needs to correspond to the secret name in the namespace, so I think the ID needs to be as third element. Maybe instead of be set by ignition we could return the path together with the ID by the operator since it is the one that generates the secret.

OK so what are the exact constraints on the path here?

[travier]

Subsequent boot does not require more interactions:

• Node initramfs opens LUKS device using Clevis PIN:

  • trustee URLs: ...
  • ressource path: /conf-cluster/<ID>/...

This is unfortunately not complete for the second boot as we need to extend the PCR(8?) with the ID before we make the attestation request. And thus this gets messy as we either have to retrieve the ID from the LUKS header to extend the PCR first (manually) or we need to teach this logic to the Clevis PIN which is really not ideal, but could be done.

Thus the subsequent boot would be:

• Read ID from LUKS header/Clevis PIN without doing the remote attestation
• Extend PCR(8?)
• Opens LUKS device using Clevis PIN, doing remote attestation:
  • trustee URLs: ...
  • ressource path: /conf-cluster/<ID>/...

[travier]

New suggested flow:

[travier]

At the machine ID creation point, the operator should be able to see if we have a machine pending inclusion in the cluster and we could potentially re-use this ID or refuse on IPI if no machine have been requested. On UPI we would always accept new ID requests.