Introduce keylime TEE attestation service support #980

tylerfanelli · 2025-09-30T02:07:41Z

TEEs are most likely found in security-sensitive environments where applications are deployed on untrusted systems such as the cloud or edge. In additions of TEE protections, some users in these environments would also like runtime attestation services to ensure that systems running sensitive applications are not compromised at any point; and if they are, that a trusted party can be notified and take action.

Keylime (whitepaper found here) is a CNCF project to measure and verify the boot and runtime environments of systems using TPMs and the Linux IMA subsystem.

For general purpose confidential VMs on AMD SEV-SNP (and eventually Intel TDX), it is recommended to use the Secure VM Service Module (SVSM) for privileged operations and device emulations to guest operating systems. One crucial service offered by the SVSM is marshaling a virtual TPM (vTPM) that a guest OS can use in a trusted manner without the respective TPM hardware.

However, since the SVSM runs at an even higher privilege than the guest OS, it must also be attested before any meaningful processing can take place in the confidential VM. As such, there has been work done to add attestation to the SVSM boot process in order to establish trust in:

The SVSM firmware responsible for bootstrapping the rest of the system.
The vTPM that is provided by the SVSM.

For SVSM attestation, the Trustee KBS is already supported. However, no one solution provides the needed TEE+firmware attestation of Trustee along with the runtime attestation services provided by Keylime. As such, to set up the required services for this combined use-case, one would need to deploy:

            ┌───────────┐                                         
  ┌───────┐ │Trustee    │ ┌────────┐      ┌────────┐ ┌─────────┐  
  │Trustee│ │Attestation│ │Trustee │      │Keylime │ │Keylime  │  
│ │KBS    │ │Service    │ │RVPS    │ │  │ │Verifier│ │Registrar│ │
│ └───────┘ └───────────┘ └────────┘ │  │ └────────┘ └─────────┘ │
│                                    │  │                        │
└─────────────────┬──────────────────┘  └────────────┬───────────┘
                  │                                  │            
                                                                  
                 TEE                             TPM+IMA          
                 boot                            runtime          
              attestation                       attestation

Although complex, this is entirely able to be automated in a cloud native environment. However, without a cloud native environment, this offers considerable complexity to deploy for general purpose VMs. Also, for legacy VM applications (i.e. not running other cloud-native software within them), the TEE attestation policies are relatively simple. Thus, there was proposal in Keylime to add simple TEE verification for the sole purpose of establishing a trusted vTPM in confidential environments. With this, legacy VMs could use SVSM for attestation and use their vTPMs in exactly the same way they always have been.

This PR adds a backend attestation service module for the Keylime TEE verifier with the optional keylime-as feature. It also creates and signs EAR tokens on behalf of the Keylime Verifier for successfully-attested clients. With this, the TEE boot+runtime attestation scenario is simplified to:

  ┌───────┐┌────────┐┌─────────┐  
  │Trustee││Keylime ││Keylime  │  
│ │KBS    ││Verifier││Registrar│ │
│ └───────┘└────────┘└─────────┘ │
│                                │
└────────────────┬───────────────┘
                 │                
                                  
                TEE               
                boot              
                 +                
              runtime             
             attestation

Legacy clients can also run legacy applications and use their vTPM in a trusted manner with SVSM.

TODO in later patch series

There still requires some proper integration between Keylime TPM and TEE policy support. As such, the EAR token marshaling still could use some improvement to report the specific verifier policies used to attest TEE evidence. Also, the Keylime TEE verifier only supports SEV-SNP at the moment, with eventual plans for TDX (contingent on SVSM support for the latter).

Xynnn007

Am I right about the dataflow about the design?

keylime verified the evidence, and it only returns Ok or Fail
If Ok, kbs-keylime will sign a ear token and return

Xynnn007 · 2025-09-30T08:12:58Z

kbs/src/attestation/keylime.rs

+                "{}/v{}.{}/verify/evidence",
+                self.config.base_url, self.config.api_version_major, self.config.api_version_minor
+            ))
+            .header(CONTENT_TYPE, "application.json")


Maybe it should be application/json?

Yes, thank you. Fixed.

tylerfanelli · 2025-09-30T14:19:24Z

Am I right about the dataflow about the design?

keylime verified the evidence, and it only returns Ok or Fail

Yes, although this is temporary. Once we're able to merge good TPM+TEE policy support in Keylime, we'll likely return the policy ID and build associated with the evidence appraisal. This is secure due to mTLS being enabled by default between the KBS and Keylime verifier.

If Ok, kbs-keylime will sign a ear token and return

Yes, the KBS signs an EAR token on behalf of the Keylime Verifier (that is, the EAR token will be signed with Keylime's key).

fitzthum

Cool. Keylime API seems kinda weak tbh, but I guess we can support it.

A few comments.

kbs/src/attestation/keylime.rs

Cargo.toml

kbs/src/attestation/keylime.rs

fitzthum · 2025-09-30T14:43:49Z

kbs/src/attestation/keylime.rs

+                "attestation_report": evidence_to_verify[0].tee_evidence,
+                "nonce": runtime.nonce,
+                "tee_pubkey_x_b64": x,
+                "tee_pubkey_y_b64": y,


So keylime has its own way of packing the nonce and tee pubkey into the report data? Does this match what we do on the attester side?

I don't think I understand the question. The KBS generates the nonce, but the Keylime Verifier recreates the freshness hash and compares it with report data. This is similar to what is done on the attester side, yes.

In the attester we populate the runtime data not just with the nonce, but with the nonce and tee public key (per the RuntimeData struct). When we validate the report data we want to make sure both of those things match. So the question is how does keylime handle that? You are separating the report data into three fields here. Will keylime recombine them in the right way to calculate the expected report data? Does it only put the nonce in the report data? Does it fail if things don't match? etc

Yes, keylime will combine them in the same way to calculate the expected report data. Both the nonce and TEE pubkey are included. It fails if the expected report data doesn't match the actual.

fitzthum · 2025-09-30T14:46:50Z

kbs/src/attestation/keylime.rs

+    client: Client,
+    cert: X509,
+    priv_key_pem: Vec<u8>,
+    pub_key: Rsa<Public>,


These keys are all used for making the JWT attestation token, right? Do we do anything to validate the response coming from keylime (besides https)? I guess keylime itself doesn't return any kind of JWT.

This group of keys serves two purposes:

The first being mTLS between the KBS/Keylime so they both are confident they are talking to the right counterpart. Since we have mTLS, we don't need to validate the response.

The second being giving the KBS the ability to sign JWTs on behalf of the Keylime Verifier. As mentioned before, the Keylime Verifier requires mTLS to communicate with. For a CVM scenario with SVSM, it is not wise to give the CVM access to the mTLS certs, so we restrict communication to the KBS, which acts as a intermediary to the Keylime Verifier.

Ok, I guess it could be more clear which is which.

kbs/src/attestation/keylime.rs

tylerfanelli · 2025-09-30T15:04:32Z

Cool. Keylime API seems kinda weak tbh, but I guess we can support it.

It is weak right now. Keylime has its own policy format it follows for TPMs, and we're trying to square that with TEE evidence. It'll be much more mature at that point. These changes coupled with coconut-svsm/svsm#828 define the e2e story for SVSM <--> KBS <--> Keylime however, which certainly helps with testing the new policy changes in Keylime.

tylerfanelli · 2025-10-01T03:12:45Z

@fitzthum After reviewing your comments, I opted to just return a simple JWT (i.e. not EAR yet) from the KBS, rather than a half-baked EAR implementation. This is only temporary, as Keylime itself is working out how to address EAR tokens internally at the moment.

I also added claims support returned from the Keylime Verifier, as I agreed that it made more sense to come from there. I'll add EAR support in another series in the future.

Would you mind re-reviewing?

mythi · 2025-10-02T06:49:57Z

kbs/src/attestation/keylime.rs

+            ));
+        }
+
+        let data = &evidence_to_verify[0];


perhaps add a check for keylime supported Tees here?

kbs/src/attestation/keylime.rs

fitzthum

Looking pretty good. Should add some docs, but this seems reasonable to me.

tylerfanelli · 2025-10-02T17:36:33Z

Looking pretty good. Should add some docs, but this seems reasonable to me.

Great. Yes, I'll add some docs showing purpose and how to set up.

tylerfanelli · 2025-10-08T03:38:50Z

@fitzthum @Xynnn007 @mythi I've cleaned up the API to open the possibility of other CPU TEE formats to be supported. Would you mind another review?

Adding documentation in the meantime.

Xynnn007

Hi @tylerfanelli there are some more code nits. Besides, there is one thing that I hope to include in this PR

How to generate the certs and keys used by keylime
What are the keys and how to set them in both keylime and KBS side

Also, a design perspective that is not a blocker. This PR looks like to let KBS play a role as a representative to sign JWT for KeylimeVerifier. From the perspective of functional decoupling, KBS should not be concerned with token issuance, leaving it to the Backend AS. Therefore, an optional refactoring direction in the future is to add an API for issuing JWTs to Keylime itself. Alternatively, a simple shim component, such as KeylimeAttestationService, can be built between Keylime and KBS. Its function is to call Keylime for authentication, issue tokens, and pass them to KBS.

Xynnn007 · 2025-10-11T05:45:52Z

Cargo.toml

 ] }
 reqwest = { version = "0.12", default-features = false, features = [
-    "default-tls",
+    "native-tls",


any reason to change this to native-tls? default-tls will use pure rust underlying crypto crates, while native-tls will use openssl

Setting the ClientBuilder's identity (for mTLS) requires either native-tls or rustls-tls. It seems like you're looking for rustls-tls? I can switch.

Side note: I'd likely prefer OpenSSL, is there a reason you want to avoid it?

Yes, rustls.

We usually want to build static compilation to ensure that, for example, a binary can be executed on all x86 platforms without having to worry about the OS. Static compilation of openssl is not simple. Although KBS and AS have not yet achieved full static compilation, it is still generally preferred to avoid openssl under the same circumstances.

Understood. I've made the change.

kbs/src/attestation/keylime.rs

Xynnn007 · 2025-10-11T06:08:43Z

kbs/src/attestation/keylime.rs

+    pub api_version_major: u8,
+    pub api_version_minor: u8,
+    /// Path of the verifier CA certificates.
+    pub cv_ca_path: String,


I noticed that the certs to be looked up are {cv_ca_path}/cacert.crt, {cv_ca_path}/server-cert.crt and some others. I recommend writing this down in a user guide documentation, including how to organize the configuration file names in this directory and the meaning of each file.

Yes, currently working on a setup document that will include this.

Xynnn007 · 2025-10-11T06:15:09Z

kbs/src/attestation/keylime.rs

+        // Build an HTTP client to communicate with the Keylime verifier. Establish an identity
+        // using the verifier's certificates.
+        let client = ClientBuilder::new()
+            .identity(


Looks like server_priv_pem and server_x509_pem is used for KBS side to make mTLS channel with keylime verifier. Some questions:

This key and cert are not the ones that keylime verifier server uses to launch TLS server, right?

Is it better to rename them to client_priv_pem and client_x509_pem?

Logically this requires the keylime verifier side needs to set a trust root that trustes this client x509 cert - or did I miss anything?

Looks like server_priv_pem and server_x509_pem is used for KBS side to make mTLS channel with keylime verifier. Some questions:

1. This key and cert are not the ones that keylime verifier server uses to launch TLS server, right?

Yes, it is. In traditional Keylime setups, the Verifier and Registrar use the server certificates, and are considered all part of the server. We're adding onto that and making the KBS also a part of this setup.

Okay, this is a bit strange, because we're using the KeylimeVerifier certificate as the client certificate to access KeylimeVerifier. When you mention Verifier using the server certificate, are you referring to the Verifier's externally exposed interface using this certificate?

So, I'm a bit skeptical about whether a client certificate is necessary. In other words, do we need to add a TrustRoot for the target KeylimeVerifier root certificate to the client logic?

Sorry, I think I misunderstood. KBS is using the verifier's client certificates for mTLS. They are not using the same certificates to communicate with each other.

However, since the KBS signs JWTs on behalf of the verifier, it also requires its public/private keypair to do that.

What is the verifier's client certificate? Do you mean that by default keylime enables mTLS and connections between KeylimeVerifier and Agent use that?

However, since the KBS signs JWTs on behalf of the verifier, it also requires its public/private keypair to do that.

Do you mean KBS should use KeylimeVerifier's key pair? I am not an expert on keylime; do you mean that KeylimeVerifier have another key pair except for https key pair?

kbs/src/attestation/keylime.rs

tylerfanelli · 2025-10-13T03:17:37Z

Also, a design perspective that is not a blocker. This PR looks like to let KBS play a role as a representative to sign JWT for KeylimeVerifier. From the perspective of functional decoupling, KBS should not be concerned with token issuance, leaving it to the Backend AS.

We want to eventually get there, but this requires a bit more thought on the Keylime side, as token issuance is not something that is built into the current architecture. This is a compromise for the moment, allowing Keylime to not worry about it.

Therefore, an optional refactoring direction in the future is to add an API for issuing JWTs to Keylime itself. Alternatively, a simple shim component, such as KeylimeAttestationService, can be built between Keylime and KBS. Its function is to call Keylime for authentication, issue tokens, and pass them to KBS.

An interesting thought. I'll have to discuss with the community more on if they forsee token issuance being useful for other areas in the project in the future. If we're the only use-case, then a middle attestation service seems like the best option. But if they forsee other uses, then we should likely just build it into Keylime directly.

Thanks for the input.

With the introduction of the Keylime one-shot attestation API and TEE extensions, the Keylime Verifier is able to attest the state of TEE attestation reports alongside TPM evidence. This allows Keylime to act as an attestation service backend for the KBS. Signed-off-by: Tyler Fanelli <[email protected]>

Create an example config file to show how one could configure the KBS to use Keylime as an alternative attestation service. Signed-off-by: Tyler Fanelli <[email protected]>

tylerfanelli · 2025-10-20T05:12:44Z

@Xynnn007 The verifier features this relies on hasn't made it into a released API version yet. Would you mind if we added this as experimental and provided the documentation once the new verifier API (with the required TEE changes) is released to add docs?

Xynnn007 · 2025-10-23T14:30:47Z

kbs/src/attestation/keylime.rs

+            Tee::Snp => (),
+            _ => bail!("invalid TEE"),
+        }
+


Suggested change

if data.tee != Tee::Snp {

bail!("invalid TEE");

}

While this makes complete sense at the moment, I want to be able to just add to this match statement as new TEE architectures become supported, like TDX.

Ok. I once thought this would be detected by cargo clippy as a lint error but it is not. Btw you might need to add feature keylime-as to be covered by Rust lint checker here

kbs/src/attestation/keylime.rs

tylerfanelli requested a review from a team as a code owner September 30, 2025 02:07

Xynnn007 reviewed Sep 30, 2025

View reviewed changes

tylerfanelli force-pushed the keylime-as branch from c669a64 to 98ef6d0 Compare September 30, 2025 14:21

fitzthum reviewed Sep 30, 2025

View reviewed changes

tylerfanelli force-pushed the keylime-as branch from 98ef6d0 to 7b18880 Compare September 30, 2025 18:37

tylerfanelli marked this pull request as draft September 30, 2025 23:52

tylerfanelli force-pushed the keylime-as branch 2 times, most recently from 3cbd5f6 to 10c51ff Compare October 1, 2025 03:10

tylerfanelli marked this pull request as ready for review October 1, 2025 03:13

tylerfanelli force-pushed the keylime-as branch from 10c51ff to d83a57d Compare October 1, 2025 03:15

mythi reviewed Oct 2, 2025

View reviewed changes

fitzthum reviewed Oct 2, 2025

View reviewed changes

kbs/src/attestation/keylime.rs Outdated Show resolved Hide resolved

fitzthum reviewed Oct 2, 2025

View reviewed changes

tylerfanelli force-pushed the keylime-as branch 3 times, most recently from f674909 to 0ab36f2 Compare October 4, 2025 03:41

tylerfanelli marked this pull request as draft October 7, 2025 02:35

tylerfanelli force-pushed the keylime-as branch 2 times, most recently from 0f0a52c to 65d1cd7 Compare October 8, 2025 01:21

tylerfanelli marked this pull request as ready for review October 8, 2025 03:36

Xynnn007 reviewed Oct 11, 2025

View reviewed changes

tylerfanelli force-pushed the keylime-as branch from 65d1cd7 to 6f46900 Compare October 14, 2025 00:34

tylerfanelli requested a review from Xynnn007 October 14, 2025 00:36

tylerfanelli force-pushed the keylime-as branch from 6f46900 to 404e1bb Compare October 14, 2025 02:30

tylerfanelli force-pushed the keylime-as branch 4 times, most recently from bc2cad6 to e812a89 Compare October 20, 2025 04:54

tylerfanelli added 2 commits October 20, 2025 00:54

kbs/config: Add example config for keylime AS

f9110a7

Create an example config file to show how one could configure the KBS to use Keylime as an alternative attestation service. Signed-off-by: Tyler Fanelli <[email protected]>

tylerfanelli force-pushed the keylime-as branch from e812a89 to f9110a7 Compare October 20, 2025 04:54

Xynnn007 reviewed Oct 23, 2025

View reviewed changes

tylerfanelli mentioned this pull request Oct 31, 2025

attestation: Update kernel module and proxy for AES-GCM, other small changes coconut-svsm/svsm#828

Open

+if data.tee != Tee::Snp {
+    bail!("invalid TEE");
+}

Introduce keylime TEE attestation service support #980

Are you sure you want to change the base?

Introduce keylime TEE attestation service support #980

Uh oh!

Conversation

tylerfanelli commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO in later patch series

Uh oh!

Xynnn007 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tylerfanelli commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fitzthum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tylerfanelli commented Sep 30, 2025

Uh oh!

tylerfanelli commented Oct 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fitzthum left a comment

Choose a reason for hiding this comment

Uh oh!

tylerfanelli commented Oct 2, 2025

Uh oh!

tylerfanelli commented Oct 8, 2025

Uh oh!

Xynnn007 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tylerfanelli Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tylerfanelli commented Sep 30, 2025 •

edited

Loading

tylerfanelli commented Sep 30, 2025 •

edited

Loading

tylerfanelli Oct 20, 2025 •

edited

Loading

tylerfanelli Oct 24, 2025 •

edited

Loading