Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .chloggen/fix-ca-cert-renewal-race.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: bug_fix

# The name of the component, or a single word describing the area of concern, (e.g. collector, target allocator, auto-instrumentation, opamp, github action)
component: target allocator

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Fix CA certificate renewal race condition by extending CA certificate duration to 1 year

# One or more tracking issues related to the change
issues: [4441]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext: |
The CA certificate now has a 1-year duration (instead of the default 90 days) to prevent race conditions
where client and server certificates could be signed by different CA versions during simultaneous renewal.
This ensures the CA remains stable while dependent certificates renew regularly.
9 changes: 6 additions & 3 deletions internal/controllers/builder_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package controllers

import (
"testing"
"time"

cmv1 "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1"
cmmetav1 "github.com/cert-manager/cert-manager/pkg/apis/meta/v1"
Expand Down Expand Up @@ -3260,9 +3261,11 @@ prometheus_cr:
Subject: &cmv1.X509Subject{
OrganizationalUnits: []string{"opentelemetry-operator"},
},
CommonName: "test-ca-cert",
IsCA: true,
SecretName: "test-ca-cert",
CommonName: "test-ca-cert",
Duration: &metav1.Duration{Duration: 8760 * time.Hour},
RenewBefore: &metav1.Duration{Duration: 2400 * time.Hour},
IsCA: true,
SecretName: "test-ca-cert",
IssuerRef: cmmetav1.ObjectReference{
Name: "test-self-signed-issuer",
Kind: "Issuer",
Expand Down
10 changes: 10 additions & 0 deletions internal/manifests/targetallocator/certificate.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package targetallocator

import (
"fmt"
"time"

cmv1 "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1"
cmmeta "github.com/cert-manager/cert-manager/pkg/apis/meta/v1"
Expand All @@ -28,6 +29,15 @@ func CACertificate(params Params) *cmv1.Certificate {
Spec: cmv1.CertificateSpec{
IsCA: true,
CommonName: naming.CACertificate(params.TargetAllocator.Name),
// Set CA certificate to 1 year (much longer than the default 90-day duration of client/server certs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the client/server certs renewal configurable?

Would it make sense to make the CA renewal shorter? / What is the lowest safe value possible for the CA renewal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

client/server certs renewal configurable?

No, when I'd looked at it, it was using cert-manager's defaults (which I had to look up, hence the comments for reference because i thought it was 90 days, but had to look it up).

I think that's fine, for simplicity

make the CA renewal shorter? / What is the lowest safe value possible for the CA renewal?

In general, I've seen CAs pretty long-lived. 1-year seems reasonable (I think even like 10 years would be ok, but guessing original author(s) were erring on shorter lifecycle for CA - I don't have that history, so I picked 1 year.

I think 1 year, combined with @swiatekm suggestion adding a renewBefore would actually eliminate the race.

Will push a change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// to prevent renewal race conditions where client and server certificates might be signed by different
// CA versions during simultaneous renewal. This ensures the CA remains stable while dependent certificates renew.
Duration: &metav1.Duration{Duration: 8760 * time.Hour}, // 1 year
// Set renewBefore to 100 days (longer than the 90-day client/server cert duration) to ensure:
// 1. CA renewal doesn't coincide with client/server renewal cycles (which occur every ~60 days at the 2/3 point of their 90-day lifetime)
// 2. The CA always has sufficient remaining validity (≥100 days) to safely issue client/server certificates with 90-day lifetimes
// 3. Client/server certificates can never outlive the CA certificate that signed them
RenewBefore: &metav1.Duration{Duration: 2400 * time.Hour}, // 100 days
Subject: &cmv1.X509Subject{
OrganizationalUnits: []string{"opentelemetry-operator"},
},
Expand Down
7 changes: 7 additions & 0 deletions internal/manifests/targetallocator/certificate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package targetallocator

import (
"testing"
"time"

"github.com/stretchr/testify/assert"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -83,6 +84,12 @@ func TestCACertificate(t *testing.T) {
assert.Equal(t, "Issuer", caCert.Spec.IssuerRef.Kind)
assert.Equal(t, []string{"opentelemetry-operator"}, caCert.Spec.Subject.OrganizationalUnits)
assert.Equal(t, tt.expectedLabels, caCert.Labels)
// Verify CA certificate has 1 year duration to prevent renewal race conditions
assert.NotNil(t, caCert.Spec.Duration)
assert.Equal(t, 8760*time.Hour, caCert.Spec.Duration.Duration)
// Verify CA certificate renewBefore is set to 100 days (longer than client/server cert duration)
assert.NotNil(t, caCert.Spec.RenewBefore)
assert.Equal(t, 2400*time.Hour, caCert.Spec.RenewBefore.Duration)
})
}
}
Expand Down
Loading