Skip to content

Commit 6acb924

Browse files
idan-starkwareidan-starkware
authored
deployment: cdk8s deployment config 2.0 (#10104)
* deployment: cdk8s deployment config 2.0 * deployment: add application config placeholders override mechanism * deployment: add hybrid configs * deployment: application config placeholder replace mechanism * deployment: service ports from common config * deployment: add podmonitoring to common config * deployment: refactor config merge flow * deployment: polish code * deployment: polish monitoring naming convention * delpoyment: update docs * deployment: common config is optional and will be ignored if not provided * deployment: prep consolidated layout for default usage * deployment: prep hybrid layout for ci testing * deployment: config 2.0 ci testing (#10143) * deployment: prep hybrid layout for ci testing * deployment: sync hybrid system tests with deployment config 2.0 * deployment: deployment-config2.0 ci testing * deployment: refactor system tests scripts * deployment: test secrets * deployment: adjust sequencer-cdk8s-test to deployment_config2.0 * deployment: sort service ports
1 parent 537e8fc commit 6acb924

File tree

82 files changed

+35655
-22
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

82 files changed

+35655
-22
lines changed

‎.github/workflows/hybrid_system_test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ on:
99
default: 10
1010
type: number
1111

12-
pull_request:
12+
# pull_request:
1313

1414
env:
1515
job_link: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

‎.github/workflows/hybrid_system_test2.yaml‎

Lines changed: 391 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/workflows/sequencer_cdk8s-test.yml‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,23 @@ name: Sequencer-Cdk8s-Test
22
on:
33
workflow_dispatch:
44

5-
push:
6-
branches:
7-
- main
8-
- main-v[0-9].**
9-
tags:
10-
- v[0-9].**
11-
paths:
12-
- ".github/workflows/sequencer_cdk8s-test.yml"
13-
- "deployments/sequencer/**"
14-
15-
pull_request:
16-
branches:
17-
- main
18-
- main-v[0-9].**
19-
paths:
20-
- ".github/workflows/sequencer_cdk8s-test.yml"
21-
- "deployments/sequencer/**"
5+
# push:
6+
# branches:
7+
# - main
8+
# - main-v[0-9].**
9+
# tags:
10+
# - v[0-9].**
11+
# paths:
12+
# - ".github/workflows/sequencer_cdk8s-test.yml"
13+
# - "deployments/sequencer/**"
14+
15+
# pull_request:
16+
# branches:
17+
# - main
18+
# - main-v[0-9].**
19+
# paths:
20+
# - ".github/workflows/sequencer_cdk8s-test.yml"
21+
# - "deployments/sequencer/**"
2222

2323
jobs:
2424
prepare:

‎.github/workflows/sequencer_cdk8s-test2.yml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
name: Sequencer CdK8s Dry Test
2+
on:
3+
workflow_dispatch:
4+
5+
push:
6+
paths:
7+
- ".github/workflows/sequencer_cdk8s-test2.yml"
8+
- "deployments/sequencer2/**"
9+
10+
pull_request:
11+
paths:
12+
- ".github/workflows/sequencer_cdk8s-test2.yml"
13+
- "deployments/sequencer2/**"
14+
15+
jobs:
16+
prepare:
17+
runs-on: ubuntu-24.04
18+
env:
19+
cluster: test
20+
namespace: test
21+
monitoring_dashboard_file: ${{ github.workspace }}/deployments/monitoring/examples/output/dashboards/sequencer_node_dashboard.json
22+
cdk8s_sequencer_path: ${{ github.workspace }}/deployments/sequencer2
23+
24+
steps:
25+
- name: Checkout sequencer
26+
uses: actions/checkout@v4
27+
28+
- name: Setup python
29+
uses: actions/[email protected]
30+
with:
31+
python-version: "3.10"
32+
cache: "pip"
33+
34+
- name: Setup Node
35+
uses: actions/[email protected]
36+
with:
37+
node-version: 22
38+
39+
- name: Install pip dependencies
40+
run: python3 -m pip install black pipenv
41+
42+
- name: Install cdk8s-cli
43+
run: npm install -g [email protected]
44+
45+
# Synthesize the CDK8s Sequencer app.
46+
- name: CDk8s synth
47+
working-directory: ${{ env.cdk8s_sequencer_path }}
48+
run: |
49+
cdk8s import
50+
pipenv install
51+
cdk8s synth --app "pipenv run python -m main --namespace ${{ env.namespace }} -l hybrid --monitoring-dashboard-file ${{ env.monitoring_dashboard_file }} --cluster ${{ env.cluster }}"
52+
53+
- name: Upload artifacts
54+
uses: actions/upload-artifact@v4
55+
with:
56+
name: cdk8s-artifacts
57+
path: |
58+
${{ env.cdk8s_sequencer_path }}/dist
59+
${{ env.cdk8s_sequencer_path }}/resources
60+
61+
validate:
62+
runs-on: ubuntu-24.04
63+
needs: prepare
64+
env:
65+
dist_path: ./cdk8s-artifacts/dist
66+
crds_path: ./cdk8s-artifacts/resources/crds
67+
steps:
68+
- name: Setup go lang
69+
uses: actions/setup-go@v5
70+
with:
71+
go-version: "stable"
72+
73+
- name: Setup kubectl-validate
74+
run: go install sigs.k8s.io/kubectl-validate@latest
75+
76+
- name: Download artifacts
77+
uses: actions/download-artifact@v4
78+
with:
79+
name: cdk8s-artifacts
80+
path: cdk8s-artifacts
81+
merge-multiple: true
82+
83+
- name: kubectl validation test for version 1.27
84+
run: kubectl validate ${{ env.dist_path }} --local-crds ${{ env.crds_path }} --version 1.27
85+
continue-on-error: true
86+
87+
- name: kubectl validation test for version 1.28
88+
run: kubectl validate ${{ env.dist_path }} --local-crds ${{ env.crds_path }} --version 1.28
89+
continue-on-error: true
90+
91+
- name: kubectl validation test for version 1.29
92+
run: kubectl validate ${{ env.dist_path }} --local-crds ${{ env.crds_path }} --version 1.29
93+
continue-on-error: false
94+
95+
- name: kubectl validation test for version 1.30
96+
run: kubectl validate ${{ env.dist_path }} --local-crds ${{ env.crds_path }} --version 1.30
97+
continue-on-error: false
98+
99+
- name: kubectl validation test for version 1.31
100+
run: kubectl validate ${{ env.dist_path }} --local-crds ${{ env.crds_path }} --version 1.31
101+
continue-on-error: false
102+
103+
- name: kubectl validation test for version 1.32
104+
run: kubectl validate ${{ env.dist_path }} --local-crds ${{ env.crds_path }} --version 1.32
105+
continue-on-error: false

‎crates/apollo_deployments/resources/testing_secrets.json‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,4 @@
66
"recorder_url": "http://dummy-recorder-service.dummy-recorder.svc.cluster.local:8080",
77
"state_sync_config.central_sync_client_config.central_source_config.http_headers": "",
88
"state_sync_config.network_config.secret_key" : "0x0101010101010101010101010101010101010101010101010101010101010101"
9-
}
10-
11-
12-
9+
}

‎deployments/sequencer2/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
__pycache__/
2+
.idea/
3+
dist/
4+
imports/

‎deployments/sequencer2/CDK8S_CONCEPTS.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# CDK8s Concepts Explained
2+
3+
## Hierarchy
4+
5+
```
6+
App
7+
├── Chart 1 (provides namespace, generates YAML directory)
8+
│ └── Construct (organizes resources)
9+
│ └── Construct (can nest deeper)
10+
│
11+
└── Chart 2 (separate namespace, separate YAML directory)
12+
└── Construct
13+
```
14+
15+
## App
16+
17+
- **One per application** - the root container
18+
- Handles synthesis (`app.synth()`) - generates YAML files
19+
- No namespace - just the root of the tree
20+
- **Example:** `app = App(...)`
21+
22+
## Chart
23+
24+
- **Purpose:** Groups resources under a **namespace** and organizes YAML output
25+
- **Rule:** Must be a direct child of `App` or another `Chart`
26+
- **Creates:** One YAML output directory per Chart
27+
- **Provides:** Namespace that all children inherit
28+
- **Inherits from:** `Chart` class (which inherits from `Construct`)
29+
- **Example:** `NodeChart` (line 20 in app.py)
30+
31+
```python
32+
class NodeChart(Chart): # ✅ Correct - inherits from Chart
33+
def __init__(self, scope: Construct, ...): # scope must be App or Chart
34+
super().__init__(scope, name, namespace=namespace)
35+
```
36+
37+
## Construct
38+
39+
- **Purpose:** Reusable component that organizes related resources
40+
- **Rule:** Can be a child of `Chart` or another `Construct`
41+
- **Does NOT:** Provide namespace (inherits from parent Chart)
42+
- **Does NOT:** Create separate YAML directory
43+
- **Inherits from:** `Construct` class
44+
- **Example:** `NodeConstruct`, `DeploymentConstruct`, `ServiceConstruct`
45+
46+
```python
47+
class NodeConstruct(Construct): # ✅ Correct - inherits from Construct
48+
def __init__(self, scope: Construct, ...): # scope can be Chart or Construct
49+
super().__init__(scope, id)
50+
# Now create children:
51+
self.service = ServiceConstruct(self, "service", ...) # 'self' is the scope
52+
```
53+
54+
## Scope Explained
55+
56+
**Scope** = the parent container in the construct tree.
57+
58+
### How scope works:
59+
60+
1. **Every construct takes `scope` as first parameter:**
61+
```python
62+
def __init__(self, scope: Construct, id: str, ...):
63+
super().__init__(scope, id) # Registers self as child of scope
64+
```
65+
66+
2. **When creating a child, pass `self` as scope:**
67+
```python
68+
# In NodeConstruct:
69+
self.service = ServiceConstruct(
70+
self, # ← 'self' (NodeConstruct) is the scope/parent
71+
"service", # ← id/name of this construct
72+
...
73+
)
74+
```
75+
76+
3. **What children inherit from scope:**
77+
- Namespace (from the parent Chart)
78+
- Name prefix (constructs get unique names like `NodeChart-NodeConstruct-service`)
79+
- Organization (resources grouped together)
80+
81+
### Scope Hierarchy Example:
82+
83+
```python
84+
# main() function:
85+
app = App(...) # Root - has no scope
86+
87+
# Inside main():
88+
NodeChart(scope=app, ...) # Chart - scope is App
89+
│
90+
└── NodeConstruct(scope=self, ...) # Construct - scope is NodeChart
91+
│
92+
├── ServiceConstruct(scope=self, ...) # scope is NodeConstruct
93+
├── DeploymentConstruct(scope=self, ...) # scope is NodeConstruct
94+
└── ConfigMapConstruct(scope=self, ...) # scope is NodeConstruct
95+
```
96+
97+
## Why Your Code Structure Works
98+
99+
### Your Current Structure:
100+
```
101+
App
102+
└── NodeChart (Chart)
103+
└── NodeConstruct (Construct)
104+
├── ServiceConstruct
105+
├── DeploymentConstruct
106+
└── ConfigMapConstruct
107+
```
108+
109+
### Why NodeConstruct is a Construct (not Chart):
110+
111+
1. **Namespace:** All resources in NodeConstruct inherit namespace from NodeChart
112+
2. **Organization:** Resources stay together in the same YAML directory
113+
3. **Flexibility:** NodeConstruct can be reused in different Charts
114+
4. **Tree Rules:** Only one level of Chart needed (NodeChart), then Constructs can nest
115+
116+
### If NodeConstruct were a Chart (WRONG):
117+
118+
```python
119+
# This would be WRONG:
120+
class NodeConstruct(Chart): # ❌ Don't do this!
121+
...
122+
```
123+
124+
Problems:
125+
- Charts can't be nested directly (only App → Chart → Chart is allowed, but creates separate namespace)
126+
- Each Chart creates separate YAML output directory
127+
- Resources wouldn't share namespace properly
128+
129+
## Quick Reference
130+
131+
| Type | Inherits From | Scope Can Be | Provides Namespace? | Creates YAML Dir? |
132+
|------|---------------|--------------|---------------------|-------------------|
133+
| **App** | - | - | ❌ | ❌ (root) |
134+
| **Chart** | `Chart` (→ `Construct`) | `App` or `Chart` | ✅ | ✅ |
135+
| **Construct** | `Construct` | `Chart` or `Construct` | ❌ (inherits) | ❌ |
136+
137+
## Naming Convention (Your Codebase)
138+
139+
- `*Chart` classes → inherit from `Chart` → provide namespace
140+
- `*Construct` classes → inherit from `Construct` → organize resources
141+

‎deployments/sequencer2/Pipfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
[[source]]
2+
name = "pypi"
3+
url = "https://pypi.org/simple"
4+
verify_ssl = true
5+
6+
[dev-packages]
7+
8+
[packages]
9+
cdk8s = "~=2.66.2"
10+
constructs = "~=10.2.70"
11+
jsonschema = "~=4.23.0"
12+
exceptiongroup = "~=1.2.2"
13+
types-jsonschema = "*"
14+
mypy = "*"
15+
black = "*"
16+
isort = "*"
17+
pydantic = "*"
18+
pyyaml = "*"
19+
types-pyyaml = "*"
20+
21+
[requires]
22+
python_version = "3.10"

0 commit comments

Comments
 (0)