Skip to content

Commit 0f16bd3

Browse files
author
Rayhan Hossain
committed
Add playground example with Azure Arc and Fleet
Signed-off-by: Rayhan Hossain <rhossain@microsoft.com>
1 parent 39b0596 commit 0f16bd3

11 files changed

Lines changed: 1850 additions & 0 deletions

File tree

docs/designs/azure-arc/azure-arc-integration-plan.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ This document outlines a two-phase plan to integrate DocumentDB Kubernetes Opera
1010

1111
> **Note:** "Kubernetes extensions" (via `az k8s-extension`) are specific to Kubernetes clusters. Azure has separate extension mechanisms for VMs, Arc servers, etc.
1212
13+
> **Interim Solution:** For immediate portal visibility without extension registration or billing integration, see [arc-hybrid-setup-with-fleet](../../../documentdb-playground/arc-hybrid-setup-with-fleet/) which uses Azure Fleet Manager + direct Helm deployment. This provides cluster tracking today while the full integration is being developed.
14+
1315
### Cluster Type Support
1416

1517
| Cluster Type | Extension Cluster Type | Arc Agent Required? |
Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# Copilot Agent Instructions: DocumentDB Fleet Hybrid Setup
2+
3+
## Overview
4+
5+
This guide helps AI agents assist users in setting up DocumentDB on a hybrid environment using Azure Fleet Manager. The setup creates visibility in Azure Portal for both cloud (AKS) and on-premises (Arc-enabled) clusters.
6+
7+
**End Result:**
8+
- Fleet Hub with 2 member clusters
9+
- AKS cluster with DocumentDB operator + instance
10+
- Kind cluster (simulating on-prem) with Arc agent, DocumentDB operator + instance
11+
- Full visibility in Azure Portal
12+
13+
**Estimated Time:** 30-45 minutes
14+
15+
## Agent Behavior
16+
17+
**CRITICAL:** This setup requires commands in **two different environments**:
18+
19+
| Environment | Used For | Commands |
20+
|-------------|----------|----------|
21+
| **WSL/Linux** | Kind, Docker, kubectl, Helm | `kind`, `docker`, `kubectl`, `helm` |
22+
| **PowerShell (Windows)** | Azure CLI | `az` |
23+
24+
**Always tell the user which environment to run each command in.**
25+
26+
> **Why dual environments?** In corporate environments with Conditional Access Policies, `az login` fails in WSL. PowerShell bypasses this limitation.
27+
28+
---
29+
30+
## Environment Setup
31+
32+
### WSL Terminal
33+
```bash
34+
# Verify tools
35+
kubectl version --client
36+
helm version --short
37+
kind --version
38+
docker ps
39+
```
40+
41+
### PowerShell Terminal
42+
```powershell
43+
# Verify and install extensions
44+
az --version # Must be 2.50.0+
45+
az extension add --name fleet --upgrade --yes
46+
az extension add --name connectedk8s --upgrade --yes
47+
az extension add --name k8s-extension --upgrade --yes
48+
```
49+
50+
**KUBECONFIG Setup (PowerShell):**
51+
```powershell
52+
$env:KUBECONFIG = "\\wsl.localhost\Ubuntu\home\$env:USERNAME\.kube\config"
53+
```
54+
55+
---
56+
57+
## Execution Instructions
58+
59+
### Phase 1: Create Fleet Hub + AKS (PowerShell)
60+
61+
**Tell user:** "Run these commands in PowerShell"
62+
63+
```powershell
64+
# Variables - user can customize these
65+
$RESOURCE_GROUP = "documentdb-fleet-rg"
66+
$LOCATION = "westus2"
67+
$FLEET_NAME = "documentdb-fleet"
68+
$AKS_CLUSTER = "documentdb-aks"
69+
70+
# Create resource group
71+
az group create --name $RESOURCE_GROUP --location $LOCATION
72+
73+
# Create Fleet hub (hubless mode)
74+
az fleet create --resource-group $RESOURCE_GROUP --name $FLEET_NAME --location $LOCATION
75+
76+
# Create AKS cluster (~5-10 minutes)
77+
az aks create `
78+
--resource-group $RESOURCE_GROUP `
79+
--name $AKS_CLUSTER `
80+
--node-count 2 `
81+
--node-vm-size Standard_D4s_v3 `
82+
--enable-managed-identity `
83+
--generate-ssh-keys
84+
85+
# Join AKS to Fleet
86+
$AKS_ID = az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query id -o tsv
87+
az fleet member create `
88+
--resource-group $RESOURCE_GROUP `
89+
--fleet-name $FLEET_NAME `
90+
--name $AKS_CLUSTER `
91+
--member-cluster-id $AKS_ID
92+
93+
# Get AKS credentials to WSL kubeconfig
94+
$env:KUBECONFIG = "\\wsl.localhost\Ubuntu\home\$env:USERNAME\.kube\config"
95+
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER --overwrite-existing
96+
```
97+
98+
**Success check:** `az fleet member list` shows `documentdb-aks`
99+
100+
### Phase 2: Install cert-manager on AKS (WSL)
101+
102+
**Tell user:** "Run these commands in WSL"
103+
104+
```bash
105+
kubectl config use-context documentdb-aks
106+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.2/cert-manager.yaml
107+
kubectl wait --for=condition=Available deployment --all -n cert-manager --timeout=300s
108+
```
109+
110+
**Success check:** All cert-manager pods Running
111+
112+
### Phase 3: Create Kind Cluster (WSL)
113+
114+
**Tell user:** "Run these commands in WSL"
115+
116+
```bash
117+
kind create cluster --name documentdb-onprem --config - <<EOF
118+
kind: Cluster
119+
apiVersion: kind.x-k8s.io/v1alpha4
120+
nodes:
121+
- role: control-plane
122+
- role: worker
123+
- role: worker
124+
EOF
125+
126+
kubectl config use-context kind-documentdb-onprem
127+
kubectl get nodes
128+
```
129+
130+
**Success check:** 3 nodes in Ready state
131+
132+
### Phase 4: Arc-Enable Kind Cluster (PowerShell)
133+
134+
**Tell user:** "Run these commands in PowerShell"
135+
136+
**IMPORTANT:** Verify kubectl context first!
137+
138+
```powershell
139+
$env:KUBECONFIG = "\\wsl.localhost\Ubuntu\home\$env:USERNAME\.kube\config"
140+
kubectl config use-context kind-documentdb-onprem
141+
kubectl get nodes # MUST show "documentdb-onprem-*" nodes, NOT AKS nodes
142+
```
143+
144+
Then Arc-enable:
145+
146+
```powershell
147+
az connectedk8s connect `
148+
--name documentdb-onprem `
149+
--resource-group $RESOURCE_GROUP `
150+
--location $LOCATION `
151+
--tags "environment=onprem" "purpose=documentdb"
152+
153+
# Verify connection
154+
az connectedk8s show --name documentdb-onprem --resource-group $RESOURCE_GROUP `
155+
--query "{name:name, connectivityStatus:connectivityStatus}" -o table
156+
157+
# Join to Fleet
158+
$ARC_ID = az connectedk8s show -g $RESOURCE_GROUP -n documentdb-onprem --query id -o tsv
159+
az fleet member create `
160+
--resource-group $RESOURCE_GROUP `
161+
--fleet-name $FLEET_NAME `
162+
--name documentdb-onprem `
163+
--member-cluster-id $ARC_ID
164+
```
165+
166+
**Success check:** `connectivityStatus` = `Connected`, Fleet has 2 members
167+
168+
### Phase 5: Install cert-manager on Kind (WSL)
169+
170+
**Tell user:** "Run these commands in WSL"
171+
172+
```bash
173+
kubectl config use-context kind-documentdb-onprem
174+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.2/cert-manager.yaml
175+
kubectl wait --for=condition=Available deployment --all -n cert-manager --timeout=300s
176+
```
177+
178+
### Phase 6: Deploy DocumentDB Operator (WSL)
179+
180+
**Tell user:** "Run these commands in WSL from the repo root"
181+
182+
```bash
183+
cd operator/documentdb-helm-chart
184+
185+
# Kind cluster
186+
kubectl config use-context kind-documentdb-onprem
187+
helm install documentdb-operator . --namespace documentdb-operator --create-namespace --wait --timeout 10m
188+
kubectl get pods -n documentdb-operator # Verify Running
189+
190+
# AKS cluster
191+
kubectl config use-context documentdb-aks
192+
helm install documentdb-operator . --namespace documentdb-operator --create-namespace --wait --timeout 10m
193+
kubectl get pods -n documentdb-operator # Verify Running
194+
```
195+
196+
**Success check:** `documentdb-operator-*` and `cnpg-*` pods Running on both clusters
197+
198+
### Phase 7: Create Service Account Token for Arc Portal (WSL)
199+
200+
**Tell user:** "Run these commands in WSL"
201+
202+
```bash
203+
kubectl config use-context kind-documentdb-onprem
204+
205+
kubectl create serviceaccount arc-portal-viewer -n default
206+
kubectl create clusterrolebinding arc-portal-viewer-binding \
207+
--clusterrole=cluster-admin \
208+
--serviceaccount=default:arc-portal-viewer
209+
210+
# Generate 1-year token
211+
kubectl create token arc-portal-viewer -n default --duration=8760h
212+
```
213+
214+
**Tell user:** "Save this token - you'll need it for Azure Portal"
215+
216+
### Phase 8: Deploy DocumentDB Instance (WSL)
217+
218+
**Tell user:** "Run these commands in WSL"
219+
220+
```bash
221+
# Kind cluster
222+
kubectl config use-context kind-documentdb-onprem
223+
kubectl create namespace app-namespace
224+
kubectl create secret generic documentdb-credentials \
225+
--namespace app-namespace \
226+
--from-literal=username=docdbuser \
227+
--from-literal=password=YourSecurePassword123!
228+
229+
kubectl apply -f documentdb-instance.yaml
230+
kubectl get pods -n app-namespace -w # Wait for 2/2 Running
231+
kubectl get documentdb -n app-namespace # Should show "Cluster in healthy state"
232+
233+
# AKS cluster
234+
kubectl config use-context documentdb-aks
235+
kubectl create namespace app-namespace
236+
kubectl create secret generic documentdb-credentials \
237+
--namespace app-namespace \
238+
--from-literal=username=docdbuser \
239+
--from-literal=password=YourSecurePassword123!
240+
241+
kubectl apply -f documentdb-instance.yaml
242+
kubectl get pods -n app-namespace -w
243+
kubectl get documentdb -n app-namespace
244+
```
245+
246+
**Success check:** `demo-documentdb-1` pod 2/2 Running, DocumentDB status "Cluster in healthy state"
247+
248+
### Phase 9: Verify in Azure Portal
249+
250+
**Tell user:** "Verify in Azure Portal"
251+
252+
1. **Fleet:** https://portal.azure.com/#view/Microsoft_Azure_Fleet/FleetMenuBlade
253+
- Should show 2 members: `documentdb-aks`, `documentdb-onprem`
254+
255+
2. **AKS:** Azure Portal → AKS → `documentdb-aks` → Workloads → Pods
256+
- Select namespace `app-namespace`
257+
- Should see `demo-documentdb-1`
258+
259+
3. **Arc:** Azure Portal → Arc → Kubernetes → `documentdb-onprem` → Kubernetes resources
260+
- Click "Sign in with service account token"
261+
- Paste token from Phase 7
262+
- Select namespace `app-namespace`
263+
- Should see `demo-documentdb-1`
264+
265+
---
266+
267+
## Error Handling
268+
269+
| Error | Cause | Solution |
270+
|-------|-------|----------|
271+
| `az login` fails in WSL | Conditional Access Policy | Use PowerShell for `az` commands |
272+
| Kind context not in PowerShell | kubeconfig not shared | Set `$env:KUBECONFIG` to WSL path |
273+
| Arc CRD conflicts | Wrong kubectl context | Verify context with `kubectl get nodes` before Arc connect |
274+
| Arc connect timeout | Network issues | Retry, check firewall allows outbound to Azure |
275+
| Arc portal "token required" | No service account | Run Phase 7 |
276+
| Pods not visible in Portal | Wrong namespace | Select `app-namespace` from dropdown |
277+
278+
---
279+
280+
## Cleanup
281+
282+
**Tell user:** "Run cleanup when done to avoid charges"
283+
284+
**In WSL:**
285+
```bash
286+
kind delete cluster --name documentdb-onprem
287+
```
288+
289+
**In PowerShell:**
290+
```powershell
291+
az group delete --name documentdb-fleet-rg --yes --no-wait
292+
```
293+
294+
---
295+
296+
## Reference
297+
298+
- Full documentation: See `README.md` in this directory
299+
- DocumentDB instance spec: See `documentdb-instance.yaml`

0 commit comments

Comments
 (0)