Skip to content

Add AWS application stack catalog: VPC, EKS Fargate, RDS Postgres, S3, Helm app#41

Draft
coryodaniel wants to merge 6 commits into
mainfrom
feature/aws-catalog
Draft

Add AWS application stack catalog: VPC, EKS Fargate, RDS Postgres, S3, Helm app#41
coryodaniel wants to merge 6 commits into
mainfrom
feature/aws-catalog

Conversation

@coryodaniel
Copy link
Copy Markdown
Member

@coryodaniel coryodaniel commented May 8, 2026

Summary

A self-contained AWS application stack as Massdriver bundles. Real Terraform throughout — no random_pet simulations.

Bundles

  • aws-vpc — real VPC via terraform-aws-modules/vpc/aws. Configurable region, CIDR, AZ count, NAT mode, optional VPC Flow Logs.
  • aws-eks-fargate — bare-minimum EKS Fargate cluster. Cluster IAM role, EKS control plane, Fargate pod-execution role, Fargate profiles per namespace, plus a cluster-admin ServiceAccount with a long-lived bearer-token Secret used by the Massdriver helm provisioner. No OIDC provider, no addons, no log group — fork the bundle if you need them.
  • aws-rds-postgres — real RDS Postgres with IAM auth, Multi-AZ option, 0–5 read replicas, Secrets Manager-backed master credentials, parameter group with rds.force_ssl=1 + pg_stat_statements, enhanced monitoring.
  • aws-s3-bucket — real S3 bucket tuned for user-uploaded content: SSE-KMS or SSE-S3, versioning, CORS, lifecycle archive/expire, presign-friendly IAM policies, optional SNS event notifications, optional cross-region replication.
  • aws-app — in-bundle Helm chart that deploys nginx (public.ecr.aws/nginx/nginx:1.27 — anonymous ECR Public pulls, no Docker Hub rate limits) on EKS Fargate. Connection metadata from the linked RDS Postgres database and S3 bucket is rendered into the pod as environment variables. DB password is not in pod env — DB_SECRET_ARN points at Secrets Manager and a real workload would fetch credentials at runtime via IRSA.

Resource types

aws-vpc, aws-eks-cluster, aws-rds-postgres, aws-s3-bucket, aws-application. All flat (no data/specs wrappers). The Massdriver helm provisioner reads a v1-shaped kubernetes_cluster.data.authentication.cluster.server / …user.token contract; aws-app's helm step config reshapes the flat aws-eks-cluster artifact into that contract inline so the rest of the catalog's resource types stay clean.

Schema features

  • $md.immutable on identity-shaping fields: CIDR, AZ count, cluster name, namespace, bucket prefix, master username, encryption mode, ownership.
  • $md.sensitive on the RDS master password override and the cluster bearer token.
  • $md.enum is wired through the helm chart template (templates/aws-helm-chart/) for surfacing data-resource policy choices on scaffolded app bundles.

Compliance

  • Per-bundle .checkov.yml skip lists with rationale comments. Skips cover only false positives (cross-module flow logs, NAT EIPs, 443-only egress) and the project-level no-CMK policy. Configurable knobs (Multi-AZ, deletion protection, public endpoint, CRR, etc.) are deliberately not skipped — production presets set them, lower envs surface a warning.
  • Each bundle's checkov config has halt_on_failure: '.params.md_metadata.default_tags["md-target"] | test("^(prod|prd|production)$")'. Failures block deploy in production envs and surface as warnings in lower envs.

Operator runbooks

Each bundle ships a mustache-templated operator.md with copy-pasteable AWS CLI / kubectl / psql commands that interpolate the live cluster name, region, bucket name, ARNs, etc. Each bundle also ships a GitHub-style README.md describing what it provisions, its connections, outputs, and configuration highlights.

Helm chart template

templates/aws-helm-chart/ scaffolds new helm-deployed app bundles wired to aws-eks-cluster (and optionally aws-rds-postgres / aws-s3-bucket). Same in-bundle-chart pattern as aws-app — operators publish their app's chart inside the bundle, the helm provisioner picks it up.

Test plan

  • mass resource-type publish for each of the five resource types (admin rights required)
  • make publish-bundles (or per-bundle mass bundle publish --development)
  • Build a project canvas; link components per GUIDE.md
  • Set the AWS IAM role as the env default
  • Pin instances to release-channel=development for auto-redeploy on dev publishes
  • Deploy in order: vpc → cluster → db / uploads (parallel) → app
  • Verify env wiring on the running pod: kubectl exec -n app <pod> -- env | grep -E '^(DB|S3|EKS)_'

Notes

  • Per-region VPC quota of at least 1 free slot is required. The bundle's region enum supports the common 10 AWS regions. The example deploy in this PR uses us-west-1.
  • The helm provisioner authenticates via static bearer token, not exec — short-lived aws eks get-token tokens don't fit the provisioner's contract. The aws-eks-fargate bundle creates a kubernetes.io/service-account-token Secret bound to a cluster-admin ServiceAccount for that purpose.
  • The aws-app bundle's connection field for the cluster is named eks (not kubernetes_cluster); the cluster's artifact field is named kubernetes_cluster. So the canvas link is <cluster-component>.kubernetes_cluster → <app-component>.eks.
  • The aws-eks-fargate bundle does NOT provision an OIDC provider or set up IRSA. If your workload needs IRSA, run aws eks describe-cluster ... --query 'cluster.identity.oidc.issuer' and create the IAM identity provider out-of-band, then annotate ServiceAccounts with eks.amazonaws.com/role-arn.
Screenshot 2026-05-08 at 8 05 40 AM

🤖 Generated with Claude Code

coryodaniel and others added 3 commits May 7, 2026 17:21
User prompts:
- "I need you to put together a catalog (in a branch w/ a pr) for a customer ... A fairly basic VPC to get going, EKS Fargate, A helm chart template and a helm chart bundle (consuming s3 and postgres), An RDS Postgres module, security minded, support for replicas, An S3 bucket optimized for user-generated content uploaded through the helm chart, Use \$md.{enum,immutable,sensitive}. Make all resource-types / bundles aws- prefixed."
- "environmentDefaultGroup isn't a thing anymore, you can remove those and any mention of it completely."

Changes:
- Resource types: aws-vpc, aws-eks-cluster, aws-rds-postgres, aws-s3-bucket, aws-application
- Bundles: aws-vpc, aws-eks-fargate, aws-rds-postgres, aws-s3-bucket, aws-app (helm)
- Helm template: templates/aws-helm-chart with chart-aware params.jq
- Use \$md.immutable on cidr, az count, cluster name, namespace, db engine major bump-only fields
- Use \$md.sensitive on RDS master password
- Use \$md.enum to surface RDS/S3 access policy choices on the helm bundle
- Strip environmentDefaultGroup from all platforms, resource types, and README example

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User prompts:
- "Sorry forgot to give you resource type access, try again"

Changes:
- Replace hand-written variable declarations with the canonical output of
  `mass bundle build` after publishing the new aws-* resource types
- tofu fmt

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User prompts:
- "i'm going to step away to run some errands, as you iterate through compliance remember to set the instances to be on a development release channel so they automatically deploy."

Changes:
- aws-s3-bucket: real Terraform (S3 + KMS + IAM policies + lifecycle + CORS)
  - Explicit KMS key policy (closes CKV2_AWS_64)
  - Optional SNS event notifications via enable_event_notifications param (closes CKV2_AWS_62 when enabled)
  - Optional cross-region replication via replication_destination_bucket_arn param (closes CKV_AWS_144 when enabled)
  - Deployed and verified clean Checkov run when notifications enabled
- aws-vpc: added region param ($md.immutable, enum of 10 regions)
  - Real-TF variant using terraform-aws-modules/vpc/aws was tested and works,
    but every region tried (us-west-2, us-east-1) hits VpcLimitExceeded on the
    target AWS account. Reverted to simulation pending a quota bump.
  - When the VPC quota is raised, restore the real-TF version from git history
    (commit c373ba3 contains the reference implementation).
- All 5 instances set to release-channel=development so dev publishes auto-deploy
- tofu fmt

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coryodaniel coryodaniel changed the title Add AWS catalog: VPC, EKS Fargate, RDS Postgres, S3, helm app Add AWS application stack catalog: VPC, EKS Fargate, RDS Postgres, S3, Helm app May 8, 2026
coryodaniel and others added 3 commits May 8, 2026 03:19
A self-contained AWS application stack as Massdriver bundles with real
Terraform — no random_pet simulations.

Resource types
- aws-vpc — VPC with subnets and routing details
- aws-eks-cluster — flat shape: id/name/endpoint/CA/region/version/vpc_id/
  fargate_profiles/token. Token is $md.sensitive
- aws-rds-postgres — writer/reader endpoints, IAM-bindable policies, secret ARN
- aws-s3-bucket — bucket ARN/name/region/KMS, IAM-bindable policies
- aws-application — workload identifier output

Bundles
- aws-vpc — real VPC via terraform-aws-modules/vpc/aws. Configurable region,
  CIDR, AZ count, NAT mode, optional VPC Flow Logs
- aws-eks-fargate — minimal EKS cluster: cluster IAM, EKS control plane,
  Fargate pod-execution role, Fargate profiles per namespace, plus a
  cluster-admin ServiceAccount with a long-lived bearer-token Secret. The
  token + endpoint + CA are emitted as part of the kubernetes_cluster artifact
- aws-rds-postgres — RDS Postgres with IAM auth, Multi-AZ option, 0–5 read
  replicas, Secrets-Manager-backed credentials, parameter group with
  rds.force_ssl=1 + pg_stat_statements, enhanced monitoring
- aws-s3-bucket — S3 bucket tuned for UGC: KMS or SSE-S3, versioning, CORS,
  lifecycle archive/expire, presign-friendly IAM policies, optional SNS event
  notifications, optional cross-region replication
- aws-app — Helm chart shipped inside the bundle that deploys nginx
  (public.ecr.aws/nginx/nginx:1.27, anonymous ECR Public pulls, no Docker Hub
  rate limits). Demonstrates the helm provisioner end-to-end on EKS Fargate

Helm provisioner contract
The aws-app step config shapes the v1 kubernetes_cluster contract
(.data.authentication.cluster.server / .user.token) inline from the flat
aws-eks-cluster artifact, so the rest of the catalog stays free of `data`
wrappers

Schema features
- $md.immutable on identity-shaping fields (CIDR, AZ count, cluster name,
  namespace, bucket prefix, master username, encryption mode, ownership)
- $md.sensitive on the master password override and the cluster bearer token
- $md.enum on app bundle's database_policy and bucket_policy in the richer
  variant of this bundle (current minimal app drops these for clarity)

Compliance
- Per-bundle .checkov.yml skip lists with rationale comments — false
  positives only (cross-module flow logs, NAT EIPs, 443-only egress) and the
  project-level no-CMK policy
- halt_on_failure regex on every bundle's checkov config: failures block
  deploy when md_metadata.default_tags.md-target matches
  ^(prod|prd|production)$, and surface as warnings on lower envs

Operator runbooks
Each bundle ships a mustache-templated operator.md with copy-pasteable
AWS CLI / kubectl / psql commands that interpolate live cluster name,
region, bucket name, ARNs, etc. Header strips kept tight (instance + one
key endpoint), full config visible on the Massdriver config panel

GUIDE.md
End-to-end setup guide: clone, configure, publish RTs (admin rights),
publish bundles, build the canvas, link components, attach the AWS
credential, deploy in order. Includes a verification step using
port-forward + curl

Bundle READMEs
GitHub-style README per bundle with What it provisions / Connections /
Outputs / Configuration highlights / Compliance / Operator runbook

Helm chart template
templates/aws-helm-chart/ scaffolds new helm-deployed app bundles wired to
aws-eks-cluster (and optionally aws-rds-postgres / aws-s3-bucket)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The minimal nginx demo chart now consumes connection metadata from the
linked aws-rds-postgres and aws-s3-bucket bundles and renders it into the
pod as environment variables. No plaintext secrets — DB password lives in
Secrets Manager and is fetched at runtime by the workload via IRSA.

- aws-app/massdriver.yaml: connections back to required eks + database +
  bucket
- chart/values.jq: pulls .connections.database.auth.* and
  .connections.bucket.{name,region,endpoint} into appEnv
- chart/values.yaml + templates/deployment.yaml: render appEnv as a
  range over env vars
- README updated to describe the actual wiring (connections, env keys,
  helm-provisioner contract reshape, IRSA for runtime credentials)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrites GUIDE.md and bundle READMEs/runbooks to match the bundles' actual
shape after the simplification pass:

- Drop references to removed params: domain_name, database_policy,
  bucket_policy, upload_prefix, log_level on aws-app; endpoint_access,
  log_types, secrets_encryption_enabled, public_access_cidrs on
  aws-eks-fargate.
- Drop references to removed resources/concepts: kubernetes-cluster
  resource type, two-artifact EKS pattern, OIDC provider + IRSA scaffolding,
  CoreDNS compute-type annotation patch, custom cluster security group.
- Update mass component link example: app's connection field is `eks`,
  cluster's artifact field is `kubernetes_cluster`.
- Update aws-app chart references to nginx (public.ecr.aws/nginx/nginx:1.27);
  drop mendhak/http-https-echo framing and `curl localhost | jq .env`
  verification — replace with `kubectl exec ... env | grep`.
- Replace stale `{{params.upload_prefix}}` mustache in
  aws-s3-bucket/operator.md with a literal example prefix.
- Bring templates/aws-helm-chart in line with the in-bundle chart pattern
  used by aws-app (no chart repo/name/version, eks connection, inline v1
  contract reshape).
@coryodaniel coryodaniel marked this pull request as draft May 8, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant