Harness CI/CD pipeline for RAG applications

This repository contains Terraform code that implements the reference architecture described in this reference architecture Harness CI/CD pipeline for RAG applications. It provides a complete infrastructure-as-code solution for setting up a continuous integration (CI) and continuous deployment (CD) pipeline designed for Retrieval-Augmented Generation (RAG) applications in Google Cloud.

With this code, you can:

• Create the multi-project GCP infrastructure described in the blog post
• Set up Harness CI/CD pipelines with all necessary connectors and configurations
• Deploy a Kubernetes delegate to connect your infrastructure with Harness
• Configure production approval policies for secure deployment workflows

Follow the instructions in this README to implement the architecture and start deploying your RAG applications using Harness CI/CD and Google Cloud Run.

Contents

• Architecture overview
• Prerequisites
• Required tokens and credentials
  • Harness account ID
  • Harness API key
  • GitHub personal access token (PAT)
  • GCP billing account ID
  • Kubernetes configuration
    • Using cloud-based Kubernetes
    • Using Minikube for local development
    • Using Kind (Kubernetes IN Docker)
    • Important notes for local Kubernetes clusters
• Quick start guide
• Detailed setup instructions
• Project structure
• IAM role configuration
• Setting up Harness
• GitHub connector
• Production approval policy
• Customization
• Troubleshooting
• Destroying Resources
• License

Architecture overview

This reference architecture describes how to implement a continuous integration (CI) and continuous deployment (CD) pipeline for a Retrieval-Augmented Generation (RAG) application in Google Cloud. The architecture uses Harness CI/CD products to deploy containers to Cloud Run services.

The architecture consists of:

• GCP Projects:

  • Harness Project: Hosts the Artifact Registry repository and the Harness service account
  • Environment Projects: Separate projects for develop, staging, and production environments

• Harness CI/CD Configuration:

  • Project: Defines the Harness project structure
  • Environments: Dev, Staging, and Production environments
  • Services: Frontend and Backend service definitions for the RAG application
  • Pipelines: CI/CD pipeline for building, testing, and deploying to different environments
  • Delegate: Kubernetes-based Harness delegate for executing pipeline tasks

Prerequisites

• Terraform 1.0.0 or newer
• Google Cloud SDK
• Harness account and API key
• Kubernetes cluster (for the Harness delegate)
• Appropriate permissions to create GCP projects and assign IAM roles

Required tokens and credentials

This project requires several tokens and credentials to function properly:

Harness account ID

To find your Harness account ID:

1. Log in to Harness
2. Look for your Account ID in the URL: https://app.harness.io/#/account/[ACCOUNT_ID]/...
3. Add this to your harness/terraform.tfvars file:

   account_id = "your_harness_account_id"

Harness API key

To generate a Harness API key:

1. Log in to Harness at https://app.harness.io
2. Click your profile icon in the top-right corner and select "User Settings"
3. Navigate to the "API Keys" tab
4. Click "Generate API Key"
5. Enter a name for your API key (for example, "Terraform Integration")
6. Select the appropriate permissions (typically "Account Administrator" for full access)
7. Click "Generate"
8. Copy the generated API key and store it securely (this key will only be shown once)
9. Add the API key to your harness/terraform.tfvars file:

   api_key = "your_harness_api_key"

GitHub personal access token (PAT)

To create a GitHub personal access token:

1. Log in to GitHub at https://github.com
2. Click your profile picture → Settings → Developer settings → Personal access tokens → Tokens (classic)
3. Click "Generate new token" → "Generate new token (classic)"
4. Add a note (for example, "Harness Integration")
5. Select scopes: at minimum, select repo (Full control of private repositories)
6. Click "Generate token"
7. Copy the generated token and store it securely (this token will only be shown once)
8. Add the token to your harness/terraform.tfvars file:

   github_pat = "your_github_personal_access_token"

GCP billing account ID

To find your GCP billing account ID:

1. Log in to GCP Console at https://console.cloud.google.com
2. Navigate to Billing
3. Select the billing account you want to use
4. Copy the Billing Account ID (displayed in the format XXXXXX-XXXXXX-XXXXXX)
5. Add it to your gcp/terraform.tfvars file:

   billing_account = "XXXXXX-XXXXXX-XXXXXX"

Kubernetes configuration

For the Harness delegate to work, you need a properly configured Kubernetes cluster. You can use a cloud-based cluster like GKE or EKS, or set up a local cluster using Minikube or Kind for development and testing.

Using cloud-based Kubernetes

1. Ensure kubectl is configured to point to the cluster where you want to deploy the delegate

2. Check access:

   kubectl cluster-info
   kubectl get nodes

3. Create kubeconfig if needed:

   # For GKE
   gcloud container clusters get-credentials [CLUSTER_NAME] --zone [ZONE] --project [PROJECT_ID]
   
   # For EKS
   aws eks update-kubeconfig --name [CLUSTER_NAME] --region [REGION]

4. Verify Terraform can access the cluster: The Terraform Kubernetes provider will use the same kubeconfig as kubectl

Using Minikube for local development

Minikube is a tool that lets you run Kubernetes locally. It's a good option for testing the Harness delegate on your laptop.

1. System Requirements:

   • 2 CPUs or more
   • 2GB of free memory
   • 20GB of free disk space
   • Internet connection
   • Container or virtual machine manager (Docker, Hyperkit, Hyper-V, KVM, Parallels, Podman, VirtualBox, or VMware Fusion/Workstation)

2. Check system resources:

   # On macOS/Linux
   sysctl -n hw.physicalcpu  # Check CPU cores
   free -m                   # Check available memory (Linux)
   vm_stat                   # Check memory stats (macOS)
   df -h                     # Check disk space
   
   # On Windows (PowerShell)
   Get-WmiObject -Class Win32_Processor | Select-Object NumberOfCores
   Get-WmiObject -Class Win32_OperatingSystem | Select-Object FreePhysicalMemory
   Get-PSDrive C | Select-Object Free

3. Install Minikube:

   # macOS (with Homebrew)
   brew install minikube
   
   # Linux
   curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
   sudo install minikube-linux-amd64 /usr/local/bin/minikube
   
   # Windows (with Chocolatey)
   choco install minikube

4. Start Minikube with sufficient resources:

   minikube start --cpus 2 --memory 4096 --disk-size 20g

5. Verify Minikube is running:

   minikube status
   kubectl cluster-info

6. Configure kubectl to use Minikube:

   kubectl config use-context minikube

Using Kind (Kubernetes IN Docker)

Kind is another tool for running local Kubernetes clusters using Docker containers as nodes.

1. System Requirements:

   • Docker installed and running
   • 2GB of RAM (4GB recommended)
   • 2 CPU cores (4 recommended)
   • 20GB of free disk space

2. Check Docker is installed and running:

   docker --version
   docker info

3. Install Kind:

   # macOS (with Homebrew)
   brew install kind
   
   # Linux/macOS
   curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-$(uname)-amd64
   chmod +x ./kind
   sudo mv ./kind /usr/local/bin/kind
   
   # Windows (with Chocolatey)
   choco install kind

4. Create a Kind cluster with sufficient resources:

   # Create a config file
   cat > kind-config.yaml <<EOF
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
   - role: control-plane
     extraPortMappings:
     - containerPort: 80
       hostPort: 80
     - containerPort: 443
       hostPort: 443
   EOF
   
   # Create the cluster with a name related to the project
   kind create cluster --name harness-delegate-cluster --config kind-config.yaml

5. Verify Kind is running:

   kind get clusters
   kubectl cluster-info

Important notes for local Kubernetes clusters

When using a local Kubernetes cluster for the Harness delegate:

1. Resource Requirements: The Harness delegate requires at least:

   • 1 CPU core
   • 2GB RAM
   • 5GB disk space

2. Network Connectivity: The delegate needs outbound internet access to connect to Harness Platform.

3. Persistence: Local clusters like Minikube and Kind are ephemeral. If you restart your computer or the cluster, you may need to redeploy the delegate.

4. Performance: For production workloads, a cloud-based Kubernetes cluster is recommended. Local clusters are best for development and testing.

5. Delegate Sizing: For local testing, you can reduce the delegate resource requirements in the Terraform configuration:

   # In harness/main.tf
   module "delegate" {
     # ...
     delegate_cpu = "0.5"
     delegate_memory = "1Gi"
     # ...
   }

Quick start guide

1. Set up GCP Infrastructure:

   # Clone repository and configure
   cd gcp
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your billing account ID
   
   # Initialize and apply
   terraform init
   terraform apply

2. Set up Harness:

   # Generate Harness variables from GCP infrastructure outputs
   chmod +x scripts/generate_harness_tfvars.sh
   ./scripts/generate_harness_tfvars.sh
   # The script will prompt you for your Harness Account ID, API Key, GitHub URL, and GitHub PAT
   # It will create terraform.tfvars file in the harness/ directory
   
   # Initialize and apply to set up Harness resources
   cd harness
   terraform init
   terraform apply

Detailed setup instructions

Step 1: Set up GCP infrastructure

1. Clone this repository

2. Navigate to the GCP directory and copy terraform.tfvars.example to terraform.tfvars:

   cd gcp
   cp terraform.tfvars.example terraform.tfvars

3. Edit terraform.tfvars to provide your GCP billing account ID and other settings:

   billing_account = "YOUR-BILLING-ACCOUNT-ID"
   prepend         = "yourorg"  # Optional prefix for project IDs

4. Initialize and apply Terraform:

   terraform init
   terraform apply

Step 2: Set up Harness

Setting up Harness involves configuring both the Harness CI/CD pipelines and the Harness Delegate (which connects your infrastructure to Harness). The Terraform code sets up both components in a single step.

1. Make the script executable and generate the Harness terraform.tfvars file from the GCP infrastructure outputs:

   chmod +x scripts/generate_harness_tfvars.sh
   ./scripts/generate_harness_tfvars.sh

   The script will:

   • Prompt you for your Harness Account ID, API Key, GitHub URL, and GitHub Personal Access Token
   • Create terraform.tfvars file in the harness/ directory

2. Initialize and apply Terraform to set up Harness resources:

   cd harness
   terraform init
   terraform apply

   This will:

   • Set up the Harness CI/CD configuration
   • Configure environments, services, and pipelines

Project structure

This repository is organized into two main components:

1. GCP Infrastructure (gcp/ directory):

   • main.tf: Main Terraform configuration for GCP resources
   • variables.tf: Input variables for the GCP configuration
   • outputs.tf: Output values from the GCP configuration

2. Harness Configuration (harness/ directory):

   • main.tf: Main Terraform configuration for Harness resources
   • variables.tf: Input variables for the Harness configuration
   • outputs.tf: Output values from the Harness configuration
   • templates/: Template files for Harness resources
   • pipelines/: Pipeline definitions

IAM role configuration

This project uses carefully selected IAM roles for the Harness service account to ensure it has the necessary permissions while following the principle of least privilege:

Role	Purpose
roles/run.admin	Deploy and manage Cloud Run services
roles/artifactregistry.admin	Push and pull container images
roles/aiplatform.admin	Manage AI Platform resources
roles/monitoring.admin	Configure monitoring resources
roles/monitoring.viewer	View monitoring data
roles/storage.objectAdmin	Manage storage objects
roles/compute.networkViewer	View network resources

Setting up Harness

Setting up Harness involves two interconnected components: the Harness CI/CD configuration (which defines your pipelines and resources) and the Harness Delegate (which connects your infrastructure to Harness).

Harness CI/CD configuration

The Harness CI/CD configuration creates all necessary resources in Harness:

• Project Creation: Sets up the Harness project structure
• Environment Definition: Creates development, staging, and production environments
• Service Configuration: Defines frontend and backend services
• Pipeline Setup: Creates the RAG Vertex AI pipeline
• Token Generation: Generates a delegate token that will be used to set up the delegate

The pipeline configuration is fully variablized, making it more flexible and easier to maintain:

• Template-based: Uses Terraform's templatefile function to replace hardcoded values
• Customizable: All aspects of the pipeline can be configured through variables
• Maintainable: Changes to project IDs or connector names only need to be made in one place

Key pipeline variables:

# Pipeline configuration
harness_image_connector = "account.harnessImage"
k8s_connector          = "k8s_delegate_connector"
github_connector       = "github_connector"
repo_name              = "your-rag-application-repo"

The pipeline is configured to use delegate tags (rag, vertex-ai, gcp) that are applied to the delegate, ensuring that pipeline tasks are executed on delegates with the appropriate capabilities:

delegateSelectors:
  - rag
  - vertex-ai
  - gcp

Harness delegate

The Harness delegate is deployed to a Kubernetes cluster using the token generated by the Harness CI/CD configuration:

1. Token Generation: The Harness module creates a delegate token
2. Delegate Deployment: The Kubernetes delegate module uses this token to deploy a delegate
3. Configuration: The delegate connects to your Harness account automatically

You can customize the delegate configuration in the Harness module:

# In harness/main.tf
module "delegate" {
  # ...
  delegate_name = "terraform-delegate"
  namespace = "harness-delegate-ng"
  # ...
}

The delegate is configured with tags that can be used as selectors in pipelines and other Harness resources. By default, it includes the tags rag, vertex-ai, and gcp, which identify its purpose and capabilities. These tags can be referenced in pipeline configurations using the delegateSelectors field:

delegateSelectors:
  - rag
  - vertex-ai
  - gcp

This ensures that your pipelines run on the appropriate delegate with the required capabilities and access.

The Terraform code sets up both the Harness CI/CD configuration and the delegate in a single step, ensuring that the delegate token is properly generated and used.

GitHub connector

The project includes a GitHub connector for accessing repositories:

• Automated Setup: The connector is automatically created using Terraform
• Configurable: All aspects of the connector can be configured through variables
• Secure: GitHub credentials are stored securely in Harness as a secret

GitHub connector configuration:

# GitHub Connector configuration
github_connector_id    = "github_connector"
github_connector_name  = "GitHub Connector"
github_url             = "https://github.com/your-username" # Your GitHub URL
github_validation_repo = "your-rag-application-repo"
github_username        = "your-github-username"
github_pat             = "your-github-personal-access-token" # Get this from GitHub

The GitHub Personal Access Token (PAT) is stored as a secret in Harness, providing secure access to your GitHub repositories. You'll need to create a PAT in GitHub with the appropriate permissions (typically repo scope) and provide it in the terraform.tfvars file.

Production approval policy

The project includes a production approval policy to ensure that all production deployments are approved:

• Automated Enforcement: The policy is automatically enforced for all pipelines
• Rego-based: Uses the Open Policy Agent (OPA) Rego language for policy definition
• Customizable: The policy can be customized to fit your specific requirements

The policy ensures that the Approval stage must be completed successfully before a production deployment can proceed:

package pipeline

# Default to deny
default allow = false

# Allow if the approval stage has been approved
allow {
  input.pipeline.stages.Approval.spec.execution.steps.Approval.output.status == "APPROVED"
}

Customization

You can customize the configuration by modifying these files:

• GCP Infrastructure:

  • gcp/variables.tf: Modify input variables
  • gcp/main.tf: Add or modify GCP resources

• Harness Configuration:

  • harness/variables.tf: Modify Harness variables
  • harness/templates/*.tpl: Modify templates for Harness resources

Troubleshooting

Common issues

1. Project ID Conflicts: Change the prepend variable or increase random_prefix_length
2. Permission Issues: Ensure your GCP account has the necessary permissions
3. Harness API Key: Verify that your API key has the required permissions
4. Delegate Connection: Check that your Kubernetes cluster can reach Harness
5. GitHub PAT Missing: If you encounter the error "Required attribute github_pat not specified", make sure you've added your GitHub Personal Access Token to your harness/terraform.tfvars file. This token is required for the GitHub connector to access repositories.
6. Timing Issues with Resource Creation: The Terraform configuration includes a 30-second delay after Harness project creation to ensure all resources are properly initialized before dependent resources are created. If you're still experiencing timing issues, you may need to increase this delay in harness/main.tf.

Getting help

If you encounter issues:

1. Check the Terraform logs for error messages
2. Verify your GCP and Harness credentials
3. Ensure all required APIs are enabled in your GCP projects

Destroying Resources

When you're done with the infrastructure and want to clean up all resources to avoid incurring costs, follow these steps:

Option 1: Using the Cleanup Script (Recommended)

The cleanup script provides a controlled, sequential destruction of resources:

cd harness
./cleanup.sh

This script will:

1. Remove resources in the correct order to handle dependencies
2. Provide clear status updates during the process
3. Handle Harness API delays automatically
4. Verify the final state when complete

Option 2: Manual Terraform Destroy

If you prefer to use Terraform directly:

# First destroy Harness resources
cd harness
terraform destroy -auto-approve

# Then destroy GCP resources
cd ../gcp
terraform destroy -auto-approve

Important Considerations

• Order Matters: It's recommended to destroy the Harness resources before destroying the GCP infrastructure
• Manual Cleanup: Some resources might require manual cleanup through the GCP Console or Harness Platform
• Persistent Data: Back up any important data before destroying resources

Code Quality

This repository includes automated code quality checks to ensure consistent style and avoid common errors:

Terraform Linting

A GitHub Actions workflow automatically runs the following checks on all Terraform files:

• terraform validate: Checks for syntax and semantic errors
• terraform fmt: Ensures consistent code formatting
• tflint: Detects potential issues and best practice violations

The workflow runs whenever .tf files are modified in pull requests or when pushed to the main branch.

Resilience Testing (Optional)

For information on how to enhance your CI/CD pipeline with resilience testing using Grafana k6, refer to the Harness blog post:

Resilience Testing Your Applications Under Load Using Grafana k6

Adding resilience testing can help you identify performance bottlenecks and stability issues before they reach production.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.