Skip to content

Commit 2e55691

Browse files
committed
added final details
1 parent cf4db7b commit 2e55691

File tree

18 files changed

+1060
-0
lines changed

18 files changed

+1060
-0
lines changed

.DS_Store

6 KB
Binary file not shown.
Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# Catalog-fed-ramp
2+
The following folder contains guidelines and examples on how to create scorecards based on custom attributes, as an example to meet FIPS compliance. See https://harness.atlassian.net/wiki/spaces/IE1/pages/22281127180/IDP+PS+Best+Practice+Implementations for case studies.
3+
4+
The scorecards are being reviewed by development team each week, and OPA policies will be eventually used to warn or enforce from deployment until dev team fixes defects to be in federal compliance.
5+
6+
The score card will be relying on several custom attributes to pass:
7+
8+
metadata.cve.warn
9+
metadata.cve.fail
10+
metadata.baseImage.compliance
11+
metadata.fips.compliance
12+
13+
These attributes are populated using 3 pipelines to test the service Dockerfile, Jira tickets, and deployed image. The pipelines run once a day for each registered service to update these flags. Timestamp attributes are stamped for each execution.
14+
15+
In this example, the pipelines are store in infra-harness repo, along with python scripts, and catalog-info.yaml. But there should be logic to allow overrides to use catalog-info.yaml residing in the service repo.
16+
17+
# Catalog
18+
Catalog file is centralized in a repo managed by Infra team, or it could be maintained by each development team owning the service. However, the pipeline that performs the checks should be owned by infra or security team.
19+
20+
Here is an example of a typical registration of catalog component.
21+
22+
```yaml
23+
# see https://developer.harness.io/docs/internal-developer-portal/catalog/how-to-create-idp-yaml/
24+
25+
apiVersion: backstage.io/v1alpha1
26+
kind: Component
27+
metadata:
28+
# name of the service
29+
name: test-service
30+
# tags - python/go/nodejs/java and fedramp are required for the scorecard to include only fedramp tags (exclusion rule does not exist yet)
31+
tags:
32+
- auto-generated
33+
- python
34+
- fedramp
35+
# useful links
36+
links:
37+
- url: https://app.harness.io/ng/account/xyz/module/cd/orgs/demo/projects/testervice/overview
38+
title: Deployment Dashboard
39+
icon: dashboard
40+
# recommended annotations
41+
annotations:
42+
# source code location for catalog-info.yaml, omit if move to service repo
43+
backstage.io/source-location: url:https://github.com/acme/test-service
44+
# service repository
45+
github.com/project-slug: acme/test-service
46+
# jira project key
47+
jira/project-key: xyz
48+
# optional jira component
49+
jira/component: xyz-service
50+
# technical doc
51+
backstage.io/techdocs-ref: url:https://github.com/acme/test-service
52+
# project url required for feature flag plugin
53+
harness.io/project-url: https://app.harness.io/ng/account/xyz/all/orgs/xyz/projects/testservice
54+
# serviceId required for listing CD and also used in OPA
55+
harness.io/cd-serviceId: testserviceid
56+
# numerate pipelines as needed (cd-serviceId list its default pipelines)
57+
harness.io/pipelines: |
58+
All in One Pipeline:
59+
Daily Security Scans: https://app.harness.io/ng/account/xyz/module/ci/orgs/demo/projects/testservice/pipelines/Daily_Security_Scan...
60+
# enumerate services as needed (cd-serviceId list its default service)
61+
harness.io/services: |
62+
test-service: http://...
63+
# optional jql warn / fail override (custom)
64+
acme/cve-warn: 'project = <<projectKey>> AND type = CVE AND "CVE SLA: Due date[Date]" >= endOfDay() AND "CVE SLA: Due date[Date]" <= 5d AND statusCategory != Done'
65+
acme/cve-fail: 'project = <<projectKey>> and type = CVE and "CVE SLA: Due date[Date]" < endOfDay() AND statusCategory != Done'
66+
67+
68+
spec:
69+
type: service
70+
lifecycle: staging
71+
# claim owner
72+
owner: devops_usergroup
73+
# optional reference to grpc definition
74+
providesApis:
75+
- test-service-grpc-api
76+
```
77+
78+
## Python scripts
79+
80+
# common.py
81+
Contains commonly used python functions used across scripts:
82+
83+
extract_image_and_tag - extracts the Dockerfile's image_name and tag. Requires stage name to be labelled.
84+
85+
fetch_catalog_attributes - reads the catalog info and extract any custom info from catalog info using annotations acme/docker-path and tags for languages
86+
87+
updateCatalogAttributes - very useful fucntion to update IDP catalog attributes
88+
89+
determine_catalog_path - determines where to pull catalog info yaml, either from service repo path ./harness/idp/catalog-info.yaml or from harness-infra repo (the repo hosting all the catalog-info.yaml owned by infra or security team)
90+
91+
# Base tag compliance check
92+
docker-basetag-check.py inspects the Dockerfile looking for specific stage tags, and inspects whether the image used matches the ones defined in compliant_image function. The results are posted back to IDP catalog.
93+
94+
# CVE check
95+
jira-cve-check-dir.py iterates through the catalog configuration file for warn / fail overrides (use default jql query defined in the code otherwise), and post queries to Jira. Results are updated into IDP catalog.
96+
97+
98+
# fetch-repo-default-branch.py
99+
Queries GITHUB to determine a service's default branch (main or master for example).
100+
101+
# checks language FIPs compliance
102+
fips-validator.py uses different logic to check if the image built by the service passes compliance for different languages. It examines the service's helm chart, determine the image built, pulls the image. It will perform MD5 tests for nodejs and python. It will check for specific attributes in the docker image for labels matching chain guard inages. Chain guard images are useful for fed compliance.
103+
104+
# experimental
105+
docker-image-fips.py is an experimental docker n docker build and execute commands. Deprecated. Uses fips subfolder for docker n docker build as an example
106+
107+
# catalog-cve-scorecard.yaml
108+
Pipeline to pull information from Jira to populate cve ticket violates into IDP
109+
110+
# dockerverifications-pipeline.yaml
111+
Pipeline to very base image used
112+
113+
# fips-pipeline.yaml
114+
Examine the helm chart in the service repo to pull the latest image, to check for FIPS compliance for different languages
115+
116+
# Custom checks for score card
117+
# Fips compliance check
118+
Catalog Info YAML
119+
Equal to TRUE
120+
jexl: <+metadata.fips.compliance>
121+
122+
# Base Image Compliance
123+
Catalog Info YAML
124+
Equal to TRUE
125+
jexl: <+metadata.baseImage.compliance>
126+
127+
# CVE Fail
128+
Catalog Info YAML
129+
Equal to 0
130+
jexl: <+metadata.cve.fail>
131+
132+
# CVE Warn
133+
Catalog Info YAML
134+
Equal to 0
135+
jexl: <+metadata.cve.warn>
Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
import os
2+
import re
3+
import argparse
4+
from datetime import datetime, timezone
5+
from common import updateCatalogAttributes, extract_image_and_tag, fetch_service_path
6+
import sys
7+
import requests
8+
import yaml
9+
import docker
10+
import subprocess
11+
12+
13+
def determine_image_language(dockerfile_path: str, fromstages: list):
14+
"""
15+
Determine the language from the matching stages
16+
:param dockerfile_path (str): Path to the Dockerfile
17+
:param fromstages (list): list of strings to filter
18+
:return: Boolean if crypto program executed successfully
19+
"""
20+
for stage in fromstages:
21+
image, tag = extract_image_and_tag(dockerfile_path, stage)
22+
print(f" stage: {stage} image: {image} tag: {tag}")
23+
# return first match
24+
if image != None and image.startswith("datarobotdev/platform-base-python"):
25+
return "python", image, tag
26+
if image != None and image.startswith("datarobotdev/platform-base-go"):
27+
return "go", image, tag
28+
if image != None and image.startswith("datarobotdev/platform-base-node"):
29+
return "nodejs", image, tag
30+
if image != None and image.startswith("datarobotdev/platform-base-java"):
31+
return "java", image, tag
32+
33+
return None, None, None
34+
35+
36+
def fips_build_check(language: str, image: str, tag: str, base_path: str):
37+
"""
38+
An example to build docker in docker to run crypto test
39+
:param language (str): Programming language (go, nodejs, python)
40+
:param image (str): The image to test
41+
:param tag (str): The tag to pull
42+
:param base_path (str): The basedirectory holding the harness-infra repo
43+
:return: Boolean if crypto program executed successfully
44+
"""
45+
client = docker.from_env()
46+
image_pull = f"{image}:{tag}"
47+
try:
48+
# pull the image
49+
image = client.images.pull(image_pull)
50+
print(f"Successfully pulled {image_pull}")
51+
52+
# grab image config and lables
53+
image = client.images.get(image_pull)
54+
labels = image.attrs.get("Config", {}).get("Labels", {})
55+
configs = image.attrs.get("Config", {})
56+
print(f"Configs: {configs}")
57+
print(f"Labels: {labels}")
58+
59+
# execute building image with test program
60+
print("Running docker build...")
61+
docker_command = [
62+
"docker",
63+
"build",
64+
"--build-arg",
65+
f"BASE_IMAGE={image_pull}",
66+
"-t",
67+
"fips-test",
68+
f"{base_path}/harness-infra/idp/fips/{language}/.",
69+
]
70+
exit_code = subprocess.call(docker_command)
71+
if exit_code == 0:
72+
print("Docker build succeeded.")
73+
else:
74+
print(f"Error occurred during Docker build. Exit code: {exit_code}")
75+
76+
# attempt to execute the built image
77+
print("Running docker container...")
78+
docker_command_run = ["docker", "run", "--rm", "fips-test"]
79+
exit_code = subprocess.call(docker_command_run)
80+
print(f"Docker run exited with code {exit_code}")
81+
82+
if exit_code == 0:
83+
print("The container ran successfully.")
84+
return True
85+
86+
print(f"An error occurred during Docker run. Exit code: {exit_code}")
87+
return False
88+
89+
except docker.errors.APIError as e:
90+
print(f"Error pulling image: {e}")
91+
except docker.errors.ImageNotFound:
92+
print(f"Image '{image_pull}' not found.")
93+
except Exception as e:
94+
print(f"An error occurred while retrieving image labels: {e}")
95+
96+
97+
def main():
98+
99+
parser = argparse.ArgumentParser(
100+
description="Process a Dockerfile for FIPS checks."
101+
)
102+
parser.add_argument(
103+
"--dockerfile_path",
104+
type=str,
105+
help="Absolute path to the Default docker path file (usually root dir of repo)",
106+
)
107+
108+
parser.add_argument(
109+
"--catalog_entity", type=str, help="example: component:default/demo-catalog-svc"
110+
)
111+
112+
parser.add_argument("--harness_account_id", type=str, help="harness account id")
113+
114+
parser.add_argument("--harness_api_token", type=str, help="API Token")
115+
116+
parser.add_argument(
117+
"--base_repo_path",
118+
type=str,
119+
help="Path to base folder holding repos",
120+
default="/harness",
121+
)
122+
123+
args = parser.parse_args()
124+
base_path = args.base_repo_path
125+
catalog_entity = args.catalog_entity
126+
dockerfile_path = args.dockerfile_path
127+
service_components = catalog_entity.split("/")
128+
catalog_svc = service_components[1]
129+
130+
# check for custom configure path in catalog-info.yaml
131+
print(f"BASE PATH: {base_path} for SERVICE : {catalog_svc}")
132+
relative_path, language_tag = fetch_service_path(base_path, catalog_svc)
133+
dockerfile_path_cfg = f"/{base_path}/{catalog_svc}/{relative_path}"
134+
135+
# if the configure path exists for dockerfile use it instead
136+
if relative_path != None and os.path.exists(dockerfile_path_cfg):
137+
dockerfile_path = dockerfile_path_cfg
138+
else:
139+
print(f" configurepath not found using default instead")
140+
141+
print(f"catalog_entity: {catalog_entity} dockerfilePath: {dockerfile_path}")
142+
143+
# determine the language based on docker base tag
144+
language, image, tag = determine_image_language(
145+
dockerfile_path, ["build-release-stage", "runtime-stage", "runner", "base"]
146+
)
147+
148+
print(f" Image supporting language {language} {image} {tag}")
149+
150+
# test the image with crypto test
151+
fips_compliance = fips_build_check(language, image, tag, base_path)
152+
153+
# use for local testing
154+
# fips_compliance = fips_build_check ("python", "python", "slim-bullseye", base_path)
155+
156+
# prepare to update the catalog
157+
harness_account_id = args.harness_account_id
158+
harness_api_token = args.harness_api_token
159+
properties_array = [
160+
{"property": "metadata.fips.compliance", "value": fips_compliance}
161+
]
162+
163+
# update the catalog
164+
# updateCatalogAttributes (harness_account_id,
165+
# harness_api_token,
166+
# catalog_entity,
167+
# "metadata.fips.compliance",
168+
# properties_array)
169+
170+
171+
if __name__ == "__main__":
172+
main()
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
ARG BASE_IMAGE=debian:bullseye-slim
2+
3+
# Stage 1: Build the Go program
4+
FROM golang:1.21 AS builder
5+
6+
# Set the working directory
7+
WORKDIR /app
8+
9+
# Copy the Go source code into the container
10+
COPY fips_check.go .
11+
COPY go.mod .
12+
13+
# Download Go modules and build the application
14+
RUN go mod tidy
15+
RUN go build -o fips-checker fips_check.go
16+
17+
# Stage 2: Testing the Go program
18+
FROM ${BASE_IMAGE} AS stage
19+
20+
# Install dependencies (OpenSSL and others if needed)
21+
RUN apt-get update && \
22+
apt-get install -y openssl gcc libssl-dev && \
23+
rm -rf /var/lib/apt/lists/*
24+
25+
# Set the working directory
26+
WORKDIR /app
27+
28+
# Copy the built Go binary from the build stage
29+
COPY --from=builder /app/fips-checker .
30+
31+
# Run the Go program
32+
CMD ["./fips-checker"]

0 commit comments

Comments
 (0)