Add job for ESCU Tests

[spanchal-crest]

No description provided.

[mgazda-splunk]

Wondering if we should discuss with Security Content team to have releases matching somehow the repo, as downloading develop and newest tool may be problematic at some point?

[spanchal-crest]

Yeah, we can

I asked this in our ESCU channel - https://splunk.slack.com/archives/C088U396U11/p1739876980416399?thread_ts=1739268536.325259&cid=C088U396U11

But Eric mentioned Unfortunately we don’t have a specific version of contentctl called out for building security_content

[mgazda-splunk]

Yes yes, they do not have, but can have :)

[mgazda-splunk]

Suggested change

	# Load the YAML content
	with open("security_content/contentctl.yml", "r") as file:
	data = yaml.safe_load(file)
	for app in data["apps"]:
	if app['appid'] == APP_ID or GITHUB_REPOSITORY in app['hardcoded_path'] or app["title"] == APP_LABEL:
	app['hardcoded_path'] = "${{ env.TA_BUILD_PATH }}"
	elif app['appid'] == "PALO_ALTO_NETWORKS_ADD_ON_FOR_SPLUNK" and APP_ID == "Splunk_TA_paloalto_networks":
	app['hardcoded_path'] = "${{ env.TA_BUILD_PATH }}"
	# Load the YAML content
	with open("security_content/contentctl.yml", "r") as file:
	data = yaml.safe_load(file)
	found = False
	for app in data["apps"]:
	if app['appid'] == APP_ID or GITHUB_REPOSITORY in app['hardcoded_path'] or APP_LABEL in app['title']:
	app['hardcoded_path'] = "${{ env.TA_BUILD_PATH }}"
	found = True
	if not found:
	exit(127)

[spanchal-crest]

Don't we need this condition as well?
elif app['appid'] == "PALO_ALTO_NETWORKS_ADD_ON_FOR_SPLUNK" and APP_ID == "Splunk_TA_paloalto_networks":

Because the name in contentctl is I guess of the vendor TA and we have Splunk TA for palo alto.
So, unless the contentctl is updated with the proper name, we will need this.

[mgazda-splunk]

Dow we need ths additonal file?

[spanchal-crest]

I think we can remove that.

[mgazda-splunk]

Is failure of test making build red? - it should.

[mgazda-splunk]

Please also rebase first on newest develop @spanchal-crest

[artemrys]

Can we move it out to a separate file / script?

[spanchal-crest]

Can I know what would be the appropriate place to put that file?

[artemrys]

How did you solve the performance aspect of running for big add-ons?

[spanchal-crest]

We have run the detections using a larger runner and it is very fast.

ESCU Tests in Windows TA using large runner (32 cores)
Using 4 containers - https://github.com/splunk/splunk-add-on-for-microsoft-windows/actions/runs/13492512521/job/37693050535 took ~52min
Using 8 containers - https://github.com/splunk/splunk-add-on-for-microsoft-windows/actions/runs/13492512521/job/37701127717 took ~35min
Using 12 containers - https://github.com/splunk/splunk-add-on-for-microsoft-windows/actions/runs/13492512521/job/37703683659 took ~35min

We have also tried other approach of splitting the tests into multiple jobs.

[artemrys]

We have also tried other approach of splitting the tests into multiple jobs.

Can you please share the results for it?

Can I run the tests locally? What would be the performance of it? I expect folks to run tests (if possible) locally before pushing code to GitHub.

[artemrys]

@mgazda-splunk we run it completely in GitHub while other types of tests we do run in AWS, right? Did you consider running it in AWS instead? This PR brings testing logic into the reusable workflow - we use https://github.com/splunk/ta-automation-k8s-manifests/blob/main/test-addon.sh for testing logic.