Kubernetes Job Creation Fails When Job ID Starts with Non-Alphanumeric Character

[hjaait-ga]

Description

We're running into a pretty nasty issue with the Kubernetes job runner where jobs fail to be created (and subsequently deleted) when the Elasticsearch-generated job ID happens to start with a dash or underscore.

The problem is that Cortex uses the raw job ID from Elasticsearch as a Kubernetes label value without sanitizing it first, but Kubernetes has strict validation rules that require label values to start and end with alphanumeric characters.

Error Message

Failure executing: DELETE at: https://kubernetes.default.svc/apis/batch/v1/namespaces/cyber-np/jobs?labelSelector=cortex-job-id%3D-HRAlJwB0OVc1KWLhGUA. 

Message: unable to parse requirement: values[0][cortex-job-id]: Invalid value: "-HRAlJwB0OVc1KWLhGUA": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

Environment

• Cortex Version: 4.0.0-1 (StrangeBee)
• Kubernetes Version: 1.28+ (AKS)
• Job Runner: Kubernetes (configured with job.runners = [kubernetes])
• Backend: Elasticsearch

Root Cause

Looking at the code in K8sJobRunnerSrv.scala, the job ID is used directly as a label value:

val labels = Map(
  "app" -> "cortex",
  "cortex-job-id" -> job.id,  // <- Direct use of Elasticsearch ID
  "cortex-job-type" -> job.`type`
)

Elasticsearch/OpenSearch generates document IDs using base64url encoding, which can produce IDs starting with - or _. While these are perfectly valid for Elasticsearch, they violate Kubernetes label validation rules:

• Labels must start with [A-Za-z0-9]
• Labels must end with [A-Za-z0-9]
• Labels can contain [-A-Za-z0-9_.] in the middle

Impact

When this happens:

1. Jobs fail to be created in Kubernetes
2. Cortex can't delete the failed jobs (same label selector issue)
3. Analyzers/responders don't execute
4. Jobs get stuck in a retry loop

It's relatively rare (depends on how often Elasticsearch generates IDs starting with these characters), but when it hits, those specific jobs are completely broken.

Proposed Solution

Sanitize the job ID before using it as a label value. A simple approach would be to prefix IDs that start with non-alphanumeric characters:

val sanitizedJobId = job.id match {
  case id if id.matches("^[A-Za-z0-9].*") => id
  case id => s"j$id"  // Prefix with 'j' if starts with non-alphanumeric
}

val labels = Map(
  "app" -> "cortex",
  "cortex-job-id" -> sanitizedJobId,
  "cortex-job-type" -> job.`type`
)

Alternatively, you could hash the job ID or use a more restrictive character set for the label while keeping the original ID stored elsewhere (annotations, perhaps?).

Reproduction

This is somewhat random since it depends on Elasticsearch ID generation, but you can reproduce it by:

1. Configure Cortex with Kubernetes job runner
2. Trigger enough analyzer/responder jobs that you eventually get an Elasticsearch doc ID starting with - or _
3. Watch the job creation fail with the validation error above

Workaround

Currently, there's no clean workaround without patching Cortex itself. The jobs simply fail and can't be recovered.

---

Let me know if you need any additional information or if I can help test a fix. Happy to provide more details about our setup if needed.

[hjaait-ga]

PS : screenshotted the error from the UI :