Skip to content

fix(ci): cosign sign container manifest by tag, not by local podman digest#539

Merged
christiangda merged 1 commit into
mainfrom
fix/cosign-sign-by-tag
May 24, 2026
Merged

fix(ci): cosign sign container manifest by tag, not by local podman digest#539
christiangda merged 1 commit into
mainfrom
fix/cosign-sign-by-tag

Conversation

@christiangda
Copy link
Copy Markdown
Contributor

Summary

Fixes the Cosign sign published container manifest (keyless / Sigstore) step of the release workflow, which fails on every release with MANIFEST_UNKNOWN: manifest unknown — most recently seen on the v0.45.0 re-attempt: Actions run 26357260675 / job 77586423614.

Error: signing [ghcr.io/slashdevops/idp-scim-sync@sha256:29254941...]:
GET https://ghcr.io/v2/.../manifests/sha256:29254941...: MANIFEST_UNKNOWN

This is the second of two CI bugs in the same job — the first (#538) restored multi-arch builds; this one fixes the signing step that was previously masked by that earlier failure.

Root cause

The step resolved the digest to sign by piping podman manifest inspect ghcr.io/…:TAG into jq -r '.digest // .manifests[0].digest'. Two compounding bugs:

  1. A manifest list's own JSON has no top-level .digest. The list's digest is computed by hashing the JSON, not stored inside it. So the // fallback always wins and returns .manifests[0].digest — the digest of the first per-arch image (arm64), not of the manifest list.
  2. Podman re-serializes manifests on push. Media-type conversion between Docker vnd.docker.distribution.manifest.v2+json and OCI vnd.oci.image.manifest.v1+json means the local podman digest does not match what GHCR stores. Cosign's lookup at the local digest therefore returns 404.

Net effect: cosign was asked to sign a digest that exists nowhere on the registry.

Fix

Switch to cosign sign --recursive ${IMAGE}:${TAG}. Cosign internally HEAD-resolves the tag to the authoritative on-registry digest and signs that digest — the signature is still stored by digest, so the resulting Sigstore artifact is identical to what the broken code intended to produce.

The classic "signing by tag races with concurrent pushes" caveat (which the original comment cited) does not apply: this job exclusively owns the v<x.y.z> and latest tags and has just pushed them sequentially in the previous step. No other actor can mutate them mid-job.

Changes

  • .github/workflows/container-image.yml — replace the broken podman manifest inspect + jq digest resolution with a single cosign sign --recursive ${IMAGE}:${TAG}.
  • docs/Whats-New.md — Unreleased entry.

Test plan

  • Merge to main.
  • (Optional) Trigger Container Image via workflow_dispatch to validate without cutting a tag.
  • On the next tag push, the Publish Container Images job:
    • Pushes the multi-arch manifest to :<tag> and :latest (already working post-fix(ci): restore multi-arch container builds in release workflow #538).
    • cosign sign --recursive succeeds for both tags.
    • cosign verify --certificate-identity-regexp '^https://github.com/slashdevops/idp-scim-sync/.*' --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/slashdevops/idp-scim-sync:<tag> succeeds locally after release.
  • (Recovery) Re-run the workflow for v0.45.0 so the manifest gets signed.

🤖 Generated with Claude Code

…igest

The cosign step of the release workflow failed with MANIFEST_UNKNOWN on
every release attempt after multi-arch builds were restored, e.g. on
v0.45.0:

  Error: signing [ghcr.io/slashdevops/idp-scim-sync@sha256:2925...]:
  GET https://ghcr.io/v2/.../manifests/sha256:2925...: MANIFEST_UNKNOWN

The previous logic resolved the digest to sign by piping
`podman manifest inspect ghcr.io/...:TAG` into
`jq -r '.digest // .manifests[0].digest'`. Two compounding bugs:

1. A manifest list's own JSON has no top-level `.digest` (the list's
   digest is computed by hashing the JSON, not stored inside it), so
   the `//` fallback always returned `.manifests[0].digest` -- which is
   the digest of the first per-arch image (arm64), not of the manifest
   list itself.
2. Podman re-serializes manifests on push (media-type conversion
   between Docker manifest.v2+json and OCI image.manifest.v1+json),
   so the locally computed digest does not match what GHCR stores.
   Cosign's lookup at that local digest therefore returned 404.

Switch to `cosign sign --recursive ${IMAGE}:${TAG}`. Cosign internally
HEAD-resolves the tag to the authoritative on-registry digest and
signs that digest, so the resulting artifact is identical to what the
broken code intended to produce. The classic "signing by tag races
with concurrent pushes" caveat does not apply: this job exclusively
owns the v<x.y.z> and latest tags and has just pushed them
sequentially in the previous step.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@christiangda christiangda self-assigned this May 24, 2026
@christiangda christiangda added this pull request to the merge queue May 24, 2026
Merged via the queue into main with commit ae9775a May 24, 2026
6 checks passed
@christiangda christiangda deleted the fix/cosign-sign-by-tag branch May 24, 2026 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant