[release-4.21] MCO-2017: Backport OSImageStreams to 4.21 by pablintino · Pull Request #5515 · openshift/machine-config-operator

pablintino · 2025-12-23T14:58:55Z

- What I did
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

- How to verify it

TBD

- Description for the changelog
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

openshift-ci-robot · 2025-12-23T14:58:59Z

@pablintino: This pull request references MCO-2017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

- What I did
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

- How to verify it

TBD

- Description for the changelog
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-12-23T14:59:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pablintino

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [pablintino]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ptalgulk01 · 2026-01-08T15:29:51Z

Pre-merge verified:

Environment Setup
OpenShift Version: 4.21.0-0-2026-01-08-104425-test-ci-ln-zjmf0hk-latest
Platform: AWS, GCP, Azure, Vsphere, FIPS on TechPreview Cluster

Verification Steps

Checking Switch

Enable the TechPreview and check OSStream enabled

status:
  featureGates:
           - name: OSStreams

Default is rhel-9

$  oc get mcp worker -ojsonpath='{.status.osImageStream.name}'
 empty
$ oc debug node/ip-10-0-71-60.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-71-60us-east-2computeinternal-debug-qqxzm ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build05.ci.openshift.org/ci-ln-qz3x1kb/stable@sha256:707f4ec6193865bb7eda34e8e9bb5f1310b00eb4342fd666ae72a55c218811aa
                   Digest: sha256:707f4ec6193865bb7eda34e8e9bb5f1310b00eb4342fd666ae72a55c218811aa
                  Version: 9.6.20251219-0 (2025-12-25T12:49:21Z)

Switch to rhel-10 (Tested on both Master and Worker pool)

 $ oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

$ oc debug node/ip-10-0-8-235.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-8-235us-east-2computeinternal-debug-tglv9 ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build05.ci.openshift.org/ci-ln-qz3x1kb/stable@sha256:8eab89184784583b739b785772d1697a73b8bd09f8dfb7b1f61dbb34d3a56bd0
                   Digest: sha256:8eab89184784583b739b785772d1697a73b8bd09f8dfb7b1f61dbb34d3a56bd0
                  Version: 10.1.20251222-0 (2025-12-26T23:56:41Z)

PIS Verification (No-issue)

Applied PIS on rhel-9 updated to rhel-10
Applied on rhel10 updated to rhel-9

Kernel Argument Verification (No-issue)

Applied the Kernel argument MC

$ oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker  
  name: 05-worker-kernelarg-selinuxpermissive  
spec:
  kernelArguments:
    - enforcing=0
EOF

Updated to rhel10.
Applied MC on rhel-10 -> switch

Custom Pool

Created custom Pool and check the switch on the node
Added rhel-10 node to custom mcp got reverted to rhel-9

Scaling Node (No-issue)

Able to Scale node with rhel-9 and rhel-10

Scaling node on Custom Machineset (No-issue)

Created the Custom MS and scaled the node for it

$ oc get machinesets.machine.openshift.io -n openshift-machine-api ppt-0701a-6m9l2-worker-us-east-1d -o yaml > ms10.yaml

Able to create custom MS and scale node for it for rhel-9 and rhel-10

Pause (No-issue)

Pause the MCP with rhel-9 -> Switch to rhel10 -> Unpause -> Switched to rhel10
Pause the MCP with rhel-10 -> Switch to rhel9-> Unpause -> Switched to rhel9

Max Unavailable (No-issue)

Kernel Argument (issue)

Applied MC on rhel-9 cluster -> Switch to rhel10 -> Error
Applied MC on rhel-10 -> Switch to rhel-9 -> MCP updated with error and modified to rhel10

oc create -f - << EOF                                                       
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker 
  name: worker-kernel-rt
spec:
  kernelType: realtime

EOF
machineconfig.machineconfiguration.openshift.io/worker-kernel-rt created

Error seen

  - lastTransitionTime: "2026-01-08T12:42:26Z"
    message: 'Node ip-10-0-62-222.us-east-2.compute.internal is reporting: "Node ip-10-0-62-222.us-east-2.compute.internal
      upgrade failure. error running rpm-ostree override remove kernel kernel-core
      kernel-modules kernel-modules-core kernel-modules-extra --install kernel-rt-core
      --install kernel-rt-modules --install kernel-rt-modules-extra --install kernel-rt-kvm:
      error: Packages not found: kernel-rt-kvm\n: exit status 1", Node ip-10-0-62-222.us-east-2.compute.internal
      is reporting: "error running rpm-ostree override remove kernel kernel-core kernel-modules
      kernel-modules-core kernel-modules-extra --install kernel-rt-core --install
      kernel-rt-modules --install kernel-rt-modules-extra --install kernel-rt-kvm:
      error: Packages not found: kernel-rt-kvm\n: exit status 1"'
    reason: 1 nodes are reporting degraded status on sync
    status: "True"
    type: NodeDegraded

Extension (No-issue)

Applied MC on rhel-9 cluster -> Switch to rhel10 -> MCP got updated to rhel10
Applied MC on rhel-10 -> Switch to rhel-9 -> MCP updated with error and modified to rhel10

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: master-ext
spec:
  config:
    ignition:
      version: 3.1.0
  extensions:
    -  usbguard
EOF

SNO (no-issue)

FIPS based cluster (No-issue)

Verified the Day-1 installation (No-issue)

Both for rhel10 and rhel-9

pablintino · 2026-01-09T11:18:22Z

/hold
Till https://issues.redhat.com/browse/OCPBUGS-71189 is resolved

ptalgulk01 · 2026-01-16T13:29:56Z

Pre-merge verification:
Environment Setup
OpenShift Version: 4.21.0-0-2026-01-15-073433-test-ci-ln-9gzhqyt-latest
Platform: AWS,proxy
Flexy Template: private-templates/functionality-testing/aos-4_21/ipi-on-gcp/versioned-installer_customer_vpc-http_proxy

Tested on proxy based cluster with and without TechPreview enabled.

$ oc debug node/ip-10-0-70-104.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-70-104us-east-2computeinternal-debug-bsscz ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                   Digest: sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                  Version: 9.6.20260112-0 (2026-01-13T02:17:59Z)

Removing debug pod ...

# Patch the rhel-10
$ oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

# wait for mcp update
$ oc debug node/ip-10-0-48-232.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-48-232us-east-2computeinternal-debug-zc24c ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                   Digest: sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                  Version: 10.1.20260106-1 (2026-01-09T10:39:31Z)

Removing debug pod ...

For extension when we apply all extension MC the MCP gets degraded when we switch to rhel-10 which is reported
Tested using manual-installation too for rhel-10 installation.

/label qe-approved
/verified by @ptalgulk01

openshift-ci-robot · 2026-01-16T13:30:04Z

@pablintino: This pull request references MCO-2017 which is a valid jira issue.

Details

In response to this:

- What I did
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

- How to verify it

TBD

- Description for the changelog
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-01-16T13:30:09Z

@ptalgulk01: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

Pre-merge verification:
Environment Setup
OpenShift Version: 4.21.0-0-2026-01-15-073433-test-ci-ln-9gzhqyt-latest
Platform: AWS,proxy
Flexy Template: private-templates/functionality-testing/aos-4_21/ipi-on-gcp/versioned-installer_customer_vpc-http_proxy

Tested on proxy based cluster with and without TechPreview enabled.

$ oc debug node/ip-10-0-70-104.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-70-104us-east-2computeinternal-debug-bsscz ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                  Digest: sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                 Version: 9.6.20260112-0 (2026-01-13T02:17:59Z)

Removing debug pod ...

# Patch the rhel-10
$ oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

# wait for mcp update
$ oc debug node/ip-10-0-48-232.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-48-232us-east-2computeinternal-debug-zc24c ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                  Digest: sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                 Version: 10.1.20260106-1 (2026-01-09T10:39:31Z)

Removing debug pod ...

For extension when we apply all extension MC the MCP gets degraded when we switch to rhel-10 which is reported
Tested using manual-installation too for rhel-10 installation.

/label qe-approved
/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sdodson · 2026-01-22T18:23:36Z

/retest-required

pablintino · 2026-01-26T16:06:02Z

/retest-required

pablintino · 2026-02-06T13:14:40Z

/retest-required

This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21. Co-authored-by: Zack Zlotnik <zzlotnik@redhat.com> Co-authored-by: Isabella Janssen <ijanssen@redhat.com> Signed-off-by: Pablo Rodriguez Nava <git@amail.pablintino.com>

pablintino · 2026-02-06T14:14:20Z

/retest-required

pablintino · 2026-02-09T08:00:26Z

/retest-required

ptalgulk01 · 2026-02-09T14:58:49Z

Pre-merge verified:

Verified on IPI based http_proxy, sno , fips and aws TP enabled cluster

Verified the following scenarios for (rhel9 -> rhel10) switch

Able to Switch

oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

oc debug node/ip-10-0-29-59.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-29-59us-east-2computeinternal-debug-9kfgt ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-pxhfcc2/stable@sha256:4a1798a3b92a794a69d56eaf78c1521a1c4d2e52fd05057072780ec19ccabd45
                   Digest: sha256:4a1798a3b92a794a69d56eaf78c1521a1c4d2e52fd05057072780ec19ccabd45
                  Version: 10.1.20260126-0 (2026-01-27T06:43:47Z)

Removing debug pod ...

Custom Pool
Created custom pool added the node and switch to rhel-10
For rhel-10 node adding after adding it to custom MCP was to switched rhel-9 after upgrade
PIS
Added the PIS switch to rhel10
Applied the PIS on rhel-10 node
Node and Machinesets adding
Added new node to cluster and switch to rhel10
Created new MS and switch the new node created to rhel10
Kernel Argument
Added the kernel arg based MC and switch to rhel-10
Added the kernel ard on rhel-10 node
Extension

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra1
  name: tc-56131-all-extensions
spec:
  config:
    ignition:
      version: 3.1.0
  extensions:
  - usbguard
  - kerberos
  - kernel-devel
  - sandboxed-containers
  - sysstat
EOF

Applied the extension MC and switch the node to rhel10

Pause
Paused the MCP updated to rhel-10 unpaused the MCP and switch was complete
Max Unavailable
Updated the MCP as max unavailable switch to rhel-10

Not supported

** KernelType**

Upgrade the PR to 4.21 nightly

Was able to upgrade the cluster to new 4.21 nightly where the osstream was rhel9
Once we have it in nightly will verify rhel-10 to rhel-10 upgrade

/verified by @ptalgulk01

openshift-ci-robot · 2026-02-09T14:59:04Z

@ptalgulk01: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

Pre-merge verified:

Verified on IPI based http_proxy, sno , fips and aws TP enabled cluster

Verified the following scenarios for (rhel9 -> rhel10) switch

Able to Switch
oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

oc debug node/ip-10-0-29-59.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-29-59us-east-2computeinternal-debug-9kfgt ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-pxhfcc2/stable@sha256:4a1798a3b92a794a69d56eaf78c1521a1c4d2e52fd05057072780ec19ccabd45
                  Digest: sha256:4a1798a3b92a794a69d56eaf78c1521a1c4d2e52fd05057072780ec19ccabd45
                 Version: 10.1.20260126-0 (2026-01-27T06:43:47Z)

Removing debug pod ...
Custom Pool
Created custom pool added the node and switch to rhel-10
For rhel-10 node adding after adding it to custom MCP was to switched rhel-9 after upgrade

PIS
Added the PIS switch to rhel10
Applied the PIS on rhel-10 node

Node and Machinesets adding
Added new node to cluster and switch to rhel10
Created new MS and switch the new node created to rhel10

Kernel Argument
Added the kernel arg based MC and switch to rhel-10
Added the kernel ard on rhel-10 node

Extension
oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
 labels:
   machineconfiguration.openshift.io/role: infra1
 name: tc-56131-all-extensions
spec:
 config:
   ignition:
     version: 3.1.0
 extensions:
 - usbguard
 - kerberos
 - kernel-devel
 - sandboxed-containers
 - sysstat
EOF
Applied the extension MC and switch the node to rhel10

Pause
Paused the MCP updated to rhel-10 unpaused the MCP and switch was complete

Max Unavailable
Updated the MCP as max unavailable switch to rhel-10

Not supported

** KernelType**

Upgrade the PR to 4.21 nightly

Was able to upgrade the cluster to new 4.21 nightly where the osstream was rhel9

Once we have it in nightly will verify rhel-10 to rhel-10 upgrade

/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pablintino · 2026-02-09T15:36:21Z

/test e2e-gcp-op-ocl

pablintino · 2026-02-09T15:36:46Z

/unhold

pablintino · 2026-02-09T20:49:18Z

/test e2e-gcp-op-ocl
Re-running as the failure seems to be related to GitHub

openshift-ci · 2026-02-10T00:50:24Z

@pablintino: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 23, 2025

openshift-ci Bot requested review from djoshy and yuqi-zhang December 23, 2025 14:59

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 23, 2025

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 9, 2026

pablintino force-pushed the mco-2017 branch from 420c2a7 to d18e2f5 Compare January 14, 2026 14:14

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 14, 2026

pablintino force-pushed the mco-2017 branch from d18e2f5 to d012c19 Compare January 14, 2026 14:57

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 14, 2026

openshift-ci Bot added the qe-approved Signifies that QE has signed off on this PR label Jan 16, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jan 16, 2026

pablintino force-pushed the mco-2017 branch from d012c19 to 0d39de2 Compare January 23, 2026 08:35

openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Jan 23, 2026

pablintino force-pushed the mco-2017 branch 2 times, most recently from 5717259 to 9305397 Compare February 4, 2026 08:41

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 5, 2026

pablintino force-pushed the mco-2017 branch from 9305397 to 014dcca Compare February 5, 2026 13:43

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 5, 2026

pablintino force-pushed the mco-2017 branch from 014dcca to 3a5e338 Compare February 6, 2026 13:41

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Feb 9, 2026

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 9, 2026

sdodson added backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Feb 10, 2026

openshift-ci Bot assigned anuragthehatter, imatza-rh, lyman9966, mhanss, ptalgulk01, rbbratta, rioliu-rh, sergiordlr and zhaozhanqi Feb 10, 2026

sdodson merged commit b59e48e into openshift:release-4.21 Feb 10, 2026
15 of 16 checks passed

Conversation

pablintino commented Dec 23, 2025

Uh oh!

openshift-ci-robot commented Dec 23, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Dec 23, 2025

Uh oh!

ptalgulk01 commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-merge verified:

Verification Steps

PIS Verification (No-issue)

Kernel Argument Verification (No-issue)

Custom Pool

Scaling Node (No-issue)

Scaling node on Custom Machineset (No-issue)

Max Unavailable (No-issue)

Kernel Argument (issue)

Extension (No-issue)

SNO (no-issue)

FIPS based cluster (No-issue)

Verified the Day-1 installation (No-issue)

Uh oh!

pablintino commented Jan 9, 2026

Uh oh!

ptalgulk01 commented Jan 16, 2026

Uh oh!

openshift-ci-robot commented Jan 16, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jan 16, 2026

Uh oh!

sdodson commented Jan 22, 2026

Uh oh!

pablintino commented Jan 26, 2026

Uh oh!

pablintino commented Feb 6, 2026

Uh oh!

pablintino commented Feb 6, 2026

Uh oh!

pablintino commented Feb 9, 2026

Uh oh!

ptalgulk01 commented Feb 9, 2026

Uh oh!

openshift-ci-robot commented Feb 9, 2026

Uh oh!

pablintino commented Feb 9, 2026

Uh oh!

pablintino commented Feb 9, 2026

Uh oh!

pablintino commented Feb 9, 2026

Uh oh!

openshift-ci Bot commented Feb 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

openshift-ci-robot commented Dec 23, 2025 •

edited by openshift-ci Bot

Loading

ptalgulk01 commented Jan 8, 2026 •

edited

Loading

openshift-ci-robot commented Jan 16, 2026 •

edited by openshift-ci Bot

Loading