Repository navigation
Conversation
|
Pipeline controller notification This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration. Use |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughThe EL9 and EL10 optional templates no longer install x86_64-specific RPMs. The FIPS templates install those RPMs from copied MicroShift repositories on x86_64 builds. The EL9 and EL10 scenarios now select AI model-serving FIPS images. ChangesAI model-serving FIPS images
Priority: ⚪ Pending latest changes Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant ReleaseScenario
participant FIPSImage
participant MicroShiftRepository
participant DNF
ReleaseScenario->>FIPSImage: select AI-enabled FIPS image
FIPSImage->>MicroShiftRepository: copy repository content on x86_64
FIPSImage->>DNF: install versioned AI model-serving RPMs
FIPSImage->>DNF: set FIPS policy and clean metadata
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Description Check |  |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Docstring Coverage | |
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files. (1 skipped: 1 … |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Stable And Deterministic Test Names | |
The PR changes no Ginkgo test titles or other dynamic test-name expressions. The only scenario-name changes use static identifiers: rhel98-bootc-brew-lrel-fips-ai and `rhel102-bootc-brew-lrel-fips-a… |
| Test Structure And Quality | |
PASS: The pull request changes only bootc containerfiles and release scenario shell scripts. It adds or modifies no Ginkgo test code or It blocks, so the listed Ginkgo quality requirements do not appl… |
| Microshift Test Compatibility | |
The pull request adds no Ginkgo e2e tests. The authoritative diff changes only containerfile image definitions and two scenario shell files that select the image. No new It(), Describe(), Context(), o… |
| Single Node Openshift (Sno) Test Compatibility | |
PASS: The pull request adds no new Ginkgo e2e tests. The authoritative diff changes containerfiles and two shell scenario image selections only; no It(), Describe(), Context(), or When() declarations … |
| Topology-Aware Scheduling Compatibility | |
PASS — The pull request changes only bootc Containerfiles and scenario image selection. It adds/removes RPM installation, FIPS image naming, and start_image values. It does not modify deployment man… |
| Ote Binary Stdout Contract | |
PASS: The pull request changes only Containerfiles and two shell scenario configuration files. It adds or removes RPM installation, repository cleanup, FIPS image naming, and start_image values. It … |
| Ipv6 And Disconnected Network Test Compatibility | |
PASS: The pull request changes bootc containerfiles and two scenario shell variables. It adds no Ginkgo e2e tests (It, Describe, Context, or When) and introduces no IPv4 assumptions or externa… |
| No-Weak-Crypto | |
The pull request does not introduce MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB mode, custom cryptography, or non-constant-time secret comparisons. The added crypto-related commands only enable the FIPS … |
| Container-Privileges | |
No custom-check violation was introduced. The PR changes only Containerfiles and scenario shell scripts. The added content contains no privileged: true, host namespace settings, SYS_ADMIN, `allowP… |
| No-Sensitive-Data-In-Logs | |
PASS. The pull request adds no logging statements or commands that print passwords, tokens, API keys, PII, session IDs, hostnames, or customer data. The changes only copy repositories, install RPMs, r… |
| Title check | |
The title clearly summarizes the main change: limiting AI model serving RPM installation to the image used by the AI model serving RF test. |
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files. (1 skipped: 1 unsupported.)
Full details: Stable And Deterministic Test Names
-
Explanation
The PR changes no Ginkgo test titles or other dynamic test-name expressions. The only scenario-name changes use static identifiers:
rhel98-bootc-brew-lrel-fips-aiandrhel102-bootc-brew-lrel-fips-ai. These contain no generated suffixes, timestamps, node or namespace names, IP addresses, UUIDs, or legacy metadata tokens.
Full details: Test Structure And Quality
-
Explanation
PASS: The pull request changes only bootc containerfiles and release scenario shell scripts. It adds or modifies no Ginkgo test code or It blocks, so the listed Ginkgo quality requirements do not apply.
Full details: Microshift Test Compatibility
-
Explanation
The pull request adds no Ginkgo e2e tests. The authoritative diff changes only containerfile image definitions and two scenario shell files that select the image. No new It(), Describe(), Context(), or When() test declarations, MicroShift API references, or unsupported feature assumptions were introduced.
Full details: Single Node Openshift (Sno) Test Compatibility
-
Explanation
PASS: The pull request adds no new Ginkgo e2e tests. The authoritative diff changes containerfiles and two shell scenario image selections only; no It(), Describe(), Context(), or When() declarations were added. Therefore the SNO multi-node compatibility check is not applicable.
Full details: Topology-Aware Scheduling Compatibility
-
Explanation
PASS — The pull request changes only bootc Containerfiles and scenario image selection. It adds/removes RPM installation, FIPS image naming, and
start_imagevalues. It does not modify deployment manifests, operators, controllers, replicas, affinity, topology spread, node selectors, tolerations, or PDBs. The topology-aware scheduling check is therefore not applicable.
Full details: Ote Binary Stdout Contract
-
Explanation
PASS: The pull request changes only Containerfiles and two shell scenario configuration files. It adds or removes RPM installation, repository cleanup, FIPS image naming, and
start_imagevalues. It does not change Go OTE binaries or process-level code such asmain(), suite setup, logging, or stdout handling.
Full details: Ipv6 And Disconnected Network Test Compatibility
-
Explanation
PASS: The pull request changes bootc containerfiles and two scenario shell variables. It adds no Ginkgo e2e tests (
It,Describe,Context, orWhen) and introduces no IPv4 assumptions or external connectivity requirements in test code.
Full details: No-Weak-Crypto
-
Explanation
The pull request does not introduce MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB mode, custom cryptography, or non-constant-time secret comparisons. The added crypto-related commands only enable the FIPS crypto policy in the new EL9 and EL10 AI images.
Full details: Container-Privileges
-
Explanation
No custom-check violation was introduced. The PR changes only Containerfiles and scenario shell scripts. The added content contains no
privileged: true, host namespace settings,SYS_ADMIN,allowPrivilegeEscalation: true, root user directive, or Kubernetes security context.
Full details: No-Sensitive-Data-In-Logs
-
Explanation
PASS. The pull request adds no logging statements or commands that print passwords, tokens, API keys, PII, session IDs, hostnames, or customer data. The changes only copy repositories, install RPMs, remove temporary files, clean DNF metadata, and select an image. Existing repository setup and cleanup do not introduce sensitive-data logging.
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
Comment @coderabbitai help to get the list of available commands.
agullon
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@agullon: This pull request explicitly references no jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: agullon The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
agullon
changed the title
While running the 5.0.0-rc.7 release tests, the x86_64 lrel jobs showed
performance degradation (memory-pressure page-cache thrash) that the arm jobs
did not. A large part of it traced to the x86_64-only microshift-ai-model-serving
(kserve) RPM being installed into the shared rhel{98,102}-bootc-brew-lrel-optional
base image. That image is the FROM base for ~39 non-AI lrel scenarios, so on x86
every one of them loaded kserve's CRDs/OpenAPI into the MicroShift apiserver,
inflating resident heap by ~940 MB (measured) for scenarios that never exercise
AI model serving and squeezing available memory on the memory-constrained x86
metal hosts.
Scope the AI RPM to the single scenario that actually tests it:
- Remove the MICROSHIFT_X86_64_RPMS block from both -brew-lrel-optional images.
- Install the AI RPM (x86_64-gated) in the lrel FIPS image, which is consumed
exclusively by the *@ai-model-serving-online-fips scenario.
- Rename the FIPS containerfiles -brew-lrel-fips -> -brew-lrel-fips-ai (el9 +
el10) to reflect that they now carry the AI RPM, and repoint start_image in
both ai-model-serving-online-fips scenarios accordingly.
suites/optional/ has no kserve references, so the optional scenario is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pre-commit.check-secrets: ENABLED
f412d25 to
d79350d
Compare
|
/auto |
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
|
/pipeline required |
|
/test e2e-aws-tests-release |
agullon
changed the title
|
/pipeline required |
|
Scheduling tests matching the |
1 similar comment
|
Scheduling tests matching the |
|
/retest |
|
/test e2e-aws-tests-cache |
agullon
changed the title
|
@agullon: This pull request references USHIFT-7587 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.1.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@agullon: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
The RPM/ostree twin of the bootc change. rhel98-brew-lrel-optional (and the image-installer ISO built from it) still installed the x86_64-only kserve RPM, and it is the start image for 19 non-AI release scenarios. No RPM/ostree scenario runs suites/ai-model-serving. On every cold start (cleanup --all, fresh database) kserve's webhook has no endpoints for several minutes. MicroShift applies manifests.d paths sequentially, retrying each for up to 10m, so 010-kserve and 050-runtimes block every later path: - 080-metrics-server lands after the 5m waitForNamespace deadline, the metrics certs are never provisioned and the 600s healthcheck fails (osconfig "Verify Cluster ID Change For New Database", lvms-standard "Log Scan", "Verify Local Host Name Resolution"). - Test manifests.d dirs are applied ~14m after restart, past the kustomize suite's 200s wait, and the retries plus sos collection exhaust the 60m robot budget (iso-standard). The bootc iso-standard twin, whose image no longer carries kserve, passed 52/52 in the same CI run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> pre-commit.check-secrets: ENABLED
agullon
changed the title
…tests
Deploy Web Server ran `oc create` and immediately `oc wait pod --selector`.
When the ReplicaSet had not created the pod yet, `oc wait` exits at once with
"no matching resources found" instead of waiting, failing the test setup.
Seen in el98-lrel@router-routes ("Edge And Passthrough Routes"): oc wait gave
up 160ms before the pod was bound.
Wait on the Deployment with `oc rollout status`, which blocks until the
replicas are available even if no pod exists yet. Both web-server manifests
name the Deployment web-server-deploy. Deploy Web Server Signed had the same
pattern, usually hidden by the preceding secret poll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pre-commit.check-secrets: ENABLED
Summary
While running the 5.0.0-rc.7 release tests, the x86_64 lrel jobs showed performance degradation (memory-pressure page-cache thrash) that the arm jobs did not. Investigation traced a large part of it to the x86_64-only
microshift-ai-model-serving(kserve) RPM being installed into the sharedrhel{98,102}-bootc-brew-lrel-optionalbase image. That image is theFROMbase for ~39 non-AI lrel scenarios, so on x86 every one of them loaded kserve's CRDs/OpenAPI into the MicroShift apiserver — inflating resident heap by ~940 MB (measured) for scenarios that never exercise AI model serving, and squeezing available memory on the already memory-constrained x86 metal hosts.This PR scopes the AI RPM to the single scenario that actually tests it, so the non-AI scenarios stay lean and the x86-only pressure is removed.
Changes
MICROSHIFT_X86_64_RPMS(AI model serving) block from the sharedrhel102-bootc-brew-lrel-optionalandrhel98-bootc-brew-lrel-optionalimages.x86_64-gated AI RPM install (re-copying the microshift repos, then cleaning up) to the lrel FIPS image, which is consumed exclusively by the*@ai-model-serving-online-fipsscenario.…-brew-lrel-fips→…-brew-lrel-fips-ai(el9 + el10) to reflect that they now carry the AI RPM.start_imageinel98-lrel@ai-model-serving-online-fips.shandel102-lrel@ai-model-serving-online-fips.shto the renamed image.RPM/ostree image (second commit)
The RPM/ostree twin
test/image-blueprints/layer4-release/group3/rhel98-brew-lrel-optional.toml(and the.image-installerISO built from it) also installed the kserve RPM on x86_64. It is the start image for 19 non-AI release scenarios; no RPM/ostree scenario runssuites/ai-model-serving.On every cold start (
cleanup --all, fresh database) the kserve webhook has no endpoints for several minutes. MicroShift appliesmanifests.dpaths sequentially with a 10m retry per path, so010-kserve/050-runtimesblock everything after them. This is the root cause of the chronice2e-aws-tests-releaseflakes onrelease-5.0:080-metrics-serverlands after the 5mwaitForNamespacedeadline, the metrics certs are never provisioned and the 600s healthcheck fails:el98-lrel@osconfig"Verify Cluster ID Change For New Database",el98-yminus1@el98-lrel@lvms-standard"Log Scan", "Verify Local Host Name Resolution".manifests.ddirs get applied ~14m after restart, past the kustomize suite wait, and the retries exhaust the 60m robot budget:el98-lrel@iso-standard.Every failing run has
Failed to provision metrics-server certs: waiting for namespace openshift-monitoring: context deadline exceededat the failure time. The bootcel98-lrel@iso-standardtwin, whose image no longer has kserve, passed 52/52 in the same CI run.Router test race (third commit)
Unrelated to kserve, but small and found in the same CI analysis.
Deploy Web Server/Deploy Web Server Signedintest/resources/router.resourceranoc createand immediatelyoc wait pod --selector, which exits with "no matching resources found" if the pod is not created yet (el98-lrel@router-routes"Edge And Passthrough Routes" failed this way, 160ms before the pod was bound). Now they wait withNamed Deployment Rollout Should Be Complete(oc rollout status). Verified with robocop androbot --dryrun suites/router/(31/31).Testing
…-brew-lrel-fipsimage names.suites/optional/has no kserve references, so theoptionalscenario is unaffected by dropping the RPM from its base image.optional(group1) builds beforefips-ai(group2), which re-copies the same RPM repos it needs.e2e-aws-tests-bootc-release-el10,-el9, and arm variants) and confirm the x86 non-AI scenarios no longer thrash whileai-model-serving-online-fipsstill passes.🤖 Generated with Claude Code