Skip to content

NO-ISSUE: wait for greenboot before service failure - #7449

Open
redhat-chai-bot wants to merge 2 commits into
openshift:mainfrom
redhat-chai-bot:fix-greenboot-reboot-handling
Open

redhat-chai-bot wants to merge 2 commits into
openshift:mainfrom
redhat-chai-bot:fix-greenboot-reboot-handling

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Wait for the current boot's greenboot health check to exit before the test deliberately makes MicroShift unhealthy.

The change is limited to one line in test/suites/greenboot/greenboot.robot.

Validation

  • focused Robot dry-run: 3 passed, 0 failed
  • make verify-rf
  • git diff --check

Summary by CodeRabbit

  • Tests
    • Updated service and pod failure checks to wait for the Greenboot health check to finish before verifying MicroShift health. This ensures MicroShift health is checked after the Greenboot check exits, rather than while it may still be running, making the order of these checks consistent in both scenarios.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 25, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@redhat-chai-bot: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

  • validate and normalize every observed boot ID
  • require exactly one boot transition for normal reboot flows
  • bind greenboot SubState, Result, and ExecMainStatus checks to the observed boot
  • fail deterministically when greenboot remediation causes an unexpected reboot
  • make intentional multi-reboot recovery and rollback flows explicit
  • cover shared SSH, OSTree, C2CC, offline qemu, TuneD, and affected suites
  • preserve existing OSTree keyword compatibility

Validation

  • make verify-rf
  • focused Robot dry-run: 60 passed, 0 failed
  • Python compile and boot-ID behavior tests
  • git diff --check upstream/main...HEAD
  • clean merge-tree check against current upstream/main

make verify-py has the same pre-existing flake8 failures on upstream main; no branch-caused Python lint findings were introduced.

Full AWS E2E was not available locally.


AI-generated. Review for accuracy.

@pmtk requested via Chai Bot

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
openshift-ci Bot requested review from agullon and pacevedom September 25, 2026 11:21
@openshift-ci

openshift-ci Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: redhat-chai-bot
Once this PR has been reviewed and has the lgtm label, please assign copejon for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: openshift/microshift
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: a58403e
  • Layers: Repository YAML (base), Central YAML (inherited)
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: e83c58e9-17f3-4caa-94e0-15ac5224f6fd
Commits

Reviewing files that changed from the base of the PR and between fcb56b7 and 06860de.

Files selected for processing: 1
  • test/suites/greenboot/greenboot.robot

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The service failure and pod failure tests now wait for the Greenboot health check to exit before checking MicroShift health.

Changes

Greenboot failure tests

Layer / File(s) Summary
Wait for Greenboot health check
test/suites/greenboot/greenboot.robot
Both tests wait for the Greenboot health check to exit before checking MicroShift health.

Priority: Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Bug fix

Suggested reviewers: pacevedom

Merge Risk: Minimal · up to 06860

The change implements the intended ordering in both tests, with no actionable merge-blocking risk remaining.

Pre-merge checks: Passed 15
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly describes the main change: waiting for Greenboot before simulating a service failure.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names Passed The pull request changes only Robot Framework test bodies. The test titles Simulate Service Failure and Simulate Pod Failure remain static and contain no dynamic values or legacy metadata tokens. …
Test Structure And Quality Passed PASS: The pull request changes only test/suites/greenboot/greenboot.robot. It adds Robot Framework keywords to two existing test cases and does not add or modify Ginkgo test code, It blocks, clust…
Microshift Test Compatibility Passed PASS: The pull request changes only test/suites/greenboot/greenboot.robot, a Robot Framework suite. It adds Wait Until Greenboot Health Check Exited to two existing Robot test cases. It does not a…
Single Node Openshift (Sno) Test Compatibility Passed PASS: The pull request changes only test/suites/greenboot/greenboot.robot, a Robot Framework suite. It adds waits to existing Simulate Service Failure and Simulate Pod Failure cases. It adds no …
Topology-Aware Scheduling Compatibility Passed PASS: The pull request changes only test/suites/greenboot/greenboot.robot. It adds waits for the Greenboot health check in two tests. It does not modify deployment manifests, operator code, controll…
Ote Binary Stdout Contract Passed PASS: The pull request changes only test/suites/greenboot/greenboot.robot, adding waits inside the Simulate Service Failure and Simulate Pod Failure test cases. It changes no OTE binary or proce…
Ipv6 And Disconnected Network Test Compatibility Passed The pull request changes only the existing Robot suite test/suites/greenboot/greenboot.robot. It adds wait keywords to two existing tests and adds no Ginkgo tests. Therefore, this check is not appli…
No-Weak-Crypto Passed PASS. The PR adds only two calls to Wait Until Greenboot Health Check Exited in test/suites/greenboot/greenboot.robot. The referenced keyword only waits for greenboot-healthcheck.service to reac…
Container-Privileges Passed PASS. The pull request changes only test/suites/greenboot/greenboot.robot by adding two Wait Until Greenboot Health Check Exited calls. The diff adds no container or Kubernetes manifest settings a…
No-Sensitive-Data-In-Logs Passed PASS: The pull request only adds calls to Wait Until Greenboot Health Check Exited in two Robot tests. The referenced keyword reconnects and checks service state; it does not log passwords, tokens, …
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Full details: Stable And Deterministic Test Names

Explanation

The pull request changes only Robot Framework test bodies. The test titles Simulate Service Failure and Simulate Pod Failure remain static and contain no dynamic values or legacy metadata tokens. The added lines only wait for the Greenboot health check to exit.

Full details: Test Structure And Quality

Explanation

PASS: The pull request changes only test/suites/greenboot/greenboot.robot. It adds Robot Framework keywords to two existing test cases and does not add or modify Ginkgo test code, It blocks, cluster operations, or Ginkgo assertions.

Full details: Microshift Test Compatibility

Explanation

PASS: The pull request changes only test/suites/greenboot/greenboot.robot, a Robot Framework suite. It adds Wait Until Greenboot Health Check Exited to two existing Robot test cases. It does not add Ginkgo tests or reference MicroShift-incompatible OpenShift APIs or features.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request changes only test/suites/greenboot/greenboot.robot, a Robot Framework suite. It adds waits to existing Simulate Service Failure and Simulate Pod Failure cases. It adds no Ginkgo e2e tests and introduces no SNO multi-node assumptions.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The pull request changes only test/suites/greenboot/greenboot.robot. It adds waits for the Greenboot health check in two tests. It does not modify deployment manifests, operator code, controllers, replicas, affinity, topology spread, node selectors, tolerations, or PDBs.

Full details: Ote Binary Stdout Contract

Explanation

PASS: The pull request changes only test/suites/greenboot/greenboot.robot, adding waits inside the Simulate Service Failure and Simulate Pod Failure test cases. It changes no OTE binary or process-level Go code, so it introduces no stdout contract violation.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

The pull request changes only the existing Robot suite test/suites/greenboot/greenboot.robot. It adds wait keywords to two existing tests and adds no Ginkgo tests. Therefore, this check is not applicable. The existing IPv4 CIDRs in Disrupt Pod Network are unchanged by the pull request.

Full details: No-Weak-Crypto

Explanation

PASS. The PR adds only two calls to Wait Until Greenboot Health Check Exited in test/suites/greenboot/greenboot.robot. The referenced keyword only waits for greenboot-healthcheck.service to reach the exited state. The diff introduces no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom cryptography, or secret comparison.

Full details: Container-Privileges

Explanation

PASS. The pull request changes only test/suites/greenboot/greenboot.robot by adding two Wait Until Greenboot Health Check Exited calls. The diff adds no container or Kubernetes manifest settings and no privileged, host namespace, SYS_ADMIN, root, or allowPrivilegeEscalation configuration. The referenced keyword only waits for the greenboot service state.

Full details: No-Sensitive-Data-In-Logs

Explanation

PASS: The pull request only adds calls to Wait Until Greenboot Health Check Exited in two Robot tests. The referenced keyword reconnects and checks service state; it does not log passwords, tokens, PII, session IDs, hostnames, or customer data. No logging statements or sensitive values were added.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@redhat-chai-bot
redhat-chai-bot force-pushed the fix-greenboot-reboot-handling branch 2 times, most recently from 00e5a37 to 07482e4 Compare September 25, 2026 12:11
@redhat-chai-bot redhat-chai-bot changed the title NO-ISSUE: make greenboot reboot handling strict NO-ISSUE: handle greenboot remediation reboots in fault test Sep 25, 2026
@redhat-chai-bot
redhat-chai-bot force-pushed the fix-greenboot-reboot-handling branch from edc61ac to ad06b6d Compare September 25, 2026 13:10
@redhat-chai-bot redhat-chai-bot changed the title NO-ISSUE: handle greenboot remediation reboots in fault test NO-ISSUE: wait for greenboot before service failure Sep 25, 2026
@redhat-chai-bot
redhat-chai-bot force-pushed the fix-greenboot-reboot-handling branch from ad06b6d to 4d866f5 Compare September 25, 2026 13:19
@pmtk

pmtk commented Sep 25, 2026

Copy link
Copy Markdown
Member

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9


Simulate Pod Failure
[Documentation] Simulate pod network failure
Wait For MicroShift Healthcheck Success

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isnt the same check needed here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Simulate Pod Failure currently calls Wait For MicroShift Healthcheck Success before disrupting the pod network, but that keyword runs microshift healthcheck; it does not wait for greenboot-healthcheck.service to exit. The same Wait Until Greenboot Health Check Exited call should precede it, matching Simulate Service Failure. This establishes the same pre-disruption ordering; it does not by itself confirm the cause of the previously observed test failure.


AI-assisted response


AI-generated. Review for accuracy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the same wait before the MicroShift healthcheck in Simulate Pod Failure in commit 543111d. make verify-rf passed, and the focused Greenboot Robot dry-run passed all 3 tests. This is static validation; live Greenboot execution was not tested.


AI-generated. Review for accuracy.

@agullon

agullon commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9

@redhat-chai-bot
redhat-chai-bot force-pushed the fix-greenboot-reboot-handling branch from 543111d to fcb56b7 Compare October 28, 2026 10:17
Ensure the previous greenboot health check has exited before disrupting the MicroShift service.
Wait for the greenboot health check to exit before simulating pod network failure, matching the service failure test.
@redhat-chai-bot
redhat-chai-bot force-pushed the fix-greenboot-reboot-handling branch from fcb56b7 to 06860de Compare October 28, 2026 11:17
@agullon

agullon commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9

@openshift-ci

openshift-ci Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-tests-bootc-arm-el9 06860de link true /test e2e-aws-tests-bootc-arm-el9
ci/prow/e2e-aws-tests 06860de link true /test e2e-aws-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Investigated the two failed checks on this revision:

  • e2e-aws-tests: standard2 failed during VM setup when router-default did not become ready within five minutes. That scenario sets SKIP_GREENBOOT=true.
  • e2e-aws-tests-bootc-arm-el9: multi-nic lost SSH connectivity (No route to host / SSH session not active). It runs the multi-NIC suite, not the modified Greenboot suite.

Neither failing scenario executes the changed test cases. Retrying the failed checks once; no code changes.

/retest


AI-generated. Review for accuracy.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants