Skip to content

fix(e2e): detect gateway workload for Kubernetes health port-forward#2198

Closed
elezar wants to merge 1 commit into
mainfrom
fix/ha-e2e-health-portforward-workload-kind
Closed

fix(e2e): detect gateway workload for Kubernetes health port-forward#2198
elezar wants to merge 1 commit into
mainfrom
fix/ha-e2e-health-portforward-workload-kind

Conversation

@elezar

@elezar elezar commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

The Kubernetes HA E2E deploys the gateway as a Deployment, but the test harness hardcoded the health port-forward to a StatefulSet. That caused the HA job to fail before the test suite could exercise the deployment.

This PR makes the minimal harness change required to select the deployed workload kind. The existing test:e2e-kubernetes label triggers the optional HA proof-of-life workflow.

HA gateway behavior is not expected to pass on current main. Full HA gateway rebalancing, peer routing, and supervisor relay handoff are being implemented in #1868. Until that PR merges, the HA suite is expected to expose those outstanding failures after this port-forward prerequisite succeeds.

Related Issue

Related to #2153, where the health port-forward failure was observed.

HA implementation: #1868.

Changes

  • Add a small kube_workload_ref helper that selects sts/ or deploy/ from the cluster.
  • Use that reference for the health port-forward in both Kubernetes E2E execution paths.

Testing

  • bash -n e2e/with-kube-gateway.sh
  • The HA workflow selects deploy/openshell and reaches the Rust E2E suite.
  • Kubernetes HA E2E passes. Failure is expected on current main until feat(kubernetes): support HA gateway rebalancing #1868 lands; this PR intentionally exposes that known gap and does not attempt to implement HA support.

Proof-of-life run for this minimal commit: https://github.com/NVIDIA/OpenShell/actions/runs/29091285118/job/86358988762

Checklist

  • Follows Conventional Commits
  • Commit is signed off (DCO)
  • Change is limited to the HA health port-forward prerequisite

@elezar elezar added the test:e2e-kubernetes Requires Kubernetes end-to-end coverage label Jul 9, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 9, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Label test:e2e-kubernetes applied, but pull-request/2198 is at {"messa while the PR head is 2f05f62. A maintainer needs to comment /ok to test 2f05f6253248659791fe4ce466646136e764b782 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@elezar elezar marked this pull request as ready for review July 9, 2026 14:12
@elezar elezar requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 9, 2026 14:12
@elezar elezar changed the title fix(e2e): detect workload kind for health port-forward in kube e2e fix(ci): add kubernetes e2e to post-merge pipeline and fix HA health port-forward Jul 9, 2026
@elezar elezar force-pushed the fix/ha-e2e-health-portforward-workload-kind branch from 7de3f4e to 01180a4 Compare July 9, 2026 18:19
Comment thread .github/workflows/release-dev.yml Outdated
image-tag: ${{ github.sha }}
runner: linux-arm64-cpu8

kubernetes-e2e:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we duplicate these or rather factor out the k8s tests?

The health port-forward was hardcoded to sts/<name>, which breaks when
the chart deploys a Deployment (workload.kind=deployment, used by the
HA E2E). Add kube_workload_ref() to probe the cluster and return the
correct resource reference at runtime.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar force-pushed the fix/ha-e2e-health-portforward-workload-kind branch from 01180a4 to 56cfba5 Compare July 9, 2026 18:43
@elezar elezar changed the title fix(ci): add kubernetes e2e to post-merge pipeline and fix HA health port-forward fix(e2e): detect gateway workload for Kubernetes health port-forward Jul 10, 2026
@elezar elezar force-pushed the fix/ha-e2e-health-portforward-workload-kind branch from 56cfba5 to c61b114 Compare July 10, 2026 12:02
@elezar

elezar commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

Closing for now. HA behavior is not expected to pass on main until #1868 lands; we can reassess whether this port-forward adjustment is still needed once that work merges.

@elezar elezar closed this Jul 10, 2026
@elezar elezar deleted the fix/ha-e2e-health-portforward-workload-kind branch July 10, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e-kubernetes Requires Kubernetes end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant