test(e2e): add API and CLI conformance profiles by elezar · Pull Request #2182 · NVIDIA/OpenShell

elezar · 2026-07-08T13:15:22Z

Summary

Adds explicit API and CLI conformance profiles to the standalone Rust e2e package without changing production code.

The API profile validates driver-independent gateway and sandbox semantics directly over gRPC. The CLI profile identifies portable user workflows that can run against any compatible gateway while remaining separate from driver-specific infrastructure coverage. Gateway restart tests now share an application-level resume scenario but remain outside baseline conformance because they require provisioner control.

Related Issue

spike: driver conformance suite for compute drivers #2183
Follow-up: test(e2e): separate VM gateway provisioning from test execution #2314
Related VM CI work: test(e2e): run VM suite in CI #2305

Changes

add an internal openshell-conformance binary under e2e/rust
add a shared API scenario engine used directly by the Rust integration test
implement API scenarios for lifecycle, not-found behavior, idempotent deletion, validation, concurrent creation, labels, streamed command execution, and process hardening
add the e2e-api-conformance profile and run it against Docker, Podman, Kubernetes, and VM gateway lanes
add the e2e-cli-conformance profile for portable smoke, sandbox lifecycle, labels, port forwarding, sync, and upload-on-create workflows
verify both CLI command paths: sandbox create -- <command> and sandbox exec
run CLI conformance in the canonical Docker-backed Rust e2e suite
extract VM overlay, writable-workspace, and TLS-key permission assertions into a VM-specific test
share the gateway restart/resume contract across Docker, Podman, and VM tests while retaining driver-specific observation hooks
keep gateway provisioning separate from scenario execution through an explicit endpoint and mTLS environment contract
use local tonic transport and output formatting so the suite does not depend on the full openshell-cli crate
keep the conformance binary internal to tests and out of release packaging

The capabilities scenario remains deferred until the gateway exposes driver capabilities through its public API. Additional CLI, security/network, and external-image conformance candidates are intentionally left for follow-up work.

Testing

Checklist

Changes are limited to e2e test code, test provisioners, and task configuration
No production code or production dependencies changed
API and CLI conformance remain explicitly separated
All commits are signed off and follow Conventional Commits
Documentation updated

copy-pr-bot · 2026-07-08T13:15:26Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

alangou · 2026-07-21T13:59:46Z

+    // ── 4. Second delete → ok, deleted: false ────────────────────────────
+    let del2 = client
+        .delete_sandbox(DeleteSandboxRequest {
+            name: sandbox_name.clone(),
+            workspace: String::new(),
+        })
+        .await
+        .into_diagnostic()
+        .wrap_err("second delete_sandbox (idempotency check) returned an error")?
+        .into_inner();


same comment #2182 (comment)

Good catch. deleted: false reflects the compute-driver contract, but the gateway returns NOT_FOUND when its sandbox record is already absent. Since this PR should document existing behavior rather than change production semantics, I’ve updated both assertions to expect NOT_FOUND. We can consider idempotent gateway deletion separately.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

github-actions · 2026-07-22T14:07:43Z

Label test:e2e applied for 2ce39c4. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

alangou · 2026-07-22T15:17:15Z

+    fn sandbox_names(&self, run_id: &str) -> Vec<String> {
+        self.sandbox_name_stems
+            .iter()
+            .map(|stem| format!("conformance-{stem}-{run_id}"))
+            .collect()
+    }


I suspect some issues with this function since sandbox names are limited by default

OpenShell/crates/openshell-server/src/grpc/mod.rs

Lines 108 to 113 in 2ce39c4

/// Maximum length for a sandbox or provider name (Kubernetes name limit).

const MAX_NAME_LEN: usize = 253;

/// Maximum length for DNS-routable names (workspace, sandbox, service).

/// Three segments plus two `--` delimiters must fit a 63-char DNS label:

/// 19 + 2 + 19 + 2 + 19 = 61.

const MAX_ROUTABLE_NAME_LEN: usize = 19;

to avoid issues maybe we can trim the string to respect the configuration

alangou · 2026-07-22T15:24:37Z

https://github.com/NVIDIA/OpenShell/actions/runs/29925827101/job/88952077297#step:11:600 it seems there is an issue with rustls but I don't see a change in this PR that could explain this

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar mentioned this pull request Jul 8, 2026

spike: driver conformance suite for compute drivers #2183

Open

elezar mentioned this pull request Jul 16, 2026

test(e2e): separate VM gateway provisioning from test execution #2314

Open

7 tasks

elezar force-pushed the feat/conformance-suite-skeleton branch from 8f15a6c to a0bb1ff Compare July 16, 2026 14:19

elezar changed the title ~~feat(cli): add openshell conformance subcommand skeleton~~ test(e2e): add API and CLI conformance profiles Jul 16, 2026

elezar force-pushed the feat/conformance-suite-skeleton branch from de26f37 to 9493e29 Compare July 17, 2026 14:00

elezar marked this pull request as ready for review July 17, 2026 14:03

elezar requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 17, 2026 14:03

elezar force-pushed the feat/conformance-suite-skeleton branch 3 times, most recently from 6fe2c16 to 8bb0509 Compare July 21, 2026 12:47

alangou reviewed Jul 21, 2026

View reviewed changes

Comment thread e2e/rust/src/conformance.rs Outdated

alangou reviewed Jul 21, 2026

View reviewed changes

elezar added 7 commits July 22, 2026 15:39

test(e2e): add API conformance suite

405067b

Signed-off-by: Evan Lezar <elezar@nvidia.com>

test(e2e): define CLI conformance profile

3a27b59

Signed-off-by: Evan Lezar <elezar@nvidia.com>

test(e2e): add command execution conformance

25e0a8b

Signed-off-by: Evan Lezar <elezar@nvidia.com>

test(e2e): add process hardening API conformance

983b79a

Signed-off-by: Evan Lezar <elezar@nvidia.com>

test(e2e): classify portable CLI workflows

5587eb1

Signed-off-by: Evan Lezar <elezar@nvidia.com>

refactor(e2e): share gateway resume scenario

a7ea34d

Signed-off-by: Evan Lezar <elezar@nvidia.com>

docs(e2e): diagram conformance test architecture

2ce39c4

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar force-pushed the feat/conformance-suite-skeleton branch from 8bb0509 to 2ce39c4 Compare July 22, 2026 13:51

elezar mentioned this pull request Jul 22, 2026

docs(examples): document external compute driver authoring and extensibility #2418

Open

7 tasks

elezar added the test:e2e Requires end-to-end coverage label Jul 22, 2026

alangou reviewed Jul 22, 2026

View reviewed changes

fix(e2e): make conformance tests runnable in CI

2176dca

Signed-off-by: Evan Lezar <elezar@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(e2e): add API and CLI conformance profiles - #2182

test(e2e): add API and CLI conformance profiles#2182
elezar wants to merge 8 commits into
mainfrom
feat/conformance-suite-skeleton

elezar commented Jul 8, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

Uh oh!

alangou Jul 21, 2026

Uh oh!

elezar Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

alangou Jul 22, 2026

Uh oh!

alangou commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	/// Maximum length for a sandbox or provider name (Kubernetes name limit).
	const MAX_NAME_LEN: usize = 253;
	/// Maximum length for DNS-routable names (workspace, sandbox, service).
	/// Three segments plus two `--` delimiters must fit a 63-char DNS label:
	/// 19 + 2 + 19 + 2 + 19 = 61.
	const MAX_ROUTABLE_NAME_LEN: usize = 19;

Uh oh!

Conversation

elezar commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

Uh oh!

alangou Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

alangou Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

alangou commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elezar commented Jul 8, 2026 •

edited

Loading