tests: unblock CLI integration nightlies after main-branch drift#5076
Merged
simonfaltum merged 5 commits intomainfrom Apr 23, 2026
Merged
tests: unblock CLI integration nightlies after main-branch drift#5076simonfaltum merged 5 commits intomainfrom
simonfaltum merged 5 commits intomainfrom
Conversation
Two unrelated main-branch regressions surfaced after the integration test trigger was re-enabled in #5034. Neither is caused by any in-flight feature PR; they just sat behind the disabled trigger. - record_cloud goldens: the host-metadata cache (#5011) regenerated most goldens but could not touch Cloud=true/Local=false suites. Regenerated workspace-file-io and pipeline-crud against e2-dogfood; changes are purely removal of redundant /.well-known/databricks-config requests. - dashboard assumptions: Lakeview behavior now varies across clouds: AWS staging includes serialized_dashboard in updates; GCP production no longer clears warehouse_id on workspace import. Loosened both assertions to Subset checks that still catch unexpected drift. Co-authored-by: Isaac
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Simplification pass: two assert.Contains + assert.Subset become assert.Subset(actual, required) + assert.Subset(allowed, actual), and the two comments merge into one. Co-authored-by: Isaac
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
andrewnester
approved these changes
Apr 23, 2026
Same post-cache drift as the other two record_cloud suites; this one is gated on RequiresUnityCatalog so it skipped on non-UC nightlies but would fail on the aws-prod-ucws-is / azure-prod-ucws-is runners. Regenerated against e2-dogfood with TEST_METASTORE_ID set. Flagged by codex review. Co-authored-by: Isaac
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
…te_one/cloud The prod-is test workspace no longer returns display_name for test-dabs-1@databricks.com and test-dabs-2@databricks.com in the permissions API response (the SPN owner still does). Predictable server-side drift, not reproducible against e2-dogfood because the SPN/user distinction differs. Mass-replacement edit per the testing rule exception. Co-authored-by: Isaac
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
…obs/delete_one/cloud" This reverts commit f05d5cd.
Member
Author
|
Reverted The earlier theory (prod-is stopped returning If it turns out to be flaky between envs, a |
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Integration test nightlies (
cli-isolated-pr.yml) have been red on every main run since 2026-04-02, when#4899temporarily disabled the trigger. The trigger was re-enabled in#5034and all accumulated failures surfaced at once. Nothing in any in-flight feature PR is to blame; this PR just clears the backlog so nightly signal goes green again.Two independent regressions:
#5011) regenerated goldens for tests that run locally, but could not touchCloud=true, Local=falsesuites.acceptance/selftest/record_cloud/{pipeline-crud,workspace-file-io}still expected the pre-cache/.well-known/databricks-configcalls.serialized_dashboardin the updated fields; GCP production no longer clearswarehouse_id. The exact-match assertions inTestDashboardAssumptions_WorkspaceImportfail differently on each cloud.Changes
Before:
record_cloudgoldens include redundant/.well-known/databricks-configGETs; dashboard test hard-codes exact updated/deleted fields.Now: goldens regenerated against e2-dogfood (only diff is removal of the cached requests); dashboard assertions use
assert.Subsetso they tolerate cross-cloud drift but still fail on anything outside the known-allowed set.acceptance/selftest/record_cloud/pipeline-crud/output.txt,acceptance/selftest/record_cloud/workspace-file-io/output.txt: rerun with-updateunderCLOUD_ENV=awsagainst e2-dogfood. Both terraform and direct variants produce identical output.integration/assumptions/dashboard_assumptions_test.go:etagandupdate_timemust appear in updated fields;serialized_dashboardis allowed;warehouse_idis the only allowed deletion. Comment points to the observed cross-cloud split so the next reader knows why.Follows the pattern of the previous Lakeview-behavior-change fix in
#4640.Test plan
make checkscleanmake lintclean (0 issues)go test ./acceptance -run 'TestAccept/selftest/record_cloud/{workspace-file-io,pipeline-crud}'passes against e2-dogfood (both terraform and direct variants)go test ./integration/assumptions -run TestDashboardAssumptions_WorkspaceImportpasses against e2-dogfood