Add validation mutator for volume artifact_path#2050
Conversation
arifact_pathartifact_path
| CreateVolumeRequestContent: &catalog.CreateVolumeRequestContent{ | ||
| CatalogName: "catalogN", | ||
| Name: "volumeN", | ||
| VolumeType: "MANAGED", |
There was a problem hiding this comment.
Is the VolumeType necessary for this test?
| // Fast mutators with only in-memory checks | ||
| JobClusterKeyDefined(), | ||
| JobTaskClusterSpec(), | ||
| SingleNodeCluster(), |
There was a problem hiding this comment.
Do all of these return errors?
Warnings currently only show up at the end of deploy, so they're not actionable (yet).
There was a problem hiding this comment.
Good point, SingleNodeCluster() returns a warning that is useful even with a deployment. The others, though indeed, are no longer useful.
I can follow up with a PR that serializes the warnings (and below like INFO) on a rolling basis in the bundle.Seq mutator. That should make the warnings useful before deployment.
There was a problem hiding this comment.
The others, though indeed, are no longer useful.
How so?
There was a problem hiding this comment.
We need to revisit how we output these diags anyway, so no need to address here.
There was a problem hiding this comment.
How so?
You'd end up with runtime errors on JobClusterKeyDefined. The validation warning has a runtime error equivalent:
.venv➜ bundle-playground git:(master) ✗ databricks bundle validate
Warning: job_cluster_key c1 is not defined
at resources.jobs.foo.tasks[0].job_cluster_key
in databricks.yml:17:28
Error: terraform apply: exit status 1
Error: cannot create job: Job cluster 'c1' is not defined in field 'job_clusters'.
with databricks_job.foo,
on bundle.tf.json line 35, in resource.databricks_job.foo:
35: }
JobTaskClusterSpec returns errors though so it's useful for returning an error early.
There was a problem hiding this comment.
We should probably promote this warning to an error though.
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Bundles: * Fix finding Python within virtualenv on Windows ([#2034](#2034)). * Include missing field descriptions in JSON schema ([#2045](#2045)). * Add validation for volume referenced from `artifact_path` ([#2050](#2050)). * Handle `${workspace.file_path}` references in source-linked deployments ([#2046](#2046)). * Set the write bit for files written during template initialization ([#2068](#2068)).
Bundles: * Fix finding Python within virtualenv on Windows ([#2034](#2034)). * Include missing field descriptions in JSON schema ([#2045](#2045)). * Add validation for volume referenced from `artifact_path` ([#2050](#2050)). * Handle `${workspace.file_path}` references in source-linked deployments ([#2046](#2046)). * Set the write bit for files written during template initialization ([#2068](#2068)).
## Changes
This PR:
1. Incrementally improves the error messages shown to the user when the
volume they are referring to in `workspace.artifact_path` does not
exist.
2. Performs this validation in both `bundle validate` and `bundle
deploy` compared to before on just deployments.
3. It runs "fast" validations on `bundle deploy`, which earlier were
only run on `bundle validate`.
## Tests
Unit tests and manually. Also, existing integration tests provide
coverage (`TestUploadArtifactToVolumeNotYetDeployed`,
`TestUploadArtifactFileToVolumeThatDoesNotExist`)
Examples:
```
.venv➜ bundle-playground git:(master) ✗ cli bundle validate
Error: cannot access volume capital.whatever.my_volume: User does not have READ VOLUME on Volume 'capital.whatever.my_volume'.
at workspace.artifact_path
in databricks.yml:7:18
```
and
```
.venv➜ bundle-playground git:(master) ✗ cli bundle validate
Error: volume capital.whatever.foobar does not exist
at workspace.artifact_path
resources.volumes.foo
in databricks.yml:7:18
databricks.yml:12:7
You are using a volume in your artifact_path that is managed by
this bundle but which has not been deployed yet. Please first deploy
the volume using 'bundle deploy' and then switch over to using it in
the artifact_path.
```
Bundles: * Fix finding Python within virtualenv on Windows ([#2034](#2034)). * Include missing field descriptions in JSON schema ([#2045](#2045)). * Add validation for volume referenced from `artifact_path` ([#2050](#2050)). * Handle `${workspace.file_path}` references in source-linked deployments ([#2046](#2046)). * Set the write bit for files written during template initialization ([#2068](#2068)).
Changes
This PR:
workspace.artifact_pathdoes not exist.bundle validateandbundle deploycompared to before on just deployments.bundle deploy, which earlier were only run onbundle validate.Tests
Unit tests and manually. Also, existing integration tests provide coverage (
TestUploadArtifactToVolumeNotYetDeployed,TestUploadArtifactFileToVolumeThatDoesNotExist)Examples:
and