Bundle: Workflow wheel location not interpolated anymore in v0.212.4

[JCZuurmond]

Describe the issue

When creating containing a reference to a wheel in the libraries:

libraries:
- whl: /dist/*.whl

The wheel location in the resulting job is not interpolated anymore:

Before:

- whl: /Workspace/Users/<user name>/.bundle/<project>/<target>/artifacts/.internal/<wheel name>.whl

Current:

- whl: /dist/*.whl

The later results in the following error when running the job:

Python wheels must be stored in UC Volumes, dbfs, s3, adls, gs or as a workspace file/local file. Make sure the URI begins with 'dbfs:', 'file:', 's3:', 'abfss:', 'gs:', 'wasbs:', '/Volumes', or '/Workspace'but the URI is '/dist/*.whl').

Steps to reproduce the behavior

1. Follow the Databricks guide on deploying a workflow with a Python wheel
2. Run the created job AND/OR looking at the YAML job definition from the Databricks workflow UI

Expected Behavior

A Databricks workflow with a library location as follows should be created:

- whl: /Workspace/Users/<user name>/.bundle/<project>/<target>/artifacts/.internal/<wheel name>.whl

Actual Behavior

A Databricks workflow with a library location as follows is created:

- whl: /dist/*.whl

This causes the job to fail as this wheel location does not exists.

OS and CLI version

Error occurs in: v0.212.4
Error does not occur in: v0.212.3

Is this a regression?

Yes, it worked in: v0.212.3

[JCZuurmond]

@pietern : Could you have a look? I expect this issue to be created in this PR

[pietern]

@JCZuurmond Thanks for reporting. To confirm, you have an absolute path /dist where you keep built wheel files?

The rationale for the change is that absolute paths may be ambiguous in interpretation; they can be local paths or remote (workspace) paths. We treat relative paths as local always, so that could be a mitigation here.

[JCZuurmond]

I used an absolute path in the bug report above. When I change it to a relative path

- whl: ./dist/*.whl

I get an error when running a bundle deploy:

$ databricks bundle deploy
Building default...
Error: file bundles\resources\dist\*.whl is referenced in libraries section but doesn't exist on the local file system

[pietern]

Can you confirm the path is relative to the configuration file where it is defined, and not the bundle root?

With a dist folder in the bundle root and configuration in bundles/resources, the relative path is ../../dist.

[JCZuurmond]

yes, that solved the issue, thanks! Now, the library path is relative to the workflow definition instead of location of execution. I like that better as well

Maybe you could improve the error messaging? The mentioned location bundles\resources\dist\*.whl does not exist within my project. It looks like something internal to bundles. If that would point to the resolved relative path location on my filesystem it would be more clear that the wheel is not found

[andrewnester]

The mentioned location bundles\resources\dist*.whl does not exist within my project

Not sure I understood what you mean here. If I understand correctly you have a top-level bundle.yml file and relative to this file you have bundles/resources folder, correct?
In that resources folder you have another yml configuration with - whl: ./dist/*.whl. When this path is resolved, it's resolved relative to the yml file location it's defined in, so to -> bundles\resources\dist\*.whl which indeed does not exists

Let me know if I misunderstood something

[JCZuurmond]

Ah you are completely right. I was thrown off because it first worked in version v0.212.3 and not in v0.212.4 anymore. That path does exists within the context of the project yes, the error message is correct. Thanks for your help.