Job task pypi libraries broken

[Tello-Wharton]

Describe the issue

(regression) pypi package libraries no longer can be included as a part of a task definition in databricks asset bundles

Configuration

resources:
  jobs:
    test-job:
      name: test-job
      max_concurrent_runs: 1
      tasks:
      - task_key: test
        libraries: 
        - pypi:
            package: requests

Steps to reproduce the behavior

Run databricks bundle deploy

Expected Behavior

A successful deployment

Actual Behavior

Error: exit status 1

Error: Missing required argument

  on bundle.tf.json line 67, in resource.databricks_job.test-job.task[0].library[0].pypi:
  67:                 "pypi": {}

The argument "package" is required, but no definition was found.

OS and CLI version

OS: Ubuntu
CLI Version: v0.241.1

Is this a regression?

This was working prior to 0.241, the hotfix in 0.241.1 merely changed the error, did not actually fix deployments

[artemsachuk]

I experience the same issue, but with maven, on both 0.241.0 and 0.241.1:

<...>
Error: Missing required argument

on bundle.tf.json line 2583, in resource.databricks_job..task[14].library[1].maven:
2583: "maven": {}

The argument "coordinates" is required, but no definition was found.
<...>

It worked prior to 0.241.0

[balexander]

I'm also experiencing the same thing, even on 0.241.1

[pietern]

Thanks for reporting folks. We're working on a fix. In the meantime, you can try to pin the CLI release to v0.240.0.

[pietern]

We just released v0.241.2 which includes a fix for the issue.

I manually confirmed that deploying a job with tasks with PyPI libraries works again.

We'll take a closer look and figure out how to be more defensive against such issues.

Many thanks for reporting the issue and bearing with us.

[pietern]

The patch release is available through the GitHub Action and our Homebrew tap.