Describe the issue
When using azure-devops-oidc authentication in Azure DevOps pipelines, CLI commands like databricks current-user me and databricks bundle validate work correctly. However, databricks bundle deploy fails during the Terraform apply phase because the Terraform subprocess doesn't receive the required SYSTEM_ACCESSTOKEN environment variable:
Error: cannot create job: failed during request visitor: azure-devops-oidc auth: SYSTEM_ACCESSTOKEN env var not found, if calling from Azure DevOps Pipeline, please set this env var following https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#systemaccesstoken. Config: host=https://adb-X.azuredatabricks.net,/ client_id=X. Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID
While exploring the codebase, I noticed the static allowlist of environment variables passed to Terraform in bundle/deploy/terraform/init.go.
To test this hypothesis, I created a wrapper script that injects all SYSTEM_* and DATABRICKS_* env vars and used DATABRICKS_TF_EXEC_PATH to point to it. This made bundle deploy work successfully.
Steps to reproduce the behavior
- Configure a Databricks service principal with Azure DevOps OIDC federation policy
- Set up an Azure DevOps pipeline like the following :
steps:
- task: Bash@3
inputs:
targetType: 'inline'
script: |
databricks bundle deploy -t dev
displayName: 'Deploy Bundle'
env:
DATABRICKS_HOST: $(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID: $(DATABRICKS_CLIENT_ID)
DATABRICKS_AUTH_TYPE: azure-devops-oidc
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
- Run the pipeline
Expected Behavior
databricks bundle deploy should pass the required Azure DevOps environment variables to Terraform so OIDC authentication works & the deployment completes successfully.
Actual Behavior
Clear and concise description of what actually happened
OS and CLI version
- Databricks CLI v0.278.0
- Ubuntu (Azure DevOps hosted agent vmImage: ubuntu-latest)
Is this a regression?
Unknown
Debug Logs
Error: cannot create job: failed during request visitor: azure-devops-oidc auth: SYSTEM_ACCESSTOKEN env var not found, if calling from Azure DevOps Pipeline, please set this env var following
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables
Describe the issue
When using azure-devops-oidc authentication in Azure DevOps pipelines, CLI commands like databricks current-user me and databricks bundle validate work correctly. However, databricks bundle deploy fails during the Terraform apply phase because the Terraform subprocess doesn't receive the required SYSTEM_ACCESSTOKEN environment variable:
While exploring the codebase, I noticed the static allowlist of environment variables passed to Terraform in bundle/deploy/terraform/init.go.
To test this hypothesis, I created a wrapper script that injects all SYSTEM_* and DATABRICKS_* env vars and used DATABRICKS_TF_EXEC_PATH to point to it. This made bundle deploy work successfully.
Steps to reproduce the behavior
Expected Behavior
databricks bundle deploy should pass the required Azure DevOps environment variables to Terraform so OIDC authentication works & the deployment completes successfully.
Actual Behavior
Clear and concise description of what actually happened
OS and CLI version
Is this a regression?
Unknown
Debug Logs
Error: cannot create job: failed during request visitor: azure-devops-oidc auth: SYSTEM_ACCESSTOKEN env var not found, if calling from Azure DevOps Pipeline, please set this env var following
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables