Skip to content

Add checkovAzurePipelines sample fixture for CKV_AZUREPIPELINES_2#165

Draft
Copilot wants to merge 6 commits into
mainfrom
copilot/promote-ckv-azurepipelines-severity
Draft

Add checkovAzurePipelines sample fixture for CKV_AZUREPIPELINES_2#165
Copilot wants to merge 6 commits into
mainfrom
copilot/promote-ckv-azurepipelines-severity

Conversation

Copy link
Copy Markdown

Copilot AI commented May 16, 2026

Deliverable #3 from #164 (split out from #163): a reusable fixture that lets reviewers verify the CKV_AZUREPIPELINES_* severity promotion (notewarning) end-to-end against the public MicrosoftSecurityDevOps@1 task output.

Pending in Guardian: the actual severity-mapping policy change lives in Microsoft's internal Guardian repo and is owned by that team — out of scope here. This PR only adds the public-side fixture so behavior can be validated once the internal change ships.

Changes

  • samples/checkovAzurePipelines/azure-pipelines.yml — minimal pipeline whose job-level container: ubuntu:latest triggers CKV_AZUREPIPELINES_1 and CKV_AZUREPIPELINES_2 when Checkov scans the pipeline file, followed by MicrosoftSecurityDevOps@1 with tools: 'checkov'.
  • samples/checkovAzurePipelines/readme.md — run instructions, expected Scans Tab behavior (visible as Warning post-Guardian-change; hidden under default filter until then), and the paste-ready AC1/AC3/AC4 jq snippets lifted from the issue body.
  • samples/readme.md — index the new sample alongside the existing ones.

Structure/style mirrors samples/IaCMapping/ and samples/trivypipeline/.

Out of scope (per spec + agent instructions)

  • src/MicrosoftSecurityDevOps/** — fix is not in the wrapper; .gdnconfig severity overrides are a separate feature.
  • Wiki update — lives in microsoft/security-devops-azdevops.wiki.git.

Validation

Ran Checkov locally against the fixture to confirm the intended rules fire:

  • Checkov version: 3.2.529
  • Command:
    checkov -f samples/checkovAzurePipelines/azure-pipelines.yml \
            --framework azure_pipelines -o sarif --output-file-path .
    jq '[.runs[0].results[] | select(.ruleId | startswith("CKV_AZUREPIPELINES_")) | .ruleId] | unique' results_sarif.sarif
  • Output:
    [
      "CKV_AZUREPIPELINES_1",
      "CKV_AZUREPIPELINES_2"
    ]

TODO: AC2 screenshot pending real ADO build run by a maintainer (cannot drive an Azure DevOps pipeline autonomously from this environment).

Copilot AI changed the title [WIP] Promote CKV_AZUREPIPELINES_* severity from note to warning Add checkovAzurePipelines sample fixture for CKV_AZUREPIPELINES_2 May 16, 2026
Copilot AI requested a review from DimaBir May 16, 2026 07:01
@DimaBir
Copy link
Copy Markdown
Contributor

DimaBir commented May 16, 2026

@copilot follow-up validation pass for this PR:

  1. Validate the fixture against Checkov locally. Install Checkov in the agent environment (pip install checkov), then run:

    checkov -f samples/checkovAzurePipelines/azure-pipelines.yml --framework azure_pipelines -o sarif --output-file-path .

    Confirm the resulting results_sarif.sarif contains CKV_AZUREPIPELINES_1 and CKV_AZUREPIPELINES_2 results against this file. Expected jq output:

    jq '[.runs[0].results[] | select(.ruleId | startswith("CKV_AZUREPIPELINES_")) | .ruleId] | unique' results_sarif.sarif
    # ["CKV_AZUREPIPELINES_1","CKV_AZUREPIPELINES_2"]
  2. Append a "Validation" section to this PR's body (edit the PR description, not a new comment) containing:

    • Checkov version used.
    • The exact jq command above and its real output.
    • A "TODO: AC2 screenshot pending real ADO build run by a maintainer." note — you can't drive an Azure DevOps pipeline autonomously, so leave the screenshot step explicitly deferred.
  3. If the rules do NOT fire (zero or one of _1/_2 missing), do not paper over it — fix the YAML and re-run, then attach evidence. The fixture's only purpose is to trigger those rules; a non-triggering fixture should not be merged.

Scope constraints (unchanged from the original assignment):

  • Do not modify src/MicrosoftSecurityDevOps/**, the wiki, or anything outside samples/checkovAzurePipelines/ and the PR description.
  • One-line commit messages, no co-author trailers.

Spec for reference: #164. Original review feedback that produced commits 733cecd, 0cd5973, 7e2fcf1, d07b0f9 is in this PR's history.

Copy link
Copy Markdown
Author

Copilot AI commented May 16, 2026

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • api0.prismacloud.io
    • Triggering command: /home/REDACTED/.local/bin/checkov checkov -f samples/checkovAzurePipelines/azure-pipelines.yml --framework azure_pipelines -o sarif --output-file-path /tmp/ (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@DimaBir
Copy link
Copy Markdown
Contributor

DimaBir commented May 16, 2026

@copilot the firewall block is on Checkov's Prisma Cloud telemetry endpoint, not on anything the scan itself needs. The scan is fully local. Retry with --skip-download, which suppresses the api0.prismacloud.io call:

checkov \
  -f samples/checkovAzurePipelines/azure-pipelines.yml \
  --framework azure_pipelines \
  -o sarif \
  --output-file-path /tmp/ \
  --skip-download

If --skip-download alone isn't enough on the bundled Checkov version, also export:

export BC_API_URL=  CHECKOV_RUN_LOCAL_ONLY=true

and re-run. Both target the same thing — preventing Checkov from phoning home — but --skip-download is the official switch.

Then complete step 2 from the previous comment: append the Validation section (Checkov version, jq output, and the AC2 TODO) to the PR description.

Do NOT request firewall allowlisting changes — the offline run is what we want; it matches how the MSDO task invokes Checkov in customer environments anyway.

Copy link
Copy Markdown
Author

Copilot AI commented May 16, 2026

@copilot the firewall block is on Checkov's Prisma Cloud telemetry endpoint, not on anything the scan itself needs. The scan is fully local. Retry with --skip-download, which suppresses the `api0.pr...

Re-ran offline with --skip-download on Checkov 3.2.529:

$ checkov -f samples/checkovAzurePipelines/azure-pipelines.yml --framework azure_pipelines -o sarif --output-file-path /tmp/ --skip-download
$ jq '[.runs[0].results[] | select(.ruleId | startswith("CKV_AZUREPIPELINES_")) | .ruleId] | unique' /tmp/results_sarif.sarif
[
  "CKV_AZUREPIPELINES_1",
  "CKV_AZUREPIPELINES_2"
]

Both _1 and _2 fire — no BC_API_URL / CHECKOV_RUN_LOCAL_ONLY fallback needed. PR description already carries the Validation section (version, jq command, real output) and the AC2 screenshot TODO deferred to a maintainer ADO run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spec: Promote CKV_AZUREPIPELINES_* severity from note to warning

2 participants