Spec: Promote CKV_AZUREPIPELINES_* severity from note to warning

[DimaBir]

Tracking issue split out from the diagnostic discussion in #163 so this can be assigned and worked independently. See #163 for the full investigation history.

Spec: Promote CKV_AZUREPIPELINES_* severity from note → warning

The implementation lives in Microsoft's internal Guardian severity-policy, but the acceptance criteria below are verifiable against the public MSDO task's output, so completion can be validated end-to-end without any internal artifacts.

Background

From #163:

• Checkov 3.2.497 produces CKV_AZUREPIPELINES_1 / _2 findings against azure-pipelines.yml.
• Guardian's Checkov severity-mapping policy maps these rule IDs to SARIF level: "note" in the published msdo.sarif.
• The SARIF Scans Tab extension's default filter hides note-level findings, so the results land in the published artifact but are invisible to users by default.
• The same mechanism downgrades CKV_AZURE_177 to warning (still visible). The asymmetry between warning (visible) and note (hidden) is the user-visible bug.

Change

In Guardian's Checkov severity-mapping policy:

• Map every rule whose ID matches ^CKV_AZUREPIPELINES_ to SARIF level: "warning".
• Prefer a prefix/wildcard rule over per-rule entries, so future additions to Checkov's azure_pipelines framework inherit the promotion without policy churn.
• Place the rule in the same policy file used for the existing CKV_AZURE_177 downgrade, for locality and consistency.

Out of scope

• Upstream Checkov default severities (owned by Prisma Cloud).
• SARIF Scans Tab's default filter — separate UX concern; will be filed as a follow-up.
• Exposing per-rule severity overrides in .gdnconfig — separate feature.

Acceptance criteria

Each criterion is a hard requirement and must have an automated check or a documented manual verification step.

AC1 — Severity in published SARIF. Running MicrosoftSecurityDevOps@1 (Checkov enabled, default policy: 'azuredevops') against a fixture repo whose azure-pipelines.yml triggers CKV_AZUREPIPELINES_2 (e.g., a step referencing a container image tagged :latest) MUST yield a msdo.sarif where every CKV_AZUREPIPELINES_* result has level == "warning". Verified by:

jq '[.runs[]
     | select(.tool.driver.name == "checkov")
     | .results[]
     | select(.ruleId | startswith("CKV_AZUREPIPELINES_"))
     | .level] | unique' /home/vsts/work/1/a/.gdn/msdo.sarif
# Expected: ["warning"]

AC2 — Scans Tab visibility. With the SARIF Scans Tab extension at its default severity filter, CKV_AZUREPIPELINES_* findings MUST appear under the checkov tool collapse without the user enabling "Notes." Verified by screenshot of a build run against the fixture from AC1 attached to the PR.

AC3 — No regression for other Checkov rules. The severity mapping for every rule that is not CKV_AZUREPIPELINES_* MUST be unchanged. Verified by capturing baseline (pre-change) and post-change severity histograms over the fixture from AC1:

jq '[.runs[]
     | select(.tool.driver.name == "checkov")
     | .results[]
     | {ruleId, level}]
    | group_by(.ruleId)
    | map({ruleId: .[0].ruleId, level: .[0].level})' /home/vsts/work/1/a/.gdn/msdo.sarif

The diff MUST contain only CKV_AZUREPIPELINES_* rows changing from note → warning.

AC4 — Build break unchanged. With MicrosoftSecurityDevOps@1 configured break: true and the default --min-severity Error, AZUREPIPELINES warnings MUST NOT break the build. Verified by running the fixture pipeline twice — once with break: true, once with break: false — both completing with succeeded status.

AC5 — Test coverage. A unit test in Guardian's severity-mapping suite MUST assert the new mapping for at least CKV_AZUREPIPELINES_1 and CKV_AZUREPIPELINES_2. Test placement and framework conventions to follow the existing tests for CKV_AZURE_177.

Deliverables

1. Policy change in Guardian (one file).
2. Unit test covering AC5.
3. Integration fixture under samples/ in this repo (microsoft/security-devops-azdevops) containing a minimal azure-pipelines.yml that triggers CKV_AZUREPIPELINES_2, with a short README.md describing the expected Scans Tab output. Submit as a separate PR against this repo so the fixture is reusable.
4. Wiki update on the Home page Checkov section noting that CKV_AZUREPIPELINES_* surface as warning-level findings.

Verification commands (paste-ready)

A reviewer should be able to copy these and verify a built artifact:

# All AZUREPIPELINES rules should be warning, none note:
jq '[.runs[] | select(.tool.driver.name=="checkov") | .results[]
     | select(.ruleId | startswith("CKV_AZUREPIPELINES_")) | .level] | unique' msdo.sarif

# Verify no rule other than AZUREPIPELINES has flipped:
jq '[.runs[] | select(.tool.driver.name=="checkov") | .results[]
     | {ruleId, level}] | unique' msdo.sarif

Definition of done

• AC1–AC5 all pass on a real CI run of the fixture pipeline.
• PR description includes the AC1/AC3 jq outputs and the AC2 screenshot.
• Linked back to Checkov tool omits Azure Pipelines results #163 and closes it on merge.

---

Original ask: #163