You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: add eMudhra/emSign CA stopgap for GitHub-hosted Windows runners - #3270
The runner trust probe (#3268, run) shows that the GitHub-hosted Windows images (windows-2022, windows-latest, windows-11-arm) still trust the compromised eMudhra intermediates (CVE-2026-111943), so schannel-based clients (git, gh, PowerShell/.NET) accept certificates issued with the compromised keys. Microsoft blocks these certificates through Windows' automatic trust-list updates, but the hosted images do not have that update.
This change adds a composite action (.github/actions/upgrade-ca-certificates) that applies the OS-appropriate fix as the first step of each affected job, before actions/checkout sends any token over HTTPS:
Windows (GitHub-hosted): adds the eight compromised intermediates and their eight Sectigo cross-signs to Cert:\LocalMachine\Disallowed, then fails the job unless Windows chain building reports every one of them as explicitly distrusted. This matches the disclosure's own Windows workaround. Distrusting only emSign ECC Root CA - G3 is not enough, because the cross-signs chain to AAA Certificate Services, which Windows trusts (thanks @rwgk).
Linux (GitHub-hosted): upgrades ca-certificates via apt-get, which removes emSign ECC Root CA - G3 from /etc/ssl/certs/ca-certificates.crt. Ubuntu's bundle does not contain AAA Certificate Services, so the cross-signs are not trusted there; verified offline against the old and fixed ca-certificates packages for 24.04. Wiring this into the remaining Linux jobs is a follow-up (security/emudhra-plan2.md).
The action is a no-op on self-hosted runners (runner.environment != 'github-hosted'), whose trust stores are managed by the runner team separately.
The 16 certificates come from the disclosure's impacted-intermediates.json, committed unmodified (SHA-256 f67b3abe9b4879f4e91874ed041beccd71257f1db41c588c724ccf12aa90c59e, excluded from end-of-file-fixer to keep it byte-identical), so nothing is downloaded at run time. Every certificate's fingerprint and key hash was recomputed locally, and crt.sh lists exactly these two certificates for each of the eight keys. The set is closed: new certificates for these keys would have to be issued by another CA, and all existing ones are revoked.
Probe changes
The probe now tests the same thing the fix does, so a green probe means the cross-sign path is closed too:
Windows: gates on all 16 certificates being explicitly distrusted (shared helper with the action). Root presence in LocalMachine\Root is now informational, because Microsoft's fix leaves the root in place.
Linux: verifies the disclosure's 14 served chains (ci/tools/runner_trust_probe_chains/, from its chains.zip) offline against the CA bundle with openssl verify. Only an issuer-lookup error counts as rejected; any other error fails the check.
Both: the live endpoint check now covers one host per path (via emSign ECC Root CA - G3 and via the AAA Certificate Services cross-sign).
What the fix does and does not cover
Fixed:git (schannel backend, which GitHub-hosted Windows runners use by default), gh, Invoke-WebRequest, .NETHttpClient, curl with WinSSL.
Not fixed: the Node.js store compiled into the runner agent (JS actions: actions/checkout's own API calls, artifact actions, etc.). For GitHub-hosted runners this requires GitHub to roll out agent v2.338.2 or later to their hosted fleet; for self-hosted runners it is tracked separately.
Jobs wired
build-wheel.yml: win-64 (windows-2022) and win-arm64 (windows-11-arm) builds.
pre-commit.yml: Linux (ubuntu-latest) and Windows (windows-latest) rows.
test-sdist-windows.yml: windows-2022.
coverage.yml (build-wheel-windows): standalone inline job on windows-2022, not a call to the reusable build-wheel.yml.
Removal
Remove the Windows step from this action (or the entire action if the Linux wiring is also done and removed) once the probe's "Compromised intermediates distrusted" check passes on all three hosted Windows images across at least 14 consecutive daily runs. Link the probe run evidence in the removal PR. Tracked by #3257.
Checklist
New or existing tests cover these changes.
The documentation is up to date with these changes.
Three review fixes:
- Wire upgrade-ca-certificates into coverage.yml's build-wheel-windows job,
which is a standalone inline job (not a call to build-wheel.yml) running
on windows-2022 with actions/checkout.
- Linux path: fail closed when openssl is absent or the CA bundle cannot be
parsed (cert_count == 0), mirroring the probe script's guard.
- Windows path: use -ErrorAction Stop when reading LocalMachine\Root (an
unreadable store is not a clean state) and keep SilentlyContinue only for
Disallowed; replace the manual SHA256::Create()/BitConverter block with
GetCertHashString('SHA256'), which is available in PS7 (shell: pwsh).
Updated automated build and validation checks to address specified certificate trust issues on hosted Windows and Linux runners.
Expanded trust checks across certificate-chain variants and added verification of expected trust-store behavior.
Adjusted workflow linting and file-processing rules to accommodate the new certificate data.
Documentation
Added guidance for reviewing captured certificate chains and interpreting trust-check results.
The pull request adds a composite action that checks and updates certificate trust on hosted Windows and Linux runners. It adds impacted intermediate-certificate data and expands Windows and Linux runner probes. Four workflows invoke the action.
Changes: Hosted runner certificate trust
Layer / File(s)
Summary
Impacted certificate data and validation .github/actions/upgrade-ca-certificates/impacted-intermediates.json, .github/actions/upgrade-ca-certificates/emudhra_intermediates.ps1, .pre-commit-config.yaml
Adds 16 impacted intermediate-certificate records. PowerShell functions parse the records, check their SHA-256 hashes, and test for explicit distrust. The pre-commit exclusion includes the JSON file.
On hosted Windows, the action adds the impacted certificates to LocalMachine\Disallowed and verifies explicit distrust. On hosted Linux, it checks the CA bundle for the emSign root fingerprint and, when present, retries the ca-certificates upgrade up to three times before verifying the bundle.
Runner trust probes and chain fixtures ci/tools/runner_trust_probe.ps1, ci/tools/runner_trust_probe.sh, ci/tools/runner_trust_probe_chains/README.txt, .spdx-ignore
The probes test impacted-certificate distrust and both endpoint paths. The shell probe also checks offline chains at a fixed verification time. The README describes the captured chains and OpenSSL checks; .spdx-ignore excludes the chain data.
Workflow invocation and lint configuration .github/workflows/build-wheel.yml, .github/workflows/pre-commit.yml, .github/workflows/test-sdist-windows.yml, .github/workflows/coverage.yml, .github/actionlint.yaml
Four workflows run the local action. Actionlint ignores missing-reference diagnostics for these action references.
Possibly related PRs
NVIDIA/cuda-python#3197: Updates certificate-related packages and CI pins for the same eMudhra/emSign incident.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
Blocking only emSign ECC Root CA - G3 on Windows leaves the eight
compromised intermediates trusted through their Sectigo cross-signs, which
chain to AAA Certificate Services. The Windows step now adds all 16
certificates from the disclosure's impacted-intermediates.json (vendored
unmodified) to LocalMachine\Disallowed and verifies that chain building
reports each one as explicitly distrusted.
The probe now gates on the same explicit-distrust check on Windows (the
root stays in LocalMachine\Root under Microsoft's own fix, so root presence
is informational), verifies the disclosure's 14 served chains offline
against the CA bundle on Linux, and tests one live host per path.
suggestion: Check the complete offline fixture set.
If one documented PEM file is omitted, the non-empty glob still passes the empty-directory check. The loop then verifies only the remaining files and can report PASS. Compare the discovered filenames with the documented fixture set before reporting success.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
My original P1 is resolved. The revised action imports all 16 intermediates and cross-signs from vendored data, then verifies explicit distrust. The offline approach also avoids runtime downloads.
Validation confirms:
The distrust check passes on windows-latest, windows-2022, and Windows ARM.
Linux probe checks correctly distinguish vulnerable, fixed, and malformed inputs.
Both pre-commit jobs pass; broader CI is still running without observed failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
rwgk
left a comment
Description
The runner trust probe (#3268, run) shows that the GitHub-hosted Windows images (
windows-2022,windows-latest,windows-11-arm) still trust the compromised eMudhra intermediates (CVE-2026-111943), so schannel-based clients (git,gh, PowerShell/.NET) accept certificates issued with the compromised keys. Microsoft blocks these certificates through Windows' automatic trust-list updates, but the hosted images do not have that update.This change adds a composite action (
.github/actions/upgrade-ca-certificates) that applies the OS-appropriate fix as the first step of each affected job, beforeactions/checkoutsends any token over HTTPS:Cert:\LocalMachine\Disallowed, then fails the job unless Windows chain building reports every one of them as explicitly distrusted. This matches the disclosure's own Windows workaround. Distrusting onlyemSign ECC Root CA - G3is not enough, because the cross-signs chain toAAA Certificate Services, which Windows trusts (thanks @rwgk).ca-certificatesviaapt-get, which removesemSign ECC Root CA - G3from/etc/ssl/certs/ca-certificates.crt. Ubuntu's bundle does not containAAA Certificate Services, so the cross-signs are not trusted there; verified offline against the old and fixedca-certificatespackages for 24.04. Wiring this into the remaining Linux jobs is a follow-up (security/emudhra-plan2.md).The action is a no-op on self-hosted runners (
runner.environment != 'github-hosted'), whose trust stores are managed by the runner team separately.The 16 certificates come from the disclosure's
impacted-intermediates.json, committed unmodified (SHA-256f67b3abe9b4879f4e91874ed041beccd71257f1db41c588c724ccf12aa90c59e, excluded fromend-of-file-fixerto keep it byte-identical), so nothing is downloaded at run time. Every certificate's fingerprint and key hash was recomputed locally, and crt.sh lists exactly these two certificates for each of the eight keys. The set is closed: new certificates for these keys would have to be issued by another CA, and all existing ones are revoked.Probe changes
The probe now tests the same thing the fix does, so a green probe means the cross-sign path is closed too:
LocalMachine\Rootis now informational, because Microsoft's fix leaves the root in place.ci/tools/runner_trust_probe_chains/, from itschains.zip) offline against the CA bundle withopenssl verify. Only an issuer-lookup error counts as rejected; any other error fails the check.emSign ECC Root CA - G3and via theAAA Certificate Servicescross-sign).What the fix does and does not cover
git(schannel backend, which GitHub-hosted Windows runners use by default),gh,Invoke-WebRequest,.NETHttpClient,curlwith WinSSL.actions/checkout's own API calls, artifact actions, etc.). For GitHub-hosted runners this requires GitHub to roll out agent v2.338.2 or later to their hosted fleet; for self-hosted runners it is tracked separately.Jobs wired
build-wheel.yml:win-64(windows-2022) andwin-arm64(windows-11-arm) builds.pre-commit.yml: Linux (ubuntu-latest) and Windows (windows-latest) rows.test-sdist-windows.yml:windows-2022.coverage.yml(build-wheel-windows): standalone inline job onwindows-2022, not a call to the reusablebuild-wheel.yml.Removal
Remove the Windows step from this action (or the entire action if the Linux wiring is also done and removed) once the probe's "Compromised intermediates distrusted" check passes on all three hosted Windows images across at least 14 consecutive daily runs. Link the probe run evidence in the removal PR. Tracked by #3257.
Checklist