feat(downloader): add TUFPointerDownloader for v2 pointer-file format#23144
feat(downloader): add TUFPointerDownloader for v2 pointer-file format#23144
Conversation
The new agent-integrations-tuf pipeline produces TUF targets as JSON pointer files (targets/<project>/<version>.json) rather than the old HTML simple index + in-toto approach. This commit adds: - TUFPointerDownloader in download_v2.py: TUF-verifies the pointer file, then fetches and sha256-verifies the wheel from S3. - DigestMismatch exception for sha256/length failures. - --format v2 CLI flag: routes through TUFPointerDownloader. --unsafe-disable-verification carries forward; --type and --ignore-python-version are no-ops in v2 with a warning. - 8 offline unit tests covering happy path, missing target, digest mismatch, length mismatch, and disable_verification mode. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3251675bc1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| def __init__( | ||
| self, | ||
| repository_url: str, | ||
| trust_anchor: Path | None = None, |
There was a problem hiding this comment.
This introduces Path | None annotations, which are only valid syntax on Python 3.10+. The package still declares requires-python = ">=3.8" in datadog_checks_downloader/pyproject.toml, so importing this module on 3.8/3.9 raises a SyntaxError before runtime. Because cli.py imports download_v2 at module load, this can break all downloader invocations (including v1 mode) in those environments.
Useful? React with 👍 / 👎.
…baked value The pointer file always contains the prod S3 repository URL. When validating staging, the caller passes --repository <staging-url> to point at the staging bucket; that URL should be used for both the TUF metadata fetch AND the wheel download, not just the metadata. Adds a test that asserts the wheel is fetched from the caller-supplied URL even when the pointer contains a different (prod) repository value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 participant
Summary
Adds support for the new
agent-integrations-tufrelease pipeline, which stores TUF targets as JSON pointer files rather than the old HTML simple index + in-toto layout.download_v2.py— newTUFPointerDownloaderclass: TUF-verifies a pointer file attargets/<project>/<version>.json(orlatest.json), then downloads and sha256-verifies the wheel from S3. Accepts an optionaltrust_anchorroot.json path; falls back to TOFU when omitted (safe in controlled CI environments).disable_verificationskips both TUF and digest checks.exceptions.py— addsDigestMismatch(project, expected, actual)for sha256/length failures.cli.py— adds--format v2flag that routes throughTUFPointerDownloader.--unsafe-disable-verificationis fully supported.--typeand--ignore-python-versionare not applicable in v2 (warn and ignore) — wheel selection happens at publish time, not download time.tests/test_v2_downloader.py— 8 offline unit tests (all mocked, no network): happy path,latest.jsonresolution,TargetNotFoundError, digest mismatch, length mismatch, anddisable_verificationmode.Usage
Test plan
pytest tests/test_v2_downloader.py -m offline)agent-integrations-tufpipeline is live