feat(canonical-formats): v1↔v2 reverse + pixel_tracker + narrowing (#741, half 2 of 2)

[bokelley]

Summary

Closes #741. Second half of the canonical-formats projection layer — lands the v1 → v2 inbound projection, the bidirectional pixel_tracker contract, the FORMAT_DECLARATION_DIVERGENT narrowing check, and 65 upstream reference fixtures + round-trip tests.

v1 → v2 inbound projection (adcp/canonical_formats/v1_to_v2.py)

project_v1_format_to_declaration(v1_format) walks the 4-step resolution order from registries/v1-canonical-mapping.json:

1. Seller-asserted canonical: annotation on the v1 file → use declared kind; thread asset_source and slots_override[] into the emitted declaration's params.
2. Registry format_id_glob lookup — uses format_id.id (3.1 ships zero literal entries; future literals land here).
3. Registry structural match — asset_types + VAST/DAAST versions + dimensions. Family-level match emits FORMAT_DECLARATION_V1_AMBIGUOUS but still produces a usable declaration.
4. Fail closed → emit FORMAT_PROJECTION_FAILED, no declaration.

Emitted declarations always carry v1_format_ref pointing back at the source v1 format_id so the half-1 v2 → v1 path round-trips cleanly.

Bidirectional pixel_tracker (pixel_tracker.py)

Implements the normative downgrade/upgrade table from core/assets/pixel-tracker-asset.json:

• downgrade_pixel_tracker (v2 → v1) collapses 7 event × 2 method combos onto the v1 {asset_type: url, url_type: tracker_pixel} shape keyed on impression_tracker / viewability_tracker / click_tracker slots. Lossy on viewability variants, custom events, and any JS method — those emit PIXEL_TRACKER_LOSSY_DOWNGRADE.
• upgrade_v1_tracker (v1 → v2) inverts via asset_id convention. ALWAYS emits PIXEL_TRACKER_UPGRADE_INFERRED because the v1 wire shape carries no explicit event/method.

Batch helpers (downgrade_pixel_trackers / upgrade_v1_trackers) deduplicate advisories so a manifest with many same-kind pixels surfaces one advisory per kind.

FORMAT_DECLARATION_DIVERGENT narrowing check (narrowing.py)

check_narrows(v2_params, v1_requirements) returns divergence records covering four kinds:

• exceeds_max — v2 numeric value exceeds v1 max
• below_min — v2 numeric value below v1 min
• not_subset — v2 enum-typed set has values outside v1 allowed set
• not_equal — exact-scalar disagreement

narrowing_advisory(declaration, *, v1_requirements, v1_format_id) wraps the divergence list in a wire-correct FORMAT_DECLARATION_DIVERGENT Error with details.divergences enumerating each failing field.

v2 silently omitting a v1-declared field is NOT a divergence (narrows into unconstrained space).

Vendored fixtures

Vendored from adcontextprotocol/adcp@main upstream:

• 14 v2 Product fixtures at tests/fixtures/canonical/<name>.json (from static/examples/products/canonical/). Real-world seller catalogs exercising the v2 → v1 path.
• 50-entry v1 reference catalog at tests/fixtures/canonical/v1-reference-formats.json (from server/src/creative-agent/reference-formats.json). Every entry carries an explicit canonical: annotation, exercising the v1 → v2 step-1 path.

Round-trip tests (tests/test_canonical_formats_roundtrip.py)

• For every v2 product fixture, project_product_to_v1 MUST emit format_ids that round-trip back to the source declaration via find_declaration_by_v1_format_id.
• For every v2 product fixture, format_options[] MUST be parseable as the hand-rolled ProductFormatDeclaration.
• v1 reference catalog MUST project with zero advisories.
• v1 catalog covers 8 distinct CanonicalFormatKind values (pinned so an upstream catalog drift trips CI).

20 new public-API names on adcp.canonical_formats

V1ToV2Projection, V1CatalogProjection, V1Tracker, PixelTrackerDowngrade, PixelTrackerUpgrade, PixelTrackerBatchResult, plus the project_v1_format_to_declaration, project_v1_catalog_to_v2, downgrade_pixel_tracker, downgrade_pixel_trackers, upgrade_v1_tracker, upgrade_v1_trackers, check_narrows, narrowing_advisory functions.

Test plan

• ruff check src/ — all checks passed
• mypy src/adcp/ — 901 files, no issues
• pytest tests/ — 5190 passed (5117 from half 1 + 73 new across 4 test files)
• 14 v2 product fixtures + 50-entry v1 catalog round-trip cleanly
• All 4 new error codes (FORMAT_DECLARATION_V1_AMBIGUOUS, FORMAT_PROJECTION_FAILED, PIXEL_TRACKER_LOSSY_DOWNGRADE, PIXEL_TRACKER_UPGRADE_INFERRED, FORMAT_DECLARATION_DIVERGENT) verified against core/error.json enum
• Public-API snapshot regenerated

Closes #741.

[aao-ipr-bot]

Request changes. ad-tech-protocol-expert returned unsound with cited wire-shape divergence on FORMAT_DECLARATION_V1_AMBIGUOUS direction-of-use and a SHOULD-NOT batch-dedup violation; code-reviewer separately flagged a Major in _v1_version_constraints that breaks the documented Pydantic-input contract. The downgrade table, upgrade defaults, narrowing semantics, and v1_format_ref shape are all sound — the unsoundness is concentrated in advisory emission and one duck-typing gap.

MUST FIX

1. FORMAT_DECLARATION_V1_AMBIGUOUS is being emitted on the wrong projection direction. src/adcp/canonical_formats/v1_to_v2.py:307-330 raises this code when the v1→v2 inbound path hits a family-level structural match. Per schemas/cache/3.1.0-beta.3/enums/error-code.json, the code describes the v2→v1 reverse direction — "an SDK detects that a product's v2 declaration cannot be unambiguously projected back to a single v1 named format." On the inbound side a family-level structural match is the expected 3.1 outcome (the registry ships 7 pure-structural entries, zero literals — see the Initial scope (3.1) note in registries/v1-canonical-mapping.json). What breaks: every v1 format in a 3.1 catalog without a canonical: annotation produces a spurious AMBIGUOUS advisory advising the seller to add an annotation they don't need on the inbound direction, and buyer-side diagnostic tooling that keys on this code being a v2→v1 problem misclassifies. Either drop the advisory on the structural-match branch or surface a v1→v2-direction code. The half-1 reverse path already uses AMBIGUOUS correctly — don't overload it here.

2. _v1_version_constraints silently returns [] for Pydantic Format inputs. src/adcp/canonical_formats/v1_to_v2.py:160-184 gates on if isinstance(v1_format, dict) else None and returns immediately. The module docstring and project_v1_format_to_declaration docstring both promise duck-typing across "raw dict, a Pydantic Format, or a duck-typed object." Sibling helpers (_v1_format_id:75, _v1_canonical_annotation:99, _v1_asset_types:123) all handle both shapes — this one diverges. What breaks: a Pydantic-typed VAST 4.x format with no canonical: annotation falls through to step 5 fail-closed instead of matching the video_vast structural entry. Tests don't catch it because every tests/test_canonical_formats_v1_to_v2.py input is a raw dict and the vendored 50-entry catalog all exits at step 1. Mirror the dict-or-getattr pattern used by the sibling helpers.

3. Batch downgrade_pixel_trackers dedup violates the spec's SHOULD-NOT. src/adcp/canonical_formats/pixel_tracker.py:336-350 deduplicates on (code, source_event, source_method). The PIXEL_TRACKER_LOSSY_DOWNGRADE metadata in core/assets/pixel-tracker-asset.json is explicit: "One advisory per downgraded asset; SDKs SHOULD NOT collapse multiple downgrades into a single advisory entry — per-asset details let the buyer's measurement-plan owner decide whether each loss is tolerable." The collapsed shape defeats that contract. Drop the dedup on the downgrade side. The upgrade-side dedup at L367-379 isn't spec-prohibited but should probably match for consistency.

Follow-ups (non-blocking — file as issues)

• Advisory details key drift from the spec's SHOULD shapes. Spec keys (format_id, original_event, original_method, original_custom_event_name, lost_fields, downgrade_target: \"url+tracker_pixel\", inferred_event, inferred_method, asset_id, product_id) vs the SDK's invented keys (v1_format_id, source_event, source_method, source_custom_event_name, lost, missing downgrade_target, source_asset_id, no product_id). SHOULD-level so wire-conformant either way, but buyer-side classifiers key on these names — mechanical rename closes the gap.
• inference_basis=\"fallback_custom_event\" at pixel_tracker.py:271 isn't in the spec enum {\"asset_id_convention\", \"default\"}. Restrict to one of the two enumerated literals.
• Structural-match call never threads width/height. v1_to_v2.py:289-294 doesn't extract dimensions from the v1 format. Latent today (registry has zero dimensions-constrained entries) — silent miss the moment a future registry entry adds one.
• load_default_registry() called once per format in project_v1_catalog_to_v2. Each call deep-copies the cached registry. Hoist the load to the catalog wrapper.
• _v1_to_v2 module docstring numbers its steps as 1-5 mirroring registries/v1-canonical-mapping.json but the registry's normative order has 6 steps with step 1 being the cross-product v1_format_ref link. Either declare step 1 out of scope explicitly or renumber.

Minor nits (non-blocking)

1. narrowing.py:148-149 — fallback short-circuits when v2 declares both forms. If v2 carries max_width=350 AND width=400 against v1's max_width=300, only the 350 is checked because the if v2_value is None fallback gates the bare-form lookup. 400>300 goes undetected. Check both forms or document the contract.
2. narrowing.py:103-109 — _is_subset does set(v1) / set(v2) which raises TypeError if a list-of-dicts slips through. v1_requirements is dict[str, Any] so the shape isn't guaranteed. Catch and treat as not_subset.
3. pixel_tracker.py:184 — hasattr(pixel, \"custom_event_name\") is always True for a generated Pydantic model with the field declared. Drop the guard.
4. pixel_tracker.py:213 — event.value if event else 'impression'!r emits event='impression' when _coerce_event returned None on invalid input. Misleading; echo the raw or label as <invalid>.

Verdict ladder: changes requested on items 1-3 above. Items 1 and 3 are direct spec divergences with a concrete adopter-visible failure mode; item 2 breaks a documented duck-typing contract. The architecture is right — three modules, single resolution-order walker, advisories typed via make_sdk_advisory — the bugs are localized.

[aao-ipr-bot]

Approving on the strength of the fix-commit's response to prior review plus the clean expert verdicts. The right architectural call: half-2 lands the inbound projection alongside the half-1 outbound path, the bidirectional pixel_tracker contract closes the asset round-trip, and the resolution order is documented at module level so future contributors don't have to reverse-engineer it.

Things I checked

• Public-API surface — 14 new exports on adcp.canonical_formats (verified against src/adcp/canonical_formats/__init__.py L87-117). Additive only, feat: prefix correct, no semver signal needed.
• _echo_identifier coverage on every seller-controlled echo path: pixel_tracker.py L219, L305, L311, L358, L364, L369; narrowing.py L293, L304; v1_to_v2.py L350, L376. _echo_set cap=32 on the only unbounded list echo (narrowing.py L243-244). security-reviewer: clean with one Low defense-in-depth note (see Follow-up 4).
• URL scheme allowlist on upgrade_v1_tracker rejects javascript: / file: / data: / ftp: / vbscript: / "". Batch helper drops rejected entries from items (pixel_tracker.py L465-466) — not just the advisory.
• Advisory codes (FORMAT_DECLARATION_V1_AMBIGUOUS, FORMAT_PROJECTION_FAILED, PIXEL_TRACKER_LOSSY_DOWNGRADE, PIXEL_TRACKER_UPGRADE_INFERRED, FORMAT_DECLARATION_DIVERGENT) all upstream-registered in schemas/cache/3.1.0-beta.3/enums/error-code.json L79-85. source=sdk + sdk_id per the multi-hop dedup contract.
• pixel_tracker.py downgrade/upgrade tables match core/assets/pixel-tracker-asset.json byte-for-byte. viewable_mrc_50 default on the viewability upgrade is spec-prescribed, not a coin flip.
• _is_numeric bool exclusion (narrowing.py L125) — bool is the only int subclass that would corrupt comparisons; no other path uses raw isinstance(_, (int, float)) on seller values.
• ad-tech-protocol-expert: sound-with-caveats (caveats in Follow-ups). code-reviewer: no blockers.

Follow-ups (non-blocking — file as issues)

1. Narrowing field-list coverage gap (narrowing.py L52-92). The fixup commit added max_file_size_mb, max_bitrate_kbps, max_wrapper_depth, etc. — but missed the carousel/responsive/native parameter surface: max_cards, min_cards, max_items, min_items, max_chars, max_size_kb, max_image_file_size_kb, and the flat headline_max_chars / description_max_chars / card_*_max_chars family on responsive_creative.json. Same class of bug the fixup was supposed to close — real v1↔v2 carousel/native pairings with divergent per-card caps will silently pass. Conversely _MAX_FIELDS/_MIN_FIELDS ship max_dpi/min_dpi and _ENUM_SUBSET_FIELDS ships plural vast_versions/daast_versions which don't appear anywhere in the 3.1.0-beta.3 canonical schemas — harmless (v1.get(field) returns None and the check skips) but odd. Worth filing.

2. Step-1 annotation/registry-params precedence inversion (v1_to_v2.py L212-218, L286-295). Docstring says "seller annotation wins on kind/slots_override/asset_source but registry defaults fill in everything else." The merge logic does the opposite for asset_source and slots — _build_declaration only threads the annotation's values when the key is NOT already in body (which was pre-populated from registry_params). Latent today (3.1 ships zero literal globs) but will fire silently the moment a literal glob lands.

3. MIGRATION_v5_to_v6.md L223 still says V1Tracker. The fixup commit renamed it to V1UrlTracker (rationale: dodge cognitive collision with V1Pattern). Copy-paste of the migration doc's import block will ImportError. One-line fix.

4. Defense-in-depth: cap fid.id echo in advisory messages (v1_to_v2.py L342, L370). FormatId.id is regex-constrained to ^[a-zA-Z0-9_-]+$ so _echo_identifier's control-char scrub is structurally unnecessary, but there's no max_length on FormatId.id and no max_length on Error.message — a seller publishing a 100KB alphanumeric id balloons every emitted advisory. _echo_identifier(fid.id) in the message f-string closes it. Same for the asset_types echo at L378 (per-element from raw dict input, no bound).

5. _v1_canonical_annotation silently swallows ValidationError (v1_to_v2.py L124-130). A seller typo (kind: \"imag\" instead of \"image\") degrades to a structural-ambiguous advisory instead of surfacing as a FORMAT_PROJECTION_FAILED with the validation error in details. Sellers won't know their annotation didn't take effect.

6. Round-trip test parametrizes over all 14 fixtures but is vacuously true on fixtures without v1_format_ref. tests/test_canonical_formats_roundtrip.py L98-145 — expected_refs becomes [] and matches result.format_ids trivially for v1-untranslatable canonicals. Either split the parametrize or assert at least N fixtures contributed non-empty refs.

7. tests/fixtures/canonical/VENDOR.md doesn't pin upstream SHAs. For a 14-fixture + 50-entry conformance corpus, omitting the source commit SHA means upstream drift is invisible until a test happens to fail.

Minor nits (non-blocking)

1. pixel_tracker.py L418 dedup-key fallback is unreachable. source_method is always set on L212 — details.get(\"source_method\", \"img\") never hits the default. Harmless.

2. v1_to_v2.py docstring numbers steps 1-4 but L364 says "Step 5: fail closed". The fixup commit re-aligned to the registry's numbering — the inline comment didn't move with it.

The third drift-cleanup commit in a row in this area still didn't catch the V1Tracker → V1UrlTracker rename in the migration doc. Worth a grep pass on rename PRs going forward.

Safe to merge.