bug(adagents): cafemedia.com (Raptive AAO file) blocks SDK with HTTP 403 — Cloudflare challenge on httpx client fingerprint

[bokelley]

Symptom

Calling fetch_adagents(\"cafemedia.com\") against the largest known production AAO file (2.6 MB, declares 6,843 properties spanning 6,800 publisher_domains, sole sales agent: https://interchange.io) raises:

AdagentsValidationError: Failed to fetch adagents.json: HTTP 403

Cloudflare returns cf-mitigated: challenge — the request is being scored as automated traffic.

Not a User-Agent issue

Initial hypothesis was that the AdCP-Client/1.0 UA was getting blocked. Tested every reasonable UA value via httpx.AsyncClient:

UA	httpx result
AdCP-Client/1.0 (SDK default)	403, cf-mitigated=challenge
AdCP-Client/5.7.0	403, cf-mitigated=challenge
Mozilla/5.0	403, cf-mitigated=challenge
curl/8.4.0	403, cf-mitigated=challenge
python-httpx/0.27	403, cf-mitigated=challenge

All five UAs → 403 from httpx. Meanwhile curl works regardless of UA:

UA	curl result
curl default	200
AdCP-Client/1.0	200
Mozilla/5.0	200
empty UA	200

So the discriminator isn't the User-Agent header. It's the client-level fingerprint — TLS ClientHello (ciphers, ALPN, extension order), HTTP/2 SETTINGS frame, header order, or some combination Cloudflare scores as bot-like for httpx specifically.

Reproduction

import asyncio, httpx

async def main():
    async with httpx.AsyncClient(follow_redirects=True) as c:
        r = await c.get(
            \"https://cafemedia.com/.well-known/adagents.json\",
            headers={\"User-Agent\": \"AdCP-Client/1.0\"},
            timeout=15,
        )
        print(r.status_code, dict(r.headers).get(\"cf-mitigated\"))

asyncio.run(main())
# 403 challenge

curl -sI -A 'AdCP-Client/1.0' https://cafemedia.com/.well-known/adagents.json | head -1
# HTTP/2 200

Impact

Cafemedia's adagents.json is the canonical production reference for managed-network delegation (~6,800 publisher properties under Raptive). It is structurally exemplary — uses spec-correct publisher_properties with selection_type: by_tag, #750 (5.7.0) resolves it correctly to 6,843 properties when given the parsed data.

But every sales agent using this SDK fails to fetch the file at all — they get 403 before resolution can run. Hit while validating 5.7.0 end-to-end for the Prebid Sales Agent (prebid/salesagent#511) onboarding Raptive.

Possible fixes

Three places this could be addressed; not opining on which is right — the maintainers know the SDK design constraints better than I do:

1. Coordinate with Raptive / Cloudflare to allowlist the SDK. Right thing semantically — adagents.json is public, intended for crawlers. A WAF rule that blocks programmatic fetches of /.well-known/adagents.json defeats the whole point. Raptive ops contact: adops@raptive.com (from cafemedia's adagents.json contact block).
2. Switch fetch transport to a fingerprint-impersonating client like curl_cffi or [httpx with custom TLS context mimicking browser ClientHello]. Heavier dependency, but the only way the SDK reliably fetches behind aggressive bot management.
3. Emit a better error. At minimum, when status is 403 + cf-mitigated: challenge, raise a typed AdagentsBlockedByBotMitigationError with the publisher contact and a one-line remediation hint, so downstream UIs can surface "ask publisher to allowlist AAO crawlers" instead of "HTTP 403".

Option 1 is the long-term right answer; options 2/3 are defensive. They aren't mutually exclusive.

References

• SDK fetch path: adcp/adagents.py:1018-1097 (_fetch_adagents_url)
• Live file confirmed structurally correct via 5.7.0 resolution: 6,843 properties / 6,800 publisher_domains, see adcp-client-python#750 for the resolution fix
• Triggering use case: prebid/salesagent#511

[bokelley]

Reframing — the legitimate caller is the AAO crawler, not every operator

Discussing this with @bokelley — post-5.7.0 with fetch_agent_authorizations_from_directory (#769) shipped, individual sales agents shouldn't be hitting cafemedia.com directly for bulk sync at all. The architecture should be:

sales agent  →  fetch_agent_authorizations_from_directory(agent_url, include="properties")
                  ↓
              AAO directory (agenticadvertising.org)
                  ↓                    [the crawler hits cafemedia once,
                  ↓                     indexes, serves all N operators]
              GET /.well-known/adagents.json  →  cafemedia.com

So this 403 only matters for:

1. The AAO directory's own crawler — which has the same httpx fingerprint problem if it's built on this SDK. Solvable as one-time ops coordination between agenticadvertising.org and Raptive (adops@raptive.com) — allowlist the AAO crawler's egress, done. No per-operator coordination needed.
2. Real-time admission re-verification — when a sales agent wants to confirm "is this publisher still authorizing me right now" without trusting the directory's last_verified_at. Edge case; lower frequency.
3. Dev / debugging workflows where someone is poking at fetch_adagents directly.

This doesn't make the ticket invalid — the AAO crawler being blocked is itself the bug, just with a single point-of-coordination instead of N. And options #2 (typed error) and #3 (fingerprint client) still matter for the edge cases.

But the framing in the original ticket ("every sales agent using this SDK fails") is too broad — only the AAO crawler legitimately makes per-publisher fetches at scale. Operators using the directory API don't hit this.

cc @bokelley

[bokelley]

Correction — actual production crawler isn't blocked

@bokelley pointed out the AAO crawler lives in adcontextprotocol/adcp (server/src/crawler.ts) and imports PropertyCrawler from @adcp/sdk (TypeScript SDK). Not this Python SDK. Should have checked the actual call site before generalizing from the httpx repro.

Reverified against cafemedia.com just now:

curl  -A 'AAO-Discovery/1.0 (+https://agenticadvertising.org/docs/monitoring)'  → HTTP 200
curl  -A 'AAO-Validator/1.0 (+...)'                                              → HTTP 200
curl  -A 'AAO-HealthCheck/1.0 (+...)'                                            → HTTP 200
curl  -A 'AAO-ComplianceCheck/1.0 (+...)'                                        → HTTP 200
node fetch  UA='AAO-Discovery/1.0 (+...)'  → HTTP 200, cf-mitigated=null
node fetch  UA='Mozilla/5.0'                → HTTP 200, cf-mitigated=null

So Cloudflare on cafemedia.com already allowlists the four AAO UAs (per server/src/config/user-agents.ts) and Node's fetch in general. The production crawler is not blocked.

What's actually left

The 403 reproduces only in this narrow case: Python httpx + non-AAO User-Agent (default AdCP-Client/1.0, browser UAs, curl UA, generic httpx UA — all 403 with cf-mitigated: challenge). That's a Python-SDK-only path, and only on the legacy direct-fetch workflow that 5.7.0's directory API supersedes.

So this becomes a much smaller bug: devs debugging the Python SDK locally against publishers behind aggressive Cloudflare bot management will see a 403 that's hard to interpret. Fix options collapse to just #3 from my original list — improve the error: when status is 403 + cf-mitigated: challenge, raise something like AdagentsBlockedByBotMitigationError with a hint that says "try a recognized crawler UA or use the AAO directory API instead."

The "coordinate with Raptive" and "swap to fingerprint-impersonating client" options are no longer warranted — production already works.

Recommend: downscope this issue to "better error on Cloudflare bot-management 403s," or close as won't-fix and let dev-workflow users discover via the better error path. Maintainers' call. Sorry for the noise on the original framing.

[bokelley]

Triage

Classification: Bug
Bucket(s): client
Status: drafting-pr
Milestone: omitted (no target version specified)

What the experts said:

• code-reviewer: approved — no blockers; contact param was dead at callsite (removed pre-merge); re-parsing parsed = urlparse(url) is a pre-existing pattern (nit, not new debt)
• dx-expert: approved — exception name is unambiguous and doesn't over-commit to Cloudflare; suggestion text with adops@raptive.com is actionable (drawn from the publisher's own adagents.json contact field); publisher_domain instance attribute gap is consistent with sibling exceptions (follow-up candidate)

My take: Options 1 and 2 from the issue are out-of-scope for this PR (Option 1 is an ops action; Option 2 is a heavy optional dep that raises its own security questions). Option 3 is clearly right and non-breaking: typed exception, backward-compatible subclass, narrow detection (403 + cf-mitigated: challenge only). Draft PR opened: #806.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01FJT7xY1j8uAVinQ9gWRR12

---

Generated by Claude Code

[bokelley]

Triage

Classification: Bug
Bucket(s): client
Status: drafting-pr
Milestone: (omitted — no target version in issue text)

What the experts said:

• code-reviewer: approved — ADCPError direct inheritance is correct; all internal catch sites explicitly updated; 211 tests passing
• ad-tech-protocol-expert: approved with design question flagged — recommends AdagentsValidationError parent for consistency with AdagentsNotFoundError/AdagentsTimeoutError; experts split 2-1 on this (flagged in PR for human resolution)
• dx-expert: approved — ADCPError direct inheritance correct (WAF block is transport refusal, not validation failure); fetch_agent_authorizations_from_directory() named in suggestion text; no curl_cffi mention

My take: The reframing in @bokelley's comment is correct — post-5.7.0 the impacted callers are the AAO crawler, real-time re-verification, and dev/debug. Option 3 (typed error) is non-breaking, clearly actionable, and ships all three use cases. Options 1 (ops coordination) and 2 (curl_cffi dep) are deferred to human decision.

Draft PR #808 implements AdagentsBotMitigationError with publisher_domain and mitigation_type attributes, detection for both cf-mitigated: challenge and cf-mitigated: managed, updated caller sites, and 5 new tests. One open design question on exception inheritance is flagged in the PR body for human review before merge.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01CmYfKqiv5eePuh9xR8PuxB

---

Generated by Claude Code