Title: bug(adagents): cafemedia.com (Raptive AAO file) blocks SDK with HTTP 403 — Cloudflare challenge on httpx client fingerprint · Issue #801 · adcontextprotocol/adcp-client-python · GitHub
Open Graph Title: bug(adagents): cafemedia.com (Raptive AAO file) blocks SDK with HTTP 403 — Cloudflare challenge on httpx client fingerprint · Issue #801 · adcontextprotocol/adcp-client-python
X Title: bug(adagents): cafemedia.com (Raptive AAO file) blocks SDK with HTTP 403 — Cloudflare challenge on httpx client fingerprint · Issue #801 · adcontextprotocol/adcp-client-python
Description: Symptom Calling fetch_adagents(\"cafemedia.com\") against the largest known production AAO file (2.6 MB, declares 6,843 properties spanning 6,800 publisher_domains, sole sales agent: https://interchange.io) raises: AdagentsValidationErro...
Open Graph Description: Symptom Calling fetch_adagents(\"cafemedia.com\") against the largest known production AAO file (2.6 MB, declares 6,843 properties spanning 6,800 publisher_domains, sole sales agent: https://interc...
X Description: Symptom Calling fetch_adagents(\"cafemedia.com\") against the largest known production AAO file (2.6 MB, declares 6,843 properties spanning 6,800 publisher_domains, sole sales agent: http...
Opengraph URL: https://github.com/adcontextprotocol/adcp-client-python/issues/801
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"bug(adagents): cafemedia.com (Raptive AAO file) blocks SDK with HTTP 403 — Cloudflare challenge on httpx client fingerprint","articleBody":"## Symptom\n\nCalling `fetch_adagents(\\\"cafemedia.com\\\")` against the largest known production AAO file (2.6 MB, declares 6,843 properties spanning 6,800 publisher_domains, sole sales agent: `https://interchange.io`) raises:\n\n```\nAdagentsValidationError: Failed to fetch adagents.json: HTTP 403\n```\n\nCloudflare returns `cf-mitigated: challenge` — the request is being scored as automated traffic.\n\n## Not a User-Agent issue\n\nInitial hypothesis was that the `AdCP-Client/1.0` UA was getting blocked. Tested every reasonable UA value via `httpx.AsyncClient`:\n\n| UA | httpx result |\n|----|---|\n| `AdCP-Client/1.0` (SDK default) | 403, cf-mitigated=challenge |\n| `AdCP-Client/5.7.0` | 403, cf-mitigated=challenge |\n| `Mozilla/5.0` | 403, cf-mitigated=challenge |\n| `curl/8.4.0` | 403, cf-mitigated=challenge |\n| `python-httpx/0.27` | 403, cf-mitigated=challenge |\n\nAll five UAs → 403 from httpx. Meanwhile **curl works regardless of UA**:\n\n| UA | curl result |\n|----|---|\n| curl default | 200 |\n| `AdCP-Client/1.0` | 200 |\n| `Mozilla/5.0` | 200 |\n| empty UA | 200 |\n\nSo the discriminator isn't the User-Agent header. It's the client-level fingerprint — TLS ClientHello (ciphers, ALPN, extension order), HTTP/2 SETTINGS frame, header order, or some combination Cloudflare scores as bot-like for httpx specifically.\n\n## Reproduction\n\n```python\nimport asyncio, httpx\n\nasync def main():\n async with httpx.AsyncClient(follow_redirects=True) as c:\n r = await c.get(\n \\\"https://cafemedia.com/.well-known/adagents.json\\\",\n headers={\\\"User-Agent\\\": \\\"AdCP-Client/1.0\\\"},\n timeout=15,\n )\n print(r.status_code, dict(r.headers).get(\\\"cf-mitigated\\\"))\n\nasyncio.run(main())\n# 403 challenge\n```\n\n```bash\ncurl -sI -A 'AdCP-Client/1.0' https://cafemedia.com/.well-known/adagents.json | head -1\n# HTTP/2 200\n```\n\n## Impact\n\nCafemedia's adagents.json is the canonical production reference for managed-network delegation (~6,800 publisher properties under Raptive). It is structurally exemplary — uses spec-correct `publisher_properties` with `selection_type: by_tag`, [#750](https://github.com/adcontextprotocol/adcp-client-python/pull/750) (5.7.0) resolves it correctly to 6,843 properties when given the parsed data.\n\nBut **every sales agent using this SDK fails to fetch the file at all** — they get 403 before resolution can run. Hit while validating 5.7.0 end-to-end for the Prebid Sales Agent ([prebid/salesagent#511](https://github.com/bokelley/salesagent/issues/511)) onboarding Raptive.\n\n## Possible fixes\n\nThree places this could be addressed; not opining on which is right — the maintainers know the SDK design constraints better than I do:\n\n1. **Coordinate with Raptive / Cloudflare to allowlist the SDK.** Right thing semantically — adagents.json is *public, intended for crawlers*. A WAF rule that blocks programmatic fetches of `/.well-known/adagents.json` defeats the whole point. Raptive ops contact: `adops@raptive.com` (from cafemedia's adagents.json `contact` block).\n2. **Switch fetch transport to a fingerprint-impersonating client** like [`curl_cffi`](https://github.com/yifeikong/curl_cffi) or [`httpx` with custom TLS context mimicking browser ClientHello]. Heavier dependency, but the only way the SDK reliably fetches behind aggressive bot management.\n3. **Emit a better error.** At minimum, when status is 403 + `cf-mitigated: challenge`, raise a typed `AdagentsBlockedByBotMitigationError` with the publisher contact and a one-line remediation hint, so downstream UIs can surface \\\"ask publisher to allowlist AAO crawlers\\\" instead of \\\"HTTP 403\\\".\n\nOption 1 is the long-term right answer; options 2/3 are defensive. They aren't mutually exclusive.\n\n## References\n\n- SDK fetch path: [`adcp/adagents.py:1018-1097`](https://github.com/adcontextprotocol/adcp-client-python/blob/main/adcp/adagents.py#L1018) (`_fetch_adagents_url`)\n- Live file confirmed structurally correct via 5.7.0 resolution: 6,843 properties / 6,800 publisher_domains, see [adcp-client-python#750](https://github.com/adcontextprotocol/adcp-client-python/pull/750) for the resolution fix\n- Triggering use case: [prebid/salesagent#511](https://github.com/bokelley/salesagent/issues/511)","author":{"url":"https://github.com/bokelley","@type":"Person","name":"bokelley"},"datePublished":"2026-05-22T17:37:01Z","url":"https://github.com/adcontextprotocol/adcp-client-python/issues/801"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ac6a235e-9238-52d5-29fb-743014575d35 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E262:154D88:CD82F:105FC1:6B00F72B |
| html-safe-nonce | 6c286cc7fbdacb761594c9d97d7d1673cce7aab7ece54962976de8df4d98775e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMjYyOjE1NEQ4ODpDRDgyRjoxMDVGQzE6NkIwMEY3MkIiLCJ2aXNpdG9yX2lkIjoiNjA1MjQyMTk5MzUyODA5NjU1NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 9a270b6295e29267eca6f3db253801497713c6e30014bada6acc81f4c0846dd2 |
| hovercard-subject-tag | issue:4504514843 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/adcontextprotocol/adcp-client-python/801/issue_layout |
| twitter:image | https://opengraph.githubassets.com/a1e8cd472b5d16588287feea8cc76c1479ff11177409bbe9c4773d29c41c5d80/adcontextprotocol/adcp-client-python/issues/801 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/a1e8cd472b5d16588287feea8cc76c1479ff11177409bbe9c4773d29c41c5d80/adcontextprotocol/adcp-client-python/issues/801 |
| og:image:alt | Symptom Calling fetch_adagents(\"cafemedia.com\") against the largest known production AAO file (2.6 MB, declares 6,843 properties spanning 6,800 publisher_domains, sole sales agent: https://interc... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | bokelley |
| hostname | github.com |
| expected-hostname | github.com |
| None | 4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc |
| turbo-cache-control | no-preview |
| go-import | github.com/adcontextprotocol/adcp-client-python git https://github.com/adcontextprotocol/adcp-client-python.git |
| octolytics-dimension-user_id | 221690821 |
| octolytics-dimension-user_login | adcontextprotocol |
| octolytics-dimension-repository_id | 1090305973 |
| octolytics-dimension-repository_nwo | adcontextprotocol/adcp-client-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 1090305973 |
| octolytics-dimension-repository_network_root_nwo | adcontextprotocol/adcp-client-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 9983dd5da06943f833c971dd5b6bc988e2858be5 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width