René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"http.client._MAXHEADERS = 100 limit no longer sufficient","articleBody":"# Bug report\n\n### Bug description:\n\nThis hard-coded sanity check for HTTP response headers is no longer sufficient to fetch a Microsoft 365 page.  \n\nhttps://github.com/python/cpython/blob/0a91456ad14bb598646f50bf8f034e8887c0c468/Lib/http/client.py#L112\n\nI do have a case open with Microsoft Support (TrackingID#2503130010002871), but it's not getting much traction as it's not causing problems with full web browsers.\n\nSteps to reproduce:\n\n```python\nimport http.client\ncon = http.client.HTTPSConnection('outlook.office365.com')\ncon.request(\"GET\", \"/owa/example.edu\")  # any domain seems to trigger\nr = con.getresponse()\n```\n\nAnd that throws a HTTPException\n```\n>>> con = http.client.HTTPSConnection('outlook.office365.com')\n>>> con.request(\"GET\", \"/owa/foo.bar\")\n>>> r = con.getresponse()\nTraceback (most recent call last):\n  File \"\u003cpython-input-8>\", line 1, in \u003cmodule>\n    r = con.getresponse()\n  File \"C:\\Users\\jmacdone\\AppData\\Local\\Programs\\Python\\Python313-arm64\\Lib\\http\\client.py\", line 1428, in getresponse\n    response.begin()\n    ~~~~~~~~~~~~~~^^\n  File \"C:\\Users\\jmacdone\\AppData\\Local\\Programs\\Python\\Python313-arm64\\Lib\\http\\client.py\", line 350, in begin\n    self.headers = self.msg = parse_headers(self.fp)\n                              ~~~~~~~~~~~~~^^^^^^^^^\n  File \"C:\\Users\\jmacdone\\AppData\\Local\\Programs\\Python\\Python313-arm64\\Lib\\http\\client.py\", line 248, in parse_headers\n    headers = _read_headers(fp)\n  File \"C:\\Users\\jmacdone\\AppData\\Local\\Programs\\Python\\Python313-arm64\\Lib\\http\\client.py\", line 226, in _read_headers\n    raise HTTPException(\"got more than %d headers\" % _MAXHEADERS)\nhttp.client.HTTPException: got more than 100 headers\n>>>\n```\n\nIt seems to be just spilling over with 101 headers. Though, not consistently.  Presumably it depends upon which load balancer node is responding.\n\n\n```bash\n$ curl --silent -D - 'https://outlook.office365.com/owa/example.edu' | grep -E \"^[a-zA-Z-]+: \" | wc -l\n```\nreturns with 96, 99, 101, etc. headers, depending on ~Microsoft's mood~ unknown factors.\n\nFor background, it's common to use `https://outlook.com/example.edu` as a domain hint (\"smart link\") to go directly to a tenant's identity provider and avoid the \"Please provide your email address\" step.  We have a nagios check for that, which broke recently as the number of  `Set-Cookie:  OpenIdConnect.token.[...]` variants continues to grow.\n\n### CPython versions tested on:\n\n3.13, 3.9, 3.11\n\n### Operating systems tested on:\n\nLinux, Windows\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-136814\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/jmacdone","@type":"Person","name":"jmacdone"},"datePublished":"2025-03-25T16:27:47Z","url":"https://github.com/python/cpython/issues/131724"}