Title: email.policy.default - gotcha with re-using parsed headers with embedded newlines · Issue #121650 · python/cpython · GitHub
Open Graph Title: email.policy.default - gotcha with re-using parsed headers with embedded newlines · Issue #121650 · python/cpython
X Title: email.policy.default - gotcha with re-using parsed headers with embedded newlines · Issue #121650 · python/cpython
Description: Bug report Bug description: I'm not sure if this is a bug, feature request, or user error. I'm happy to re-file once I know which If a parsed email header contains a correctly quoted newline, setting an email header to that value will in...
Open Graph Description: Bug report Bug description: I'm not sure if this is a bug, feature request, or user error. I'm happy to re-file once I know which If a parsed email header contains a correctly quoted newline, setti...
X Description: Bug report Bug description: I'm not sure if this is a bug, feature request, or user error. I'm happy to re-file once I know which If a parsed email header contains a correctly quoted newlin...
Opengraph URL: https://github.com/python/cpython/issues/121650
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"email.policy.default - gotcha with re-using parsed headers with embedded newlines","articleBody":"# Bug report\n\n### Bug description:\n\n*I'm not sure if this is a bug, feature request, or user error. I'm happy to re-file once I know which*\n\nIf a parsed email header contains a correctly quoted newline, setting an email header to that value will include a newline.\n\n```python\nfrom email import message_from_string\nfrom email.policy import default\n\nemail_in = \"\"\"\\\nTo: incoming+tag@me.example.com\nFrom: External Sender \u003csender@them.example.com>\nSubject: Here's an =?UTF-8?Q?embedded_newline=0A?=\nContent-Type: text/html; charset=UTF-8\nContent-Transfer-Encoding: quoted-printable\nMIME-Version: 1.0\n\n\u003chtml>\n\u003chead>\u003ctitle>An embeded newline\u003c/title>\u003c/head>\n\u003cbody>\n \u003cp>I sent you an embedded newline in the subject. How do you like that?!\u003c/p>\n\u003c/body>\n\u003c/html>\n\"\"\"\n\nmsg = message_from_string(email_in, policy=default)\nmsg = message_from_string(email_in, policy=default)\nfor header, value in msg.items():\n del msg[header]\n msg[header] = value\nemail_out = str(msg)\nprint(email_out)\n```\n\nOutput is:\n```email\nTo: incoming+tag@me.example.com\nFrom: External Sender \u003csender@them.example.com>\nSubject: Here's an embedded newline\n\nContent-Type: text/html; charset=\"UTF-8\"\nContent-Transfer-Encoding: quoted-printable\nMIME-Version: 1.0\n\n\u003chtml>\n\u003chead>\u003ctitle>An embeded newline\u003c/title>\u003c/head>\n\u003cbody>\n \u003cp>I sent you an embedded newline in the subject. How do you like that?!\u003c/p>\n\u003c/body>\n\u003c/html>\n\n```\nAn email parser will interpret the newline as the start of the message. In this case, the `Content-Type` and other MIME headers will not be processed, and the email treated as plain text. In other cases, required headers like `To` may not be processed and the email will not be delivered.\n\nI'd expect an error on setting the value, an error on serializing the `EmailMessage` to a string, the subject to retain the original encoding, or the newline to be quoted in the serialized version. \n\nNow that we know the behavior, we can process the headers (embed or strip trailing newlines). However, you may see this is a bug, a needed feature, or missing documentation.\n\nMore info:\n\n`subject`'s type is a `email.headerregistry._UniqueUnstructuredHeader`. It has a `name`, so it is assigned without checking (`email.policy.EmailPolicy.header_store_parse()`).\n\nThe `_parse_tree`, returned by `email._header_value_parser.get_unstructured()`, is:\n```\nUnstructuredTokenList([ValueTerminal(\"Here's\"), WhiteSpaceTerminal(' '), ValueTerminal('an'), WhiteSpaceTerminal(' '), EncodedWord([ValueTerminal('embedded'), WhiteSpaceTerminal(' '), ValueTerminal('newline\\n')])])\n```\n\nA user encountered this for our email relaying service https://relay.firefox.com (https://github.com/mozilla/fx-private-relay/issues/4841). An incoming email to a service address is matched to a user. We re-write the email headers and forward the email to the user's \"real\" address. \n\nA real email has this subject header:\n\n```\nSubject: The All Over Piercings Wishlist of =?UTF-8?Q?John=2E=0A?=\n```\n\nThis is from a European website https://www.alloverpiercings.com. You can create a wishlist and send it to an email address. The subject appears correctly encoded to me, to allow for non-ASCII usernames, with the unfortunate embedded newline. When forwarding this email, using something similar to the code above (but with more header modifications and additions), the embedded newline is turned into a real newline. The rest of the email headers are treated as part of the body. Since the `Content-Type` and other MIME headers are not processed as headers, the email is treated as a plain text email.\n\n\n### CPython versions tested on:\n\n3.11, 3.12\n\n### Operating systems tested on:\n\nmacOS\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-121812\n* gh-122233\n* gh-122484\n* gh-122599\n* gh-122608\n* gh-122609\n* gh-122610\n* gh-122611\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/jwhitlock","@type":"Person","name":"jwhitlock"},"datePublished":"2024-07-12T17:17:12Z","url":"https://github.com/python/cpython/issues/121650"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:eb0b943d-236b-c20a-584a-3e7cc58ab9da |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C89A:3B60C0:101D994:159B38C:6B005C9C |
| html-safe-nonce | 12ce96f475d53b6108b15f17a64cd0fae7d948f69ad54ea1c587e98a81dd7d20 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDODlBOjNCNjBDMDoxMDFEOTk0OjE1OUIzOEM6NkIwMDVDOUMiLCJ2aXNpdG9yX2lkIjoiMjA2MDMzNTU1MDcyMzgwODQxMiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 2ccc6daa54dcb2b6b44d849eddaee920db81b1ab6f46cdf82e28d2ee5f17b68c |
| hovercard-subject-tag | issue:2406021585 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/121650/issue_layout |
| twitter:image | https://opengraph.githubassets.com/b1d35164d4c2906de34871adc48b3253800c81d68c89e864f0091a129398fc68/python/cpython/issues/121650 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/b1d35164d4c2906de34871adc48b3253800c81d68c89e864f0091a129398fc68/python/cpython/issues/121650 |
| og:image:alt | Bug report Bug description: I'm not sure if this is a bug, feature request, or user error. I'm happy to re-file once I know which If a parsed email header contains a correctly quoted newline, setti... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | jwhitlock |
| hostname | github.com |
| expected-hostname | github.com |
| None | 29be975efff6507a5d135de9c7d634cf7ff8f0c55222a64386ab3202209aaf73 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | eacd927ba6112e3b01ba99c762bd699a4a4f5df7 |
| ui-target | canary-1 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width