René's URL Explorer Experiment

Title: gh-74668: Fix encoded unicode in url byte string by roaanv · Pull Request #93757 · python/cpython · GitHub

Open Graph Title: gh-74668: Fix encoded unicode in url byte string by roaanv · Pull Request #93757 · python/cpython

X Title: gh-74668: Fix encoded unicode in url byte string by roaanv · Pull Request #93757 · python/cpython

Description: When urllib.parse.parse_qs is called with a byte string, an encoding can be provided which is used to decode the byte string. However, when parsing, parse.py uses 'ascii' encoding to re-encode the parsed data. This breaks utf-8 encoded URLs received as byte strings. This change uses the encoding passed to parse_qs to re-encode the parsed data. Caveat This is probably not the correct solution, but gets us closer to a working implementation that, at worst case, can be dictated by the caller of parse_qs. My understanding of the problem is as follows. parse_qs detects the qs is a byte string and decodes it according to the encoding parameter. After parsing the decoded input, it then re-encodes it (because it detected the input was a byte string), but instead of using the value of the encoding parameter, it uses 'ascii'. The decoding and encoding thus uses different encoders. This PR fixes that, in that it uses the encoding parameter value to re-encode the parsed data. However, this is not a complete solution. The problem is that there are in essence 2 encodings involved. One encoder used to encode/decode the byte string and another to encode/decode the URL (which is utf-8 encoded). Take b"a=a’b" as an example. This is a valid ascii encoded byte string and can be decoded with an ascii decoder. However, after decoding and parsing, it will not produce valid ascii. I.e. parsing of a’b will produce a’b which can not be ascii encoded. A possible solution would be to pass a reencoding parameter, but since it's unlikely callers will have different encoding and reencoding parameters, this PR opts for reusing the encoding parameter. Issue: gh-74668

Open Graph Description: When urllib.parse.parse_qs is called with a byte string, an encoding can be provided which is used to decode the byte string. However, when parsing, parse.py uses 'ascii' encoding to re-enc...

X Description: When urllib.parse.parse_qs is called with a byte string, an encoding can be provided which is used to decode the byte string. However, when parsing, parse.py uses 'ascii' encoding t...

Opengraph URL: https://github.com/python/cpython/pull/93757

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width