René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Refresh big5hkscs mapping to HKSCS-2016","articleBody":"While working on #84508 I noticed that the mapping for `big5hkscs` codec has not been updated in a while. The current version in CPython reflects the Big-5 mappings for *HKSCS-2004*.\r\n\r\nSince then, there have been [some updates](https://www.ccli.gov.hk/en/hkscs/what_is_hkscs.html):\r\n\r\n- *HKSCS-2008* adds 68 code points to the Big-5 encoding scheme\r\n- *HKSCS-2016* adds no code points to Big-5 (it's Unicode-only), but since new characters have been added to Unicode, the mapping can change\r\n- after 2016, at least one mapped code point has been changed in an amendment\r\n\r\nI can update the script and generate the mapping using the latest data available on the [CCLI](https://www.ccli.gov.hk/en/download/) website, since I was already looking into this.\r\n\r\nIf we care about refreshing `big5hkscs` at all, there are a couple questions about compatibility. In case mapping a Big-5 code X used to map to Unicode code point A (in HKSCS-2004), and is changed to map to B (in later versions):\r\n\r\n1) should we: decode X to A, or to B?\r\n\r\n2) should we: encode B to X, A to X, or both?\r\n\r\n---\r\n\r\nE.g. right now the Big-5 sequence 9D73 round-trips:\r\n\r\n```python\r\n\u003e\u003e\u003e x = bytes.fromhex('9D73')\r\n\u003e\u003e\u003e x.decode('big5hkscs') == '\\u4ca4'\r\nTrue\r\n\u003e\u003e\u003e '\\u4ca4'.encode('big5hkscs') == x\r\nTrue\r\n```\r\n\r\nIf we followed the new HKSCS-2016 mapping with no compatibility provisions, this round-trip would instead go through the newly mapped character `\\u9fd0`. This might be fine for some users, but it might break compatibility for others. So the questions are about what kind of compatibility we want to guarantee.\r\n\r\n---\r\n\r\nRelated question which should not block this issue. For the web platform, WHATWG defines a [Big5 encoding](https://encoding.spec.whatwg.org/#legacy-multi-byte-chinese-(traditional)-encodings) which includes HKSCS extensions, and already overlaps 99% with `big5hkscs`, but is incompatible in some cases. Since one of the users of the CPython CJK codecs is html5lib, this means that html5lib does not comply with the web platform specifications. Should CPython be concerned with this, since it already provides the codec and the mapping tables, and it could provide a web-compatible codec with just a few fixups? Or does this belong in third-party libraries?","author":{"url":"https://github.com/sorcio","@type":"Person","name":"sorcio"},"datePublished":"2022-05-26T20:13:44.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/93271/cpython/issues/93271"}