René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Clarity on units of string","articleBody":"I'll start by saying that I'm not well-versed in WebAssembly specifications or proposals, but I happened to come across this proposal, and I'm quite interested in Unicode strings and how they're represented in different programming languages and VMs.\r\n\r\nThis might be just a wording issue, but I think the current description of [\"What's a string?\"](https://github.com/WebAssembly/stringref/blob/a64917cd5346f8704e614c4825ebf05737ac5e64/proposals/stringref/Overview.md?plain=1#L50) could be problematic:\r\n\r\n\u003e Therefore **we define a string to be a sequence of unicode scalar values and isolated surrogates**. The code units of a Java or JavaScript string can be interpreted to encode such a sequence, in the [WTF-16](https://simonsapin.github.io/wtf-8/) encoding form.\r\n\r\nIf this is taken to mean that a string is a sequence of Unicode code points (ie, \"Unicode scalar values\" and \"isolated surrogates\", basically any integer from `0x0` to `0x10FFFF`), this does not correspond with \"WTF-16\" or JavaScript strings, since there are sequences of isolated surrogates that can't be distinctly encoded in WTF-16.\r\n\r\neg, the sequence `[U+D83D, U+DCA9]` (high surrogate, low surrogate) doesn't have an encoding form in WTF-16. Interpreting the WTF-16/UTF-16 code unit sequence `\u003cD83D DCA9\u003e` produces the sequence `[U+1F4A9]` (a Unicode scalar value, `💩`). There are 1048576 occurrences of such sequences (one for every code point outside of the BMP), where the \"obvious\" encoding is already used to encode a USV.\r\n\r\n```js\r\n\u003e \"\\uD83D\\uDCA9\"\r\n'💩'\r\n\u003e \"\\u{1F4A9}\"\r\n'💩'\r\n\u003e [...\"\\uD83D\\uDCA9\"].length\r\n1\r\n\u003e [...\"\\u{1F4A9}\"].length\r\n1\r\n```\r\n\r\nNote that this is different to how strings work in Python 3, where a string is indeed a sequence of any Unicode code points:\r\n```python\r\n\u003e\u003e\u003e \"\\U0000D83D\\U0000DCA9\"\r\n'\\ud83d\\udca9'\r\n\u003e\u003e\u003e \"\\U0001F4A9\"\r\n'💩'\r\n\u003e\u003e\u003e len(\"\\U0000D83D\\U0000DCA9\")\r\n2\r\n\u003e\u003e\u003e len(\"\\U0001F4A9\")\r\n1\r\n```\r\n\r\nIf this proposal is suggesting that strings work the same way as in Python 3, I think implementations will likely[0] resort to using UTF-32 in some cases, as I believe Python implementations do (I think Python implementations usually use UTF-32[1] essentially for all strings, though they will switch between using 8-bit, 16-bit and 32-bit arrays depending on the range of code points used). Other than Python, I'm not actually sure what language implementations would benefit from such a string representation.\r\n\r\nAs a side note, the section in question also lumps together Python (presumably 3) and Rust, though this might result from a misunderstanding that should hopefully be explained above. Rust strings are meant to be[2] valid UTF-8, hence they correspond to sequences of Unicode scalar values, but as explained above, Python strings can be any distinct sequence of Unicode code points.\r\n\r\n---\r\n\r\n[0] Another alternative would be using a variation of WTF-8 that preserves UTF-16 surrogates instead of normalising them to USVs on concatenation, though this seems a bit crazy.\r\n\r\n[1] Technically this is an extension of UTF-32, since UTF-32 itself doesn't allow encoding of code points in the surrogate range.\r\n\r\n[2] This is at least true at some API level, though technically the representation of strings in Rust is allowed to contain arbitrary bytes, where it is up to libraries to avoid emitting invalid UTF-8 to safe code: https://github.com/rust-lang/rust/issues/71033","author":{"url":"https://github.com/Maxdamantus","@type":"Person","name":"Maxdamantus"},"datePublished":"2023-10-21T02:36:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":16},"url":"https://github.com/65/stringref/issues/65"}