René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Replace malformed new_unicode_decode_error / new_unicode_encode_error helpers with the _real variants","articleBody":"## Summary\n\n`VirtualMachine::new_unicode_decode_error` (and its sibling `new_unicode_encode_error`) construct a `UnicodeDecodeError` that is **structurally invalid**: it has none of the five attributes that a `UnicodeDecodeError` must have (`encoding`, `object`, `start`, `end`, `reason`), and its `args` is a 1‑tuple holding a plain message string. A correct helper — `new_unicode_decode_error_real` — already exists, and both macro-generated helpers are already marked for removal:\n\n```rust\n// crates/vm/src/vm/vm_new.rs\n// TODO: remove \u0026 replace with new_unicode_decode_error_real\ndefine_exception_fn!(fn new_unicode_decode_error, unicode_decode_error, UnicodeDecodeError);\n\n// TODO: remove \u0026 replace with new_unicode_encode_error_real\ndefine_exception_fn!(fn new_unicode_encode_error, unicode_encode_error, UnicodeEncodeError);\n```\n\nThis issue tracks replacing the remaining call sites of the broken helpers with the `_real` variants and removing the broken helpers.\n\n## Why the macro version is wrong\n\n`new_unicode_decode_error` is generated by the `define_exception_fn!` macro, which just calls `new_exception_msg`:\n\n```rust\npub fn $fn_name(\u0026self, msg: impl Into\u003cWtf8Buf\u003e) -\u003e PyBaseExceptionRef {\n    let err = self.ctx.exceptions.$attr.to_owned();\n    self.new_exception_msg(err, msg.into())   // -\u003e new_exception(type, vec![msg_str])\n}\n```\n\n`new_exception` builds the exception via `PyBaseException::new(args, vm)`, which only stores `args`. It **does not run `slot_init`**, so `PyUnicodeDecodeError`'s initializer — the only thing that sets `encoding`/`object`/`start`/`end`/`reason` — never runs. The resulting object is a `UnicodeDecodeError` with `args = (\"some message\",)` and no unicode-error attributes at all.\n\n`new_unicode_decode_error_real` instead passes the full 5-tuple and sets every attribute, matching what `UnicodeDecodeError(encoding, object, start, end, reason)` produces.\n\nNote that in CPython this object cannot even be constructed — `UnicodeDecodeError` requires exactly 5 arguments:\n\n```\n\u003e\u003e\u003e UnicodeDecodeError('csv not utf8')\nTypeError: function takes exactly 5 arguments (1 given)\n```\n\n## Reproduction\n\n`csv.reader(..., quoting=csv.QUOTE_STRINGS)` reaches `crates/stdlib/src/csv.rs:1113`, which uses the broken helper:\n\n```python\nimport csv\nr = csv.reader(['\\udcff'], quoting=csv.QUOTE_STRINGS)\ntry:\n    next(r)\nexcept UnicodeDecodeError as e:\n    print(\"str(e) =\", repr(str(e)))\n    print(\"args   =\", e.args)\n    for attr in (\"encoding\", \"object\", \"start\", \"end\", \"reason\"):\n        try:\n            print(\" \", attr, \"=\", repr(getattr(e, attr)))\n        except AttributeError:\n            print(\" \", attr, \"-\u003e AttributeError\")\n```\n\nRustPython output:\n\n```\nstr(e) = ''\nargs   = ('csv not utf8',)\n  encoding -\u003e AttributeError\n  object -\u003e AttributeError\n  start -\u003e AttributeError\n  end -\u003e AttributeError\n  reason -\u003e AttributeError\n```\n\nTwo concrete symptoms:\n\n1. **The diagnostic message is silently lost.** `UnicodeDecodeError.__str__` returns an empty string when the `object` attribute is missing, so `str(e)` is `''` instead of the message that was passed in.\n2. **Attribute access raises `AttributeError`.** Any error handler that inspects `e.encoding` / `e.start` / `e.reason` (as the `codecs` error-handler machinery and normal user code do) breaks. A genuine `UnicodeDecodeError` always has these attributes.\n\nFor comparison, a well-formed instance (CPython, and RustPython via `new_unicode_decode_error_real`):\n\n```\nstr(e) = \"'utf-8' codec can't decode byte 0xff in position 0: invalid start byte\"\nargs   = ('utf-8', b'\\xff', 0, 1, 'invalid start byte')\nencoding = utf-8 | object = b'\\xff' | start = 0 | end = 1 | reason = 'invalid start byte'\n```\n\n## Call sites to convert\n\nCorrect (already using `_real`) — for reference:\n\n- `crates/vm/src/codecs.rs:805`\n- `crates/vm/src/stdlib/_codecs.rs:1023`, `:1114`, `:1125`\n- `crates/capi/src/pyerrors.rs:353`\n\nBroken — `new_unicode_decode_error(msg)`:\n\n- `crates/stdlib/src/csv.rs:1113`, `:1239`, `:1415`, `:1461`, `:1510`, `:1584`\n- `crates/vm/src/stdlib/nt.rs:218`, `:722`, `:724`, `:726`, `:938` (Windows)\n- `crates/stdlib/src/socket.rs:2616`, `:2637`\n- `crates/vm/src/stdlib/_codecs.rs:483`, `:499`, `:608`, `:624` (mbcs/oem, Windows)\n- `crates/vm/src/stdlib/posix.rs:1281`, `:1737`\n- `crates/vm/src/function/fspath.rs:129`\n- `crates/vm/src/stdlib/os.rs:134`\n\nBroken — `new_unicode_encode_error(msg)` (same defect, `UnicodeEncodeError` side; `_real` variant is `new_unicode_encode_error_real`):\n\n- `crates/stdlib/src/array.rs:641`, `:1700`\n- `crates/stdlib/src/socket.rs:2630`\n- `crates/vm/src/stdlib/_codecs.rs:397`, `:431`, `:521`, `:555` (mbcs/oem, Windows)\n\n## Plan\n\n1. At each broken decode call site, supply the real `(encoding, object, start, end, reason)` values and call `new_unicode_decode_error_real`. Most of these already hold the source bytes and the failing offset (or can compute them from `Utf8Error::valid_up_to`); where only a generic reason is available, pass `\"utf-8\"` as encoding, the original bytes as `object`, and the reason string. Do the same for the encode call sites with `new_unicode_encode_error_real` and `(encoding, object: str, start, end, reason)`.\n2. Remove `new_unicode_decode_error` and `new_unicode_encode_error` once their last callers are gone.\n\n## Related\n\n- #8345 — separate but adjacent: even the *correctly* constructed `UnicodeDecodeError` leaks its five attributes into `__dict__` instead of storing them as struct fields.\n\n---\n\n*Investigated and drafted by Claude; reviewed before filing.*\n","author":{"url":"https://github.com/youknowone","@type":"Person","name":"youknowone"},"datePublished":"2026-07-23T03:22:55.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/8352/RustPython/issues/8352"}