René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Source and editable installs fail in some locales","articleBody":"Methods of installing GitPython that run code from `setup.py` fail in some locales. This does not affect installing from a wheel, but it does affect installing from an sdist, or installing from a local directory, including the editable install procedure recommended for development in the readme. The same problem happens when building GitPython. Building or installing using the old method of running `setup.py` directly is also affected. The error is of this form, though the codec will not always be `gbk`:\r\n\r\n```text\r\nUnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 473: illegal multibyte sequence\r\n```\r\n\r\nI am unsure if this ever happens in practice on Unix-like systems, whose locales are usually UTF-8. However, it happens on Windows systems in which `README.md` cannot be decoded using the system's active ANSI code page. This is a rarely-changed systemwide setting, so changing the user preferred languages, display language, or input method are not workarounds. I discovered this on a Simplified Chinese (zh-CN) build of Windows Server 2022 while using it to test some WSL-related test helper logic in #1745. Such a system uses ANSI code page 936. `README.md` is UTF-8, but it currently happens that it can be decoded with code page 1252, which Windows builds for Western European languages use as their ANSI code page. I expect encodings other than cp936 to fail as well.\r\n\r\nWith the PyPI sdist for GitPython 3.1.40 (on Python 3.12 x86-64, though I expect all supported versions to be affected):\r\n\r\n```text\r\n(.venv) C:\\Users\\Administrator\\gptest\u003e pip install --no-binary GitPython GitPython\r\nCollecting GitPython\r\n  Using cached GitPython-3.1.40.tar.gz (200 kB)\r\n  Installing build dependencies ... done\r\n  Getting requirements to build wheel ... error\r\n  error: subprocess-exited-with-error\r\n\r\n  × Getting requirements to build wheel did not run successfully.\r\n  │ exit code: 1\r\n  ╰─\u003e [18 lines of output]\r\n      Traceback (most recent call last):\r\n        File \"C:\\Users\\Administrator\\gptest\\.venv\\Lib\\site-packages\\pip\\_vendor\\pyproject_hooks\\_in_process\\_in_process.py\", line 353, in \u003cmodule\u003e\r\n          main()\r\n        File \"C:\\Users\\Administrator\\gptest\\.venv\\Lib\\site-packages\\pip\\_vendor\\pyproject_hooks\\_in_process\\_in_process.py\", line 335, in main\r\n          json_out['return_val'] = hook(**hook_input['kwargs'])\r\n                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n        File \"C:\\Users\\Administrator\\gptest\\.venv\\Lib\\site-packages\\pip\\_vendor\\pyproject_hooks\\_in_process\\_in_process.py\", line 118, in get_requires_for_build_wheel\r\n          return hook(config_settings)\r\n                 ^^^^^^^^^^^^^^^^^^^^^\r\n        File \"C:\\Users\\Administrator\\AppData\\Local\\Temp\\pip-build-env-8j1uxmuv\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py\", line 325, in get_requires_for_build_wheel\r\n          return self._get_build_requires(config_settings, requirements=['wheel'])\r\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n        File \"C:\\Users\\Administrator\\AppData\\Local\\Temp\\pip-build-env-8j1uxmuv\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py\", line 295, in _get_build_requires\r\n          self.run_setup()\r\n        File \"C:\\Users\\Administrator\\AppData\\Local\\Temp\\pip-build-env-8j1uxmuv\\overlay\\Lib\\site-packages\\setuptools\\build_meta.py\", line 311, in run_setup\r\n          exec(code, locals())\r\n        File \"\u003cstring\u003e\", line 20, in \u003cmodule\u003e\r\n      UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 473: illegal multibyte sequence\r\n      [end of output]\r\n\r\n  note: This error originates from a subprocess, and is likely not a problem with pip.\r\nerror: subprocess-exited-with-error\r\n\r\n× Getting requirements to build wheel did not run successfully.\r\n│ exit code: 1\r\n╰─\u003e See above for output.\r\n\r\nnote: This error originates from a subprocess, and is likely not a problem with pip.\r\n```\r\n\r\nA workaround is to pass `-X utf8` to `python`. For example, `python -X utf8 -m pip install ...` can be used for installation.\r\n\r\nThe fix should be straightforward. Importing `setup.py` confirms that the specific cause is reading `README.md`:\r\n\r\n```text\r\n(.venv) C:\\Users\\Administrator\\repos\\GitPython [main ≡]\u003e python -c 'import setup'\r\nTraceback (most recent call last):\r\n  File \"\u003cstring\u003e\", line 1, in \u003cmodule\u003e\r\n  File \"C:\\Users\\Administrator\\repos\\GitPython\\setup.py\", line 20, in \u003cmodule\u003e\r\n    long_description = rm_file.read()\r\n                       ^^^^^^^^^^^^^^\r\nUnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 477: illegal multibyte sequence\r\n```\r\n\r\nIt can be fixed by passing `encoding=\"utf-8\"`. **I have proposed this change in #1748.**\r\n\r\ngitdb and smmap are unaffected, because [gitdb does not open files to read those data in `setup.py`](https://github.com/gitpython-developers/gitdb/blob/ec58b7e55c8fec2479290a22faa293799f6805fc/setup.py#L24), while [smmap does open `README.md` but passes `encoding=\"utf-8\"`](https://github.com/gitpython-developers/smmap/blob/f1ace75be355fdec927793e462b9b12bf6ec9520/setup.py#L14).","author":{"url":"https://github.com/EliahKagan","@type":"Person","name":"EliahKagan"},"datePublished":"2023-11-28T23:15:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/1747/GitPython/issues/1747"}