René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"TarFile filters fail in non-UTF-8 locales","articleBody":"# Bug report\n\n`test_tarfile` files in non-UTF-8 locales. For example:\n```\n$ LC_ALL=uk_UA ./python -m test -vuall test_tarfile -m 'NoneInfoExtractTests_*' -m test_data_filter -m test_tar_filter\n```\n\u003cdetails\u003e\n\n```\n======================================================================\nERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Data)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/serhiy/py/cpython/Lib/test/test_tarfile.py\", line 3264, in setUpClass\n    tar.extractall(cls.control_dir, filter=cls.extraction_filter)\n    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2389, in extractall\n    tarinfo = self._get_extract_tarinfo(member, filter_function, path)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2441, in _get_extract_tarinfo\n    tarinfo = filter_function(tarinfo, path)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 842, in data_filter\n    new_attrs = _get_filtered_attrs(member, dest_path, True)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 782, in _get_filtered_attrs\n    target_path = os.path.realpath(os.path.join(dest_path, name))\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 405, in realpath\n    return _realpath(filename, strict, sep, curdir, pardir, getcwd)\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 452, in _realpath\n    st_mode = lstat(newpath).st_mode\n              ~~~~~^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/encodings/koi8_u.py\", line 12, in encode\n    return codecs.charmap_encode(input,errors,encoding_table)\n           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to \u003cundefined\u003e\nencoding with 'koi8-u' codec failed\n\n======================================================================\nERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Default)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/serhiy/py/cpython/Lib/test/test_tarfile.py\", line 3264, in setUpClass\n    tar.extractall(cls.control_dir, filter=cls.extraction_filter)\n    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2389, in extractall\n    tarinfo = self._get_extract_tarinfo(member, filter_function, path)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2441, in _get_extract_tarinfo\n    tarinfo = filter_function(tarinfo, path)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 842, in data_filter\n    new_attrs = _get_filtered_attrs(member, dest_path, True)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 782, in _get_filtered_attrs\n    target_path = os.path.realpath(os.path.join(dest_path, name))\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 405, in realpath\n    return _realpath(filename, strict, sep, curdir, pardir, getcwd)\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 452, in _realpath\n    st_mode = lstat(newpath).st_mode\n              ~~~~~^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/encodings/koi8_u.py\", line 12, in encode\n    return codecs.charmap_encode(input,errors,encoding_table)\n           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to \u003cundefined\u003e\nencoding with 'koi8-u' codec failed\n\n======================================================================\nERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_FullyTrusted)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/serhiy/py/cpython/Lib/test/test_tarfile.py\", line 3264, in setUpClass\n    tar.extractall(cls.control_dir, filter=cls.extraction_filter)\n    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2397, in extractall\n    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),\n    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n                      numeric_owner=numeric_owner)\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2460, in _extract_one\n    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),\n    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n                         set_attrs=set_attrs,\n                         ^^^^^^^^^^^^^^^^^^^^\n                         numeric_owner=numeric_owner)\n                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2543, in _extract_member\n    self.makefile(tarinfo, targetpath)\n    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2589, in makefile\n    with bltn_open(targetpath, \"wb\") as target:\n         ~~~~~~~~~^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/encodings/koi8_u.py\", line 12, in encode\n    return codecs.charmap_encode(input,errors,encoding_table)\n           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to \u003cundefined\u003e\nencoding with 'koi8-u' codec failed\n\n======================================================================\nERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Tar)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/serhiy/py/cpython/Lib/test/test_tarfile.py\", line 3264, in setUpClass\n    tar.extractall(cls.control_dir, filter=cls.extraction_filter)\n    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2389, in extractall\n    tarinfo = self._get_extract_tarinfo(member, filter_function, path)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 2441, in _get_extract_tarinfo\n    tarinfo = filter_function(tarinfo, path)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 836, in tar_filter\n    new_attrs = _get_filtered_attrs(member, dest_path, False)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 782, in _get_filtered_attrs\n    target_path = os.path.realpath(os.path.join(dest_path, name))\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 405, in realpath\n    return _realpath(filename, strict, sep, curdir, pardir, getcwd)\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 452, in _realpath\n    st_mode = lstat(newpath).st_mode\n              ~~~~~^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/encodings/koi8_u.py\", line 12, in encode\n    return codecs.charmap_encode(input,errors,encoding_table)\n           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeEncodeError: 'charmap' codec can't encode characters in position 112-118: character maps to \u003cundefined\u003e\nencoding with 'koi8-u' codec failed\n\n======================================================================\nERROR: test_data_filter (test.test_tarfile.TestExtractionFilters.test_data_filter)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/serhiy/py/cpython/Lib/test/test_tarfile.py\", line 4086, in test_data_filter\n    filtered = tarfile.data_filter(tarinfo, '')\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 842, in data_filter\n    new_attrs = _get_filtered_attrs(member, dest_path, True)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 782, in _get_filtered_attrs\n    target_path = os.path.realpath(os.path.join(dest_path, name))\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 405, in realpath\n    return _realpath(filename, strict, sep, curdir, pardir, getcwd)\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 452, in _realpath\n    st_mode = lstat(newpath).st_mode\n              ~~~~~^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/encodings/koi8_u.py\", line 12, in encode\n    return codecs.charmap_encode(input,errors,encoding_table)\n           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeEncodeError: 'charmap' codec can't encode characters in position 69-75: character maps to \u003cundefined\u003e\nencoding with 'koi8-u' codec failed\n\n======================================================================\nERROR: test_tar_filter (test.test_tarfile.TestExtractionFilters.test_tar_filter)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/serhiy/py/cpython/Lib/test/test_tarfile.py\", line 4076, in test_tar_filter\n    filtered = tarfile.tar_filter(tarinfo, '')\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 836, in tar_filter\n    new_attrs = _get_filtered_attrs(member, dest_path, False)\n  File \"/home/serhiy/py/cpython/Lib/tarfile.py\", line 782, in _get_filtered_attrs\n    target_path = os.path.realpath(os.path.join(dest_path, name))\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 405, in realpath\n    return _realpath(filename, strict, sep, curdir, pardir, getcwd)\n  File \"/home/serhiy/py/cpython/Lib/posixpath.py\", line 452, in _realpath\n    st_mode = lstat(newpath).st_mode\n              ~~~~~^^^^^^^^^\n  File \"/home/serhiy/py/cpython/Lib/encodings/koi8_u.py\", line 12, in encode\n    return codecs.charmap_encode(input,errors,encoding_table)\n           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeEncodeError: 'charmap' codec can't encode characters in position 69-75: character maps to \u003cundefined\u003e\nencoding with 'koi8-u' codec failed\n\n----------------------------------------------------------------------\n```\n\n\u003c/details\u003e\n\nThis happens because they use `os.path.realpath()` for paths in a tar archive, which uses `os.stat()`, which fails with unexpected `UnicodeEncodeError` if the path in a tar archive can't be encoded in the current filesystem encoding. This error should be handled at some level, either in `os.path.realpath()` or in `tarfile`. `os.stat()` can also raise `ValueError` if the path contain null bytes. Don't know if this is relevant here, we should test.\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-134147\n* gh-134195\n* gh-134196\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/serhiy-storchaka","@type":"Person","name":"serhiy-storchaka"},"datePublished":"2025-05-11T11:23:42.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/133890/cpython/issues/133890"}