René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Use-after-free in `_Py_strhex_impl` via re-entrant `sep.__len__` in `bytearray.hex`","articleBody":"### What happened?\n\n`_Py_strhex_impl` calls `PyObject_Length(sep)` before hexlifying, and a crafted separator with `__len__` that clears the `bytearray` frees its storage while the loop continues to read from the original buffer, triggering a use-after-free.\n\n**Proof of Concept:**\n```python\nt = bytearray(b'\\xAA')\n\nclass S(bytes):\n    def __len__(self):\n        t.clear()\n        return 1\n\nt.hex(S(b':'))\n```\n\n```python\nba = bytearray(b'A' * 1024)\nmv = memoryview(ba)\n\nclass BadSep(bytes):\n    def __len__(self):\n        mv.release()\n        ba.clear()\n        return 1\n\nmv.hex(BadSep(b':'))\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eAffected Versions\u003c/strong\u003e\u003c/summary\u003e\n\n| Python Version | Status | Exit Code |\n|---|---|---|\n| `Python 3.9.24+ (heads/3.9:111bbc15b26, Oct 27 2025, 21:34:13) ` | ASAN | 1 |\n| `Python 3.10.19+ (heads/3.10:014261980b1, Oct 27 2025, 21:19:00) [Clang 18.1.3 (1ubuntu1)]` | ASAN | 1 |\n| `Python 3.11.14+ (heads/3.11:88f3f5b5f11, Oct 27 2025, 21:20:35) [Clang 18.1.3 (1ubuntu1)]` | ASAN | 1 |\n| `Python 3.12.12+ (heads/3.12:8cb2092bd8c, Oct 27 2025, 21:27:07) [Clang 18.1.3 (1ubuntu1)]` | ASAN | 1 |\n| `Python 3.13.9+ (heads/3.13:9c8eade20c6, Oct 27 2025, 21:28:49) [Clang 18.1.3 (1ubuntu1)]` | ASAN | 1 |\n| `Python 3.14.0+ (heads/3.14:2e216728038, Oct 27 2025, 21:30:55) [Clang 18.1.3 (1ubuntu1)]` | ASAN | 1 |\n| `Python 3.15.0a1+ (heads/main:f5394c257ce, Oct 27 2025, 21:32:37) [Clang 18.1.3 (1ubuntu1)]` | ASAN | 1 |\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eVulnerable Code\u003c/strong\u003e\u003c/summary\u003e\n\n```c\nstatic PyObject *_Py_strhex_impl(const char* argbuf, const Py_ssize_t arglen,\n                                 PyObject* sep, int bytes_per_sep_group,\n                                 const int return_bytes)\n{\n    assert(arglen \u003e= 0);\n\n    Py_UCS1 sep_char = 0;\n    if (sep) {\n        // Reentrant Call to obj's __len__ method\n        Py_ssize_t seplen = PyObject_Length((PyObject*)sep);\n        if (seplen \u003c 0) {\n            return NULL;\n        }\n        if (seplen != 1) {\n            PyErr_SetString(PyExc_ValueError, \"sep must be length 1.\");\n            return NULL;\n        }\n        if (PyUnicode_Check(sep)) {\n            if (PyUnicode_KIND(sep) != PyUnicode_1BYTE_KIND) {\n                PyErr_SetString(PyExc_ValueError, \"sep must be ASCII.\");\n                return NULL;\n            }\n            sep_char = PyUnicode_READ_CHAR(sep, 0);\n        }\n        else if (PyBytes_Check(sep)) {\n            sep_char = PyBytes_AS_STRING(sep)[0];\n        }\n        else {\n            PyErr_SetString(PyExc_TypeError, \"sep must be str or bytes.\");\n            return NULL;\n        }\n        if (sep_char \u003e 127 \u0026\u0026 !return_bytes) {\n            PyErr_SetString(PyExc_ValueError, \"sep must be ASCII.\");\n            return NULL;\n        }\n    }\n    else {\n        bytes_per_sep_group = 0;\n    }\n\n    unsigned int abs_bytes_per_sep = Py_ABS(bytes_per_sep_group);\n    Py_ssize_t resultlen = 0;\n    if (bytes_per_sep_group \u0026\u0026 arglen \u003e 0) {\n        /* How many sep characters we'll be inserting. */\n        resultlen = (arglen - 1) / abs_bytes_per_sep;\n    }\n    /* Bounds checking for our Py_ssize_t indices. */\n    if (arglen \u003e= PY_SSIZE_T_MAX / 2 - resultlen) {\n        return PyErr_NoMemory();\n    }\n    resultlen += arglen * 2;\n\n    if ((size_t)abs_bytes_per_sep \u003e= (size_t)arglen) {\n        bytes_per_sep_group = 0;\n        abs_bytes_per_sep = 0;\n    }\n\n    PyObject *retval;\n    Py_UCS1 *retbuf;\n    if (return_bytes) {\n        /* If _PyBytes_FromSize() were public we could avoid malloc+copy. */\n        retval = PyBytes_FromStringAndSize(NULL, resultlen);\n        if (!retval) {\n            return NULL;\n        }\n        retbuf = (Py_UCS1 *)PyBytes_AS_STRING(retval);\n    }\n    else {\n        retval = PyUnicode_New(resultlen, 127);\n        if (!retval) {\n            return NULL;\n        }\n        retbuf = PyUnicode_1BYTE_DATA(retval);\n    }\n\n    /* Hexlify */\n    Py_ssize_t i, j;\n    unsigned char c;\n\n    if (bytes_per_sep_group == 0) {\n        for (i = j = 0; i \u003c arglen; ++i) {\n            assert((j + 1) \u003c resultlen);\n            c = argbuf[i];\n            retbuf[j++] = Py_hexdigits[c \u003e\u003e 4];\n            retbuf[j++] = Py_hexdigits[c \u0026 0x0f];\n        }\n        assert(j == resultlen);\n    }\n    else {\n        /* The number of complete chunk+sep periods */\n        Py_ssize_t chunks = (arglen - 1) / abs_bytes_per_sep;\n        Py_ssize_t chunk;\n        unsigned int k;\n\n        if (bytes_per_sep_group \u003c 0) {\n            i = j = 0;\n            for (chunk = 0; chunk \u003c chunks; chunk++) {\n                for (k = 0; k \u003c abs_bytes_per_sep; k++) {\n                    c = argbuf[i++];\n                    retbuf[j++] = Py_hexdigits[c \u003e\u003e 4];\n                    retbuf[j++] = Py_hexdigits[c \u0026 0x0f];\n                }\n                retbuf[j++] = sep_char;\n            }\n            while (i \u003c arglen) {\n                c = argbuf[i++];\n                retbuf[j++] = Py_hexdigits[c \u003e\u003e 4];\n                retbuf[j++] = Py_hexdigits[c \u0026 0x0f];\n            }\n            assert(j == resultlen);\n        }\n        else {\n            i = arglen - 1;\n            j = resultlen - 1;\n            for (chunk = 0; chunk \u003c chunks; chunk++) {\n                for (k = 0; k \u003c abs_bytes_per_sep; k++) {\n                    // Crash: argbuf has been freed\n                    c = argbuf[i--];\n                    retbuf[j--] = Py_hexdigits[c \u0026 0x0f];\n                    retbuf[j--] = Py_hexdigits[c \u003e\u003e 4];\n                }\n                retbuf[j--] = sep_char;\n            }\n            while (i \u003e= 0) {\n                c = argbuf[i--];\n                retbuf[j--] = Py_hexdigits[c \u0026 0x0f];\n                retbuf[j--] = Py_hexdigits[c \u003e\u003e 4];\n            }\n            assert(j == -1);\n        }\n    }\n\n#ifdef Py_DEBUG\n    if (!return_bytes) {\n        assert(_PyUnicode_CheckConsistency(retval, 1));\n    }\n#endif\n\n    return retval;\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eSanitizer Output\u003c/strong\u003e\u003c/summary\u003e\n\n```\n==3528017==ERROR: AddressSanitizer: heap-use-after-free on address 0x51900003838f at pc 0x5d3d20ff03ec bp 0x7ffc59d34240 sp 0x7ffc59d34230\nREAD of size 1 at 0x51900003838f thread T0\n    #0 0x5d3d20ff03eb in _Py_strhex_impl Python/pystrhex.c:122\n    #1 0x5d3d20ff075d in _Py_strhex_with_sep Python/pystrhex.c:163\n    #2 0x5d3d20bd3e00 in bytearray_hex_impl Objects/bytearrayobject.c:2534\n    #3 0x5d3d20bd3f94 in bytearray_hex Objects/clinic/bytearrayobject.c.h:1714\n    #4 0x5d3d20c17703 in method_vectorcall_FASTCALL_KEYWORDS Objects/descrobject.c:421\n    #5 0x5d3d20bf7f19 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:169\n    #6 0x5d3d20bf800c in PyObject_Vectorcall Objects/call.c:327\n    #7 0x5d3d20e7628e in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1620\n    #8 0x5d3d20eba08c in _PyEval_EvalFrame Include/internal/pycore_ceval.h:121\n    #9 0x5d3d20eba380 in _PyEval_Vector Python/ceval.c:2001\n    #10 0x5d3d20eba630 in PyEval_EvalCode Python/ceval.c:884\n    #11 0x5d3d20fb183c in run_eval_code_obj Python/pythonrun.c:1365\n    #12 0x5d3d20fb1a58 in run_mod Python/pythonrun.c:1459\n    #13 0x5d3d20fb28af in pyrun_file Python/pythonrun.c:1293\n    #14 0x5d3d20fb5555 in _PyRun_SimpleFileObject Python/pythonrun.c:521\n    #15 0x5d3d20fb582b in _PyRun_AnyFileObject Python/pythonrun.c:81\n    #16 0x5d3d21006a82 in pymain_run_file_obj Modules/main.c:410\n    #17 0x5d3d21006ce9 in pymain_run_file Modules/main.c:429\n    #18 0x5d3d210084e7 in pymain_run_python Modules/main.c:691\n    #19 0x5d3d21008b77 in Py_RunMain Modules/main.c:772\n    #20 0x5d3d21008d63 in pymain_main Modules/main.c:802\n    #21 0x5d3d210090e8 in Py_BytesMain Modules/main.c:826\n    #22 0x5d3d20a8c655 in main Programs/python.c:15\n    #23 0x74a66442a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58\n    #24 0x74a66442a28a in __libc_start_main_impl ../csu/libc-start.c:360\n    #25 0x5d3d20a8c584 in _start (/home/jackfromeast/Desktop/entropy/targets/cpythonxx/3.15/python+0x2df584) (BuildId: f7e252f8868b92f2840a64868d70018a726f8bd2)\n\n0x51900003838f is located 1039 bytes inside of 1049-byte region [0x519000037f80,0x519000038399)\nfreed by thread T0 here:\n    #0 0x74a6648fc778 in realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85\n    #1 0x5d3d20cbf3b9 in _PyMem_RawRealloc Objects/obmalloc.c:85\n    #2 0x5d3d20cc15cb in _PyMem_DebugRawRealloc Objects/obmalloc.c:3010\n    #3 0x5d3d20cc190a in _PyMem_DebugRealloc Objects/obmalloc.c:3108\n    #4 0x5d3d20ce83ce in PyMem_Realloc Objects/obmalloc.c:1063\n    #5 0x5d3d20bd0c9e in bytearray_resize_lock_held Objects/bytearrayobject.c:258\n    #6 0x5d3d20bde2e6 in PyByteArray_Resize Objects/bytearrayobject.c:278\n    #7 0x5d3d20be0028 in bytearray_clear_impl Objects/bytearrayobject.c:1260\n    #8 0x5d3d20be0049 in bytearray_clear Objects/clinic/bytearrayobject.c.h:227\n    #9 0x5d3d20c1760b in method_vectorcall_NOARGS Objects/descrobject.c:448\n    #10 0x5d3d20bf7f19 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:169\n    #11 0x5d3d20bf800c in PyObject_Vectorcall Objects/call.c:327\n    #12 0x5d3d20e7628e in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1620\n    #13 0x5d3d20eba08c in _PyEval_EvalFrame Include/internal/pycore_ceval.h:121\n    #14 0x5d3d20eba380 in _PyEval_Vector Python/ceval.c:2001\n    #15 0x5d3d20bf7a52 in _PyFunction_Vectorcall Objects/call.c:413\n    #16 0x5d3d20d0a652 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:169\n    #17 0x5d3d20d0a767 in vectorcall_unbound Objects/typeobject.c:3033\n    #18 0x5d3d20d2ba5d in vectorcall_method Objects/typeobject.c:3104\n    #19 0x5d3d20d2c302 in slot_sq_length Objects/typeobject.c:10279\n    #20 0x5d3d20bc889f in PyObject_Size Objects/abstract.c:66\n    #21 0x5d3d20fefc16 in _Py_strhex_impl Python/pystrhex.c:15\n    #22 0x5d3d20ff075d in _Py_strhex_with_sep Python/pystrhex.c:163\n    #23 0x5d3d20bd3e00 in bytearray_hex_impl Objects/bytearrayobject.c:2534\n    #24 0x5d3d20bd3f94 in bytearray_hex Objects/clinic/bytearrayobject.c.h:1714\n    #25 0x5d3d20c17703 in method_vectorcall_FASTCALL_KEYWORDS Objects/descrobject.c:421\n    #26 0x5d3d20bf7f19 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:169\n    #27 0x5d3d20bf800c in PyObject_Vectorcall Objects/call.c:327\n    #28 0x5d3d20e7628e in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1620\n    #29 0x5d3d20eba08c in _PyEval_EvalFrame Include/internal/pycore_ceval.h:121\n\npreviously allocated by thread T0 here:\n    #0 0x74a6648fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69\n    #1 0x5d3d20cbf36b in _PyMem_RawMalloc Objects/obmalloc.c:63\n    #2 0x5d3d20cbe73c in _PyMem_DebugRawAlloc Objects/obmalloc.c:2887\n    #3 0x5d3d20cc1736 in _PyMem_DebugRawRealloc Objects/obmalloc.c:2963\n    #4 0x5d3d20cc190a in _PyMem_DebugRealloc Objects/obmalloc.c:3108\n    #5 0x5d3d20ce83ce in PyMem_Realloc Objects/obmalloc.c:1063\n    #6 0x5d3d20bd0c9e in bytearray_resize_lock_held Objects/bytearrayobject.c:258\n    #7 0x5d3d20bde2e6 in PyByteArray_Resize Objects/bytearrayobject.c:278\n    #8 0x5d3d20bde8ea in bytearray___init___impl Objects/bytearrayobject.c:978\n    #9 0x5d3d20bdf363 in bytearray___init__ Objects/clinic/bytearrayobject.c.h:102\n    #10 0x5d3d20d1e4a7 in type_call Objects/typeobject.c:2460\n    #11 0x5d3d20bf7d0b in _PyObject_MakeTpCall Objects/call.c:242\n    #12 0x5d3d20bf7fb3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:167\n    #13 0x5d3d20bf800c in PyObject_Vectorcall Objects/call.c:327\n    #14 0x5d3d20e7628e in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1620\n    #15 0x5d3d20eba08c in _PyEval_EvalFrame Include/internal/pycore_ceval.h:121\n    #16 0x5d3d20eba380 in _PyEval_Vector Python/ceval.c:2001\n    #17 0x5d3d20eba630 in PyEval_EvalCode Python/ceval.c:884\n    #18 0x5d3d20fb183c in run_eval_code_obj Python/pythonrun.c:1365\n    #19 0x5d3d20fb1a58 in run_mod Python/pythonrun.c:1459\n    #20 0x5d3d20fb28af in pyrun_file Python/pythonrun.c:1293\n    #21 0x5d3d20fb5555 in _PyRun_SimpleFileObject Python/pythonrun.c:521\n    #22 0x5d3d20fb582b in _PyRun_AnyFileObject Python/pythonrun.c:81\n    #23 0x5d3d21006a82 in pymain_run_file_obj Modules/main.c:410\n    #24 0x5d3d21006ce9 in pymain_run_file Modules/main.c:429\n    #25 0x5d3d210084e7 in pymain_run_python Modules/main.c:691\n    #26 0x5d3d21008b77 in Py_RunMain Modules/main.c:772\n    #27 0x5d3d21008d63 in pymain_main Modules/main.c:802\n    #28 0x5d3d210090e8 in Py_BytesMain Modules/main.c:826\n    #29 0x5d3d20a8c655 in main Programs/python.c:15\n\nSUMMARY: AddressSanitizer: heap-use-after-free Python/pystrhex.c:122 in _Py_strhex_impl\nShadow bytes around the buggy address:\n  0x519000038100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd\n  0x519000038180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd\n  0x519000038200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd\n  0x519000038280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd\n  0x519000038300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd\n=\u003e0x519000038380: fd[fd]fd fd fa fa fa fa fa fa fa fa fa fa fa fa\n  0x519000038400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n  0x519000038480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00\n  0x519000038500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00\n  0x519000038580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00\n  0x519000038600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00\nShadow byte legend (one shadow byte represents 8 application bytes):\n  Addressable:           00\n  Partially addressable: 01 02 03 04 05 06 07 \n  Heap left redzone:       fa\n  Freed heap region:       fd\n  Stack left redzone:      f1\n  Stack mid redzone:       f2\n  Stack right redzone:     f3\n  Stack after return:      f5\n  Stack use after scope:   f8\n  Global redzone:          f9\n  Global init order:       f6\n  Poisoned by user:        f7\n  Container overflow:      fc\n  Array cookie:            ac\n  Intra object redzone:    bb\n  ASan internal:           fe\n  Left alloca redzone:     ca\n  Right alloca redzone:    cb\n==3528017==ABORTING\n```\n\u003c/details\u003e\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-143209\n* gh-143219\n* gh-143220\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/jackfromeast","@type":"Person","name":"jackfromeast"},"datePublished":"2025-12-26T21:20:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/143195/cpython/issues/143195"}