René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`_io.TextIOWrapper.write`: write during flush causes `pending_bytes` length mismatch","articleBody":"# Crash report\n\n### What happened?\n\nBisected to #24592.\nSimple repro:\n```python\nimport _io\n\nclass MyIO(_io.BytesIO):\n    def __init__(self):\n        _io.BytesIO.__init__(self)\n        self.writes = []\n\n    def write(self, b):\n        self.writes.append(b)\n        tw.write(\"c\")\n        return len(b)\n\nbuf = MyIO()\ntw = _io.TextIOWrapper(buf)\n\nCHUNK_SIZE = 8192\n\ntw.write(\"a\" * (CHUNK_SIZE - 1))\ntw.write(\"b\" * 2)\n\ntw.flush()\n\nassert b''.join(tw.buffer.writes) == b\"a\" * (CHUNK_SIZE - 1) + b\"b\" * 2 + b\"c\"\n```\nOn debug build it causes C assertion failure:\n```\npython: ./Modules/_io/textio.c:1582: _textiowrapper_writeflush: Assertion `PyUnicode_GET_LENGTH(pending) == self-\u003epending_bytes_count' failed.\n\nProgram received signal SIGABRT, Aborted.\n__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50\n50\t../sysdeps/unix/sysv/linux/raise.c: No such file or directory.\n(gdb) bt\n#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50\n#1  0x00007ffff7c84537 in __GI_abort () at abort.c:79\n#2  0x00007ffff7c8440f in __assert_fail_base (fmt=0x7ffff7dfb688 \"%s%s%s:%u: %s%sAssertion `%s' failed.\\n%n\", \n    assertion=0x555555a24d40 \"PyUnicode_GET_LENGTH(pending) == self-\u003epending_bytes_count\", file=0x555555a25332 \"./Modules/_io/textio.c\", line=1582, \n    function=\u003coptimized out\u003e) at assert.c:92\n#3  0x00007ffff7c93662 in __GI___assert_fail (assertion=assertion@entry=0x555555a24d40 \"PyUnicode_GET_LENGTH(pending) == self-\u003epending_bytes_count\", \n    file=file@entry=0x555555a25332 \"./Modules/_io/textio.c\", line=line@entry=1582, \n    function=function@entry=0x555555a256b0 \u003c__PRETTY_FUNCTION__.9\u003e \"_textiowrapper_writeflush\") at assert.c:101\n#4  0x00005555559102b9 in _textiowrapper_writeflush (self=self@entry=0x7ffff77896d0) at ./Modules/_io/textio.c:1582\n#5  0x000055555591065d in _io_TextIOWrapper_flush_impl (self=0x7ffff77896d0) at ./Modules/_io/textio.c:3092\n#6  0x0000555555910791 in _io_TextIOWrapper_flush (self=\u003coptimized out\u003e, _unused_ignored=\u003coptimized out\u003e) at ./Modules/_io/clinic/textio.c.h:1105\n#7  0x0000555555693483 in method_vectorcall_NOARGS (func=0x7ffff7731250, args=0x7ffff7fc1070, nargsf=\u003coptimized out\u003e, kwnames=\u003coptimized out\u003e)\n    at Objects/descrobject.c:447\n#8  0x0000555555680d7c in _PyObject_VectorcallTstate (tstate=0x555555be4678 \u003c_PyRuntime+294136\u003e, callable=0x7ffff7731250, args=0x7ffff7fc1070, \n    nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:168\n#9  0x0000555555680e97 in PyObject_Vectorcall (callable=callable@entry=0x7ffff7731250, args=args@entry=0x7ffff7fc1070, nargsf=\u003coptimized out\u003e, \n    kwnames=kwnames@entry=0x0) at Objects/call.c:327\n#10 0x000055555580876d in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555be4678 \u003c_PyRuntime+294136\u003e, frame=0x7ffff7fc1020, throwflag=throwflag@entry=0)\n    at Python/generated_cases.c.h:813\n...\n```\n\nIf `_io.TextIOWrapper.write()` tries to store more than `self-\u003echunk_size` data in `self-\u003epending_bytes`, it calls `_textiowrapper_writeflush()`:\nhttps://github.com/python/cpython/blob/b48a3dbff4d70e72797e67b46276564fc63ddb89/Modules/_io/textio.c#L1726-L1733\n`_textiowrapper_writeflush()` flushes `self-\u003epending_bytes` contents to wrapped buffer through `write()` method call:\nhttps://github.com/python/cpython/blob/b48a3dbff4d70e72797e67b46276564fc63ddb89/Modules/_io/textio.c#L1621-L1628\nThe problem is that call to `write()` method can cause `_io.TextIOWrapper.write()` call (directly, as in repro, or from other thread), which re-sets `self-\u003epending_bytes` and `self-\u003epending_bytes_count` values.\n\n### CPython versions tested on:\n\n3.10, 3.11, 3.12, CPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.14.0a0 (heads/main:e94dbe4ed8, May 24 2024, 00:47:49) [GCC 10.2.1 20210110]\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-119507\n* gh-119964\n* gh-119965\n* gh-120314\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/chgnrdv","@type":"Person","name":"chgnrdv"},"datePublished":"2024-05-24T13:34:26.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/119506/cpython/issues/119506"}