René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Unfixable potential race conditions on file descriptors in CPython","articleBody":"Python allows multiple threads to concurrently access file descriptors through [files](https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files) and [sockets](https://docs.python.org/3/library/socket.html). Race conditions at the Python level can lead to unexpected behavior, even with the global interpreter lock.\r\n\r\nThread sanitizer reports these [races](https://github.com/google/sanitizers/wiki/ThreadSanitizerPopularDataRaces#race-on-a-file-descriptor) in some of our tests that exercise this behavior. We cannot fix these potential race conditions without introducing potential deadlocks.\r\n\r\nFor example, consider:\r\n\r\n```python\r\nimport threading\r\n\r\nsecrets = open(\"secrets.txt\", \"w\");\r\n\r\ndef thread1():\r\n    secrets.write(\"a secret\")\r\n\r\ndef thread2():\r\n    secrets.close()\r\n\r\ndef thread3():\r\n    with open(\"log.txt\", \"w\") as log:\r\n        log.write(\"log message\")\r\n\r\nthreading.Thread(target=thread1).start()\r\nthreading.Thread(target=thread2).start()\r\nthreading.Thread(target=thread3).start()\r\n```\r\n\r\nIf you are particularly unlucky, then the file descriptor for `secrets` may be closed by `thread2` and **reused** as the file descriptor for `log` just before `thread1` writes to it. In other words, `thread1` may write \"a secret\" to a completely different file or socket than it intended.\r\n\r\nThis can happen, **even with the GIL**, because the GIL is released around [`write()`](https://github.com/python/cpython/blob/9c08f40a613d9aee78de4ce4ec3e125d1496d148/Python/fileutils.c#L1951-L1972) and [`close()`](https://github.com/python/cpython/blob/9c08f40a613d9aee78de4ce4ec3e125d1496d148/Modules/_io/fileio.c#L122-L128). In other words, the following can happen:\r\n\r\n1. The `secrets.write()` call releases the GIL, but before it actually calls the C `write()` function on the file descriptor....\r\n2. The `secrets.close()` call closes the file descriptor\r\n3. The `open(\"log.txt\", \"w\")` re-uses the same file descriptor number \r\n4. The `secrets.write()` call now `write()` on the wrong file descriptor\r\n\r\nNote that we must release the GIL before calling `write()` or `close()` because these functions potentially block.","author":{"url":"https://github.com/colesbury","@type":"Person","name":"colesbury"},"datePublished":"2024-07-09T17:44:41.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/121544/cpython/issues/121544"}