René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"3.12 regression: interpreter shouldn't shut down when there are non-daemon threads left","articleBody":"# Bug report\r\n\r\n### Bug description:\r\n\r\nThis is related to #113964, but trying to address the bigger problem of what \"interpreter shutdown\" is, and trying to make a stronger case for restoring the old behavior.\r\n\r\nTL;DR: This was an undocumented change. The old behavior was useful. Documentation is contradictory. Users are struggling and are recommended to downgrade.\r\n\r\n### Undocumented change\r\n\r\nIn Python 3.11 and previous versions, `daemon=False` threads could spawn new threads even after the main thread has exited.\r\n\r\nThis changed in Python 3.12, which throws \"**RuntimeError: can't create new thread at interpreter shutdown**\" when a `daemon=False` thread tries to spawn new threads after the main thread has exited.\r\n\r\nI could not find this change to be documented in 3.12 release notes, it appears to be an unintentional change.\r\n\r\n### Reproducer\r\n\r\n```python\r\n\"\"\"Python 3.11: Successfully spawns 20 subthreads. Python 3.12: RuntimeError.\"\"\"\r\nfrom threading import Thread, current_thread\r\nfrom time import sleep\r\n\r\ndef subthread():\r\n    print(f\"Thread {current_thread().name} spawned...\")\r\n\r\ndef thread():\r\n    for i in range(20):\r\n        Thread(target=subthread, daemon=False).start()\r\n        sleep(0.001)\r\n\r\nThread(target=thread, daemon=False).start()\r\n```\r\n\r\n### Old behavior was useful\r\n\r\nWhile not `join()`ing the launched threads is considered an antipattern, at the very least the old behavior was useful for throw-away scripts that launch a bunch of threads, and have Python exit automatically once all work was finished.\r\n\r\n### Documentation is contradictory\r\n\r\nThe [documentation says this about daemon threads](https://docs.python.org/3/library/threading.html#thread-objects):\r\n\r\n\u003e A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program **exits when only daemon threads are left**.\r\n\r\nI think a useful interpretation of \"exits when only daemon threads are left\" also implies \"the Python program starts shutting down when only daemon threads are left\", and does not shut down before. Because what else could \"exits\" mean if not \"shuts down\"?\r\n\r\nAlthough this is in conflict with [the definition of \"interpreter shutdown\"](https://docs.python.org/3/glossary.html#term-interpreter-shutdown) that states:\r\n\r\n\u003e The main reason for interpreter shutdown is that the `__main__` module or the script being run has finished executing.\r\n\r\nThis interpetation is also not being respected by `sys.is_finalizing()` API. This is supposed to \"Return True if the Python interpreter is shutting down\", but it returns False in this situation even after the main thread has exited and thread spawning fails: #114570. That's another indication that this change was unintentional.\r\n\r\n### Users are struggling\r\n\r\nPython has a reputation for breakage across version upgrades. I hope there is motivation among CPython developers to change that. I think such unintentional changes that break users, should be treated as regressions to be fixed.\r\n\r\nThere are three posts on StackOverflow [[1]](https://stackoverflow.com/questions/77631477/runtimeerror-cant-create-new-thread-at-interpreter-shutdown-python-3-12) [[2]](https://stackoverflow.com/questions/77862839/runtimeerror-cant-create-new-thread-at-interpreter-shutdown-while-it-works-per) [[3]](https://stackoverflow.com/questions/77738120/runtimeerror-cant-create-new-thread-at-interpreter-shutdown), where downgrading Python is the dominant \"solution\". And when you have threads spawning more threads, the \"proper fix\" is often not a simple matter of adding a `Thread.join()` somewhere, but may require re-architecting the whole program to keep track of what all threads are doing.\r\n\r\nRelated issues: #113964, #114570.\r\n\r\n### CPython versions tested on:\r\n\r\n3.12.2, 3.11.7\r\n\r\n### Operating systems tested on:\r\n\r\nLinux, macOS","author":{"url":"https://github.com/intgr","@type":"Person","name":"intgr"},"datePublished":"2024-02-15T17:58:12.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":10},"url":"https://github.com/115533/cpython/issues/115533"}