René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition)","articleBody":"The ``test_interpreter_shutdown()`` test of ``test_concurrent_futures.test_shutdown`` has a race condition. On purpose, the test doesn't wait until the executor completes (!). Moreover, it **expects** the executor to always be able to submit its job, and the job to complete successfully! It's a very optimistic bet.\r\n\r\nSee also issue #107219: test_concurrent_futures: **test_crash_big_data()** hangs randomly on **Windows**.\r\n\r\n---\r\n\r\nWhen Python is shutting down, ``Py_Finalize()`` quickly blocks the creation of new threads in ``_thread.start_new_thread()``:\r\n\r\nhttps://github.com/python/cpython/blob/3bfa24e29f286cbc1f42bdb4d2b1c0c9d643c8d6/Modules/_threadmodule.c#L1161-L1165\r\n\r\nThis exception was added recently (last June) by commit ce558e69d4087dd3653207de78345fbb8a2c7835: see issue gh-104690 for the rationale.\r\n\r\nThe multiprocessing executor spawns ``_ExecutorManagerThread`` thread which runs its \"main loop\" in its ``run()`` method:\r\n\r\nhttps://github.com/python/cpython/blob/3bfa24e29f286cbc1f42bdb4d2b1c0c9d643c8d6/Lib/concurrent/futures/process.py#L335-L378\r\n\r\nIt tries to submit new jobs to the worker process through a queue, but oops, the Python main thread is finalizing (called ``Py_Finalizing()``)! There is not notification system to notify threads that Python is being finalized.\r\n\r\n---\r\n\r\nMoreover, there are **3** \"finalization\" states:\r\n\r\n* ``interp-\u003efinalizing`` -- used by ``_thread.start_new_thread()`` to block thread creation during Python finazlization\r\n* ``runtime-\u003e_finalizing`` -- used by ``sys.is_finalizing()``, ``Py_IsFinalizing()`` and ``_PyRuntimeState_GetFinalizing(runtime)``\r\n* ``interp-\u003e_finalizing`` -- used by ``ceval.c`` to decide if a Python thread \"must exit\" or not, as soon as it's set, all Python threads must exit as soon as they **attempt to acquire the GIL**\r\n\r\nThese 3 states at not set at the same time.\r\n\r\n1. Calling ``Py_Finalize()`` sets ``interp-\u003efinalizing`` to 1 as soon as possible: so spawning new threads is *immediately* blocked (which is a good thing to get a reliable finalization!)\r\n2. ``Py_Finalize()`` calls ``threading._shutdown()`` which blocks until all non-daemon threads completes\r\n3. ``Py_Finalize()`` calls ``atexit`` callbacks\r\n4. And only then, ``Py_Finalize()`` sets ``runtime-\u003e_finalizing`` *and* ``interp-\u003e_finalizing`` to the Python thread state (``tstate``) which calls ``Py_Finalize()``\r\n\r\nThe delay between (1) and (4) can be quite long, a thread can take several milliseconds, if not seconds, to complete.\r\n\r\n---\r\n\r\nCan ``multiprocessing`` or ``concurrent.futures`` check if Python is finalizing or be notified? Well, did you hear about [Time-of-check to time-of-use](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use) race conditions? Even if it would be possible, I don't think that we can \"check\" if it's safe to spawn a thread *just before* spawning a thread, since the main thread can decide to finalize Python \"at any time\". It will become even more tricky with Python nogil ;-)\r\n\r\nSo what's left? Well, ``multiprocessing`` and ``concurrent.futures`` should be optimistic, call Python functions and only *then* check for exceptions. Depending on the exceptions, they can decide how to handle it. I would suggest to exit as soon as possible, and try to cleanup resources if possible.\r\n\r\nAnother option would be to make ``multiprocessing`` and ``concurrent.futures`` more determistic. Rather than spawning threads and processes in the background \"on demand\" and hope that everything will be fine, add more synchronization to \"wait\" until everything is ready to submit jobs. I think that I already tried this approach in the past, but @pitrou didn't like it since it made some workloads slower. You may not always need to actually submits jobs. You may not always need all threads and processes.\r\n\r\nWell, I don't know even these complex modules to tell which option is the least bad :-)\r\n\r\nFinally, as usually, I beg you to make these APIs less magical, and enforce more explicit resources management! It shouldn't even be possible to not wait until an executor complete. It should be enforced by emitting loudly ``ResourceWarning`` warnings :-) Well, that's my opinion. I know that it's not shared by @pitrou :-)\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-109809\n* gh-109810\n* gh-110126\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/vstinner","@type":"Person","name":"vstinner"},"datePublished":"2023-09-07T02:15:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/109047/cpython/issues/109047"}