René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Nested multiprocessing leads to `AttributeError: is_fork_ctx` with `forkserver` or `spawn` methods","articleBody":"# Bug report\n\n### Checklist\n\n- [X] I am confident this is a bug in CPython, not a bug in a third-party project\n- [X] I have searched the [CPython issue tracker](https://github.com/python/cpython/issues?q=is%3Aissue+sort%3Acreated-desc),\nand am confident this bug has not been reported before\n\n\n### CPython versions tested on:\n\n3.11\n\n### Operating systems tested on:\n\nLinux\n\n### Output from running 'python -VV' on the command line:\n\nPython 3.11.5 (main, Aug 26 2023, 00:26:34) [GCC 12.2.1 20220924]\n\n### A clear and concise description of the bug:\n\nUsing nested `multiprocessing` (i.e. spawn a child process inside a child process) is broken as of Python 3.11.5, leading to an attribute error.\n\n```python\nProcess Process-1:\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/multiprocessing/process.py\", line 314, in _bootstrap\n    self.run()\n  File \"/usr/local/lib/python3.11/multiprocessing/process.py\", line 108, in run\n    self._target(*self._args, **self._kwargs)\n  File \"/io/pyi_multiprocessing_nested_process.py\", line 15, in process_function\n    process.start()\n  File \"/usr/local/lib/python3.11/multiprocessing/process.py\", line 121, in start\n    self._popen = self._Popen(self)\n                  ^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/multiprocessing/context.py\", line 224, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/multiprocessing/context.py\", line 288, in _Popen\n    return Popen(process_obj)\n           ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py\", line 32, in __init__\n    super().__init__(process_obj)\n  File \"/usr/local/lib/python3.11/multiprocessing/popen_fork.py\", line 19, in __init__\n    self._launch(process_obj)\n  File \"/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py\", line 47, in _launch\n    reduction.dump(process_obj, fp)\n  File \"/usr/local/lib/python3.11/multiprocessing/reduction.py\", line 60, in dump\n    ForkingPickler(file, protocol).dump(obj)\n  File \"/usr/local/lib/python3.11/multiprocessing/synchronize.py\", line 106, in __getstate__\n    if self.is_fork_ctx:\n       ^^^^^^^^^^^^^^^^\nAttributeError: 'Lock' object has no attribute 'is_fork_ctx'\nResults: [1]\n```\n\nMinimal code example below. Invoke with argument `fork`, `forkserver` or `spawn`. `fork` will work. `forkserver` and `spawn` will both raise the above error. All three variants work with Python 3.11.4.\n\n```python\nimport sys\nimport multiprocessing\n\n\ndef nested_process_function(queue):\n    print(\"Running nested sub-process!\")\n    queue.put(2)\n\n\ndef process_function(queue):\n    print(\"Running sub-process!\")\n    queue.put(1)\n\n    process = multiprocessing.Process(target=nested_process_function, args=(queue,))\n    process.start()\n    process.join()\n\n\ndef main(start_method):\n    multiprocessing.set_start_method(start_method)\n    queue = multiprocessing.Queue()\n\n    process = multiprocessing.Process(target=process_function, args=(queue,))\n    process.start()\n    process.join()\n\n    results = []\n    while not queue.empty():\n        results.append(queue.get())\n\n    print(f\"Results: {results}\")\n    assert results == [1, 2]\n\n\nif __name__ == '__main__':\n    if len(sys.argv) != 2:\n        raise SystemExit(f\"Usage: {sys.argv[0]} fork|forkserver|spawn\")\n\n    main(sys.argv[1])\n```\n\nI believe that the source of this regression is https://github.com/python/cpython/commit/34ef75d3ef559288900fad008f05b29155eb8b59 which adds the attribute `is_fork_ctx` to `multiprocessing.Lock()` but doesn't update the pickle methods (`__getstate__()` and `__setstate__()`) so after being serialised and deserialised, the `Lock()` object looses that attribute.\n\nThe following patch, adding `is_fork_ctx` to the pickle methods, makes the above work again.\n```diff\ndiff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py\nindex 2328d33212..9c5c2aada6 100644\n--- a/Lib/multiprocessing/synchronize.py\n+++ b/Lib/multiprocessing/synchronize.py\n@@ -109,10 +109,11 @@ def __getstate__(self):\n                                    'not supported. Please use the same context to create '\n                                    'multiprocessing objects and Process.')\n             h = sl.handle\n-        return (h, sl.kind, sl.maxvalue, sl.name)\n+        return (h, sl.kind, sl.maxvalue, sl.name, self.is_fork_ctx)\n \n     def __setstate__(self, state):\n-        self._semlock = _multiprocessing.SemLock._rebuild(*state)\n+        self._semlock = _multiprocessing.SemLock._rebuild(*state[:4])\n+        self.is_fork_ctx = state[4]\n         util.debug('recreated blocker with handle %r' % state[0])\n         self._make_methods()\n```\n\n```[tasklist]\n### Tasks\n```\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-108568\n* gh-108691\n* gh-108692\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/bwoodsend","@type":"Person","name":"bwoodsend"},"datePublished":"2023-08-26T18:14:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/108520/cpython/issues/108520"}