René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"\"Spawn\" locks and semaphores are crashing during pickling in correct \"spawn\" process launch. ","articleBody":"# Bug report\r\n\r\n### Bug description:\r\n\r\n\"Spawn\" locks and semaphores are crashing during the process argument pickling. \r\n\r\nThis isssue is new as of python3.11, earlier versions do not have this issue. The code change that triggered the issue seems to be this pull request  https://github.com/python/cpython/pull/108378 that was backported to python3.11 and python3.12, but unfortunately does not seem to have correct semantics.\r\n\r\n```python\r\nimport multiprocessing as mp\r\n\r\n\r\ndef sub_task(lock):\r\n    with lock['lock']:\r\n        pass\r\n\r\n\r\ndef main():\r\n    ctx = mp.get_context(\"spawn\")\r\n    lock = ctx.Lock()\r\n    lock_dict = {\"lock\":lock}\r\n    proc = ctx.Process(target=sub_task, args=(lock_dict,))\r\n    proc.start()\r\n    proc.join()\r\n\r\n```\r\nGives the error:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/ben/work/training/pipeline-lib/test/test_execution.py\", line 59, in raises_from\r\n    yield\r\n  File \"/home/ben/work/training/pipeline-lib/test/test_execution.py\", line 354, in test_single_worker_unexpected_exit\r\n    execute(tasks, parallelism)\r\n  File \"/home/ben/work/training/pipeline-lib/pipeline_lib/execution.py\", line 49, in execute\r\n    execute_mp(tasks, \"spawn\", inactivity_timeout=inactivity_timeout)\r\n  File \"/home/ben/work/training/pipeline-lib/pipeline_lib/mp_execution.py\", line 658, in execute_mp\r\n    process.start()\r\n  File \"/usr/lib/python3.11/multiprocessing/process.py\", line 121, in start\r\n    self._popen = self._Popen(self)\r\n                  ^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/multiprocessing/context.py\", line 288, in _Popen\r\n    return Popen(process_obj)\r\n           ^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/multiprocessing/popen_spawn_posix.py\", line 32, in __init__\r\n    super().__init__(process_obj)\r\n  File \"/usr/lib/python3.11/multiprocessing/popen_fork.py\", line 19, in __init__\r\n    self._launch(process_obj)\r\n  File \"/usr/lib/python3.11/multiprocessing/popen_spawn_posix.py\", line 47, in _launch\r\n    reduction.dump(process_obj, fp)\r\n  File \"/usr/lib/python3.11/multiprocessing/reduction.py\", line 60, in dump\r\n    ForkingPickler(file, protocol).dump(obj)\r\n  File \"/usr/lib/python3.11/multiprocessing/synchronize.py\", line 107, in __getstate__\r\n    raise RuntimeError('A SemLock created in a fork context is being '\r\nRuntimeError: A SemLock created in a fork context is being shared with a process in a spawn context. This is not supported. Please use the same context to create multiprocessing objects and Process.\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"\u003cfrozen runpy\u003e\", line 198, in _run_module_as_main\r\n  File \"\u003cfrozen runpy\u003e\", line 88, in _run_code\r\n  File \"/home/ben/work/training/pipeline-lib/test/test_execution.py\", line 641, in \u003cmodule\u003e\r\n    test_single_worker_unexpected_exit(\"process-spawn\")\r\n  File \"/home/ben/work/training/pipeline-lib/test/test_execution.py\", line 353, in test_single_worker_unexpected_exit\r\n    with raises_from(pipeline_lib.pipeline_task.TaskError):\r\n  File \"/usr/lib/python3.11/contextlib.py\", line 158, in __exit__\r\n    self.gen.throw(typ, value, traceback)\r\n  File \"/home/ben/work/training/pipeline-lib/test/test_execution.py\", line 66, in raises_from\r\n    raise AssertionError(f\"expected error of type {err_type} got error {err}\")\r\nAssertionError: expected error of type \u003cclass 'pipeline_lib.pipeline_task.TaskError'\u003e got error A SemLock created in a fork context is being shared with a process in a spawn context. This is not supported. Please use the same context to create multiprocessing objects and Process.\r\nTraceback (most recent call last):\r\n  File \"\u003cstring\u003e\", line 1, in \u003cmodule\u003e\r\n  File \"/usr/lib/python3.11/multiprocessing/spawn.py\", line 122, in spawn_main\r\n    exitcode = _main(fd, parent_sentinel)\r\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/multiprocessing/spawn.py\", line 132, in _main\r\n    self = reduction.pickle.load(from_parent)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/multiprocessing/synchronize.py\", line 115, in __setstate__\r\n    self._semlock = _multiprocessing.SemLock._rebuild(*state)\r\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nFileNotFoundError: [Errno 2] No such file or directory\r\n```\r\n\r\n\r\n### CPython versions tested on:\r\n\r\n3.11\r\n\r\n### Operating systems tested on:\r\n\r\nLinux","author":{"url":"https://github.com/ben-black-tec","@type":"Person","name":"ben-black-tec"},"datePublished":"2024-10-19T22:47:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/125738/cpython/issues/125738"}