René's URL Explorer Experiment

Title: bpo-36867: Create the resource_tracker before launching SharedMemoryManagers by pierreglaser · Pull Request #13276 · python/cpython · GitHub

Open Graph Title: bpo-36867: Create the resource_tracker before launching SharedMemoryManagers by pierreglaser · Pull Request #13276 · python/cpython

X Title: bpo-36867: Create the resource_tracker before launching SharedMemoryManagers by pierreglaser · Pull Request #13276 · python/cpython

Description: TLDR; The resource_tracker needs to be created before SharedMemoryManager processes are launched (which is not the case for managers created using fork) Take this example below: from multiprocessing.managers import SharedMemoryManager from multiprocessing import resource_tracker smm = SharedMemoryManager() smm.start() sl = smm.ShareableList(range(10)) smm.shutdown() This simple and legitimate python scripts results in a very noisy output: /home/pierreglaser/repos/cpython/Lib/multiprocessing/resource_tracker.py:198: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/pierreglaser/repos/cpython/Lib/multiprocessing/resource_tracker.py:211: UserWarning: resource_tracker: '/psm_3bfa242d': [Errno 2] No such file or directory: '/psm_3bfa242d' warnings.warn('resource_tracker: %r: %s' % (name, e)) Here is why it happens: smm.start() launches a new manager process. At this point in the main process, no resource_tracker is created if the manager is launched using fork. sl = smm.ShareableList(range(10)) does two things: First, it creates a new shared_memory object from the parent process. At creation, a resource_tracker.register call is done -> resource_tracker.ensure_running is done -> a new resource_tracker process is created. In addition, this resource_tracker now tracks sl. However, since the manager process was created before, it does not know that a new resource_tracker process was just created in the parent process.. Once sl is created, the SharedMemoryManager asks its SharedMemoryTracker ( which lives in the manager process) to track it. smm.shutdown shuts down the manager. It thus asks the SharedMemoryTracker to destroy sl, which it tracks. To do so, it first opens it (using a SharedMemory(name)) call, which itself triggers a resource_tracker.register call -> resource_tracker.ensure_running call in the manager process, where no resource_tracker exists yet! Thus, another ResourceTracker instance /process is created and starts tracking sl immediatly after, it unlinks sl. This sends an unregister call to the newly created resource_tracker. When the script ends however, nothing has been done to tell the original resource_tracker created in the parent process that sl was properly unlinked. The resource_tracker thus thinks a leak happened, and tells the user (first line of my output)). However, there was no leak, as sl was properly unlinked by the manager. Therefore, the cleanup attempt of the resource_tracker fails, resulting the second error. The solution to this problem is simply to create the right before the manager process is created. https://bugs.python.org/issue36867

Open Graph Description: TLDR; The resource_tracker needs to be created before SharedMemoryManager processes are launched (which is not the case for managers created using fork) Take this example below: from multiprocessin...

X Description: TLDR; The resource_tracker needs to be created before SharedMemoryManager processes are launched (which is not the case for managers created using fork) Take this example below: from multiprocessin...

Opengraph URL: https://github.com/python/cpython/pull/13276

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width