René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[Bug]: Memory not freed as expected after plotting heavy plot involving looping","articleBody":"### Solution\r\nhttps://github.com/matplotlib/matplotlib/issues/27138#issuecomment-1771141850\r\n\r\n### Bug summary\r\n\r\nI work with a large 1D dataset. For a statistical analysis, I am using the bootstrap method, resampling it many times.\r\n\r\nI am interested in **looping over all cases** in order to put together on a single figure a specific result for all resamplings. \r\n**Memory issues take place though (e.g. not freed before the very end of the script, or even leaks).**\r\n\r\nHere I document some things that at least partially address the issue. None is fully satisfactory though. \r\nI am running the same script both from Python and as a Jupyter notebook (synchronised via jupytext). I am trying to get rid of the memory issues in both cases (the RAM usage easily reaches 16–32 GB once I start playing with enough data).\r\n\r\n### Code for reproduction\r\n\r\n```python\r\nimport matplotlib.pyplot as plt\r\nimport numpy as np\r\n\r\nda = np.sort(np.random.random(int(3e6))) # Beware: please lower this if your system has less than 32 GB\r\n\r\ndef custom_plot(da, **kwargs):\r\n    \"\"\"da is a 1-D ordered xarray.DataArray or numpy array (containing tons of data)\"\"\"\r\n    plt.yscale('log')\r\n    n = len(da)\r\n    return plt.plot(da, 1 - (np.arange(1, n+1) / (n+1)), **kwargs);\r\n\r\ndef resampling_method(da, case):\r\n    \"\"\"\r\n    A complex thing in reality but for the MWE, let us simply return da itself.\r\n    It will lead to the same memory problem.\r\n    \"\"\"\r\n    return da\r\n\r\nplt.figure(figsize=(15, 8), dpi=150)\r\nplt.ylim((1e-10,1))\r\n\r\nfor case in np.arange(50):\r\n    custom_plot(resampling_method(da, case)) # each time getting the curve for a different resampling of da\r\ncustom_plot(da) # curve for the original da\r\n\r\nplt.savefig(\"output.png\")\r\nplt.show()\r\n\r\nimport gc\r\ngc.collect();\r\n\r\nprint(\"Technically the programme would continue with more calculations.\")\r\nprint(\"Notice how the memory won't be freed however until the entire script is finished.\")\r\nimport time\r\ntime.sleep(120) # This simulates the fact that the programme continues with other calculations.\r\nprint(\"Now the programme exits\")\r\n```\r\n\r\n\r\n### Actual outcome\r\n\r\nMemory issues are taking place no matter what I've tried so far. Depending on what is being attempted, it can lead to the memory either not being freed after the plot has been shown/is closed, or even memory leaks and massive swap usage.\r\n\r\n### Expected outcome\r\n\r\nMemory freed well before the end of the programme. I would expect it to be freed soon after the figure is closed.\r\n\r\n### Additional information\r\n\r\nNB: I did also try many other things (incl. `plt.cla` and the like), as well as changing backend (notably \"Agg\" and \"Qt5Agg\") but that did not solve the problem in the slightest, so I won't document them.\r\n\r\n#### Things that have some effect\r\n\r\n1) If you do `plt.show()`\r\n- It will show the plot as a new window when run from the terminal with Python but the memory usage related to the figure won't be freed after that. It will remain in use until the end of the entire script.\r\n- it will be freed in Jupyter soon after displaying the figure however.\r\n```python\r\nplt.show()\r\nimport gc\r\ngc.collect();\r\n```\r\n\r\n2) If you use `block=False`, `time.sleep` and `close('all')`, the memory will be freed after the plot has been created both with Jupyter and Python. However, in Python, a window will be created (stealing focus) and nothing will ever appear in it (it will be closed after 5 seconds). It'd therefore be tempting to comment out `plt.show(block=False)` but if you do, Jupyter will no longer clear the memory...\r\n```python\r\nplt.show(block=False) ## If you comment this out, then Jupyter will not clear the memory..\r\nimport time\r\ntime.sleep(5)\r\nplt.close('all')\r\nimport gc\r\ngc.collect();\r\n```\r\n\r\n3) Given what precedes, let us check whether Jupyter or Python is being used.\r\n```python\r\n\r\ndef type_of_script():\r\n    \"\"\"source: https://stackoverflow.com/a/47428575/452522\"\"\"\r\n    try:\r\n        ipy_str = str(type(get_ipython()))\r\n        if 'zmqshell' in ipy_str:\r\n            return 'jupyter'\r\n        if 'terminal' in ipy_str:\r\n            return 'ipython'\r\n    except:\r\n        return 'terminal'\r\n\r\nif type_of_script() == \"jupyter\":\r\n    plt.show(block=False) ## If you comment this out, then Jupyter will not clear the memory..\r\nelse:\r\n    pass\r\nimport time\r\ntime.sleep(5)\r\nplt.close('all')\r\nimport gc\r\ngc.collect();\r\n```\r\nWith this:\r\n- Jupyter will create a file and will also display the figure inline.\r\n- Python will only create a file and won't try to show a window\r\nBoth will clean the memory after that figure has been closed (or 5 seconds after rather).\r\n**This is the most satisfactory one... not exactly nice though.**\r\nShould one want to have the figure displayed when running python from CLI however, I haven't found a method\r\nwhere the memory wouldn't remain in use until the very end of the entire script.\r\n\r\n#### Some further notes:\r\n- There are known memory issues with matplotlib and looping, such as: http://datasideoflife.com/?p=1443\r\nbut here I do not create a at each iteration of a loop, but accumulate plots from a loop and plot the end result. The solution put forward there (i.e. use `plt.close(fig)` does not work in this case).\r\n\r\n- This is also distinct from https://github.com/matplotlib/matplotlib/issues/20300\r\n\r\n### Operating system\r\n\r\nUbuntu\r\n\r\n### Matplotlib Version\r\n\r\n3.7.3\r\n\r\n### Matplotlib Backend\r\n\r\nmodule://matplotlib_inline.backend_inline (default)\r\n\r\n### Python version\r\n\r\n3.8.10\r\n\r\n### Jupyter version\r\n\r\n6.5.2\r\n\r\n### Installation\r\n\r\npip","author":{"url":"https://github.com/spacescientist","@type":"Person","name":"spacescientist"},"datePublished":"2023-10-18T12:47:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":22},"url":"https://github.com/27138/matplotlib/issues/27138"}