René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Dict memory leak in Python 3.12.x / msgpack 1.0.x","articleBody":"Hello,\r\n\r\nI think I've found a second Python 3.12 memory leak (having read the issue for the other one, and tried upgrading to msgpack 1.0.8 to fix).\r\n\r\nThis issue occurs with dicts, and key uniqueness seems to be a major consideration. (A bunch of dicts with the exact same keys don't cause the problem -- some kind of key caching issue maybe?)\r\n\r\nHere's a repro:\r\n\r\n```\r\nimport psutil\r\nimport msgpack\r\n\r\n# Set up a list of packed dicts\r\np = [\r\n    msgpack.packb(\r\n        {f'{p}.{r}': True for r in range(1000)}\r\n    )\r\n    for p in range(1000)\r\n]\r\n\r\n# Check initial state\r\nprint(f'initial mem: {psutil.Process().memory_info().rss}')\r\n\r\n# Loop and unpack\r\ni = 0\r\nfor x in p:\r\n    y = msgpack.unpackb(x)\r\n    i += 1\r\n    if i % 100 == 0:\r\n        print(f'subsequent mem: {psutil.Process().memory_info().rss}')\r\n```\r\n\r\nOutput, msgpack 1.0.8 on Python 3.12.3:\r\n```\r\ninitial mem: 24571904\r\nsubsequent mem: 34029568\r\nsubsequent mem: 42627072\r\nsubsequent mem: 47493120\r\nsubsequent mem: 60088320\r\nsubsequent mem: 64684032\r\nsubsequent mem: 69550080\r\nsubsequent mem: 89878528\r\nsubsequent mem: 94744576\r\nsubsequent mem: 99340288\r\nsubsequent mem: 104206336\r\n```\r\n\r\nOutput, msgpack 0.6.2 on Python 3.12.3 or msgpack 1.0.8 on Python 3.11.9 (slightly different numbers but same effect):\r\n```\r\ninitial mem: 22761472\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\nsubsequent mem: 23089152\r\n```\r\n\r\np.s. apologies for using `psutil` in the script, I didn't remember until later that it isn't a Python built-in.\r\n","author":{"url":"https://github.com/amachanic","@type":"Person","name":"amachanic"},"datePublished":"2024-05-24T20:15:58.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/612/msgpack-python/issues/612"}