René's URL Explorer Experiment

Title: fix: avoid memory leak when decoding invalid nested arrays by KowalskiThomas · Pull Request #671 · msgpack/msgpack-python · GitHub

Open Graph Title: fix: avoid memory leak when decoding invalid nested arrays by KowalskiThomas · Pull Request #671 · msgpack/msgpack-python

X Title: fix: avoid memory leak when decoding invalid nested arrays by KowalskiThomas · Pull Request #671 · msgpack/msgpack-python

Description: What is this PR? This PR fixes a memory leak that was detected (accidentally) through fuzzing. The leak happens in some cases when trying to decode invalid data. When decoding an array, the unpacker uses a stack and pushes a new list to that stack for every nested array element. If it eventually reaches a point where the element to decode is problematic, it returns -2 which results in FormatError being raised, but unpack_clear lacks some memory freeing logic (only relevant in the problematic case -- the outermost list is freed but not the other ones) and the objects are lost forever. The leak can directly be observed by running the following reproducer, which pushes several nested lists, and eventually one undefined format byte. It results in returning an error, without freeing the intermediate list objects. The logic is run under tracemalloc, with explicit GC calls to avoid transient still-alive objects that would be freed at some point. The deeper the nesting, the more objects are leaked. import gc import tracemalloc import msgpack._cmsgpack as _c KWARGS = {"raw": False, "strict_map_key": False, "max_array_len": 1 << 20, "max_map_len": 1 << 20} N = 10000 for depth in range(1, 15): data = bytes([0x91] * depth + [0xC1]) gc.collect() tracemalloc.start() s1 = tracemalloc.take_snapshot() for _ in range(N): try: _c.unpackb(data, **KWARGS) except: pass gc.collect() s2 = tracemalloc.take_snapshot() tracemalloc.stop() objs = sum(s.count_diff for s in s2.compare_to(s1, "lineno") if s.count_diff > 0) print(f"depth={depth}: {objs / N:.1f} leaked obj/call")

Open Graph Description: What is this PR? This PR fixes a memory leak that was detected (accidentally) through fuzzing. The leak happens in some cases when trying to decode invalid data. When decoding an array, the unpacke...

X Description: What is this PR? This PR fixes a memory leak that was detected (accidentally) through fuzzing. The leak happens in some cases when trying to decode invalid data. When decoding an array, the unpacke...

Opengraph URL: https://github.com/msgpack/msgpack-python/pull/671

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width