René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Low decompression speed in multithreading","articleBody":"As far as i understand, the GIL should be dropped when calling the underlying LZ4 C library during compression and decompression, see [lz4.block.decompress](https://github.com/python-lz4/python-lz4/blob/58df0834b57d485f2483bf5ccf1007c313b25557/lz4/block/_block.c#L355-L361)\r\n\r\nI only see a minor speedup when using multiple threads for decompression using python-lz4 4.3.2 on a 6 core Intel i5-8400 on Debian 11 and also not on a Amd Ryzen 5900X on Windows 10, neither with Python 3.11.1 nor 3.8.10.\r\n\r\nCompression speed seems to increase almost linearly with the number of threads.\r\n\r\nThe following code gives me about 4500MB/s decompression speed (slight underestimation due to some overhead from starting the threads) when using 6 threads and ~4300MB/s when using 1 thread on an AMD Ryzen 5900X on Windows 10.  Using lz4.frame yields similar results. Using py-lz4framed instead gives me about 13000MB/s using 6 threads and ~8300MB/s on 1 thread (not sure the compression settings are the same, but at the very least there is some speedup for multithreading).\r\n\r\n\r\n```python\r\nimport os\r\nimport threading\r\nimport time\r\nimport lz4.block\r\n\r\nsize_mb = 2000\r\nn_threads = 6\r\n\r\ninput_data = size_mb * 1024 * os.urandom(1024)\r\ncompressed = lz4.block.compress(input_data)\r\ninput_data = None\r\n\r\n\r\ndef decompress(data):\r\n    start_time = time.perf_counter()\r\n    thread_start_time = time.thread_time()\r\n    lz4.block.decompress(data)\r\n    stop_time = time.perf_counter()\r\n    thread_stop_time = time.thread_time()\r\n    print(f\"{threading.current_thread()} Decompression took {(stop_time-start_time)*1000:.3f}ms \"\r\n          f\"(Thread time: {(thread_stop_time-thread_start_time)*1000:.3f}ms)\\n\")\r\n\r\n\r\nthreads = [threading.Thread(name=str(i), target=decompress, args=(compressed,)) for i in range(n_threads)]\r\n\r\nstart_thread_time = time.perf_counter()\r\n[t.start() for t in threads]\r\n[t.join() for t in threads]\r\ndone_thread_time = time.perf_counter()\r\nduration = done_thread_time - start_thread_time\r\nMBs = n_threads*size_mb/duration\r\nprint(f\"Total time: {duration}s : {MBs}MB/s\")\r\n```\r\n","author":{"url":"https://github.com/Dalbasar","@type":"Person","name":"Dalbasar"},"datePublished":"2023-01-02T21:26:52.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/269/python-lz4/issues/269"}