René's URL Explorer Experiment

Title: gh-95534: Improve gzip reading speed by 10% by rhpvorderman · Pull Request #97664 · python/cpython · GitHub

Open Graph Title: gh-95534: Improve gzip reading speed by 10% by rhpvorderman · Pull Request #97664 · python/cpython

X Title: gh-95534: Improve gzip reading speed by 10% by rhpvorderman · Pull Request #97664 · python/cpython

Description: For motivation check the github issue. #95534 Performance figures before (the used gzip file is the tar source distribution from github): ./python -m pyperf timeit -s "import gzip; g=gzip.open('cpython-3.10.7.tar.gz', 'rb'); it=iter(lambda:g.read(128*1024), b'');" "for _ in it: pass" ..................... Mean +- std dev: 301 ms +- 2 ms after: ./python -m pyperf timeit -s "import gzip; g=gzip.open('cpython-3.10.7.tar.gz', 'rb'); it=iter(lambda:g.read(128*1024), b'');" "for _ in it: pass" ..................... Mean +- std dev: 270 ms +- 1 ms Performance tests where run with all optimizations enabled. I found that --enable-optimizations did not influence the result. So it can be verified without all the PGO stuff. Change summary: There is now a gzip.READ_BUFFER_SIZE constant that is 128KB. Other programs that read in 128KB chunks: pigz and cat. So this seems best practice among good programs. Also it is faster than 8 kb chunks. a zlib._ZlibDecompressor was added. This is the _bz2.BZ2Decompressor ported to zlib. Since the zlib.Decompress object is better for in-memory decompression, the _ZlibDecompressor is hidden. It only makes sense in file decompression, and that is already implemented now in the gzip library. No need to bother the users with this. The ZlibDecompressor uses the older Cpython arrange_output_buffer functions, as those are faster and more appropriate for the use case. GzipFile.read has been optimized. There is no longer a unconsumed_tail member to write back to padded file. This is instead handled by the ZlibDecompressor itself, which has an internal buffer. _add_read_data has been inlined, as it was just two calls. EDIT: While I am adding improvements anyway, I figured I could add another one-liner optimization now to the python -m gzip application. That read chunks in io.DEFAULT_BUFFER_SIZE previously, but has been updated now to use READ_BUFFER_SIZE chunks. Results: before: Benchmark 1: cat cpython-3.10.7.tar.gz | ./python -m gzip -d > /dev/null Time (mean ± σ): 389.1 ms ± 12.0 ms [User: 372.7 ms, System: 19.9 ms] Range (min … max): 370.9 ms … 410.2 ms 20 runs After: Benchmark 1: cat cpython-3.10.7.tar.gz | ./python -m gzip -d > /dev/null Time (mean ± σ): 320.5 ms ± 12.1 ms [User: 306.4 ms, System: 17.6 ms] Range (min … max): 300.0 ms … 339.1 ms 20 runs For comparison: pigz, the fastest zlib utilizing gzip decompressor, on a single thread. (igzip is faster, but utilizes ISA-L). Benchmark 1: cat cpython-3.10.7.tar.gz | pigz -p 1 -d > /dev/null Time (mean ± σ): 293.8 ms ± 8.4 ms [User: 288.5 ms, System: 17.0 ms] Range (min … max): 277.5 ms … 302.7 ms 20 runs If we take the pure C pigz program as baseline, the amount of python overhead is reduced drastically from 30% to 10%. Issue: gh-95534

Open Graph Description: For motivation check the github issue. #95534 Performance figures before (the used gzip file is the tar source distribution from github): ./python -m pyperf timeit -s "import gzip; g=gzip.open...

X Description: For motivation check the github issue. #95534 Performance figures before (the used gzip file is the tar source distribution from github): ./python -m pyperf timeit -s "import gzip; g=gzip....

Opengraph URL: https://github.com/python/cpython/pull/97664

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width