René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Faster decompression of gzip files","articleBody":"# Pitch\n\nDecompressing gzip streams is an extremely common practice. Most web browsers support gzip decompression, as such most (virtually all) servers return gzip compressed data (when the gzip support is advertised via headers). Tar.gz files are an extremely common way to archive files. Zip files use internal gzip compression. \n\nSpeeding this up by a non-trivial amount is therefore very advantageous.\n\n# Feature or enhancement\n\nThe current gzip reading pipeline can be improved quite a lot. This is the current way of doing things:\n\n- read `io.DEFAULT_BUFFER_SIZE` of data from a _PaddedFile object\n- feed it to a `zlib.decompressobj()` using the `decompress(raw_data, size)` function.\n- Internally decompress always starts with a 16KB buffer, regardless of the requested size. When the output data is 64KB big, it will need to resize 2 times.\n- The decompressed data is returned, anything not returned is saved in an unconsumed_tail object.\n- The unconsumed_tail is used to rewind the _PaddedFile object to the correct position.\n- The decompressed data length and crc32 are taken and these are used to update the GzipReader state.\n\nThis has some severe disadvantages when reading large blocks:\n- Gzip compresses between 50 and 90% in most cases. This means that the decompress function is going to return anywhere   between 16 and 80KB maximum. When 128 KB is requested from the read function this means the 128 KB is not going to be filled, leading to unnecessary read requests.\n-  In the above case say the typical return size is 37kb. That means due to the DEFAULT_BUFFER_SIZE in zlib being 16KB there will be two calls to resize the memory of the return object. (16-\u003e32-\u003e64). EDIT: actually it is three, since the end product is also resized to fit the contents (16-\u003e32-\u003e64-\u003e37).\n\nThis also has some severe disadvantages when reading small blocks.\n- When reading individual lines from a file (quite common) this actually queries a io.BufferedReader instance which reads from the _GzipReader in io.DEFAULT_BUFFER_SIZE chunks.\n- This means only 8KB requested. But still, 8KB is read from the _PaddedFile object. With typical compression rates this means anywhere between 4KB to 7KB of unconsumed tail will be returned. This creates a new object. Meaning allocating memory again. The same data is reread in the next iteration. This means in case of a 70% decompression rate, the same data is memcpy'd around 2 to 3 times.\n- Continuous rewinding of the _PaddedFile object\n\n## How to improve this:\n\n- Use the structure of the _bz2.Bz2Decompressor. Submitted data is copied to an internal buffer, data from this internal buffer is used to return decompress calls. This has the advantage of not endlessly allocating and destroing uncompressed_tail objects.\n- Read 128KB at once from the _PaddedFile object. And only read this data when the decompressor's `needs_input` attribute is True. This prevents querying the _PaddedFile object too much.\n- When decompress is called and a maxsize is given (say 128KB) allocate maxsize immediately, instead of allocating only 16KB and resizing later. \n\nThis prevents a lot of calls to the Python memory allocator. \n\nThis restructuring has already been implemented in [python-isal](https://github.com/pycompression/python-isal). That project is a modification of zlibmodule.c to use the [ISA-L](https://github.com/intel/isa-l) optimizations. While this did improve speed, I also looked at other ways to improve the performance. By restructuring the gzip module and the zlib code the Python overhead was significantly reduced.\n\nRelevant code:\n- [_GzipReader.read function](https://github.com/pycompression/python-isal/blob/v1.0.0/src/isal/igzip.py#L253)\n- [Modified BZ2Decompressor code](https://github.com/pycompression/python-isal/blob/v1.0.0/src/isal/igzip_libmodule.c)\n- [Code that sets the buffer size to the maximum size](https://github.com/pycompression/python-isal/blob/v1.0.0/src/isal/igzip_libmodule.c#L67)\n\nMost of this code can be seemlessly copied back into CPython. Which I will do when I have the time. This can best be done after the 3.11 release I think.\n\n# Previous discussion\n\nNA. This is a performance enhancement, so not necessarily a new feature, but also not a bug.\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-97664\n* gh-137923\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/rhpvorderman","@type":"Person","name":"rhpvorderman"},"datePublished":"2022-08-01T13:54:11.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/95534/cpython/issues/95534"}