René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"zipfile regression: When writing a zip64 entry to an unseekable file, the local file header compressed/uncompressed fields are not set to 0","articleBody":"# Bug report\r\n\r\nWhen creating a zip file with a zip64 entry in a streaming way (unseekable file), my understanding from the spec is that the headers should be set up so that:\r\n\r\n1. The local file header general flags field has bit 3 set (§4.4.4)\r\n2. The local file header crc32, compressed size, and uncompressed size fields are set to 0 (§4.4.4)\r\n3. The local file header has a zip64 (`0x0001`) extra record so that data descriptor sizes are interpreted as 64-bit integers (§4.3.9.2)\r\n4. A data descriptor header exists (§4.3.9.1)\r\n\r\n`zipfile` normally does all this, but (2) seems to have broken after #103861/#103863. With that change, the local file header's compressed and uncompressed sizes are set to `0xffffffff` instead of `0x00000000`. (`0xffffffff` is correct for the usual case where files are seekable and data descriptors are not used.)\r\n\r\nFrom initial testing, it seems like this is all that's needed to fix the issue:\r\n\r\n```diff\r\n--- zipfile.py\t2023-06-08 22:29:05.000000000 -0400\r\n+++ zipfile2.py\t2023-06-28 17:50:45.841709476 -0400\r\n@@ -463,8 +463,9 @@\r\n             fmt = '\u003cHHQQ'\r\n             extra = extra + struct.pack(fmt,\r\n                                         1, struct.calcsize(fmt)-4, file_size, compress_size)\r\n-            file_size = 0xffffffff\r\n-            compress_size = 0xffffffff\r\n+            if not (self.flag_bits \u0026 _MASK_USE_DATA_DESCRIPTOR):\r\n+                file_size = 0xffffffff\r\n+                compress_size = 0xffffffff\r\n             min_version = ZIP64_VERSION\r\n \r\n         if self.compress_type == ZIP_BZIP2:\r\n```\r\n\r\n# To reproduce\r\n\r\n```python\r\nimport zipfile\r\n\r\n\r\nclass UnseekableFile:\r\n    def __init__(self, fp):\r\n        self.fp = fp\r\n\r\n    def write(self, data):\r\n        return self.fp.write(data)\r\n\r\n    def flush(self):\r\n        self.fp.flush()\r\n\r\n\r\nwith open('test.zip', 'wb') as f_raw:\r\n    with zipfile.ZipFile(UnseekableFile(f_raw), 'w') as z:\r\n        with z.open('foobar', 'w', force_zip64=True) as f:\r\n            f.write(b'Hello, world!')\r\n```\r\n\r\nThe resulting file looks like this, as reported by [zipdetails](https://github.com/pmqs/zipdetails) (green=good, red=problematic).\r\n\r\n```diff\r\n 0000 LOCAL HEADER #1       04034B50\r\n 0004 Extract Zip Spec      2D '4.5'\r\n 0005 Extract OS            00 'MS-DOS'\r\n+0006 General Purpose Flag  0008\r\n+     [Bit  3]              1 'Streamed'\r\n 0008 Compression Method    0000 'Stored'\r\n 000A Last Mod Time         00210000 'Mon Dec 31 19:00:00 1979'\r\n+000E CRC                   00000000\r\n-0012 Compressed Length     FFFFFFFF\r\n-0016 Uncompressed Length   FFFFFFFF\r\n 001A Filename Length       0006\r\n 001C Extra Length          0014\r\n 001E Filename              'foobar'\r\n+0024 Extra ID #0001        0001 'ZIP64'\r\n+0026   Length              0010\r\n+0028   Uncompressed Size   0000000000000000\r\n+0030   Compressed Size     0000000000000000\r\n 0038 PAYLOAD               Hello, world!\r\n \r\n+0045 STREAMING DATA HEADER 08074B50\r\n+0049 CRC                   EBE6C6E6\r\n+004D Compressed Length     000000000000000D\r\n+0055 Uncompressed Length   000000000000000D\r\n \r\n 005D CENTRAL HEADER #1     02014B50\r\n 0061 Created Zip Spec      2D '4.5'\r\n 0062 Created OS            03 'Unix'\r\n 0063 Extract Zip Spec      2D '4.5'\r\n 0064 Extract OS            00 'MS-DOS'\r\n 0065 General Purpose Flag  0008\r\n      [Bit  3]              1 'Streamed'\r\n 0067 Compression Method    0000 'Stored'\r\n 0069 Last Mod Time         00210000 'Mon Dec 31 19:00:00 1979'\r\n 006D CRC                   EBE6C6E6\r\n 0071 Compressed Length     0000000D\r\n 0075 Uncompressed Length   0000000D\r\n 0079 Filename Length       0006\r\n 007B Extra Length          0000\r\n 007D Comment Length        0000\r\n 007F Disk Start            0000\r\n 0081 Int File Attributes   0000\r\n      [Bit 0]               0 'Binary Data'\r\n 0083 Ext File Attributes   01800000\r\n 0087 Local Header Offset   00000000\r\n 008B Filename              'foobar'\r\n \r\n 0091 END CENTRAL HEADER    06054B50\r\n 0095 Number of this disk   0000\r\n 0097 Central Dir Disk no   0000\r\n 0099 Entries in this disk  0001\r\n 009B Total Entries         0001\r\n 009D Size of Central Dir   00000034\r\n 00A1 Offset to Central Dir 0000005D\r\n 00A5 Comment Length        0000\r\n Done\r\n```\r\n\r\n# Your environment\r\n\r\n- CPython versions tested on: 3.11.4\r\n- Operating system and architecture: Alpine Linux x86_64","author":{"url":"https://github.com/chenxiaolong","@type":"Person","name":"chenxiaolong"},"datePublished":"2023-06-28T21:53:19.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/106218/cpython/issues/106218"}