Title: Data corruption when reading from two files in parallel in an uncompressed .zip · Issue #127847 · python/cpython · GitHub
Open Graph Title: Data corruption when reading from two files in parallel in an uncompressed .zip · Issue #127847 · python/cpython
X Title: Data corruption when reading from two files in parallel in an uncompressed .zip · Issue #127847 · python/cpython
Description: Bug report Bug description: I ran into a data corruption bug that seems to be triggered by interleaving reads/seeks from different files inside of an uncompressed zip file. As far as I can tell from the docs, this is allowed by zipfile. ...
Open Graph Description: Bug report Bug description: I ran into a data corruption bug that seems to be triggered by interleaving reads/seeks from different files inside of an uncompressed zip file. As far as I can tell fro...
X Description: Bug report Bug description: I ran into a data corruption bug that seems to be triggered by interleaving reads/seeks from different files inside of an uncompressed zip file. As far as I can tell fro...
Opengraph URL: https://github.com/python/cpython/issues/127847
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Data corruption when reading from two files in parallel in an uncompressed .zip","articleBody":"# Bug report\n\n### Bug description:\n\nI ran into a data corruption bug that seems to be triggered by interleaving reads/seeks from different files inside of an uncompressed zip file. As far as I can tell from the docs, this is allowed by `zipfile`. It works correctly in Python 3.7 and 3.9, but fails in 3.12.\n\nI'm attaching a somewhat convoluted testcase (still working on a simpler one). It parses a dBase IV database by reading records from a .dbf file, and for each record, reading a corresponding record from a .dbt file.\n\nWhen run using Python 3.9, you will see a bunch of data printed out. When run using Python 3.12, you will get an exception `ValueError: Invalid dBase IV block: b'PK\\x03\\x04\\n\\x00\\x00\\x00'`. That block does not appear in the input file at all. (Though, when tested with a larger input, I got a block of bytes that appeared in the _wrong_ file.)\n\nFor some context, [here is a workaround](https://github.com/dimaryaz/jdmtool/commit/97ca5b168b2db5675878db676413d8c66e464584) I used in my project: I changed it to read the .dbf file first, then the .dbt.\n\n\nTestcase:\n\n```python\n#!/usr/bin/env python3\n\nimport datetime\nimport pathlib\nimport struct\nimport zipfile\n\nfrom dataclasses import dataclass\nfrom typing import Any, BinaryIO, List, Tuple\n\n\nZIP_PATH = pathlib.Path(__file__).parent / 'notams.zip'\n\n\n@dataclass\nclass DbfHeader:\n SIZE = 32\n VERSION = 3\n\n info: int\n last_update: datetime.date\n num_records: int\n header_bytes: int\n record_bytes: int\n\n @classmethod\n def from_bytes(cls, data: bytes):\n info, year, month, day, num_records, header_bytes, record_bytes = struct.unpack('\u003c4BIHH20x', data)\n version = info & 0x3\n if version != cls.VERSION:\n raise ValueError(f\"Unsupported DBF version: {version}\")\n\n return cls(info, datetime.date(year + 1900, month, day), num_records, header_bytes, record_bytes)\n\n\n@dataclass\nclass DbfField:\n SIZE = 32\n\n name: str\n type: str\n length: int\n\n @classmethod\n def from_bytes(cls, data: bytes):\n name, typ, length = struct.unpack('\u003c11sc4xB15x', data)\n return cls(name.rstrip(b'\\x00').decode(), typ.decode(), length)\n\n\nclass DbfFile:\n @classmethod\n def read_header(cls, fd: BinaryIO) -> Tuple[DbfHeader, List[DbfField]]:\n header = DbfHeader.from_bytes(fd.read(DbfHeader.SIZE))\n num_fields = (header.header_bytes - 33) // 32\n fields = [DbfField.from_bytes(fd.read(DbfField.SIZE)) for _ in range(num_fields)]\n if fd.read(1) != b'\\x0D':\n raise ValueError(\"Missing array terminator\")\n return header, fields\n\n @classmethod\n def read_record(cls, fd: BinaryIO, fields: List[DbfField]) -> List[Any]:\n fd.read(1)\n\n values = []\n for field in fields:\n data = fd.read(field.length).decode('latin-1').strip(' ')\n if field.type == 'C':\n value = data\n elif field.type == 'D':\n s = data.strip(' ')\n if s:\n value = datetime.datetime.strptime(data, '%Y%m%d').date()\n else:\n value = None\n elif field.type == 'L':\n if len(data) != 1:\n raise ValueError(f\"Incorrect length: {data!r}\")\n if data in 'YyTt':\n value = True\n elif data in 'NnFf':\n value = False\n elif data == '?':\n value = None\n else:\n raise ValueError(f\"Incorrect boolean: {data!r}\")\n elif field.type in ('M', 'N'):\n value = int(data) if data else None\n else:\n raise ValueError(f\"Unsupported field: {field.type}\")\n values.append(value)\n return values\n\n\n@dataclass\nclass DbtHeader:\n SIZE = 512\n\n next_free_block: int\n dbf_filename: str\n reserved: int\n block_length: int\n\n @classmethod\n def from_bytes(cls, data: bytes):\n next_free_block, dbf_filename, reserved, block_length = struct.unpack('\u003cI4x8sIH490x', data)\n return cls(next_free_block, dbf_filename.decode('latin-1'), reserved, block_length)\n\n\nclass DbtFile:\n DBT3_BLOCK_SIZE = 512\n DBT4_BLOCK_START = b'\\xFF\\xFF\\x08\\x00'\n\n @classmethod\n def read_header(cls, fd: BinaryIO) -> DbtHeader:\n fd.seek(0)\n block = fd.read(DbtHeader.SIZE)\n return DbtHeader.from_bytes(block)\n\n @classmethod\n def read_record(cls, fd: BinaryIO, header: DbtHeader, idx: int) -> str:\n fd.seek(header.block_length * idx)\n block_start = fd.read(8)\n if block_start[0:4] != cls.DBT4_BLOCK_START:\n raise ValueError(f\"Invalid dBase IV block: {block_start}\")\n length = int.from_bytes(block_start[4:8], 'little')\n data = fd.read(length - len(block_start))\n\n return data.decode('latin-1')\n\n\ndef main():\n with zipfile.ZipFile(ZIP_PATH) as z:\n with z.open('notams.dbf') as dbf_in, z.open('notams.dbt') as dbt_in:\n dbf_header, dbf_fields = DbfFile.read_header(dbf_in)\n dbt_header = DbtFile.read_header(dbt_in)\n\n for _ in range(dbf_header.num_records):\n record = DbfFile.read_record(dbf_in, dbf_fields)\n print(record)\n memo = DbtFile.read_record(dbt_in, dbt_header, record[3])\n print(memo)\n\n\nif __name__ == '__main__':\n main()\n```\n\nInput file:\n[notams.zip](https://github.com/user-attachments/files/18103952/notams.zip)\n\n\n### CPython versions tested on:\n\n3.9, 3.12\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-127856\n* gh-128225\n* gh-128226\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/dimaryaz","@type":"Person","name":"dimaryaz"},"datePublished":"2024-12-12T00:43:04Z","url":"https://github.com/python/cpython/issues/127847"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e7fd256d-4375-4b65-1c53-467572d7552c |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 9280:D1D28:1DF0D95:2796FEF:6B0231C2 |
| html-safe-nonce | 84d6486940c7e759a4315ff6b2dd073e093d94c5c3efdbdc0aeed339e7e59d49 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MjgwOkQxRDI4OjFERjBEOTU6Mjc5NkZFRjo2QjAyMzFDMiIsInZpc2l0b3JfaWQiOiI2ODUyMjQ4NzQ3NzYyNjU5Nzc4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | b1ddcf92b263809e24a47b76ea6426bf40cf9ca78aae7fc972009342d03a8fe9 |
| hovercard-subject-tag | issue:2734372341 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/127847/issue_layout |
| twitter:image | https://opengraph.githubassets.com/a2d9449c2072e61f67f000ceca5c322023de6a93d29dae21772b44200682d4df/python/cpython/issues/127847 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/a2d9449c2072e61f67f000ceca5c322023de6a93d29dae21772b44200682d4df/python/cpython/issues/127847 |
| og:image:alt | Bug report Bug description: I ran into a data corruption bug that seems to be triggered by interleaving reads/seeks from different files inside of an uncompressed zip file. As far as I can tell fro... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | dimaryaz |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width