René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Corrupt .pyc files stay on disk after failed writes","articleBody":"# Bug report\n\n### Bug description:\n\nIf writing a `.pyc` file fails (in my case due to a file size limit imposed by `ulimit`), it can leave corrupt data sitting on disk.  This causes a crash the _second_ time you run the program, when the interpreter tries to load the corrupt `.pyc` file instead of the original `.py` file:\n\n```\nroot@e7138ea2e2b5:/mnt# python3 crashme.py --limit 1024 --import\nSetting ulimit to 1024...\nImporting a \"big\" library...\nroot@e7138ea2e2b5:/mnt# python3 crashme.py --limit 1024 --import\nSetting ulimit to 1024...\nImporting a \"big\" library...\nTraceback (most recent call last):\n  File \"/mnt/crashme.py\", line 17, in \u003cmodule\u003e\n    from fakelib import bigfile\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1360, in _find_and_load\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1331, in _find_and_load_unlocked\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 935, in _load_unlocked\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 991, in exec_module\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 1124, in get_code\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 753, in _compile_bytecode\nEOFError: marshal data too short\n```\n\nThis was a very hard bug to track down, and it seems like there's a chance to catch this and handle it gracefully.  If I do a too-large write myself, I get an `OSError: [Errno 27] File too large`, and I'd imagine the interpreter can see something similar.  Ideally (in my opinion - feedback appreciated!), the interpreter would notice this and then:\n- Log a warning.\n- Delete the corrupt file.\n- Set `sys.dont_write_bytecode = True` (this is the workaround I've been using).\n\nHere's the code I've been using to test this (`bigfile` can be pretty much anything, but it does seem to have to be part of a module before the interpreter will write a `.pyc` file for it).\n\n```python\nimport argparse\nimport resource\nimport tempfile\n\nparser = argparse.ArgumentParser()\nparser.add_argument('-l', '--limit',  type=int)\nparser.add_argument('-i', '--import', action='store_true', dest='impoort')\nparser.add_argument('-w', '--write',  type=int)\nargs = parser.parse_args()\n\nif args.limit is not None:\n    print('Setting ulimit to %d...' % args.limit)\n    resource.setrlimit(resource.RLIMIT_FSIZE, (args.limit, args.limit))\n\nif args.impoort:\n    print('Importing a \"big\" library...')\n    from fakelib import bigfile\n\nif args.write is not None:\n    print('Writing a %d byte file...' % args.write)\n    with tempfile.TemporaryFile() as file:\n        file.write(b'a' * args.write)\n```\n\nTested in Python 3.10 on OSX and Python 3.12 in an Ubuntu 24 container.\n\n### CPython versions tested on:\n\n3.10, 3.12\n\n### Operating systems tested on:\n\nLinux, macOS\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-126627\n* gh-126809\n* gh-126810\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/xavierholt","@type":"Person","name":"xavierholt"},"datePublished":"2024-11-08T22:41:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/126606/cpython/issues/126606"}