Title: Bytecode compilation output depends on order of files compiled · Issue #129724 · python/cpython · GitHub
Open Graph Title: Bytecode compilation output depends on order of files compiled · Issue #129724 · python/cpython
X Title: Bytecode compilation output depends on order of files compiled · Issue #129724 · python/cpython
Description: This is minimal reproduction of this downstream bug report: astral-sh/uv#10619 The output of compileall.compile_file depends on the order in which the files are compiled. This means compilation is non-deterministic if builds are distribu...
Open Graph Description: This is minimal reproduction of this downstream bug report: astral-sh/uv#10619 The output of compileall.compile_file depends on the order in which the files are compiled. This means compilation is ...
X Description: This is minimal reproduction of this downstream bug report: astral-sh/uv#10619 The output of compileall.compile_file depends on the order in which the files are compiled. This means compilation is ...
Opengraph URL: https://github.com/python/cpython/issues/129724
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bytecode compilation output depends on order of files compiled","articleBody":"This is minimal reproduction of this downstream bug report: https://github.com/astral-sh/uv/issues/10619\n\nThe output of `compileall.compile_file` depends on the order in which the files are compiled. This means compilation is non-deterministic if builds are distributed over a process pool.\n\nThis becomes a problem when building docker images, where you usually bytecode compile ahead of time for faster startup, and where the hash of the image depends on all files in the image, including the `.pyc` files.\n\nSpecifically, the output of \n\n```python\na = {\"foo\", 2, 3}\n\ndef f():\n b = {\"foo\", 2, 3}\n```\n\nis different if we previously compiled another file with \n\n```python\nimport foo\n```\n\nReproducer script:\n\n```shell\n#!/bin/bash\n\nset -e\n\nscript=$(cat \u003c\u003c EOF\nimport compileall\nimport sys\n\nfor path in sys.argv[1:]:\n compileall.compile_file(path)\nEOF\n)\n\ncat \u003c\u003c EOF > a.py\nimport foo\nEOF\n\ncat \u003c\u003c EOF > b.py\na = {\"foo\", 2, 3}\n\ndef f():\n b = {\"foo\", 2, 3}\nEOF\n\n# Both files\nrm -rf __pycache__\npython3.14 -c \"$script\" a.py b.py\nsha256sum __pycache__/b.cpython-314.pyc\n\n# For debugging\ncp __pycache__/b.cpython-314.pyc b1.cpython-314.pyc\n\n# Single file only\nrm -rf __pycache__\npython3.14 -c \"$script\" b.py\nsha256sum __pycache__/b.cpython-314.pyc\n\n# For debugging\ncp __pycache__/b.cpython-314.pyc b2.cpython-314.pyc\n\n```\n\nThis is caused be different refcounts in the marshalled files:\n\n```python\nimport marshal\nimport sys\n\nwith open(\"b1.cpython-313.pyc\", \"rb\") as f:\n f.read(16) # Skip header\n pyc1 = marshal.load(f)\n\nwith open(\"b2.cpython-313.pyc\", \"rb\") as f:\n f.read(16) # Skip header\n pyc2 = marshal.load(f)\n\nprint(sys.getrefcount(pyc1.co_consts[0]))\nprint(sys.getrefcount(pyc2.co_consts[0]))\n```\n\nThis prints `2` and `3`.\n\nThe original report is from 3.13, i've reproduced it with 3.14.0a4. It happens at least on linux and windows.\n\n\n### CPython versions tested on:\n\n3.14\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-156862\n* gh-159192\n* gh-159193\n* gh-159290\n* gh-159291\n* gh-160628\n* gh-161457\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/konstin","@type":"Person","name":"konstin"},"datePublished":"2025-02-06T12:42:34Z","url":"https://github.com/python/cpython/issues/129724"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:a31a80c1-e4c4-8fb5-c17b-628f898befc0 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D7EC:3F8AB1:4189009:532F603:6B02086E |
| html-safe-nonce | 0d41aaf3d49de5a018d12454fba933480b68a609d55f550e2e284386076c6318 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEN0VDOjNGOEFCMTo0MTg5MDA5OjUzMkY2MDM6NkIwMjA4NkUiLCJ2aXNpdG9yX2lkIjoiODU4MDM2ODI1NDIyMzEyNDU5MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | d27084945088c594eeda9c9180c1d6c7f565fab674078d9d4701616363be14ce |
| hovercard-subject-tag | issue:2835430488 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/129724/issue_layout |
| twitter:image | https://opengraph.githubassets.com/661be5202b1debee5408ef1276cd160867231de92fa275150dc03258868eb4cc/python/cpython/issues/129724 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/661be5202b1debee5408ef1276cd160867231de92fa275150dc03258868eb4cc/python/cpython/issues/129724 |
| og:image:alt | This is minimal reproduction of this downstream bug report: astral-sh/uv#10619 The output of compileall.compile_file depends on the order in which the files are compiled. This means compilation is ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | konstin |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width