Title: gh-159188: Do not immortalize constants in compile() of an AST by gpshead · Pull Request #159198 · python/cpython · GitHub
Open Graph Title: gh-159188: Do not immortalize constants in compile() of an AST by gpshead · Pull Request #159198 · python/cpython
X Title: gh-159188: Do not immortalize constants in compile() of an AST by gpshead · Pull Request #159198 · python/cpython
Description: On the free-threaded build, the marshal output for code compiled from an AST depended on which code objects had been created earlier in the process:
SRC = "def f(a, b):\n return g(a=1, b=2)\n"
# optionally first: compile(ast.parse("def other(a, b):\n return h(a=3, b=4)\n"), ...)
d_src = marshal.dumps(compile(SRC, 'm.py', 'exec'))
d_ast = marshal.dumps(compile(ast.parse(SRC), 'm.py', 'exec'))
free-threaded main
len(d_src)
len(d_ast)
nothing compiled before
230
230
an unrelated AST compiled before
230
237
The default build gives 230 in every case, and so does the free-threaded build with this PR.
Cause and change
The compiler makes a tuple of names (co_names, co_localsplusnames) the same object as an equal constant: keyword names, a from list, __static_attributes__. On the free-threaded build, intern_constants() replaces each constant by the equal immortal object of a per-interpreter table, if there is one, but does not replace the tuple of names. compile() of a source string turns that off with tstate->suppress_co_const_immortalization (gh-118527). The branch that handles AST objects returned before the flag was set.
flowchart TB
subgraph before["main, free-threaded"]
direction TB
b0["compiler: constant ('a', 'b')
is the same tuple as co_names"]
b0 -- "compile(ast)" --> b2["immortalization
not suppressed"]
b0 -- "compile(str)" --> b1["immortalization
suppressed"]
b2 --> b4{"equal tuple already
in the immortal table?"}
b4 -- yes --> b6["constant replaced, co_names not:
two tuples, marshal writes both"]
b4 -- no --> b5["still one tuple"]
b1 --> b3["still one tuple
marshal: object + reference"]
end
subgraph after["this PR"]
direction TB
a0["compiler: constant ('a', 'b')
is the same tuple as co_names"]
a0 -- "compile(ast)" --> a1["immortalization
suppressed"]
a0 -- "compile(str)" --> a1
a1 --> a3["still one tuple
marshal: object + reference"]
end
Loading
The change is the same increment/decrement pair around _PyAST_Compile() in builtin_compile_impl() that already surrounds _Py_CompileString().
Effects
Tools that compile an AST and write a .pyc (assertion rewriters, SourceLoader.source_to_code() given an AST) get reproducible output. Compiling the standard library from ASTs in one process, 179 of 2067 files gave different bytes depending on file order; now none do.
Code from compile(ast) no longer has immortal constants, as is already the case for compile(str). It gets them when it is later loaded from a .pyc. Tools that compile many ASTs stop growing the immortal table: for 10000 distinct functions, +141286 memory blocks on main, +606 with this PR.
The GIL enabled build is not affected.
Not covered
exec() of a string, Py_CompileString*(), PyRun_*() and marshal.loads() do not set the flag and keep the same dependence on history. None of them is on a path that writes .pyc files.
Tests
Every TestConstantsIdentity case (added by gh-159193) is now also compiled from an AST. test_constants_not_immortal (free-threaded only) checks that a tuple constant from compile() of a string or an AST is not immortal even after an equal one entered the table.
Authored using Claude, reviewed by me.
Issue: gh-159188
Open Graph Description: On the free-threaded build, the marshal output for code compiled from an AST depended on which code objects had been created earlier in the process: SRC = "def f(a, b):\n return g(a=1, b=2)...
X Description: On the free-threaded build, the marshal output for code compiled from an AST depended on which code objects had been created earlier in the process: SRC = "def f(a, b):\n return g(a=1, ...
Opengraph URL: https://github.com/python/cpython/pull/159198
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:b697b982-beba-7887-1e9a-e405009b3b90 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9594:38E318:3664BA2:44D45B0:6B021BD8 |
| html-safe-nonce | a4565b680871117434f98c474d8904f627c412bb897ad79624f081a40f42894c |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5NTk0OjM4RTMxODozNjY0QkEyOjQ0RDQ1QjA6NkIwMjFCRDgiLCJ2aXNpdG9yX2lkIjoiNzI0MDA4NDYyODU3NjczNjg4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 11237962089c5dc4cb7f84aff7c8b75e040a9e5791ecd8e131e3305fbd20b235 |
| hovercard-subject-tag | pull_request:4822332486 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/159198/files |
| twitter:image | https://avatars.githubusercontent.com/u/68491?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/68491?s=400&v=4 |
| og:image:alt | On the free-threaded build, the marshal output for code compiled from an AST depended on which code objects had been created earlier in the process: SRC = "def f(a, b):\n return g(a=1, b=2)... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width