Title: gh-129724: Make marshalled code output deterministic by khmyznikov · Pull Request #156862 · python/cpython · GitHub
Open Graph Title: gh-129724: Make marshalled code output deterministic by khmyznikov · Pull Request #156862 · python/cpython
X Title: gh-129724: Make marshalled code output deterministic by khmyznikov · Pull Request #156862 · python/cpython
Description: Summary This makes marshalled code-object output deterministic in free-threaded builds, including when multiple threads compile modules concurrently. Fixes #129724. Fixes #156504. The standard build retains current main compiler-cache and marshal behavior. Problem Marshal decides whether to add an object to its reference table using _PyObject_IsUniquelyReferenced(). In a free-threaded build, that result can depend on object ownership and shared-reference state. The same code object can therefore receive different FLAG_REF placements depending on thread timing and prior accesses. Because a .pyc file contains the marshalled code object after its 16-byte header, this makes otherwise identical parallel compilations produce byte-different .pyc files. Running a free-threaded interpreter with -X gil=1 does not avoid the problem because it does not change the free-threaded object's reference representation. Approach For a top-level code object in a free-threaded build, marshal now uses deterministic reference tracking: supported objects are registered independently of their live reference counts; aliases and recursive containers continue to round-trip correctly; set and frozenset elements are ordered by deterministic marshal encodings; set elements with identical encodings but distinguishable alias graphs are rejected rather than ordered by process-specific identity; detailed marshal errors are preserved instead of being overwritten by generic container errors. In free-threaded builds, the compiler also avoids sharing rebuildable constant containers across code units. Frozensets and slices use a code-unit-local cache instead of the compiler-wide cache. This preserves same-code-unit deduplication while preventing compilation order from changing the object graph later passed to marshal. Valid source can create distinct NaN constants whose marshal encodings compare equal. In free-threaded builds, constant folding therefore leaves NaN-containing set displays as runtime BUILD_SET operations instead of creating an ambiguously ordered constant. Marshal versions 0 through 2 do not have a reference table, so their reference-tracking behavior is unchanged. The new ambiguous-set rejection applies to every marshal version when deterministic code-object mode is active. Generic non-code-object marshaling retains the existing path. Compatibility Standard builds retain current main sharing behavior. Baseline and candidate bytes matched for both tested compilation histories. Current main can itself produce different standard-build bytes after an unrelated compilation changes frozenset sharing. This pre-existing behavior is outside the free-threaded scope of this change. Free-threaded output changes to a deterministic canonical representation but unmarshals to equivalent code. Payload size was unchanged in the current benchmark workload: 68,172 bytes. A 24-module synthetic package was unchanged at 1,067,465 bytes. In real-wheel integration tests, total .pyc size increased by 77 bytes for pip across 17 of 404 files and by 63 bytes for Plotly across 4 of 1,507 files. The largest per-file change was 23 bytes, and each total change was below 0.002%. Repeated constants in one code unit remain deduplicated. Free-threaded cross-code-unit identity sharing of rebuildable constant containers is no longer relied on. Determinism results Testing used x64 Release standard and free-threaded interpreters built from current main (42de93a62e) and the core implementation (eae9b877a7) on the same 8-vCPU Windows VM. The final compatibility follow-up is 3f056a9782; it narrows standard-build compiler behavior without changing the free-threaded marshal algorithm. Free-threaded mode Operation Current main This change GIL disabled (default) marshal.dumps(code) Different on attempt 1 0 differences / 20 attempts GIL disabled (default) py_compile Different on attempt 1 0 differences / 20 attempts -X gil=1 marshal.dumps(code) Different on attempt 1 0 differences / 20 attempts -X gil=1 py_compile Different on attempt 1 0 differences / 20 attempts That is 80 candidate stress attempts with no differing output. The exact final tree also completed 10 attempts for each combination locally: 40 additional attempts with no differing output. Performance The same 8-vCPU VM ran eight paired current-main/candidate rounds, alternating process order. Ratios below are the median paired candidate/baseline ratio; values above 1 are slower. Free-threaded measurement Candidate/current main Repeated marshal.dumps(code) 1.513x Compile + marshal 0.981x Generic unique-value marshal 1.001x Eight-thread, 32-file compile + marshal batch 0.995x Peak memory during code marshal 1.353x marshal.loads(code_payload) 1.054x Peak memory during marshal.loads 1.045x Cold import of a 24-module pyc-only package 1.014x Synthetic package bytes 1.000x The stable tradeoff is the synthetic repeated free-threaded marshal.dumps(code) case: about 51% more time and 35% more peak memory because deterministic mode records all supported references. marshal.loads is about 5% slower with 4.5% more peak memory. Compile-plus-marshal was neutral, and the synthetic cold-import result was about 1.4% slower. Generic and threaded measurements had wider VM variance and should not be interpreted as performance improvements. Integrated pip workload The change was tested end-to-end with pip PR 14286, which parallelizes Windows wheel bytecode compilation. The benchmark copy removed only pip's guard that keeps free-threaded builds serial. Both CPython variants and both GIL modes used eight worker threads. Each timing is the median of six balanced runs on the same eight-vCPU Windows VM with Defender antivirus, real-time, IOAV, and behavior monitoring enabled. The benchmark fixed SOURCE_DATE_EPOCH=1704067200 and PYTHONHASHSEED=0 to isolate thread-order effects. GIL Volume Wheel Serial Parallel Speedup Off NTFS pip, 404 Python files 2.749 s 1.077 s 2.55x Off NTFS Plotly, 1,507 Python files 7.745 s 2.130 s 3.64x Off Dev Drive pip 1.318 s 0.798 s 1.65x Off Dev Drive Plotly 2.516 s 1.747 s 1.44x On NTFS pip 2.652 s 1.535 s 1.73x On NTFS Plotly 7.867 s 2.841 s 2.77x On Dev Drive pip 1.336 s 1.390 s 0.96x On Dev Drive Plotly 2.507 s 2.540 s 0.99x Current main produced serial/parallel artifact differences in 14 of 48 full-matrix parallel installs. The deterministic build produced zero differences in all 48. The exact-final follow-up used four counterbalanced runs per wheel, filesystem, and GIL mode. Current main differed in 9 of 32 parallel installs; exact 3f056a9782 differed in 0 of 32, and its 32 serial repeats were identical. Together, the full and exact-final matrices successfully unmarshalled 305,760 measured .pyc payloads. Across the six-run matrix, aggregate serial timing for the fixed runtime was about 1% slower than current main and within run-to-run variance, with no consistent parallel penalty. In the exact-final matrix, candidate/current-main peak committed/pagefile ratios ranged from 0.998-1.004 serial and 0.960-1.020 parallel, so there was no consistent whole-process memory increase attributable to this change. Absolute fixed-runtime peak committed usage was 133.5-134.8 MiB serial and 239.6-300.6 MiB parallel; peak working set was 98.7-107.5 MiB serial and 169.3-196.5 MiB parallel. These whole-process measurements do not replace the marshal-only peak-memory result. Default GIL-off PythonT retained a 1.44x-3.64x speedup. With the GIL re-enabled, NTFS still benefited substantially, while the trusted Dev Drive was neutral to slightly slower. The pip implementation used eight workers on a VM with eight logical CPUs; these measurements do not establish the optimal worker count elsewhere. WSL cross-platform confirmation The exact candidate was also built with GCC 15.2 on x86-64 Ubuntu 26.04 WSL2 and tested on native ext4 storage. The CPython diff has no OS- or CPU-architecture-specific branch; its conditional behavior uses only Py_GIL_DISABLED. This run therefore adds Linux ABI/toolchain coverage, but not Arm64 coverage. The standalone reproducer found a current-main difference on attempt 1 for both marshal and py_compile, with the GIL both disabled and enabled. Exact 3f056a9782 completed all 40 corresponding attempts with no difference. Focused test_marshal, test_compile, and test_code runs passed 313 tests in each GIL mode. For an integrated check, the pip PR copy was temporarily enabled on Linux and free-threaded builds. Six counterbalanced runs used four workers: GIL Wheel Serial Parallel Speedup Off pip 0.955 s 0.562 s 1.70x Off Plotly 1.629 s 0.882 s 1.85x On pip 0.964 s 1.171 s 0.82x On Plotly 1.608 s 2.081 s 0.77x Current main differed from serial output in 13 of 24 parallel installs. The candidate differed in 0 of 24, and all 91,728 measured .pyc payloads unmarshalled successfully. Candidate/current-main geometric-mean timing ratios were 0.993 serial and 1.018 parallel. The approximately 1.8% aggregate parallel difference varied by workload and is small relative to the GIL-off pip gain. Peak virtual size was effectively identical; geometric-mean peak RSS ratios were 1.001 serial and 1.014 parallel. The size effect matched Windows: pip .pyc files grew by 77 total bytes and Plotly by 63, with a maximum per-file change of 23 bytes. The result also confirms the pip policy distinction: default GIL-off PythonT benefits on ext4, while GIL-on PythonT was 21-29% slower with four workers. Alternatives measured Borrowing deterministic hashtable keys improved direct dumps by about 5% but did not reduce memory and regressed compile-plus-marshal in the sample. A graph-counting two-pass writer reduced reader peak memory by about 27% but made direct dumps another 39% slower than register-all. A one-probe hashtable API was neutral for direct dumps in stable VM rounds and about 6% slower in the threaded batch, while adding broad internal API surface. Skipping immortal small ints made direct dumps about 2% slower and did not change measured peak memory. The set-sort ordinal is normally a cached small integer, so replacing it with a temporary set or dict would add hashing and table allocation to save one tuple slot. That was not retained as a promising optimization. Validation Exact-final-tree x64 Release standard and free-threaded builds completed with no warnings. x64 Debug standard and free-threaded builds completed with no warnings. Tools/patchcheck/patchcheck.py passed. Standard default test suite: 49,911 tests in the parallel run plus the isolated Windows-flaky modules, all successful. Free-threaded default test suite: 50,178 tests in the parallel run plus the isolated Windows-flaky modules, all successful. Free-threaded -X gil=1 focused suite: 1,812 tests successful. Exact-final-tree tests for the changed compiler/marshal behavior passed in standard, free-threaded GIL-disabled, and free-threaded GIL-enabled modes. Debug refleak checks for test_marshal, test_compile, and test_code passed in both standard and free-threaded builds with -R 3:3. Regression tests cover compile order, thread interleavings, aliases, recursion, sets, ambiguous encodings, NaNs, marshal versions, cross-code-unit isolation, and same-code-unit deduplication. One test_device_encoding assertion was excluded from the final clean host run because the automation pseudo-console reports stdin as a TTY while os.device_encoding(0) is None. Current main fails the same isolated assertion in that environment. AI-assisted development disclosure GitHub Copilot was used to assist investigation, implementation, test drafting, and benchmark organization. Issue: gh-129724
Open Graph Description: Summary This makes marshalled code-object output deterministic in free-threaded builds, including when multiple threads compile modules concurrently. Fixes #129724. Fixes #156504. The standard buil...
X Description: Summary This makes marshalled code-object output deterministic in free-threaded builds, including when multiple threads compile modules concurrently. Fixes #129724. Fixes #156504. The standard buil...
Opengraph URL: https://github.com/python/cpython/pull/156862
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:12042e2c-248d-ba61-7b22-649b5243eb5b |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | B578:2AF0D8:931F03:B84BD7:6B02171B |
| html-safe-nonce | 7d5f52009d1d1fe6bc41025ed41d82dbf9daa89ab3514864dc01f5b9d76a9ba5 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNTc4OjJBRjBEODo5MzFGMDM6Qjg0QkQ3OjZCMDIxNzFCIiwidmlzaXRvcl9pZCI6IjU4MTk1MjU0MTE5MjI2NDY4MTEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | ae203a87ffea315f8d96f550b59a6ffe7870f9a9d2cbc79102ff28f417ed2bb1 |
| hovercard-subject-tag | pull_request:4427389625 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/156862/files |
| twitter:image | https://avatars.githubusercontent.com/u/6115884?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/6115884?s=400&v=4 |
| og:image:alt | Summary This makes marshalled code-object output deterministic in free-threaded builds, including when multiple threads compile modules concurrently. Fixes #129724. Fixes #156504. The standard buil... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width