Title: Align weak-reference collection with Python 3.15 finalization by youknowdot · Pull Request #9168 · RustPython/RustPython · GitHub
Open Graph Title: Align weak-reference collection with Python 3.15 finalization by youknowdot · Pull Request #9168 · RustPython/RustPython
X Title: Align weak-reference collection with Python 3.15 finalization by youknowdot · Pull Request #9168 · RustPython/RustPython
Description: Align weak-reference collection with Python 3.15 finalization Summary Extract the weakref/finalizer lifecycle work from #8954 onto main's existing collector. Follow CPython free-threading: retain a safe handle to the original static stripe after the referent is cleared and protect callback ownership with that existing stripe, without a separate callback lock. Unlink and steal under the stripe, then release callback ownership after unlocking. Trace the owned callback edge without tracing the weak referent. Keep callback ownership after a GC unlink until the weakref is cleared or destroyed. Clear callback-bearing weakrefs before finalizers and avoid invoking callbacks owned by unreachable weakrefs. Keep callback-free weakrefs usable during finalization, recheck resurrection, then close weakref access before clearing payloads. The final liveness check and weakref closure share one stop-the-world phase. Publish intrusive nodes only after insertion and unlink them before publishing a null referent. Run callbacks and release Python owners outside the stripe lock. Refresh weakref.py, test_weakref.py, test_gc.py and test_finalization.py through the repository updater from CPython 3.15.0 d220409. The finalization suite carries the changed 3.15 resurrection expectations and callback-bearing controls. No hand-edited canonical assertions or duplicate snippets. The corresponding canonical sequence is in Python/gc.c and Objects/weakrefobject.c. Review core.rs callback ownership/list operations together with gc_state.rs finalization and resurrection ordering. Scope and related work Base: main 9a16d28. The source batch contains two native lifecycle files, the four related weakref/GC/finalization library files, the complete canonical IO test-package migration, and updater dependency-path metadata/tests. No dependency, heap architecture, embedding API, interpreter isolation, workflow or unrelated teardown changes. Open #8966 also reports the missing callback traversal edge and includes a simpler stripe-protected traversal with unrelated teardown changes. Open #8939 addresses related lifetime issues as part of its much larger collector/embedding redesign; #8944 contains separate interpreter-local visibility work. This extraction uses the existing main architecture and leaves those authors' branches unchanged. The merged #9094 stop-the-world transaction prerequisite is already on the base. Canonical IO test closure Initial hosted Ubuntu and macOS runs exposed a stale PyIOTest.test_garbage_collection: its warning recorder retained WarningMessage.source, keeping the file alive during finalization. The corrected callback-free weakref now properly observes that surviving owner. CPython 3.15's canonical test ignores ResourceWarning without recording it and keeps the collection/content assertions intact. Migrate all twelve Lib/test/test_io/ files from the same pinned CPython source with scripts/update_lib migrate --legacy-map, replacing the seven legacy standalone suites. All ten existing guards are preserved, including five Windows GC skips and inherited expected-failure wrappers. No selected-method backport, assertion edits, forwarding shims, new guards or duplicated tests are introduced. _pyio.py already matches the pinned source. The updater's io and file aliases now point to the complete package, with regression tests for both mappings. Validation for earlier head 429c7718 Matching default release build passed with locked dependencies. Source and executable/dependency hashes were checked before execution and again after all gates. Cargo and Python runs were serialized, with two build jobs and incremental compilation disabled. Whole test_weakref: 138 cases, 13 existing skips, one retained expected failure. Whole test_finalization: 18 cases, five existing skips. Whole test_weakset: 46 cases, all passed. Selected test_gc: 60 cases, 16 existing skips, one retained expected failure; the exact local exclusion is below. All four module runs returned success, without unexpected successes, timeouts or altered-environment results. Workspace native tests: 1,359 passed. Workspace doctests: 24 passed, 18 ignored. These use the documented wasm/venvlauncher/C-API exclusions. Separate crates/capi native tests: 123 passed, four ignored; its doctest gate passed with zero cases. Strict workspace and separate C-API all-target Clippy passed with -D warnings. The nonthreading VM compile check passed with defaults disabled and only compiler enabled. Canonical IO repair: all six selected submodules passed, totaling 911 cases (35 existing skips, four retained expected failures). The focused six-case GC regression also passed. All four lifecycle suites were repeated against the repaired tree with the same successful counts above. No unexpected successes, timeouts or altered-environment results. Updater tooling: all 98 migrate, copy-lib and dependency-mapping tests passed. Four unconditional expected-failure guards were retired through updater metadata only after their unchanged bodies produced fresh unexpected successes; the final-source reruns above passed. Independent source review found no confirmed ownership, unlink-order or stop-the-world blocker in the bounded patch. Local validation did not include Miri or cross-platform execution; hosted results are listed below. No human manual verification is claimed. Normal commit hooks passed without bypasses; the committed tree is clean and matches the tested source bytes. Local test_gc coverage excludes exactly PythonFinalizationTests.test_warnings_fini using the standard test runner. That test creates an asyncio AF_UNIX socketpair, which this execution environment previously denied. Its canonical source is unchanged. A selected local run is not full-module coverage. The exact-head hosted runs below supply full-module GC coverage with the existing guards. Existing crash/race, CPU-resource, CPython-only and missing-C-API guards remain in effect unless fresh evidence explicitly supports a permitted retirement. Local IO verification uses the exact canonical general, bufferedio, textio, memoryio, file and univnewlines submodules. General excludes *IOTest.test_large_file_ops (the >2 GiB sparse-file cases) via the standard runner. Direct submodule selection avoids package-wide discovery. test_fileio is not imported locally because it performs an import-time strace capability probe; test_largefile and test_signals are also left to hosted CI because of their socket, large-file and signal effects. The exact-head hosted runs below supply full-package test_io coverage with the existing guards. Canonical source and guards are unchanged by these local selections. The IO closure changes only tests and updater metadata, so its runtime gates reuse the already sealed default release, which does not freeze the standard library. A separate manifest verifies every native/production/build input is unchanged and binds the repaired test tree and exact executable/ELF hashes without replacing the original build provenance. CPython free-threading correction CPython's free-threading design uses the referent stripe for mutable weakref state and retains its lock pointer after the referent is cleared. This correction follows that design: PyWeak keeps the original address solely as a stripe key, never dereferences it, and uses the same stripe for callback access. The separate callback PyRwLock is removed. Ordinary wr() upgrade is unchanged. RustPython collects callbacks and clears payloads after resuming threads, and gc.get_referents() traverses outside stop-the-world. Those paths therefore retain stripe synchronization rather than copying CPython's lockless stopped-world GC operations. Stripe critical sections neither run Python, release Python references nor suspend; callback drops remain outside the lock. No canonical tests, guards or snippets change in this correction. Revision f8a8d643a7 passed a fresh default release build, both workspace/C-API Clippy gates, 1,359 workspace native tests, 123 C-API tests (four ignored), 24 workspace doctests (18 ignored), C-API doctests and the nonthreading VM compile check. The unchanged canonical suites passed again: weakref 138, finalization 18, weakset 46 and selected GC 60 with the exact pre-existing local exclusion documented above. The six canonical IO garbage-collection cases also passed. Normal commit hooks passed, and source/executable/dependency hashes were reverified. No new guards or test changes. These local gates tested the existing branch based on 9a16d284, not a new merge with current main. All eight required checks and all 25 jobs in CI run 39008837863 passed for head f8a8d643a7682d613d5fa9d804d71581191263a6. The older hosted results below apply only to head 429c7718. The completed Ubuntu, macOS and Windows logs explicitly confirm full test_io, test_gc, test_weakref, test_finalization and test_weakset passes. All three checked out merge 2ed8eaf03a87a01be8456c47281ba929cf671236 of this head into main fc9f4a95416bf7ffcc43747c256c2c4a35f996d9. They used -u all, empty SKIPS and no IO/GC selectors; Windows retains its unrelated launcher-version exclusion. Existing platform/resource/CPython guards still apply. Hosted results for earlier head 429c7718 All eight required checks passed for head 429c7718e703b139fadc1b12f417c6bb9d76f351 in CI run 38971792428: lint, three native-test platforms, three snippets/CPython platforms and WASI. The workflow completed successfully. The completed Ubuntu, macOS and Windows logs explicitly show full test_io, test_gc, test_weakref, test_finalization and test_weakset passing. Each checked out merge 7a2bdb7528da48d3bfbfbc1c741561344a3074f1 of the exact head into base 9a16d284. The runs used -u all, empty SKIPS and no IO/GC selectors; Windows retains its unrelated launcher-version exclusion. Existing platform/resource/CPython guards still apply. AI assistance OpenAI Codex extracted and reviewed the source changes, prepared the canonical migration, and performed the listed checks. The exact model version is unavailable. No human review or manual-use verification is claimed here; this is a draft for maintainer review. Summary by CodeRabbit Bug Fixes Improved weak-reference behavior during garbage collection. Callback-free weak references remain usable while finalizers run, while callback-bearing references are cleared at the appropriate time. References to objects that are resurrected during collection are preserved, preventing weak references from being cleared prematurely when those objects remain alive after collection. Weak-reference updates are also coordinated to avoid exposing partially linked or cleared references. Weakref publication ordering Addressed the follow-up review about GC exposing a half-built weakref. PyWeak now opts out of automatic allocation-time tracking; WeakRefList::add explicitly tracks it only after list insertion, referent publication, and stripe release. Generic reuse losers remain untracked. The constructor’s strong root prevents collection, but the old ordering still allowed gc.get_objects to observe a dead-then-live transition. A native allocator-boundary regression checks real allocation headers across ref/proxy/callable-proxy/subclass payloads and verifies completed constructors are tracked. It runs no Python program. Baseline failure is established from the previous allocator/track_new_object source path, not claimed as an executed baseline repro. Canonical GC-during-ref/proxy-creation tests remain unchanged. CPython free-threaded gc.get_objects stops the world; RustPython’s enumeration does not. Deferred tracking follows RustPython’s existing allocation pattern while preserving the publication invariant, without adding another lock. Implemented in 51e0ffe. Fresh final-source local validation passed: workspace native 1,360; C-API 123 (4 existing ignores); workspace doctests 24 (18 existing ignores); C-API doctests; both Clippy gates with warnings denied; nonthreading VM compilation; canonical test_weakref (138), test_finalization (18), test_weakset (46), selected test_gc (60), and all six existing IO garbage-collection regressions. Existing skips/expected failures and the documented local test_warnings_fini exclusion are unchanged. The new allocator regression also checks GC-list membership directly. One initial test-fixture layout assertion was corrected before these final gates. Hosted verification for 51e0ffe is complete: CI run 39014010518 passed all 25 jobs. The full Ubuntu, macOS and Windows Python jobs each passed test_gc, test_io, test_weakref, test_finalization and test_weakset with all resources and empty SKIPS. Windows retains only its unrelated existing launcher-version exclusion. All additional checks are successful or intentionally skipped. Local tests retain the existing 9a16d28 base. Hosted CI tested merge a3b5ff1, whose parents are current main fc9f4a9 and head 51e0ffe. The PR is mergeable and awaiting review. Earlier hosted results above belong to f8a8d64. CPython lock-handle alignment Revision 5cbebf40c018a8ff8d694f44df124eea83f4b28f follows CPython free-threading more directly: PyWeak.weakrefs_lock retains a safe reference to the selected existing static stripe, rather than saving and rehashing the referent address. CPython likewise stores PyMutex *weakrefs_lock, initializes it from the referent's stripe, and uses it after referent clearing. The nonthreaded handle is zero-sized. Stripe selection, acquire/release ordering, fork reset, deferred GC publication and callback-owner release outside the lock remain unchanged. This adds no new locks or unsafe code. The earlier address-key description above records the previous revision. Fresh final-source validation passed: 1,360 workspace native tests, 123 C-API tests (4 existing ignores), 24 workspace doctests (18 existing ignores), C-API doctests, both Clippy gates with warnings denied, nonthreading VM compilation, canonical weakref (138), finalization (18), weakset (46), selected GC (60), and six canonical IO-GC regressions. Existing guards and the documented local test_warnings_fini exclusion remain unchanged. Normal commit hooks passed. Hosted verification is complete for this head: CI run 39037908394 passed all 25 jobs. Full Ubuntu, macOS and Windows Python logs each confirm test_gc, test_io, test_weakref, test_finalization and test_weakset passed. All 30 reported checks are successful, with one intentional pyperformance skip. Prior hosted results above apply to older revisions.
Open Graph Description: Align weak-reference collection with Python 3.15 finalization Summary Extract the weakref/finalizer lifecycle work from #8954 onto main's existing collector. Follow CPython free-threading: ret...
X Description: Align weak-reference collection with Python 3.15 finalization Summary Extract the weakref/finalizer lifecycle work from #8954 onto main's existing collector. Follow CPython free-threading:...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9168
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:1e319787-665d-3d95-f91f-6dfe737e5cc5 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9E40:19E4BB:356F4FF:442D61F:6B01E955 |
| html-safe-nonce | ed6e6c2d74859b8fe4412a51dc746d93a6a7771f43d719ef6e8699a51f021f3f |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5RTQwOjE5RTRCQjozNTZGNEZGOjQ0MkQ2MUY6NkIwMUU5NTUiLCJ2aXNpdG9yX2lkIjoiMTgxNjU4NjY4MTk3ODA1NDk5NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 00f27834afb155f0e35521ca0f236ee566be7439add535518499abbe07c6384a |
| hovercard-subject-tag | pull_request:4894267557 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9168/files |
| twitter:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| og:image:alt | Align weak-reference collection with Python 3.15 finalization Summary Extract the weakref/finalizer lifecycle work from #8954 onto main's existing collector. Follow CPython free-threading: ret... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width