René's URL Explorer Experiment


Title: gh-91351: Fix some bugs in importlib handling of re-entrant imports by exarkun · Pull Request #94504 · python/cpython · GitHub

Open Graph Title: gh-91351: Fix some bugs in importlib handling of re-entrant imports by exarkun · Pull Request #94504 · python/cpython

X Title: gh-91351: Fix some bugs in importlib handling of re-entrant imports by exarkun · Pull Request #94504 · python/cpython

Description: Fixes #91351 Update of #94342 See #91351 for details about the problem. Re-entrancy is always tricky and given the requirements of _bootstrap.py (to operate with re-entrancy and multi-threading and to do so without exposing any of the details to application code doing an import) I think this goes double. This PR does a few things to achieve better safety in the face of re-entrancy: Switch some data structures to those that support atomic operation so that they are consistent in case of asynchronous re-entrancy (eg from the garbage collector or a signal handler). Use RLock and add more RLock-like behavior to prevent deadlocks in the re-entrant case. Update the deadlock detection algorithm to support the fact that one thread might be "blocked on" acquiring the module lock for more than one module at a time. I'm not quite sure I believe this new version of the code is 100% correct with respect to re-entrancy but it does fixes mishandling of two specific cases: A re-entrant import is performed between the time _blocking_on is populated and cleaned up inside _ModuleLock.acquire. Previous this failed with a KeyError (as described in the linked issue). A re-entrant import is performed while the module lock is held inside _ModuleLock.acquire. Previously this failed by deadlocking. This PR also does not include any new unit tests. I have a small stand-alone program which can reproduce both of these but only with the assistance of some additional instrumentation inside _bootstrap.py to make sure the re-entrancy happens at the interesting times. If adding this kind of instrumentation is acceptable then it may be possible to turn this program into some unit tests. Compared to the previous PR, this one is a bit simpler because it uses RLock in a place where main currently uses a regular Lock. For reference, here is a stand-alone reproducer. This one very reliably reproduces the KeyError problem (by exercising the codepath forever until it hits it). I haven't managed to create a deterministic reproducer for the deadlock case - except with the assistance of instrumentation inside _bootstrap.py itself. import sys, socket, gc class Cycle: pass def a_cycle(): c = Cycle() c.cycle = c c.s = socket.socket() def main(): while True: # import a module that socket.__del__ is going to import to exercise # re-entrant _ModuleLock.lock handling a_cycle() import linecache del sys.modules["linecache"] main() Issue: gh-91351

Open Graph Description: Fixes #91351 Update of #94342 See #91351 for details about the problem. Re-entrancy is always tricky and given the requirements of _bootstrap.py (to operate with re-entrancy and multi-threading and...

X Description: Fixes #91351 Update of #94342 See #91351 for details about the problem. Re-entrancy is always tricky and given the requirements of _bootstrap.py (to operate with re-entrancy and multi-threading and...

Opengraph URL: https://github.com/python/cpython/pull/94504

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:569da09c-290e-d219-bb43-495cc5094023
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idA4C2:AA16D:6FDC1:9CB12:696A0C49
html-safe-nonce18232606ffa335c832db4e4b2df479ec7bb6f30f1cb1abc572f38f202bc05fa4
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNEMyOkFBMTZEOjZGREMxOjlDQjEyOjY5NkEwQzQ5IiwidmlzaXRvcl9pZCI6Ijg3MTg3OTIwNDc5NjQzMjY5ODUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac87a76fe12aa923739a32f983ef332c470e0712d72d92515070c61dd162006771
hovercard-subject-tagpull_request:985396885
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/94504/files
twitter:imagehttps://avatars.githubusercontent.com/u/254565?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/254565?s=400&v=4
og:image:altFixes #91351 Update of #94342 See #91351 for details about the problem. Re-entrancy is always tricky and given the requirements of _bootstrap.py (to operate with re-entrancy and multi-threading and...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None699227a00bbb7fe1eec276d2ae1c3a93068bc5ba483bd9dc4b2a27a8f4f2f595
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release7266b2d935baa1c6474b16dd9feaa5ca30607261
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/94504/files#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F94504%2Ffiles
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F94504%2Ffiles
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/94504/files
Reloadhttps://github.com/python/cpython/pull/94504/files
Reloadhttps://github.com/python/cpython/pull/94504/files
python https://github.com/python
cpythonhttps://github.com/python/cpython
Please reload this pagehttps://github.com/python/cpython/pull/94504/files
Notifications https://github.com/login?return_to=%2Fpython%2Fcpython
Fork 33.9k https://github.com/login?return_to=%2Fpython%2Fcpython
Star 71.1k https://github.com/login?return_to=%2Fpython%2Fcpython
Code https://github.com/python/cpython
Issues 5k+ https://github.com/python/cpython/issues
Pull requests 2.1k https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects 31 https://github.com/python/cpython/projects
Security Uh oh! There was an error while loading. Please reload this page. https://github.com/python/cpython/security
Please reload this pagehttps://github.com/python/cpython/pull/94504/files
Insights https://github.com/python/cpython/pulse
Code https://github.com/python/cpython
Issues https://github.com/python/cpython/issues
Pull requests https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects https://github.com/python/cpython/projects
Security https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
brettcannonhttps://github.com/brettcannon
python:mainhttps://github.com/python/cpython/tree/main
PrivateStorageio:91351-importlib-reentrancyhttps://github.com/PrivateStorageio/cpython/tree/91351-importlib-reentrancy
Conversation 57 https://github.com/python/cpython/pull/94504
Commits 35 https://github.com/python/cpython/pull/94504/commits
Checks 0 https://github.com/python/cpython/pull/94504/checks
Files changed https://github.com/python/cpython/pull/94504/files
Please reload this pagehttps://github.com/python/cpython/pull/94504/files
gh-91351: Fix some bugs in importlib handling of re-entrant imports https://github.com/python/cpython/pull/94504/files#top
Show all changes 35 commits https://github.com/python/cpython/pull/94504/files
2a4d4f3 Add some implementation documentation exarkun Jun 27, 2022 https://github.com/python/cpython/pull/94504/commits/2a4d4f3d4b48bbfa12f1a461d1a8422ef4d99d1d
c9e33c7 Move _blocking_on management into a context manager exarkun Jun 27, 2022 https://github.com/python/cpython/pull/94504/commits/c9e33c796160df6d5817b6fa6e887b0455c361ea
c5912bf Support re-entrant imports in _BlockingOnManager exarkun Jun 30, 2022 https://github.com/python/cpython/pull/94504/commits/c5912bf5a949fb5086406d5015895e78653ae06b
27ae2f0 Apply formatting and comment changes from review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/27ae2f07ceb5e16191bdc8cb46b821edef7b742e
1fcc78b Merge remote-tracking branch 'origin/main' into 91351-importlib-reent… exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/1fcc78b634661b7e53105857980f310289fdcd21
5fd15d8 Rename _BlockingOnManager.tid as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/5fd15d836aaf5545236473318c684d9c7d9c3f62
7a24f2c flip the first two arguments to _has_deadlock as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/7a24f2c50a98aa277fb656de5964f43c57fec946
08892b4 Mark up parameters following PEP 257 as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/08892b4c957eca6e575f7228a282a8836d7e17ef
59b53c0 rename the `_blocking_on` parameter as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/59b53c01870cf318995381a30b7e63d27ee684a5
20007c5 further document motivation for `blocking_on` parameter as suggested … exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/20007c598beb4453c4ea9e1d2db5ca3fe03c9ccb
bad1d3c Rename more _has_deadlocked parameters as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/bad1d3c5e703d8fe0490076ddcce2f518ee84470
2821fcf Treat None and [] the same for this case as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/2821fcf6f01d542eda64659e7ab30c719e5e2897
95c73cb update old comment to refer to new names as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/95c73cb6269f37058a9fb7f02f7c6a60e7e8b0b3
49ff9dd Make _ModuleLock.count a list of True as suggested by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/49ff9ddf03f2b9ae74e06ccc6bfa7e3022dbfad7
cd174a8 Adjust the check for a module lock being released as suggest by review exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/cd174a82dbb68ff27d1d6f6e8bb5a54958e20b03
719b181 Finish the _BlockingOnManager.tid renaming exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/719b1812fef19e99fa60dda7fcb2e7442cc18a97
6e809cd Fix renaming of `_blocking_on` parameter to `_has_deadlocked` exarkun Nov 28, 2022 https://github.com/python/cpython/pull/94504/commits/6e809cdabab3b6b240b9a96b450a4b367732a9a6
74cbccd Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/74cbccdd305c46402487aeb06620f3a4971c1a53
c579533 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/c57953362ecf8dd89493c6d5f35aa45e6cc72eae
decb70b Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/decb70b3983d207a33f1187fb0ee3fb26c51711d
3c91cc3 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/3c91cc32863280383a15b8a1ef571d39c6482a4c
e032ae2 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/e032ae2fc3e95e16d8b1bbbf740d9f70d185205f
dba393a Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/dba393a7cec5e02889745c112f5580a7a611a0c4
25d554b Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/25d554bcac3645099571876f91f639447617154f
44157a9 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/44157a98aa08e1c6b8e4dcd5abef504e79d8d2c1
f99ed46 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/f99ed46b7f209c5554246d2f779e7fbf0ba9dda0
b6d21f8 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/b6d21f8e468f65ccf3723ecf1d21d193570f9974
5643442 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/5643442f7e69ef8569aed9d0c5b379ace269182e
92036a8 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/92036a8b11406e9d95c047170b1e929f790ad689
bf14ce2 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/bf14ce28bd056cd1a72f97be931c30984612bde0
1cc6033 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/1cc6033dfbb3f3f92f5a2d44a22c4b0ff1d2d079
ab40737 Apply review suggestion exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/ab4073768fb32dda48f9c4f52a782053654143fc
36082eb news blurb exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/36082eb81057fe16edc41831846229ce726b4cde
55fc599 Merge branch 'main' into 91351-importlib-reentrancy exarkun Jan 6, 2023 https://github.com/python/cpython/pull/94504/commits/55fc5995529bb2dfaa02125c4249bc6ba08d9b1d
b35f0a8 Apply suggestions from code review exarkun Jan 14, 2023 https://github.com/python/cpython/pull/94504/commits/b35f0a833df3ce93ad335c55e8fcffadf2666431
Clear filters https://github.com/python/cpython/pull/94504/files
Please reload this pagehttps://github.com/python/cpython/pull/94504/files
Please reload this pagehttps://github.com/python/cpython/pull/94504/files
_bootstrap.py https://github.com/python/cpython/pull/94504/files#diff-dba786ae26d5505c4c367a93b1179c2f80f2433d983b84bacb9001fb552f0761
2023-01-06-09-22-21.gh-issue-91351.iq2vZ_.rst https://github.com/python/cpython/pull/94504/files#diff-f6122560aba1eca3974133e8f0cd5417f2490bb8317a7411e101c8e9038bfdae
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L260
Lib/importlib/_bootstrap.pyhttps://github.com/python/cpython/pull/94504/files#diff-dba786ae26d5505c4c367a93b1179c2f80f2433d983b84bacb9001fb552f0761
View file https://github.com/PrivateStorageio/cpython/blob/b35f0a833df3ce93ad335c55e8fcffadf2666431/Lib/importlib/_bootstrap.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/94504/{{ revealButtonHref }}
https://github.com/python/cpython/pull/94504/files#diff-dba786ae26d5505c4c367a93b1179c2f80f2433d983b84bacb9001fb552f0761
https://github.com/python/cpython/pull/94504/files#diff-dba786ae26d5505c4c367a93b1179c2f80f2433d983b84bacb9001fb552f0761
https://github.com/python/cpython/pull/94504/files#diff-dba786ae26d5505c4c367a93b1179c2f80f2433d983b84bacb9001fb552f0761
Misc/NEWS.d/next/Core and Builtins/2023-01-06-09-22-21.gh-issue-91351.iq2vZ_.rsthttps://github.com/python/cpython/pull/94504/files#diff-f6122560aba1eca3974133e8f0cd5417f2490bb8317a7411e101c8e9038bfdae
View file https://github.com/PrivateStorageio/cpython/blob/b35f0a833df3ce93ad335c55e8fcffadf2666431/Misc/NEWS.d/next/Core%20and%20Builtins/2023-01-06-09-22-21.gh-issue-91351.iq2vZ_.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/94504/{{ revealButtonHref }}
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.