René's URL Explorer Experiment

Title: gh-159963: Don't hold parent package import locks while a submodule executes by gpshead · Pull Request #159965 · python/cpython · GitHub

Open Graph Title: gh-159963: Don't hold parent package import locks while a submodule executes by gpshead · Pull Request #159965 · python/cpython

X Title: gh-159963: Don't hold parent package import locks while a submodule executes by gpshead · Pull Request #159965 · python/cpython

Description: Since gh-83065, the first import of pkg.mod kept pkg's module lock until pkg/mod.py had finished executing. This shipped in 3.15.0. Threads needing only the finished pkg were blocked for that long, and concurrent first imports could raise _DeadlockError or a spurious "partially initialized module" ImportError, or hang, with no import cycle involved. Parent packages are now imported, or waited for, before the submodule's lock is taken, and only that lock is held while it executes. The original gh-83065 deadlock stays fixed. importlib.import_module() now returns the partially initialized module in a cross-thread circular import, as the import statement already did, instead of raising _DeadlockError. This part is not the 3.15.0 regression: import_module() has raised _DeadlockError there since at least 3.12 (3.12, 3.13, 3.14 and main all do), so it can be judged separately for the backport. Two threads, step by step The scenario from test_concurrent_first_imports_of_sibling_packages: nothing is imported yet, a/leaf.py does from b.leaf import X and b/__init__.py does import a. The dependencies are a.leaf → b.leaf → b → a, so there is no import cycle. Thread 2 starts while thread 1 is still inside a/__init__.py. Before (3.15.0)After (this PR) sequenceDiagram participant T1 as Thread 1
import a.leaf participant T2 as Thread 2
import b.leaf Note over T1: 1. acquire locks a and a.leaf
run a/__init__.py Note over T2: 2. acquire locks b and b.leaf
run b/__init__.py (import a) T2-->>T1: 3. wait for lock a Note over T1: 4. a/__init__.py returns
lock a is NOT released
run a/leaf.py (from b.leaf import X) T1--xT2: 5. needs lock b.leaf, but T2 is waiting on T1 Note over T1: 6. _DeadlockError Loading sequenceDiagram participant T1 as Thread 1
import a.leaf participant T2 as Thread 2
import b.leaf Note over T1: 1. acquire lock a
run a/__init__.py Note over T2: 2. acquire lock b
run b/__init__.py (import a) T2-->>T1: 3. wait for lock a Note over T1: 4. a/__init__.py returns
release lock a Note over T1: 5. acquire lock a.leaf
run a/leaf.py (from b.leaf import X) Note over T2: 6. a is ready, b/__init__.py returns, release lock b
acquire lock b.leaf, run b/leaf.py T1-->>T2: 7. wait for lock b.leaf Note over T2: 8. release lock b.leaf
import b.leaf done Note over T1: 9. X imported, release lock a.leaf
import a.leaf done Loading Before, thread 1 keeps lock a after a/__init__.py has returned, so thread 2, which needs only the finished a, stays blocked while holding the locks thread 1 needs next. After, each thread holds one module lock at a time and both imports complete. Steps 5 to 8 of the right-hand side can interleave differently (thread 1 may load b.leaf itself); either way neither thread waits on a lock while holding one the other needs. authored using Claude Fable Issue: gh-159963

Open Graph Description: Since gh-83065, the first import of pkg.mod kept pkg's module lock until pkg/mod.py had finished executing. This shipped in 3.15.0. Threads needing only the finished pkg were blocked for that...

X Description: Since gh-83065, the first import of pkg.mod kept pkg's module lock until pkg/mod.py had finished executing. This shipped in 3.15.0. Threads needing only the finished pkg were blocked for ...

Opengraph URL: https://github.com/python/cpython/pull/159965

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width