René's URL Explorer Experiment

Title: gh-127266: avoid data races when updating type slots by nascheme · Pull Request #131174 · python/cpython · GitHub

Open Graph Title: gh-127266: avoid data races when updating type slots by nascheme · Pull Request #131174 · python/cpython

X Title: gh-127266: avoid data races when updating type slots by nascheme · Pull Request #131174 · python/cpython

Description: Avoid data races when updating type slots and type flags by "stopping-the-world" first. This is only needed if the type is potentially exposed to multiple threads. There are a couple places this happens: type_setattro() when the assigned item is a "dunder" method that corresponds to one or more slot assigning to __bases__ _abc.c calling _PyType_SetFlags() to set the collection flags (SEQUENCE or MAPPING) Two examples of code that causes this: dataclasses will assign dunder methods after the type object has been created, when the decorator runs. That triggers this and since the slots are assigned one-by-one, we stop for each assignment. Not great. Another example is unittest MagicMock. Again, it assigns dunder methods after the class is defined. In order to assess the performance impact of this change, I've measured the following things. The time it takes to run the unit test suite does not seem significantly worse, even when running tests serially using -j 1. Comparing the pyperformance results before and after this change shows no significant change. See link below for the chart. Running with TSAN enabled and running the --tsan and --tsan-parallel --parallel-threads=4 tests show no warnings. I also have a modified pyperformance test suite that I can run in parallel and that shows no TSAN warnings related to this as well. Additional changes after feedback from Sam: Replace the critical sections that use TYPE_LOCK with a simple mutex instead. To avoid dead-locks, the code was analyzed for re-entrancy and adjusted as needed. There are a couple places where the mutex is dropped before calling something that's possibly re-entrant. Remove some locking where it is not needed. Slots can only be changed when the world is stopped (e.g. reading tp_mro no longer needs to hold the lock). Create type_lookup_ex() as a lower-level version of _PyType_LookupStackRefAndVersion that optionally acquires the TYPE_LOCK or requires that it is already held. Add a new type flag that is used for debug enabled builds. It tries to check if a type is potentially exposed to multiple threads (and therefore needs stop-the-world before modifying slots). This flag could perhaps be removed before the final merge of this PR (since type flag bits are kind of in short supply). Benchmark using pyperformance Issue: gh-127266

Open Graph Description: Avoid data races when updating type slots and type flags by "stopping-the-world" first. This is only needed if the type is potentially exposed to multiple threads. There are a couple pl...

X Description: Avoid data races when updating type slots and type flags by "stopping-the-world" first. This is only needed if the type is potentially exposed to multiple threads. There are a c...

Opengraph URL: https://github.com/python/cpython/pull/131174

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width