René's URL Explorer Experiment

Title: gh-123471: Make itertools.chain thread-safe by eendebakpt · Pull Request #135689 · python/cpython · GitHub

Open Graph Title: gh-123471: Make itertools.chain thread-safe by eendebakpt · Pull Request #135689 · python/cpython

X Title: gh-123471: Make itertools.chain thread-safe by eendebakpt · Pull Request #135689 · python/cpython

Description: The itertools.chain has two attributes that are mutated duration iteration, making it non-thread safe. Options to make it thread safe i) Use a lock (simple, but adds a performance penalty for the single-threaded case) ii) Make the iterator thread safe using atomics. For the second option we can stop clearing the lz->source on exhaustion (this is a standard mitigation to avoid setting lz->source to zero (and doing a decref) while another thread is still using lz->source). To signal the iterator that lz->source is exhausted we can either add a new attribute, or replace the while (lz->source != NULL) with while (true). This creates a slower path the the iterator is exhausted, but typically once the iterator is exhausted performance is not relevant any longer. The lz->active is more problematic. Once lz->active is exhausted, we need to update this with a new one (e.g. PyIter_Next(lz->source). But lz->active only has a single reference count. So updating in one thread means having to decref the old lz->active. Possible mitigations: a) atomically load-and-incref the lz->active in chain_next b) add a new attribute that contains all the exhausted values of lz->active and clear this once the chainobject itself is deallocated. The second option is complex and has some performance issues (e.g. more incref/decrefs, keeping references to exhausted iterators longer in memory), so in this PR we pick the first option. Issue: gh-123471

Open Graph Description: The itertools.chain has two attributes that are mutated duration iteration, making it non-thread safe. Options to make it thread safe i) Use a lock (simple, but adds a performance penalty for the s...

X Description: The itertools.chain has two attributes that are mutated duration iteration, making it non-thread safe. Options to make it thread safe i) Use a lock (simple, but adds a performance penalty for the s...

Opengraph URL: https://github.com/python/cpython/pull/135689

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width