René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BackgroundConsumer destruction issues","articleBody":"We [recently merged a PR](https://github.com/googleapis/python-api-core/pull/770) that clears some variables that were causing a memory leak in the bidirectional streaming classes. It seems like after getting that fix in, it exposed a deeper issue: the [background thread isn't exiting as expected](https://github.com/googleapis/python-api-core/blob/34babf1a08219ff74c80d797352da760a760cd22/google/api_core/bidi.py#L719C1-L720C1).\n\n### Quick Fix\n\nWhen this happens, it's leading to a null pointer exception [here](https://github.com/googleapis/python-api-core/blob/34babf1a08219ff74c80d797352da760a760cd22/google/api_core/bidi.py#L667), now that the _on_response can be empty. So as a quick fix, we should add a None check there\n\n### Deeper Issue\n\nThe root issue is that the BackgroundConsumer's background thread is getting stuck calling [_on_response](https://github.com/googleapis/python-api-core/blob/34babf1a08219ff74c80d797352da760a760cd22/google/api_core/bidi.py#L667) while being torn down. In the case of Firestore, it looks like the deadlock is the fault of [the Watch class](https://github.com/googleapis/python-firestore/blob/main/google/cloud/firestore_v1/watch.py#L61), which tries to re-enter a lock. But I think we should consider changing the BackgroundConsumer to close quicker, without doing that final on_response call at all. Is it really necessary to call that callback while being closed?\n\n\n### More Context\nhttps://github.com/googleapis/python-firestore/issues/985#issuecomment-2622815245\n\nLet me know if you want me to submit a PR","author":{"url":"https://github.com/daniel-sanche","@type":"Person","name":"daniel-sanche"},"datePublished":"2025-02-01T00:27:37.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/788/python-api-core/issues/788"}