René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"asyncio: Use strong references for free-flying tasks","articleBody":"In #88831 @vincentbernat pointed out that CPython only keeps weak references in [`_all_tasks`](https://github.com/python/cpython/blob/090721721b373c50544d297b56c217cf15992cbe/Lib/asyncio/tasks.py#L933), so a reference to a `Task` returned by `loop.create_task` has to be kept to be sure the task will not be killed with a \"Task was destroyed but it is pending!\" at some random point in time.\r\n\r\nWhen shielding a task from cancellation with `await shield(something())`, `something` continues to run when the containing coroutine is cancelled. As soon as that happens, `something()` is free-flying, i.e. there's no reference from user code anymore. `shield` itself has a bunch of circular strong references, but these shouldn't keep CPython from garbage-collecting the task. Hence, here the same problem occurs and the task might be killed unpredictably. Additionally, when running coroutines in parallel with `gather` and `return_exceptions=False`, an exception in one of the coroutines will leave remaining tasks free-flying. Also in this case, the remaining tasks might be killed unpredictably.\r\n\r\nHence, a warning in the documentation for `create_task` unfortunately does not suffice to solve the problem. Additionally, it has been brought up in #88831 that an API for fire-and-forget tasks (i.e. when the user doesn't want to keep a reference) would be nice.\r\n\r\nAs solution, I suggest to either\r\n\r\n(1) introduce a further `_pending_tasks` set to keep strong references to all pending tasks. This would be the simplest solution also with respect to the API. In fact, a lot of dicussions on Stack Overflow (e.g., [here](https://stackoverflow.com/a/65921782), [here](https://stackoverflow.com/a/63425909), [here](https://stackoverflow.com/a/56356487)) already rely on this behavior (throwing away the reference returned by `create_task`), although it's wrong currently. Since the behaviour for free-flying tasks is unpredictable currently, it should not introduce any compatibility issues when making it predictable by preventing them from being garbage-collected.\r\n\r\n(2) make sure there's always a chain of strong references from the most basic futures to the running tasks awaiting something. A quick `grep` resulted in potential problems, e.g., [here](https://github.com/python/cpython/blob/090721721b373c50544d297b56c217cf15992cbe/Lib/asyncio/selector_events.py#L57) and [here](https://github.com/python/cpython/blob/090721721b373c50544d297b56c217cf15992cbe/Lib/asyncio/streams.py#L198). This does not seem like a very robust approach, though.\r\n\r\n(3) introduce the concept of _background tasks_, i.e., tasks the user does not want to hold references to. The interface could look like suggested in https://github.com/python/cpython/issues/88831#issuecomment-1105619239 . Tasks from `shield` and `gather` could be automatically converted to such background tasks. Clearly, it would add complexity to the API, but the distinction between normal tasks and background tasks might potentially be beneficial also for other purposes. E.g., one might add an API call that waits for all background tasks to be completed.\r\n\r\nMy preferred solution would be (1).\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-121264\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/alexhartl","@type":"Person","name":"alexhartl"},"datePublished":"2022-04-24T19:25:49.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":52},"url":"https://github.com/91887/cpython/issues/91887"}