René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"asyncio.run_coroutine_threadsafe leaves underlying cancelled asyncio task running","articleBody":"# Bug report\r\n\r\nWhen an `asyncio` task is created from another thread via `asyncio.run_coroutine_threadsafe`, a `concurrent.futures.Future` is returned, which wraps an underlying `asyncio.Task`. \r\n\r\nIn a typical use `future.result()` or `future.exception()` can be used to wait for completion. However, when the returned future is cancelled, no built-in mechanism is provided to wait for the underlying `asyncio` task to complete, leaving it running when application is terminated, possibly accessing dangling data.\r\n\r\nThis is an example that reproduces what is described above. There's no way to make it smaller without losing some aspect of what's going on.\r\n\r\nThe output of this code is as follows:\r\n```\r\nAbout to create an asyncio task\r\nasyncio task is set up\r\nasyncio task is running\r\nTask cancelled (callback)\r\nCancelled?: True\r\nDone?: True\r\nCannot access task.result()\r\nCannot access task.exception()\r\nAll done. No app code should run after this.\r\nasyncio task is cancelled; rethrowing...   \u003c-- runs after the app is shut down\r\nSetting the asyncio task event             \u003c-- same\r\n```\r\nPerhaps `concurrent.futures.Future.cancelled()` should have a timeout to wait for the underlying `asyncio` task to complete or some additional method provided for that.\r\n\r\nHere's the sample code.\r\n```python\r\nimport asyncio\r\nimport concurrent.futures\r\nimport threading\r\n\r\nclass MyApp:\r\n    def __init__(self) -\u003e None:\r\n        self.task: concurrent.futures.Future[int]|None = None\r\n\r\n        self.task_event = asyncio.Event()\r\n        self.task_event.clear()\r\n\r\n    # An asyncio task that runs in the specified event loop\r\n    async def process_task(self) -\u003e int:\r\n        try:\r\n            self.task_event.clear()\r\n            print(\"asyncio task is running\")\r\n            await asyncio.sleep(10)\r\n            print(\"asyncio task is finished\")\r\n            return 123\r\n        except asyncio.CancelledError:\r\n            print(\"asyncio task is cancelled; rethrowing...\")\r\n            raise\r\n        finally:\r\n            print(\"Setting the asyncio task event\")\r\n            self.task_event.set()\r\n\r\n    # Called immediately from task.cancel(), before the underlying\r\n    # asyncio task completes.\r\n    def task_done(self, future: concurrent.futures.Future[int]) -\u003e None:\r\n        if future.cancelled():\r\n            print(\"Task cancelled (callback)\")\r\n        elif future.exception() is not None:\r\n            print(\"Task error %s (callback)\" % future.exception())\r\n        else:\r\n            print(\"Task is done with result %d (callback)\" % future.result())\r\n\r\n    def start_async_task(self, thread_event: threading.Event) -\u003e None:\r\n\r\n        print(\"About to create an asyncio task\")\r\n\r\n        self.task = asyncio.run_coroutine_threadsafe(self.process_task(), asyncio.get_running_loop())\r\n        self.task.add_done_callback(self.task_done)\r\n\r\n        print(\"asyncio task is set up\")\r\n        thread_event.set()\r\n\r\n    # This method is supposed to be the last call to the application,\r\n    # which shuts down all services and cancels all outstanding work.\r\n    async def shutdown(self) -\u003e None:\r\n\r\n        # Calls the done callback immediately and returns before the\r\n        # future cancels the underlying asyncio task, rendering the\r\n        # future unusable via standard methods.\r\n        self.task.cancel()\r\n\r\n        # This event compensates for lack of ability to wait for the\r\n        # underlying asyncio task. Without this wait, asyncio task\r\n        # runs after the code that manages all this is gone.\r\n        #\r\n        # Perhaps Future.cancel() should provide this functionality?\r\n        #\r\n        #print(\"Waiting for a task to complete\")\r\n        #await self.task_event.wait()\r\n\r\n        # returns True, while underlying asyncio task is still running\r\n        print(\"Cancelled?: %s\" % self.task.cancelled())\r\n        print(\"Done?: %s\" % self.task.done())\r\n\r\n        # supposed to wait for result, but because it's cancelled, will throw\r\n        try:\r\n            print(\"Result?: %d\" % self.task.result())\r\n        except concurrent.futures.CancelledError:\r\n            print(\"Cannot access task.result()\")\r\n\r\n        # or wait for exception, but because it's cancelled, will throw as well\r\n        try:\r\n            print(\"Exception?: %s\" % str(self.task.exception()))\r\n        except concurrent.futures.CancelledError:\r\n            print(\"Cannot access task.exception()\")\r\n\r\nasync def main() -\u003e None:\r\n    myapp = MyApp()\r\n\r\n    thread_event = threading.Event()\r\n    thread_event.clear()\r\n\r\n    # emulates some worker thread (e.g. APScheduler)\r\n    runner = threading.Thread(target=myapp.start_async_task, args=[thread_event])\r\n    runner.run()\r\n\r\n    thread_event.wait()\r\n\r\n    await asyncio.sleep(1)\r\n\r\n    await myapp.shutdown()\r\n\r\n    await asyncio.sleep(0)\r\n\r\n    print(\"All done. No app code should run after this.\")\r\n\r\nasyncio.run(main())\r\n```\r\nIf the `wait` call under _Waiting for a task to complete_ is uncommented, which would simulate having waiting implemented in `Future.cancelled()`, it would work as expected.\r\n\r\n# Your environment\r\n\r\n- CPython versions tested on: 3.10.10\r\n- Operating system and architecture: Microsoft Windows \\[Version 10.0.19045.3086\\]\r\n\r\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-141696\n* gh-142358\n* gh-142359\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/gh-andre","@type":"Person","name":"gh-andre"},"datePublished":"2023-06-15T19:53:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":14},"url":"https://github.com/105836/cpython/issues/105836"}