René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Cancel futures if concurrent.futures.Executor is used as a context manager and there is an exception?","articleBody":"I bumped into a bit of a rough edge related to how `concurrent.futures.Executor` instances behave when they're used as context managers and there is an exception raised from the with statement body.\n\nI had some code like this (reduced to the bare minimum):\n\n```\nwith concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:\n    futures = {\n        executor.submit(check_status, item): item\n        for item in to_check\n    }\n    for index, future in enumerate(concurrent.futures.as_completed(futures)):\n        print(f\"{index}/{len(futures)} {future.result()}\")\n```\n\nNow, as it happens `check_status` liked to raise an exception from time to time and this would be fine if the application stopped because of that but it kept going but in a weird way. What I saw was `check_status` kept producing logs but the `print` calls in the snippet above stopped being executed.\n\nThis is because of how `Executor` implements `__exit__` – it waits for all pending futures to complete before propagating the exception:\n\n```\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        self.shutdown(wait=True)\n        return False\n```\n\nNo in my case I started with thousands of futures so it could take hours for the process to finally crash.\n\nSince Python 3.9 the `shutdown` method has the `cancel_futures` parameter (implemented in https://github.com/python/cpython/pull/18057): https://docs.python.org/3.9/library/concurrent.futures.html#concurrent.futures.Executor.shutdown.\n\nI'm proposing changing the behavior of the context manager `__exit__` to cancel futures if there is an exception:\n\n```\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        self.shutdown(wait=True, cancel_futures=exc_val is not None)\n        return False\n```\n\nThis would allow the in-flight futures to finish but any further processing wouldn't happen and the exception would be reported in a (more) timely fashion.\n\ncc @brianquinlan @pitrou as I saw you participate in the `cancel_futures` PR, if there's someone else I should ping here for visibility please let me know.","author":{"url":"https://github.com/jstasiak","@type":"Person","name":"jstasiak"},"datePublished":"2025-03-08T11:29:09.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/130975/cpython/issues/130975"}