René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Reduce asyncio heapq overhead","articleBody":"# Feature or enhancement\r\n\r\n### Proposal:\r\n\r\nThe `heapq` calls in base_events.py represents quite a bit of asyncio scheduling overhead because they have to run `__lt__` quite often in `TimerHandle` https://github.com/python/cpython/blob/0fd97e46c75bb3060485b796ca597b13af7e6bec/Lib/asyncio/events.py#L128\r\n\r\nhttps://github.com/python/cpython/blob/0fd97e46c75bb3060485b796ca597b13af7e6bec/Lib/asyncio/base_events.py#L1968\r\n![heapify](https://github.com/user-attachments/assets/f91833d3-3c7b-45fe-9fde-012fce8684c6)\r\n\r\nhttps://github.com/python/cpython/blob/0fd97e46c75bb3060485b796ca597b13af7e6bec/Lib/asyncio/base_events.py#L815\r\n![heappush](https://github.com/user-attachments/assets/762df6a1-ddf4-4e85-8987-71ece0cc49c4)\r\n\r\n\r\nhttps://github.com/python/cpython/blob/0fd97e46c75bb3060485b796ca597b13af7e6bec/Lib/asyncio/base_events.py#L1975\r\n![heappop](https://github.com/user-attachments/assets/13951125-990c-4709-828d-494e5bf9337e)\r\n\r\n\r\nAvoiding running `__lt__` can speed up processing `call_at`s by ~10%\r\n\r\n\r\n### Has this already been discussed elsewhere?\r\n\r\nThis is a minor feature, which does not need previous discussion elsewhere\r\n\r\n### Links to previous discussion of this feature:\r\n\r\nWrapping `TimerHandle` in a tuple starting with `when` avoids the `__lt__` call.  Thank you to whoever wrote the `heapq` docs for help getting there https://docs.python.org/3/library/heapq.html#basic-examples \r\n6f80b4c8b7be\r\n\r\n\r\nExample benchmark\r\n```python\r\nfrom asyncio import TimerHandle\r\nimport heapq\r\nimport timeit\r\n\r\n\r\ndef callback():\r\n    \"\"\"This is the callback function that will be called when the timer expires.\"\"\"\r\n\r\n\r\nclass MockLoop:\r\n    def get_debug(self):\r\n        return False\r\n\r\n\r\nloop = MockLoop()\r\n\r\n\r\ndef heap_tuple():\r\n    scheduled = []\r\n    when = 1\r\n\r\n    for _ in range(100):\r\n        when += 1\r\n        handle = TimerHandle(when, callback, (), loop)\r\n        heapq.heappush(scheduled, (when, handle))\r\n\r\n    while scheduled:\r\n        when, handle = heapq.heappop(scheduled)\r\n\r\n\r\ndef heap_handle():\r\n    scheduled = []\r\n    when = 1\r\n\r\n    for _ in range(100):\r\n        when += 1\r\n        handle = TimerHandle(when, callback, (), loop)\r\n        heapq.heappush(scheduled, handle)\r\n\r\n    while scheduled:\r\n        handle = heapq.heappop(scheduled)\r\n\r\n\r\nprint(\"wrap when, TimerHandle in tuple\", timeit.timeit(heap_tuple))\r\nprint(\"bare TimerHandle\", timeit.timeit(heap_handle))\r\n```\r\n\r\n```\r\n% python3 bench/timer_handle_heap.py\r\nwrap when, TimerHandle in tuple 34.082984749999014\r\nbare TimerHandle 49.678519583001616\r\n```\r\n\r\n\u003c!-- gh-linked-prs --\u003e\r\n### Linked PRs\r\n* gh-122882\r\n\u003c!-- /gh-linked-prs --\u003e\r\n","author":{"url":"https://github.com/bdraco","@type":"Person","name":"bdraco"},"datePublished":"2024-08-10T12:58:39.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/122881/cpython/issues/122881"}