René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Optimize XXXSelector for many iterations of the event loop","articleBody":"split out from https://github.com/python/cpython/pull/106555#issuecomment-1632025959\r\n\r\nThe current `EpollSelector` can be sped up a bit. This makes quite a difference when there are 100000+ iterations of the event loop per minute (the use case being receiving bluetooth data from multiple sources) since selectors have to run every iteration.\r\n\r\noriginal: 11.831302762031555\r\nnew: 9.579423972172663\r\n\r\n```\r\nimport timeit\r\nimport math\r\nimport select\r\nimport os\r\nfrom selectors import EpollSelector, EVENT_WRITE, EVENT_READ\r\n\r\n\r\nclass OriginalEpollSelector(EpollSelector):\r\n    def select(self, timeout=None):\r\n        if timeout is None:\r\n            timeout = -1\r\n        elif timeout \u003c= 0:\r\n            timeout = 0\r\n        else:\r\n            # epoll_wait() has a resolution of 1 millisecond, round away\r\n            # from zero to wait *at least* timeout seconds.\r\n            timeout = math.ceil(timeout * 1e3) * 1e-3\r\n        # epoll_wait() expects `maxevents` to be greater than zero;\r\n        # we want to make sure that `select()` can be called when no\r\n        # FD is registered.\r\n        max_ev = max(len(self._fd_to_key), 1)\r\n        ready = []\r\n        try:\r\n            fd_event_list = self._selector.poll(timeout, max_ev)\r\n        except InterruptedError:\r\n            return ready\r\n        for fd, event in fd_event_list:\r\n            events = 0\r\n            if event \u0026 ~select.EPOLLIN:\r\n                events |= EVENT_WRITE\r\n            if event \u0026 ~select.EPOLLOUT:\r\n                events |= EVENT_READ\r\n\r\n            key = self._key_from_fd(fd)\r\n            if key:\r\n                ready.append((key, events \u0026 key.events))\r\n        return ready\r\n\r\n\r\nNOT_EPOLLIN = ~select.EPOLLIN\r\nNOT_EPOLLOUT = ~select.EPOLLOUT\r\n\r\nclass NewEpollSelector(EpollSelector):\r\n    def select(self, timeout=None):\r\n        if timeout is None:\r\n            timeout = -1\r\n        elif timeout \u003c= 0:\r\n            timeout = 0\r\n        else:\r\n            # epoll_wait() has a resolution of 1 millisecond, round away\r\n            # from zero to wait *at least* timeout seconds.\r\n            timeout = math.ceil(timeout * 1e3) * 1e-3\r\n        # epoll_wait() expects `maxevents` to be greater than zero;\r\n        # we want to make sure that `select()` can be called when no\r\n        # FD is registered.\r\n        max_ev = len(self._fd_to_key) or 1\r\n        ready = []\r\n        try:\r\n            fd_event_list = self._selector.poll(timeout, max_ev)\r\n        except InterruptedError:\r\n            return ready\r\n        \r\n        fd_to_key = self._fd_to_key\r\n        for fd, event in fd_event_list:\r\n            key = fd_to_key.get(fd)\r\n            if key:\r\n                ready.append(\r\n                    (\r\n                        key,\r\n                        (\r\n                            (event \u0026 NOT_EPOLLIN and EVENT_WRITE)\r\n                            | (event \u0026 NOT_EPOLLOUT and EVENT_READ)\r\n                        )\r\n                        \u0026 key.events,\r\n                    )\r\n                )\r\n        return ready\r\n\r\n\r\noriginal_epoll = OriginalEpollSelector()\r\nnew_epoll = NewEpollSelector()\r\n\r\n\r\nfor _ in range(512):\r\n    r, w = os.pipe()\r\n    os.write(w, b\"a\")\r\n    original_epoll.register(r, EVENT_READ)\r\n    new_epoll.register(r, EVENT_READ)\r\n\r\n\r\noriginal_time = timeit.timeit(\r\n    \"selector.select()\",\r\n    number=100000,\r\n    globals={\"selector\": original_epoll},\r\n)\r\nnew_time = timeit.timeit(\r\n    \"selector.select()\",\r\n    number=100000,\r\n    globals={\"selector\": new_epoll},\r\n)\r\n\r\nprint(\"original: %s\" % original_time)\r\nprint(\"new: %s\" % new_time)\r\n```\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-106754\n* gh-106864\n* gh-106879\n* gh-106884\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/bdraco","@type":"Person","name":"bdraco"},"datePublished":"2023-07-14T19:48:48.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/106751/cpython/issues/106751"}