René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add optimized versions of isdir / isfile on Windows","articleBody":"I went down this rabbit hole when someone mentioned that `isfile`/`isdir`/`exists` all make a rather expensive `os.stat` call on Windows (which is actually a long wrapper around a number of system calls on Windows), rather than the simpler and more direct call to `GetFileAttributeW`.\r\n\r\nI noticed that at one point there was a [version of `isdir`](https://github.com/python/cpython/commits/9c669ccc77c85eac245d460bab510a38b20d9a08) that does exactly this.  At the time, this claimed a 2x speedup.  \r\n\r\nHowever, this C implementation of `isdir` was removed as part of a large set of changes in df2d4a6f3, and as a result, `isdir` got faster.\r\n\r\nWith the following benchmark:\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eisdir benchmark\u003c/summary\u003e\r\n\r\n```\r\nimport os.path\r\nimport timeit\r\n\r\n\r\nfor i in range(100):\r\n    os.makedirs(f\"exists{i}\", exist_ok=True)\r\n\r\n\r\ndef test_exists():\r\n    for i in range(100):\r\n        os.path.isdir(f\"exists{i}\")\r\n\r\n\r\ndef test_extinct():\r\n    for i in range(100):\r\n        os.path.isdir(f\"extinct{i}\")\r\n\r\n\r\nprint(timeit.timeit(test_exists, number=100))\r\nprint(timeit.timeit(test_extinct, number=100))\r\n\r\n\r\nfor i in range(100):\r\n    os.rmdir(f\"exists{i}\")\r\n```\r\n\r\n\u003c/details\u003e\r\n\r\nI get the following with df2d4a6f3:\r\n\r\n```\r\nexists: 0.18694799999957468\r\ndoesn't exist: 0.08418370000072173\r\n```\r\n\r\nand with the prior commit:\r\n\r\n```\r\nexists: 0.25393609999991895\r\ndoesn't exist: 0.08511730000009265\r\n```\r\n\r\nSo, from this, I'd conclude that the idea of replacing calls to `os.stat` with calls to `GetFileAttributeW` would not bear fruit, but @zooba should probably confirm I'm benchmarking the right thing and making sense.\r\n\r\nIn any event, we should probably remove the [little vestige](https://github.com/python/cpython/blob/main/Lib/ntpath.py#LL854-L862C9) that imports this fast path that was removed:\r\n\r\n```python\r\ntry:\r\n    # The genericpath.isdir implementation uses os.stat and checks the mode\r\n    # attribute to tell whether or not the path is a directory.\r\n    # This is overkill on Windows - just pass the path to GetFileAttributes\r\n    # and check the attribute from there.\r\n    from nt import _isdir as isdir\r\nexcept ImportError:\r\n    # Use genericpath.isdir as imported above.\r\n    pass\r\n```\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-101324\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/mdboom","@type":"Person","name":"mdboom"},"datePublished":"2023-01-20T18:01:32.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":15},"url":"https://github.com/101196/cpython/issues/101196"}