René's URL Explorer Experiment

Title: GH-102613: Fast recursive globbing in `pathlib.Path.glob()` by barneygale · Pull Request #104512 · python/cpython · GitHub

Open Graph Title: GH-102613: Fast recursive globbing in `pathlib.Path.glob()` by barneygale · Pull Request #104512 · python/cpython

X Title: GH-102613: Fast recursive globbing in `pathlib.Path.glob()` by barneygale · Pull Request #104512 · python/cpython

Description: This PR introduces a 'walk-and-match' strategy for handling glob patterns that include a non-terminal ** wildcard, such as **/*.py. For this example, the previous implementation recursively walked directories using os.scandir() when it expanded the ** component, and then scanned those same directories again when expanded the *.py component. This is wasteful. In the new implementation, any components following a ** wildcard are used to build a re.Pattern object, which is used to filter the results of the recursive walk. A pattern like **/*.py uses half the number of os.scandir() calls; a pattern like **/*/*.py a third, etc. This new algorithm does not apply if either: The follow_symlinks argument is set to None (its default), or The pattern contains .. components. In these cases we fall back to the old implementation. This PR also replaces selector classes with selector functions. These generators directly yield results rather calling through to their successors. A new internal Path._glob() method takes care to chain these generators together, which simplifies the lazy algorithm and slightly improves performance. It should also be easier to understand and maintain. Performance for the original #102613 repro case, with 400 nested a/ directories, and matching treatment of symlinks and hidden files: $ ../python -m timeit -s 'import glob' 'print(glob.glob("**/*", recursive=True, include_hidden=True))' 5 loops, best of 5: 66.2 msec per loop $ ../python -m timeit -s 'from pathlib import Path' 'print(list(Path(".").rglob("**/*", follow_symlinks=True)))' 10 loops, best of 5: 22.7 msec per loop # before this PR 10 loops, best of 5: 16.5 msec per loop # after this PR These results were from an SSD. The improvement will be greater for slow storage (e.g. network-mounted volumes). Issue: gh-102613

Open Graph Description: This PR introduces a 'walk-and-match' strategy for handling glob patterns that include a non-terminal ** wildcard, such as **/*.py. For this example, the previous implementation recursively...

X Description: This PR introduces a 'walk-and-match' strategy for handling glob patterns that include a non-terminal ** wildcard, such as **/*.py. For this example, the previous implementation rec...

Opengraph URL: https://github.com/python/cpython/pull/104512

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width