René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`os.walk()` no longer supports late edits to `dirnames`","articleBody":"Prior to the fix for #89727, it was possible influence which subdirectories were visited by `os.walk()` **even after** the walk had descended into siblings of those subdirectories. Such \"late\" modifications no longer have any effect.\r\n\r\nThis was [reported](https://github.com/python/cpython/issues/89727#issuecomment-1369298204) by @pochmann:\r\n\r\n\u003e I think you changed `os.walk`'s behavior. The recursive one supported late modifications of subdirs lists, due to being lazier. The iterative one doesn't (if I'm not mistaken... I can't test).\r\n\u003e \r\n\u003e With topdown, you can modify a directory's subdirs, for example remove one and it won't be visited. Usually you'd do that while your walk is on that parent directory. But with the recursive one, it was possible to do it later. With the iterative one, I don't think that works anymore.\r\n\u003e \r\n\u003e Here's a demo where I remove the second subdir after having walked onto the first:\r\n\u003e \r\n\u003e ```python\r\n\u003e import os\r\n\u003e \r\n\u003e # Create demo dir with three subdirs\r\n\u003e os.makedirs('demo/a')\r\n\u003e os.makedirs('demo/b')\r\n\u003e os.makedirs('demo/c')\r\n\u003e \r\n\u003e # Walk onto \"demo\"\r\n\u003e walk = os.walk('demo')\r\n\u003e demo = next(walk)\r\n\u003e print(demo)\r\n\u003e \r\n\u003e # Walk onto first subdir\r\n\u003e first = next(walk)\r\n\u003e print(first)\r\n\u003e \r\n\u003e # Remove the second subdir\r\n\u003e del demo[1][1]\r\n\u003e \r\n\u003e # Walk onto the remaining subdirs\r\n\u003e for x in walk:\r\n\u003e     print(x)\r\n\u003e ```\r\n\u003e \r\n\u003e Output ([Try it online!](https://tio.run/##bZCxDoJADIb3e4pGB2HBGBdj4uQbuDgYh0OKXJArOarg02MLaqLhckPv69/@vdZPLsivN3Xoe1fVFBioMWYO@4CWETKsCDIXoHVcABcBEZp7KqQx1CSVLVHjaKHCpV3EEzSdpBeh4nO0txLIM8FM8cy0CnYyRaLRKBbpMMgOPHYcaSI2dXCeI@V/jXIx4feUZnxMFA6JofIgLR4on5Ov4YV89qnN8DYs4LQ6y/01UXXAyjrv/PW7kpwCdOA8qNPWgJzRrYv7/gU)), note that the second subdir wasn't walked:\r\n\u003e \r\n\u003e ```python\r\n\u003e ('demo', ['c', 'a', 'b'], [])\r\n\u003e ('demo/c', [], [])\r\n\u003e ('demo/b', [], [])\r\n\u003e ```\r\n\u003e \r\n\u003e [Excerpt from the recursive one](https://github.com/python/cpython/blob/a7a450f84a087421603170c2dad226bb881d4d9a/Lib/os.py#L412-L419):\r\n\u003e \r\n\u003e ```python\r\n\u003e         for dirname in dirs:\r\n\u003e             new_path = join(top, dirname)\r\n\u003e             yield from _walk(new_path, topdown, onerror, followlinks)\r\n\u003e ```\r\n\u003e \r\n\u003e The iteration of `dirs` and the walking of the subdirs are intertwined. After walking a subdir, the paused iteration of `dirs` resumes. That allows the late modifications of `dirs` to have an effect.\r\n\u003e \r\n\u003e [Excerpt from the iterative one](https://github.com/python/cpython/blob/73097d91a64620ae7f620705864b84234d85cc82/Lib/os.py#L418-L425):\r\n\u003e \r\n\u003e ```python\r\n\u003e             for dirname in reversed(dirs):\r\n\u003e                 new_path = join(top, dirname)\r\n\u003e                 stack.append(new_path)\r\n\u003e ```\r\n\u003e \r\n\u003e This eagerly puts all subdirs onto the stack, before they're getting walked, and then `dirs` is never used again. So modifying `dirs` during the subdirs walking doesn't have an effect anymore.","author":{"url":"https://github.com/barneygale","@type":"Person","name":"barneygale"},"datePublished":"2023-03-22T20:17:08.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/102932/cpython/issues/102932"}