Title: On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself · Issue #8211 · PowerShell/PowerShell · GitHub
Open Graph Title: On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself · Issue #8211 · PowerShell/PowerShell
X Title: On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself · Issue #8211 · PowerShell/PowerShell
Description: Note: cmd.exe's rd /s and .NET's [System.IO.Directory]::Delete() are equally affected: see here and here. Note: The problem occurs only on Windows (and there also on Windows PowerShell). Update: Starting with Windows 10 version 1909, (at...
Open Graph Description: Note: cmd.exe's rd /s and .NET's [System.IO.Directory]::Delete() are equally affected: see here and here. Note: The problem occurs only on Windows (and there also on Windows PowerShell). Update: St...
X Description: Note: cmd.exe's rd /s and .NET's [System.IO.Directory]::Delete() are equally affected: see here and here. Note: The problem occurs only on Windows (and there also on Windows PowerShell). Up...
Opengraph URL: https://github.com/PowerShell/PowerShell/issues/8211
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself","articleBody":"\u003csup>Note: `cmd.exe`'s `rd /s` and .NET's `[System.IO.Directory]::Delete()` are equally affected: see [here](https://github.com/Microsoft/console/issues/309) and [here](https://github.com/dotnet/corefx/issues/33603).\u003c/sup>\r\n\r\nNote: The problem occurs only on _Windows_ (and there also on _Windows PowerShell_).\r\n\r\n---\r\n\r\n_Update_: \r\n\r\n**Starting with Windows 10 version `1909`, (at least) build `18363.657` (I don't know that Windows _Server_ version and build that corresponds to; run `winver.exe` to check your version and build), the `DeleteFile` Windows API function now exhibits _synchronous_ behavior, which implicitly solves the problems with PowerShell's `Remove-Item` and .NET's `System.IO.File.Delete` / `System.IO.Directory.Delete` (but, curiously, _not_ with `cmd.exe`'s `rd /s`).** \r\n\r\nThus, only older Windows 10 versions and older Windows versions in general are affected.\r\n\r\n---\r\n\r\n**The Windows API functions `DeleteFile()` and `RemoveDirectory()` functions are _inherently asynchronous_** (emphasis added):\r\n\r\n> The DeleteFile function **marks a file for deletion** on close. Therefore, the file **deletion does not occur until the last handle to the file is closed**. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED.\"\r\n\r\n> The RemoveDirectory function **marks a directory for deletion on close**. Therefore, the directory is **not removed until the last handle to the directory is closed.**\"\r\n\r\nPowerShell fails to account for this asynchronous behavior, which has two implications:\r\n\r\n* Problem (a): **Trying to delete a nonempty directory (which invariably requires recursive deletion of its _content_ first) may _fail_** - infrequently, but it does happen.\r\n\r\n* Problem (b): **Trying to _recreate_ a successfully deleted directory or a file _immediately afterwards_ may fail** - intermittently (easier to provoke than the other problem, but a more exotic use case overall).\r\n * Note: You could argue that it's not PowerShell's responsibility to create a synchronous experience in this case, given that the underlying system API is asynchronous; however, it certainly makes for more robust, predictable scripting if synchronicity is ensured.\r\n\r\nAs an aside: `cmd.exe`'s `rd /s` exhibits the same problem, [as does .NET Core's `[System.IO.Directory]::Delete()`](https://github.com/dotnet/corefx/issues/33603) (which PowerShell does _not_ use).\r\n\r\nProblem (a) is due to using depth-first recursion without accounting for the asynchronous deletion behavior; the problem, along with the solution, is described in [this YouTube video](https://youtu.be/uhRWMGBjlO8?t=455) (starts at 7:35).\r\n\r\nImportant: \r\n\r\n* The repros below use the functions defined at the bottom, which must be defined first.\r\n* The tests created temp. dir. `$HOME/tmpdir` - remove it manually afterwards, if still present.\r\n\r\nSteps to reproduce\r\n------------------\r\n\r\nSetup: Paste and submit the function definitions below at the prompt.\r\n\r\nProblem (a):\r\n\r\n```powershell\r\nAssert-ReliableDirRemoval ps\r\n```\r\n\r\nProblem (b):\r\n\r\n```powershell\r\nAssert-SyncDirRemoval ps\r\n```\r\n\r\nExpected behavior\r\n-----------------\r\n\r\nThe functions should loop _indefinitely_ (terminate them with `Ctrl+C`), emitting a `.` in each iteration.\r\n\r\nActual behavior\r\n---------------\r\n\r\n**_Eventually_** - and that there is no _predictable_ time frame is indicative of the problem - an error will occur, on the **order of _minutes_ or even longer**.\r\n\r\nProblem (a):\r\n\r\n```none\r\nReliable dir-tree-removal test: Repeatedly creating and removing a directory subtree, using ps for removal.\r\n.........Remove-Item : Directory C:\\Users\\jdoe\\tmpDir\\sub\\sub cannot be removed because it is not empty.\r\n```\r\n\r\nThat is, recursive removal of the target dir's _content_ failed due to async timing issues.\r\n\r\nProblem (b):\r\n\r\n```none\r\nSynchronous dir-removal test: Repeatedly creating and removing a directory, using ps for removal.\r\n.................New-Item : An item with the specified name C:\\Users\\jdoe\\tmpDir already exists.\r\n```\r\n\r\nThat is, recreating the target dir failed, because the prior removal hadn't yet completed.\r\n\r\n\r\nAux. function definitions:\r\n-----------------------\r\n\r\n### Functions `Assert-ReliableDirRemoval` and `Assert-SyncDirRemoval`\r\n\r\nYou can paste the entire code at the prompt in order to define these functions.\r\n\r\n```powershell\r\nfunction Assert-ReliableDirRemoval {\r\n param([parameter(Mandatory)] [ValidateSet('cmd', '.net', 'ps')] [string] $method)\r\n Write-Host \"Reliable dir-tree-removal test: Repeatedly creating and removing a directory subtree, using $method for removal.\"\r\n # Treat all errors as fatal.\r\n $ErrorActionPreference = 'Stop'\r\n Set-Location $HOME # !! Seemingly, a dir. in the user's home dir. tree reproduces the symptom fastest - $env:TEMP does not.\r\n # Remove a preexisting directory first, if any.\r\n Remove-Item -EA Ignore -Recurse tmpDir\r\n While ($true) {\r\n Write-Host -NoNewline .\r\n # Create a subdir. tree.\r\n $null = New-Item -Type Directory tmpDir, tmpDir/sub, tmpDir/sub/sub, tmpDir/sub/sub/sub\r\n # Fill each subdir. with 1000 empty test files.\r\n \"tmpDir\", \"tmpDir/sub\", \"tmpDir/sub/sub\", \"tmpDir/sub/sub/sub\"| ForEach-Object {\r\n $dir = $_\r\n 1..1e3 | ForEach-Object { $null > \"$dir/$_\" }\r\n }\r\n # Now remove the entire dir., which invariably involves deleting its contents\r\n # recursively first.\r\n switch ($method) {\r\n 'ps' { Remove-Item -Recurse tmpDir }\r\n '.net' { [System.IO.Directory]::Delete((Convert-Path tmpDir), $true) }\r\n 'cmd' { cmd /c rd /s /q tmpDir; if ($LASTEXITCODE) { Throw } }\r\n # !! If rd /s fails during recursive removal due to async issues, it emits a stderr line, but \r\n # !! does NOT report a nonzero exit code. We detect this case below.\r\n }\r\n # Does the dir. unexpectedly still exist?\r\n # This can happen for two reasons:\r\n # - The removal of the top-level directory itself has been requested, but isn't complete yet.\r\n # - Removing the content of the top-level directory failed quietly.\r\n while (Test-Path tmpDir) {\r\n if ($output = Get-ChildItem -EA Ignore -Force -Name tmpDir) { # Still has content?\r\n Throw \"Deletion failed quietly or with exit code 0, tmpDir still has content: $output\"\r\n } else {\r\n # Top-level directory removal isn't complete yet.\r\n # Wait for removal to complete, so we can safely recreate the directory in the next iteration.\r\n # This loop should exit fairly quickly.\r\n Write-Host -NoNewline !; Start-Sleep -Milliseconds 100\r\n } \r\n } \r\n }\r\n}\r\n\r\nfunction Assert-SyncDirRemoval { \r\n param([parameter(Mandatory)] [ValidateSet('cmd', '.net', 'ps')] [string] $method)\r\n Write-Host \"Synchronous dir-removal test: Repeatedly creating and removing a directory, using $method for removal.\"\r\n # Treat all errors as fatal.\r\n $ErrorActionPreference = 'Stop'\r\n Set-Location $HOME # !! Seemingly, a dir. in the user's home dir. tree reproduces the symptom fastest - $env:TEMP does not.\r\n While ($true) {\r\n Write-Host -NoNewline .\r\n # Remove the test dir., which invariably involves deleting its contents recursively first.\r\n # Note: This could itself fail intermittently, but with just 10 files and no subdirs. is unlikely to.\r\n if (Test-Path tmpDir) {\r\n switch ($method) {\r\n 'ps' { Remove-Item -Recurse tmpDir }\r\n '.net' { [System.IO.Directory]::Delete((Convert-Path tmpDir), $true) }\r\n 'cmd' { cmd /c rd /s /q tmpDir; if ($LASTEXITCODE) { Throw } }\r\n # !! If rd /s fails during recursive removal due to async issues, it emits a stderr line, but \r\n # !! does NOT report a nonzero exit code. We detect this case below.\r\n }\r\n }\r\n # (Re-)create the dir. with 10 empty test files.\r\n # If the previous removal hasn't fully completed yet, this will fail.\r\n # Note: [System.IO.Directory]::Delete() could have quietly failed to remove the contents of the dir.\r\n # due to async timing accidents, but, again, this is unlikely with 1- files and \r\n try {\r\n $null = New-Item -Type Directory tmpDir\r\n } catch {\r\n # Handle the case where removal failed\r\n # quietly due to async vagaries while removing the dir's *content*, which\r\n # can happen with [System.IO.Directory]::Delete().\r\n # Note that if removal succeeded but is pending - the scenario we're trying to\r\n # get to occur - Get-ChildItem reports an access-denied error, which we ignore\r\n # in favor of reporting the original error from the dir. re-creation attempt.\r\n if ($output = Get-ChildItem -EA Ignore -Force -Name tmpDir) { # Still has content?\r\n Write-Warning \"Ignoring failed content removal, retrying...\"\r\n # Simply try again\r\n Start-Sleep -Milliseconds 500; Remove-item tmpDir -Recurse\r\n continue\r\n }\r\n # Re-creation failed due to async removal.\r\n Throw $_\r\n }\r\n 1..10 | ForEach-Object { $null > tmpDir/$_ }\r\n }\r\n}\r\n```\r\n\r\n\r\nEnvironment data\r\n----------------\r\n\r\n```none\r\nPowerShell Core 6.2.0-preview.1 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.165)\r\nWindows PowerShell v5.1.17134.228 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.345)\r\n```\r\n","author":{"url":"https://github.com/mklement0","@type":"Person","name":"mklement0"},"datePublished":"2018-11-08T15:10:13Z","url":"https://github.com/PowerShell/PowerShell/issues/8211"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:16516ae8-82f9-5b42-647a-24417f0398c8 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 9DB2:8310E:752CA:A121E:6AFDA997 |
| html-safe-nonce | fd74da298f64f53ec8ee49ce70c34ccf2a73737b49162b68709d62d47a01d1d8 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5REIyOjgzMTBFOjc1MkNBOkExMjFFOjZBRkRBOTk3IiwidmlzaXRvcl9pZCI6IjY3NDU3OTYxMjY4ODAzNDE5OSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | b73282c10967ef569c9596cf6bd67aca168f5cf84fa180f0ec9ca32c1f60f43a |
| hovercard-subject-tag | issue:378774469 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/PowerShell/PowerShell/8211/issue_layout |
| twitter:image | https://opengraph.githubassets.com/b008c62b31e1db7ba379151f2a34ffb02048721bdda1b7855d82110b59c148d5/PowerShell/PowerShell/issues/8211 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/b008c62b31e1db7ba379151f2a34ffb02048721bdda1b7855d82110b59c148d5/PowerShell/PowerShell/issues/8211 |
| og:image:alt | Note: cmd.exe's rd /s and .NET's [System.IO.Directory]::Delete() are equally affected: see here and here. Note: The problem occurs only on Windows (and there also on Windows PowerShell). Update: St... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | mklement0 |
| hostname | github.com |
| expected-hostname | github.com |
| None | bc86424ef34a73b386237872800e922d149d65ba907f63d6f150a2dcb8433bea |
| turbo-cache-control | no-preview |
| go-import | github.com/PowerShell/PowerShell git https://github.com/PowerShell/PowerShell.git |
| octolytics-dimension-user_id | 11524380 |
| octolytics-dimension-user_login | PowerShell |
| octolytics-dimension-repository_id | 49609581 |
| octolytics-dimension-repository_nwo | PowerShell/PowerShell |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 49609581 |
| octolytics-dimension-repository_network_root_nwo | PowerShell/PowerShell |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 0ddf53eb35c87f1f8df13dbf75c3bdadbfc6e969 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width