René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Fix/Skip/Remove Flaky Tests","articleBody":"### Prerequisites\r\n\r\n- [X] Write a descriptive title.\r\n- [X] Make sure you are able to repro it on the [latest released version](https://github.com/PowerShell/PowerShell/releases)\r\n- [X] Search the existing issues.\r\n- [X] Refer to the [FAQ](https://github.com/PowerShell/PowerShell/blob/master/docs/FAQ.md).\r\n- [X] Refer to [Differences between Windows PowerShell 5.1 and PowerShell](https://learn.microsoft.com/powershell/scripting/whats-new/differences-from-windows-powershell).\r\n\r\n### Steps to reproduce\r\n\r\nWhen submitting a PR there are some tests which are currently only sometimes passing and can potentially fail. This is incredibly frustrating for people submitting PRs because it looks like your PR is failing and could give a false impression that the PR is breaking something when it isn't or is not ready for review. It is also only possible to close and reopen your PR which runs the whole test suite again which could then encounter another flaky test and just wastes time to run.\r\n\r\nEither these tests should be fixed up so they can run consistently without failure or skipped/removed because a sometimes failing test is worse than no test at all.\r\n\r\nThree tests that I've consistently come across as flaky are:\r\n\r\n+ [Test-Connection.MTUSizeDetect.MTUSizeDetect works](https://github.com/PowerShell/PowerShell/blob/338d1dcfa1ed6369980cc2b8c953aa73b1f86bfc/test/powershell/Modules/Microsoft.PowerShell.Management/Test-Connection.Tests.ps1#L264-L280) - https://dev.azure.com/powershell/PowerShell/_build/results?buildId=145814\u0026view=ms.vss-test-web.build-test-results-tab\u0026runId=2175688\u0026resultId=107348\u0026paneView=debug\r\n\r\n```\r\nValue cannot be null. (Parameter 'replyResult')\r\n\r\nat \u003cScriptBlock\u003e, /Users/runner/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/Test-Connection.Tests.ps1: line 274\r\n274:             $result = Test-Connection $testAddress -MtuSize\r\n```\r\n\r\n+ [Clipboard cmdlet tests.Text.Get-Clipboard returns what is in Set-Clipboard](https://github.com/PowerShell/PowerShell/blob/338d1dcfa1ed6369980cc2b8c953aa73b1f86bfc/test/powershell/Modules/Microsoft.PowerShell.Management/Clipboard.Tests.ps1#L19-L23) - https://dev.azure.com/powershell/PowerShell/_build/results?buildId=145814\u0026view=ms.vss-test-web.build-test-results-tab\u0026runId=2175688\u0026resultId=107348\u0026paneView=debug\r\n\r\n```\r\nProcess must exit before requested information can be determined.\r\n\r\nat \u003cScriptBlock\u003e, /Users/runner/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/Clipboard.Tests.ps1: line 21\r\n21:             Set-Clipboard -Value $guid\r\n``` \r\n\r\n+ [Group policy settings tests.Group policy settings tests.Module logging policy test](https://github.com/PowerShell/PowerShell/blob/338d1dcfa1ed6369980cc2b8c953aa73b1f86bfc/test/powershell/engine/Basic/GroupPolicySettings.Tests.ps1#L59-L107) - https://dev.azure.com/powershell/PowerShell/_build/results?buildId=145862\u0026view=ms.vss-test-web.build-test-results-tab\u0026runId=2175951\u0026resultId=100001\u0026paneView=debug\r\n\r\n```\r\nExpected $true, but got $false.\r\n\r\nat \u003cScriptBlock\u003e, D:\\a\\1\\s\\test\\powershell\\engine\\Basic\\GroupPolicySettings.Tests.ps1: line 94\r\n94:                         Should -BeTrue\r\n```\r\n\r\n### Expected behavior\r\n\r\n```console\r\nTests pass consistently\r\n```\r\n\r\n\r\n### Actual behavior\r\n\r\n```console\r\nSome tests fail\r\n```\r\n\r\n\r\n### Error details\r\n\r\n_No response_\r\n\r\n### Environment data\r\n\r\n```powershell\r\nCI\r\n```\r\n\r\n\r\n### Visuals\r\n\r\n_No response_","author":{"url":"https://github.com/jborean93","@type":"Person","name":"jborean93"},"datePublished":"2024-02-26T23:43:07.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/21274/PowerShell/issues/21274"}