René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"WebRequestPSCmdlet does not check existence of Location header when Authorization is present","articleBody":"The code introduced in https://github.com/PowerShell/PowerShell/pull/3885 seems to have introduced a regression bug which was quite hard to identify as the error message points to issues with the URI and not to anything related to redirection. [The code introduced in the pull request initializes an URI using the Location Header, without first checking if the Location Header is present in the response of a redirect.](https://github.com/PowerShell/PowerShell/blob/d20d53eaacafd4645ea668bd46e62093d3f3e503/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L1310). The [HTTP RFC 7231](https://tools.ietf.org/html/rfc7231#section-6.4) does not specify the Location Header to be mandatory and thus it's existence must be checked.\r\n\r\nI discovered this while checking strange PowerShell errors when issuing a GET or HEAD request against a AWS S3 Bucket while connecting to the wrong region. AWS S3 does respond with HTTP 301 but without a Location header.\r\n\r\nThe issue only occurs if the Server responds with a redirect HTTP Status Code 3XX and without the Location header and if the client has specified the Authorization Header in the initial request.\r\n\r\nThe issue can probably be fixed by modifying [line 1290](https://github.com/PowerShell/PowerShell/blob/d20d53eaacafd4645ea668bd46e62093d3f3e503/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L1290) to `if (stripAuthorization \u0026\u0026 IsRedirectCode(response.StatusCode) \u0026\u0026 response.Headers.Location)`.\r\n\r\nI'd be glad to create a pull request to fix this issue if my analyzis is correct and the proposed change is the correct way to fix the issue.\r\n\r\nActual behavior\r\n---------------\r\n\r\nThe following simple HEAD request demonstrates the unexpected error message:\r\n\r\n```powershell\r\nInvoke-WebRequest -Uri https://s3.amazonaws.com/test-301-redirection/ -Method HEAD -Headers @{Authorization=\"test\"}                       \r\nInvoke-WebRequest : Value cannot be null.\r\nParameter name: uri\r\nAt line:1 char:1\r\n+ Invoke-WebRequest -Uri https://s3.amazonaws.com/test-301-redirection/ ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+ CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], ArgumentNullException\r\n+ FullyQualifiedErrorId : System.ArgumentNullException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand\r\n```\r\n\r\nExpected behavior\r\n-----------------\r\n\r\nThe response should be\r\n\r\n```powershell\r\nInvoke-WebRequest -Uri https://s3.amazonaws.com/test-301-redirection/ -Method HEAD -Headers @{Authorization=\"test\"}                                                          \r\nInvoke-WebRequest : Response status code does not indicate success: 301 (Moved Permanently).\r\nAt line:1 char:1\r\n+ Invoke-WebRequest -Uri https://s3.amazonaws.com/test-301-redirection/ ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+ CategoryInfo          : InvalidOperation: (Method: HEAD, R...rShell/6.0.2\r\n}:HttpRequestMessage) [Invoke-WebRequest], HttpResponseException\r\n+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand\r\n```\r\n\r\nA workaround is, to specify the `PreserveAuthorizationOnRedirect` Parameter which will skip the section introduced in https://github.com/PowerShell/PowerShell/pull/3885 :\r\n\r\n```powershell\r\nInvoke-WebRequest -Uri https://s3.amazonaws.com/test-301-redirection/ -Method HEAD -Headers @{Authorization=\"test\"} -PreserveAuthorizationOnRedirect                                                                                                                                                                                             \r\nInvoke-WebRequest : Response status code does not indicate success: 301 (Moved Permanently).\r\nAt line:1 char:1\r\n+ Invoke-WebRequest -Uri https://s3.amazonaws.com/test-301-redirection/ ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+ CategoryInfo          : InvalidOperation: (Method: HEAD, R...rShell/6.0.2\r\n}:HttpRequestMessage) [Invoke-WebRequest], HttpResponseException\r\n+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand\r\n```\r\n\r\nEnvironment data\r\n----------------\r\n\r\n```powershell\r\n$PSVersionTable                                                                                                                           \r\n\r\nName                           Value                                                                                                                                                                  \r\n----                           -----                                                                                                                                                                  \r\nPSVersion                      6.0.2                                                                                                                                                                  \r\nPSEdition                      Core                                                                                                                                                                   \r\nGitCommitId                    v6.0.2                                                                                                                                                                 \r\nOS                             Darwin 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64                                                          \r\nPlatform                       Unix                                                                                                                                                                   \r\nPSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                \r\nPSRemotingProtocolVersion      2.3                                                                                                                                                                    \r\nSerializationVersion           1.1.0.1                                                                                                                                                                \r\nWSManStackVersion              3.0                                           \r\n```\r\n","author":{"url":"https://github.com/ffeldhaus","@type":"Person","name":"ffeldhaus"},"datePublished":"2018-04-02T17:29:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/6543/PowerShell/issues/6543"}