René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"WebCmdlets should read the encoding of Content-Type application/json per RFC","articleBody":"Steps to reproduce\r\n------------------\r\n\r\n```powershell\r\nInvoke-RestMethod 'http://api.forismatic.com/api/1.0/?method=getQuote\u0026format=json\u0026lang=ru' -verbose\r\n```\r\n\r\nExpected behavior\r\n-----------------\r\n\r\nIt should **detect** the utf-8 encoding, and produce the same output as this:\r\n\r\n```posh\r\n$resp = Invoke-WebRequest 'http://api.forismatic.com/api/1.0/?method=getQuote\u0026format=json\u0026lang=ru'\r\n$char = $resp.RawContentStream.ToArray()\r\n$str = [Text.Encoding]::UTF8.GetString($char)\r\nConvertFrom-Json $str\r\n```\r\n\r\ni.e. something like this:\r\n\r\n```none\r\nVERBOSE: GET http://api.forismatic.com/api/1.0/?method=getQuote\u0026format=json with 0-byte payload\r\nVERBOSE: received 536-byte response of content type application/json\r\nVERBOSE: Content encoding: utf-8\r\n\r\nquoteText   : Именно внутренний диалог прижимает к земле людей в повседневной жизни. Мир для нас такой-то и такой-то или этакий и этакий лишь потому, что мы сами себе говорим о нем, что он такой-то и такой-то или этакий и этакий.\r\nquoteAuthor : Карлос Кастанеда\r\nsenderName  :\r\nsenderLink  :\r\nquoteLink   : http://forismatic.com/ru/6309006412/\r\n```\r\n\r\nActual behavior\r\n---------------\r\n\r\nIt falls back to iso-8859-1 encoding and produces gobbledygook with a lot of Ð's in it.  Also, it utterly fails to produce a number for the `#-byte response` string in the verbose output.\r\n\r\n```none\r\nVERBOSE: GET http://api.forismatic.com/api/1.0/?method=getQuote\u0026format=json with 0-byte payload\r\nVERBOSE: received -byte response of content type application/json\r\nVERBOSE: Content encoding: iso-8859-1\r\n\r\nquoteText   : Ко всякому прибежищу обращаются люди, мучимые страхом: к горам\r\n              и к лесам, к деревьям в роще, к гробницам.\r\nquoteAuthor : Будда Гаутама\r\nsenderName  :\r\nsenderLink  :\r\nquoteLink   : http://forismatic.com/ru/804c7d14d9/\r\n```\r\n\r\nDiscussion\r\n----------\r\n\r\nWhen calling an HTTP endpoint that returns a header: `Content-Type: application/json` the WebCmdlets are **incorrectly** defaulting to `iso-8859-1` rather than a proper unicode encoding, and are *disregarding* the application/json RFC's simple specification for how to determine the content encoding.\r\n\r\n1. The JSON standard [ECMA-404 (PDF)](http://ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf) clearly states that JSON must be unicode\r\n2. The [application/json RFC (in section 3)](https://tools.ietf.org/html/rfc4627#section-3) clearly indicates how the encoding should be determined from the first 4 bytes of the content.\r\n\r\n**NOTE:** Please don't work around this by just defaulting to utf-8.  I'm sure that 90% of the time, you could probably get away with that, but it's not actually correct, and the RFC implementation is _trivial_.\r\n\r\n**ALSO NOTE:** The WebCmdlets *do* respect the [`; charset=utf-8`](https://github.com/PowerShell/PowerShell/blob/f5f3fab1ea8a51dd2e5d34e0e580ab39176eb696/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/StreamHelper.cs#L442) attribute if it's present on the content-type header -- which makes sense, but isn't technically standards compliant for an `application/*` content-type, as far as I can tell.\r\n\r\nTo get started: [`ProcessResponse`](https://github.com/PowerShell/PowerShell/blob/f5f3fab1ea8a51dd2e5d34e0e580ab39176eb696/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/InvokeRestMethodCommand.CoreClr.cs#L54) and [`TryGetEncoding`](https://github.com/PowerShell/PowerShell/blob/f5f3fab1ea8a51dd2e5d34e0e580ab39176eb696/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/StreamHelper.cs#L411) \r\n\r\nSee also #5528 which was a specific instance of this problem. @lipkau was incorrectly convinced by early responders that the problem was in the webserver, but it's actually in PowerShell's cmdlets. If you invoke the rest API against the Atlassian wiki, you can see the problem happening in the Verbose stream:\r\n\r\n```posh\r\n$r = IRM $url -Credential $mycred -Authentication basic -Verbose\r\nVERBOSE: GET https://powershell.atlassian.net/wiki/rest/api/content/13009245?expand=space,version with 0-byte payload\r\nVERBOSE: received -byte response of content type application/json\r\nVERBOSE: Content encoding: iso-8859-1\r\n```\r\n\r\nThe content is **actually** correctly utf-8 encoded (as you could tell from the positions of the nulls in the first 4 bytes), and iso-8859-1 is _**never** a valid encoding for application/json_, period.\r\n\r\n```plain\r\nPS C:\\Program Files\\PowerShell\\6.0.0-rc\u003e $PSVersionTable\r\n\r\nName                           Value\r\n----                           -----\r\nPSVersion                      6.0.0-rc\r\nPSEdition                      Core\r\nGitCommitId                    v6.0.0-rc\r\nOS                             Microsoft Windows 10.0.15063\r\nPlatform                       Win32NT\r\nPSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}\r\nPSRemotingProtocolVersion      2.3\r\nSerializationVersion           1.1.0.1\r\nWSManStackVersion              3.0\r\n```","author":{"url":"https://github.com/Jaykul","@type":"Person","name":"Jaykul"},"datePublished":"2017-11-22T22:33:19.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":15},"url":"https://github.com/5530/PowerShell/issues/5530"}