René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"WebClient needs to expose the timeout property (or improve Invoke-WebRequest implementation) when downloading large files ","articleBody":"I have been having some issues interacting with REST full APIs, specifically with our product (Pexip) which is writing with Python and Django and uses Apache. Some of these issues are outlined in #2112 and #4274 and this involves a similar an linked problem. I also added a comment to https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-5.1 that outlines this issue that I have repeated below:\r\n\r\n# Attempting to download a large file, GETting from a REST API.\r\n## The Basic Invoke-WebRequest\r\n\r\n```\r\n    Invoke-WebRequest -Method Get -Uri $resource -cred $cred -OutFile $Destination -TimeoutSec 1800\r\n```\r\n\r\nThis would be the ideal. I can set timeout values and seemingly achieve what I want in one line. Unfortunately, this completely locked up the machine the script was running on. Basically, all the memory was consumed and after an hour or so, I had to end the script – of the 3.5 GB file, I think 200 MB was downloaded! What I needed is a way to stream the remote file to a local file without loading everything into memory.\r\n\r\n## Using WebClient\r\nThis seemed a lot better, however, the WebClient has no timeout property!!!\r\n\r\n```\r\n    $wc = New-Object System.Net.WebClient     \r\n    $wc.Credentials = New-Object System.Net.NetworkCredential($User, $Pass)\r\n    $wc.Headers.Add(\"Authorization\", \"Basic $encoded\")\r\n    $wc.DownloadFile($resource, $DestinationPath)\r\n```\r\n\r\nThis is what I ended up using and in order to get the timeout to work, I used an extended class – luckily someone clever than I has achieved this for PowerShell using a C# class (https://koz.tv/setup-webclient-timeout-in-powershell/) although I assume the native classes in PS could also be used – but why?\r\n\r\nI have also found that I must add the additional Authorisation Headers, otherwise the client will get challenged then end up download the file effectively twice. Using Fiddler and watching the Ethernet stats, you can see this happen. Indeed, this seemingly occurred for all methods\r\n\r\n## Using HTTP client.\r\nWell I tried and failed dismally. However, I seem to create the Httpclient, it only ever seemed to stream the remote file into memory before dumping to a file. I know what I wanted to do, but do not have the wherewithal to do it.\r\n","author":{"url":"https://github.com/swinster","@type":"Person","name":"swinster"},"datePublished":"2017-08-26T15:49:34.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":10},"url":"https://github.com/4679/PowerShell/issues/4679"}