Title: Request for help: how to use DotNet HttpClient to async upload large files using multipart/form-data within PowerShell · Issue #13721 · PowerShell/PowerShell · GitHub
Open Graph Title: Request for help: how to use DotNet HttpClient to async upload large files using multipart/form-data within PowerShell · Issue #13721 · PowerShell/PowerShell
X Title: Request for help: how to use DotNet HttpClient to async upload large files using multipart/form-data within PowerShell · Issue #13721 · PowerShell/PowerShell
Description: I was asked in #4129 to open a new issue to discuss my request for help. I originally posted this question on Stackoverflow, and have had no responses. I'm not sure if this is a question for the PowerShell or DotNetCore team, but I'm sta...
Open Graph Description: I was asked in #4129 to open a new issue to discuss my request for help. I originally posted this question on Stackoverflow, and have had no responses. I'm not sure if this is a question for the Po...
X Description: I was asked in #4129 to open a new issue to discuss my request for help. I originally posted this question on Stackoverflow, and have had no responses. I'm not sure if this is a question for th...
Opengraph URL: https://github.com/PowerShell/PowerShell/issues/13721
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Request for help: how to use DotNet HttpClient to async upload large files using multipart/form-data within PowerShell","articleBody":"I was asked in #4129 to open a new issue to discuss my request for help. I originally posted this question on [Stackoverflow](https://stackoverflow.com/questions/63711832/c-sharp-netcore-httpclient-postasync-multipart-form-data-with-file-attachment-a), and have had no responses. I'm not sure if this is a question for the PowerShell or DotNetCore team, but I'm starting here.\r\n\r\nI am in the process of working on updating some PowerShell code that uses `System.Net.HttpWebRequest` to handle file transfers to and from a web service. The `System.IO.FileStream` class has a major problem with handling file streams being transferred to a web service using `HttpWebRequest` and the object (file) is greater than 2GB. [It was stated](https://github.com/PowerShell/PowerShell/issues/4129#issuecomment-617542425) in #4129 this was a breaking change, and will not be fixed by the .NetCore team, [and ask the question here](https://github.com/PowerShell/PowerShell/issues/4129#issuecomment-672008052). Essentially, one runs into the following exception in PowerShellCore/PowerShell7:\r\n\r\n```powershell\r\nMethodInvocationException: C:\\Users\\clynch\\Documents\\WindowsPowerShell\\modules\\modulename\\modulename.psm1:35180\r\n Line |\r\n35180 | $rs.write($readbuffer, 0, $bytesRead)\r\n | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n | Exception calling \"Write\" with \"3\" argument(s): \"Stream was too long.\"\r\n```\r\nI have the following verified and working PowerShell code I am trying to convert to C# async class method using `HttpClient` to asynchronously upload files to an Apache/Spring-based REST API service when using PowerShell 5.x, or within PowerShell7/PowerShellCore only when the file is less than 2GB in size:\r\n\r\n```powershell\r\n$Hostname = \"somewebserver\"\r\n$Method = \"POST\"\r\n$File = Get-ChildItem C:\\Directory\\file.iso\r\n$FSOpenMode = [System.IO.FileMode]::Open\r\n$FSRead = [System.IO.FileAccess]::Read\r\n\r\n# Read file using FileStream\r\n$fs = [IO.FileStream]::new($File.FullName, $FSOpenMode, $FSRead)\r\n\r\n[void]$fs.FlushAsync()\r\n\r\ntry\r\n{\r\n\r\n # Set the URL that will be the upload destination\r\n $url = \"https://{0}/upload?uploadfilename={1}\" -f $Hostname, $File.Name\r\n\r\n $_DispositionContentType = \"application/octet-stream\"\r\n\r\n [System.Net.httpWebRequest]$uploadRequest = [System.Net.WebRequest]::Create($uri)\r\n $uploadRequest.Method = $Method\r\n\r\n $boundary = \"---------------------------\" + [DateTime]::Now.Ticks.ToString(\"x\")\r\n [byte[]]$BoundaryBytes = [System.Text.Encoding]::UTF8.GetBytes(\"`r`n--\" + $boundary + \"`r`n\");\r\n $disposition = \"Content-Disposition: form-data; name=`\"file`\"; filename=`\"{0}`\";`r`nContent-Type: {1}`r`n`r`n\" -f $File.Name, $_DispositionContentType\r\n [byte[]]$ContentDispBytes = [System.Text.Encoding]::UTF8.GetBytes($disposition);\r\n [byte[]]$EndBoundaryBytes = [System.Text.Encoding]::UTF8.GetBytes(\"`r`n--\" + $boundary + \"--`r`n\")\r\n\r\n $uploadRequest.Timeout = 1200000\r\n $uploadRequest.ContentType = \"multipart/form-data; boundary={0}\" -f $boundary\r\n $uploadRequest.Headers.Item(\"auth\") = \"SessionID\"\r\n $uploadRequest.Headers.Item(\"uploadfilename\") = $File.Name\r\n $uploadRequest.AllowWriteStreamBuffering = $true\r\n $uploadRequest.SendChunked = $true\r\n $uploadRequest.ContentLength = $BoundaryBytes.length + $ContentDispBytes.length + $File.Length + $EndBoundaryBytes.Length\r\n $uploadRequest.Headers.Item(\"ContentLength\") = $BoundaryBytes.length + $ContentDispBytes.length + $File.Length + $EndBoundaryBytes.Length\r\n\r\n $rs = $uploadRequest.GetRequestStream()\r\n [void]$rs.FlushAsync()\r\n\r\n [byte[]]$readbuffer = [byte[]]::new(4096 * 1024)\r\n $rs.write($BoundaryBytes, 0, $BoundaryBytes.Length);\r\n $rs.write($ContentDispBytes, 0, $ContentDispBytes.Length);\r\n\r\n # This is used to keep track of the file upload progress.\r\n $numBytesToRead = $fs.Length\r\n [int64]$numBytesRead = 0\r\n\r\n $_sw = [System.Diagnostics.Stopwatch]::StartNew()\r\n $_progresssw = [System.Diagnostics.Stopwatch]::StartNew()\r\n\r\n while ($bytesRead = $fs.Read($readbuffer, 0, $readbuffer.length))\r\n {\r\n\r\n [void]$fs.Flush()\r\n\r\n $rs.write($readbuffer, 0, $bytesRead)\r\n\r\n [void]$fs.Flush()\r\n [void]$rs.Flush()\r\n\r\n # Keep track of where we are at clearduring the read operation\r\n $_numBytesRead += $bytesRead\r\n\r\n # Flush the buffer every 200ms and 1MB written\r\n if ($_progresssw.Elapsed.TotalMilliseconds -ge 200 -and $_numBytesRead % 100mb -eq 0)\r\n {\r\n\r\n [void]$rs.flush()\r\n\r\n }\r\n\r\n # Use the Write-Progress cmd-let to show the progress of uploading the file.\r\n [Int]$_percent = [math]::floor(($_numBytesRead / $fs.Length) * 100)\r\n\r\n # Elapsed time to calculat throughput\r\n [Int]$_elapsed = $_sw.ElapsedMilliseconds / 1000\r\n\r\n if ($_elapsed -ne 0 )\r\n {\r\n\r\n [single]$_transferrate = [Math]::Round(($_numBytesRead / $_elapsed) / 1mb)\r\n\r\n }\r\n\r\n else\r\n {\r\n\r\n [single]$_transferrate = 0.0\r\n\r\n }\r\n\r\n $status = \"({0:0}MB of {1:0}MB transferred @ {2}MB/s) Completed {3}%\" -f ($_numBytesRead / 1MB), ($numBytesToRead / 1MB), $_transferrate, $_percent\r\n\r\n # Handle how poorly Write-Progress adds latency to the file transfer process by only refreshing the progress at a specifc interval\r\n if ($_progresssw.Elapsed.TotalMilliseconds -ge 500)\r\n {\r\n\r\n if ($_numBytesRead % 1mb -eq 0)\r\n {\r\n\r\n Write-Progress -activity \"Upload File\" -status (\"Uploading '{0}'\" -f $File.Name) -CurrentOperation $status -PercentComplete $_percent\r\n\r\n }\r\n\r\n }\r\n\r\n }\r\n\r\n\r\n $fs.close()\r\n\r\n # Write the endboundary to the file's binary upload stream\r\n $rs.write($EndBoundaryBytes, 0, $EndBoundaryBytes.Length)\r\n\r\n $rs.close()\r\n\r\n $_sw.stop()\r\n $_sw.Reset()\r\n\r\n Write-Progress -activity \"Upload File\" -status (\"Uploading '{0}'\" -f $File.Name) -Complete\r\n\r\n}\r\n\r\ncatch [System.Exception]\r\n{\r\n\r\n if ($fs)\r\n {\r\n\r\n $fs.close()\r\n\r\n }\r\n\r\n if ($_sw.IsRunning)\r\n {\r\n\r\n $_sw.Stop()\r\n $_sw.Reset()\r\n\r\n }\r\n\r\n # Dispose if still exist\r\n if ($rs)\r\n {\r\n\r\n $rs.close()\r\n\r\n }\r\n\r\n Throw $_\r\n\r\n}\r\n\r\ntry\r\n{\r\n\r\n # Indicate we are waiting for the API to process the uploaded file in the write progress display\r\n Write-Progress -activity \"Upload File\" -status (\"Uploading '{0}'\" -f $File.Name) -CurrentOperation \"Waiting for completion response from appliance.\" -percentComplete $_percent\r\n\r\n # Get the response from the API on the progress of identifying the file\r\n [Net.httpWebResponse]$WebResponse = $uploadRequest.getResponse()\r\n $uploadResponseStream = $WebResponse.GetResponseStream()\r\n\r\n # Read the response & convert to JSON\r\n $reader = [System.IO.StreamReader]::new($uploadResponseStream)\r\n $responseJson = $reader.ReadToEnd()\r\n\r\n $uploadResponse = ConvertFrom-Json $responseJson\r\n\r\n $uploadResponseStream.Close()\r\n\r\n $uploadRequest = $Null\r\n\r\n # Finalize write progress display\r\n Write-Progress -activity \"Upload File\" -CurrentOperation \"Uploading $Filename \" -Completed\r\n\r\n}\r\n\r\ncatch [Net.WebException]\r\n{\r\n\r\n if ($null -ne $_.Exception.Response)\r\n {\r\n\r\n Try\r\n {\r\n\r\n # Need to see if Response is not empty\r\n\r\n $sr = [IO.StreamReader]::new($_.Exception.Response.GetResponseStream())\r\n\r\n }\r\n\r\n Catch\r\n {\r\n\r\n $PSCmdlet.ThrowTerminatingError($_)\r\n\r\n }\r\n\r\n $errorObject = $sr.readtoEnd() | ConvertFrom-Json\r\n\r\n # dispose if still exist\r\n if ($rs)\r\n {\r\n\r\n $rs.close()\r\n\r\n }\r\n\r\n if ($fs)\r\n {\r\n\r\n $fs.close()\r\n\r\n }\r\n\r\n $sr.close()\r\n\r\n # This New-ErrorRecord is an internal function that generates a new System.Management.Automation.ErrorRecord object to then throw\r\n $ErrorRecord = New-ErrorRecord HPEOneview.Appliance.UploadFileException $errorObject.ErrorCode InvalidResult 'Upload-File' -Message $errorObject.Message -InnerException $_.Exception\r\n\r\n Throw $ErrorRecord\r\n\r\n }\r\n\r\n else\r\n {\r\n\r\n Throw $_\r\n\r\n }\r\n\r\n}\r\n```\r\n\r\nThe following working C# code is the result of some research I have done to try to understand how HttpClient works with multipart/form-data uploads:\r\n\r\n``` csharp\r\npublic class HttpClientUpload : IDisposable\r\n{\r\n\r\n private readonly string _uploadUrl;\r\n private readonly string _sourceFilePath;\r\n private readonly string _authToken;\r\n\r\n // THIS IS ONLY HERE FOR TESTING\r\n private static HttpClientHandler _handler = new HttpClientHandler() { ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => { return true; } };\r\n private HttpClient _httpClient = new HttpClient(_handler)\r\n {\r\n Timeout = TimeSpan.FromDays(1)\r\n };\r\n\r\n public delegate void ProgressChangedHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);\r\n\r\n public event ProgressChangedHandler ProgressChanged;\r\n\r\n public HttpClientUpload(string uploadUrl, string sourceFilePath, string authToken)\r\n {\r\n _uploadUrl = uploadUrl;\r\n _sourceFilePath = sourceFilePath;\r\n _authToken = authToken;\r\n }\r\n\r\n // Borrowed code from https://stackoverflow.com/questions/16416601/c-sharp-httpclient-4-5-multipart-form-data-upload\r\n public void UploadAsync()\r\n {\r\n\r\n var Path = _sourceFilePath;\r\n var Appliance = _uploadUrl;\r\n var AuthToken = _authToken;\r\n\r\n var m_CancellationSource = new CancellationTokenSource();\r\n var token = m_CancellationSource.Token;\r\n var fileInfo = new System.IO.FileInfo(Path);\r\n var uri = $\"https://{Appliance}/rest/firmware-bundles?uploadfilename={fileInfo.Name}\";\r\n\r\n // If the file extension is CRL, we need to use a different disposition/mime type declaration.\r\n var DispositionContentType = fileInfo.Extension == \"crl\" ? \"application/pkix-crl\" : \"application/octet-stream\";\r\n\r\n _httpClient.DefaultRequestHeaders.Add(\"User-Agent\", $\"Custom User Agent ({Environment.OSVersion.ToString()})\");\r\n _httpClient.DefaultRequestHeaders.Add(\"X-API-Version\", \"1800\");\r\n _httpClient.DefaultRequestHeaders.Add(\"auth\", AuthToken);\r\n _httpClient.DefaultRequestHeaders.Add(\"uploadfilename\", fileInfo.Name);\r\n _httpClient.DefaultRequestHeaders.Add(\"accept-language\", \"en_US\");\r\n _httpClient.DefaultRequestHeaders.Add(\"accept-encoding\", \"gzip, deflate\");\r\n\r\n using (var content = new MultipartFormDataContent(\"---------------------------\" + DateTime.Now.Ticks.ToString(\"x\")))\r\n {\r\n\r\n content.Headers.Add(\"Content-Disposition\", $\"form-data; name=\\\"file\\\"; filename=\\\"{fileInfo.Name}\\\"\");\r\n\r\n content.Headers.ContentLength = fileInfo.Length;\r\n\r\n FileStream fs = File.OpenRead(Path);\r\n\r\n var streamContent = new StreamContent(fs);\r\n streamContent.Headers.Add(\"Content-Type\", $\"{DispositionContentType}\");\r\n streamContent.Headers.Add(\"Content-Disposition\", $\"form-data; name=\\\"file\\\"; filename=\\\"{fileInfo.Name}\\\"\");\r\n content.Add(streamContent, \"file\", fileInfo.FullName);\r\n\r\n Task\u003cHttpResponseMessage> message = _httpClient.PostAsync(uri, content);\r\n\r\n var input = message.Result.Content.ReadAsStringAsync();\r\n Console.WriteLine(input.Result);\r\n Console.WriteLine(\"Press enter to continue...\");\r\n Console.Read();\r\n\r\n }\r\n\r\n }\r\n\r\n public void Stop()\r\n {\r\n\r\n _httpClient.CancelPendingRequests();\r\n\r\n throw new OperationCanceledException(\"File upload cancelled.\");\r\n\r\n }\r\n\r\n public void Dispose()\r\n {\r\n\r\n _httpClient?.Dispose();\r\n\r\n }\r\n\r\n}\r\n```\r\nThe code above can be invoked from PowerShell as such, using a compiled DotNet class library that is imported into the users PowerShell runspace or using `Add-Type`:\r\n\r\n```powershell\r\n$Hostname = \"somewebserver\"\r\n$AuthToken = \"SessionID\"\r\n$FileName = 'C:\\directory\\somefile.iso'\r\n$Task = ([HttpClientUpload]::new($Hostname, $FileName, $AuthToken)).UploadAsync()\r\n```\r\n\r\nThe file appears to be transferred to the target, but the API endpoint generates an error. Not an HTTP error, but a JSON response indicating that this webservice couldn't parse the file. I get this in our API application log file:\r\n\r\n> caught apache common fileuploadexception ex=Processing of multipart/form-data request failed. Stream ended unexpectedly while uploading file somefile.iso\r\n\r\nI know that this is caused by the fact that the request from the client must append a byte array of an endboundary. Because without it, I would get the very same message with the original PowerShell code I currently use if I omit the following line(s):\r\n\r\n```powershell\r\n# Write the endboundary to the file's binary upload stream\r\n$rs.write($EndBoundaryBytes, 0, $EndBoundaryBytes.Length)\r\n```\r\n\r\nSo, the TL;DR to my post. My questions are:\r\n\r\n1. How does one ensure that a multipart/form-data request that needs to include a binary stream of a file contains the start and end boundary for the `HttpClient` request? I have found no Microsoft documentation that outlines the requirements to properly construct an `HttpClient` object with the necessary IO stream to send this multipart/form-data request to a web service.\r\n1. Any guidance on how to then chunk the file stream in order to report buffer transfer progress? I intend to read this from PowerShell after the async task has begun, in order to use Write-Progress back to the user.\r\n\r\nPowerShell environments:\r\n\r\n```powershell\r\n# Windows 10\r\n\r\nName Value\r\n---- -----\r\nPSVersion 7.0.3\r\nPSEdition Core\r\nGitCommitId 7.0.3\r\nOS Microsoft Windows 10.0.19041\r\nPlatform Win32NT\r\nPSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}\r\nPSRemotingProtocolVersion 2.3\r\nSerializationVersion 1.1.0.1\r\nWSManStackVersion 3.0\r\n```\r\n\r\n```powershell\r\n# Ubuntu 18.04 WSL2 container\r\nPS /mnt/c/Users/clynch> $PSVersiontable\r\n\r\nName Value\r\n---- -----\r\nPSVersion 7.0.1\r\nPSEdition Core\r\nGitCommitId 7.0.1\r\nOS Linux 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58…\r\nPlatform Unix\r\nPSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}\r\nPSRemotingProtocolVersion 2.3\r\nSerializationVersion 1.1.0.1\r\nWSManStackVersion 3.0\r\n```","author":{"url":"https://github.com/ChrisLynchHPE","@type":"Person","name":"ChrisLynchHPE"},"datePublished":"2020-09-30T20:26:58Z","url":"https://github.com/PowerShell/PowerShell/issues/13721"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:32000339-6c29-b65a-57a4-f51c6a091d5b |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | BEBC:3CA547:1FEF08:2AFC56:6AFD44E7 |
| html-safe-nonce | 42c4477b5c2a565d9fe61fc04aae06558c1469d89344dd6bad1271740562da61 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRUJDOjNDQTU0NzoxRkVGMDg6MkFGQzU2OjZBRkQ0NEU3IiwidmlzaXRvcl9pZCI6IjIyOTM0MTE1NzYwNjI3NTYwNzEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | ee1cbb7acabef951251d1e2ceb1413322a95c6a22e4a63c33809eff421a36107 |
| hovercard-subject-tag | issue:712256949 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/PowerShell/PowerShell/13721/issue_layout |
| twitter:image | https://opengraph.githubassets.com/513a8b0388ccde8a84ca538e5aef5e9742e12661475380731a9c86601e8602dc/PowerShell/PowerShell/issues/13721 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/513a8b0388ccde8a84ca538e5aef5e9742e12661475380731a9c86601e8602dc/PowerShell/PowerShell/issues/13721 |
| og:image:alt | I was asked in #4129 to open a new issue to discuss my request for help. I originally posted this question on Stackoverflow, and have had no responses. I'm not sure if this is a question for the Po... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | ChrisLynchHPE |
| hostname | github.com |
| expected-hostname | github.com |
| None | bdbacc25b921224ff0befcb9e629560a1255be51aec9991d6e3f95b5e9151474 |
| turbo-cache-control | no-preview |
| go-import | github.com/PowerShell/PowerShell git https://github.com/PowerShell/PowerShell.git |
| octolytics-dimension-user_id | 11524380 |
| octolytics-dimension-user_login | PowerShell |
| octolytics-dimension-repository_id | 49609581 |
| octolytics-dimension-repository_nwo | PowerShell/PowerShell |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 49609581 |
| octolytics-dimension-repository_network_root_nwo | PowerShell/PowerShell |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 556dd37bb0aceffb25724e151b5d858b3265b40b |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width