Title: Import-CSV has very high filesize-to-memory consumption ratio of around 50 · Issue #7603 · PowerShell/PowerShell · GitHub
Open Graph Title: Import-CSV has very high filesize-to-memory consumption ratio of around 50 · Issue #7603 · PowerShell/PowerShell
X Title: Import-CSV has very high filesize-to-memory consumption ratio of around 50 · Issue #7603 · PowerShell/PowerShell
Description: Steps to reproduce Importing a 150 MB CSV file with 50k rows, 700 columns and 5 +/- 3 characters per entry results in a memory consumption of 7GB of the PowerShell process. This is a ratio of around 50 when compared to the file size. I h...
Open Graph Description: Steps to reproduce Importing a 150 MB CSV file with 50k rows, 700 columns and 5 +/- 3 characters per entry results in a memory consumption of 7GB of the PowerShell process. This is a ratio of aroun...
X Description: Steps to reproduce Importing a 150 MB CSV file with 50k rows, 700 columns and 5 +/- 3 characters per entry results in a memory consumption of 7GB of the PowerShell process. This is a ratio of aroun...
Opengraph URL: https://github.com/PowerShell/PowerShell/issues/7603
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Import-CSV has very high filesize-to-memory consumption ratio of around 50","articleBody":"Steps to reproduce\r\n------------------\r\nImporting a 150 MB CSV file with 50k rows, 700 columns and 5 +/- 3 characters per entry results in a memory consumption of 7GB of the PowerShell process. This is a ratio of around 50 when compared to the file size. I have seen this ratio also with a similar file of smaller size (1.5MB) and therefore think this is a more generic issue not specific to the CSV file structure itself.\r\n```powershell\r\n$csv = Import-Csv $pathToCsvFiile\r\n```\r\n\r\nNote that this does not happen if I do not assign to a variable or pipe to `Out-Null`\r\n\r\nExpected behavior\r\n-----------------\r\n\r\nFilesize to memory ratio is reasonable (around 10)\r\n\r\nActual behavior\r\n---------------\r\n\r\nFilesize to memory ratio is around 50\r\n\r\nEnvironment data\r\n----------------\r\n\r\nLatest daily build and RC1\r\n\r\n```powershell\r\n\u003e $PSVersionTable\r\nName Value\r\n---- -----\r\nPSVersion 6.1.0-preview.761\r\nPSEdition Core\r\nGitCommitId 6.1.0-preview.761\r\nOS Microsoft Windows 6.3.9600\r\nPlatform Win32NT\r\nPSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}\r\nPSRemotingProtocolVersion 2.3\r\nSerializationVersion 1.1.0.1\r\nWSManStackVersion 3.0\r\n\r\n\r\nName Value\r\n---- -----\r\nPSVersion 6.1.0-rc.1\r\nPSEdition Core\r\nGitCommitId 6.1.0-rc.1\r\nOS Microsoft Windows 6.3.9600\r\nPlatform Win32NT\r\nPSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}\r\nPSRemotingProtocolVersion 2.3\r\nSerializationVersion 1.1.0.1\r\nWSManStackVersion 3.0\r\n\r\n\r\n```\r\n","author":{"url":"https://github.com/bergmeister","@type":"Person","name":"bergmeister"},"datePublished":"2018-08-22T13:25:49.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":22},"url":"https://github.com/7603/PowerShell/issues/7603"}
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:2096cbd8-c580-e503-63d9-91179f91e66a |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | CF40:26C765:473728:604A92:6A52332D |
| html-safe-nonce | 9a8dad05996d913088bc1227afae337e5282f9358aef08ed46e90b63b567e335 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRjQwOjI2Qzc2NTo0NzM3Mjg6NjA0QTkyOjZBNTIzMzJEIiwidmlzaXRvcl9pZCI6IjYyNjA2MzM4NDc1ODg4NjA3MTciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 8bbbc697837799141a1ea37fe5a50c0fabe9c7a178e720b5bdff89bd478d0764 |
| hovercard-subject-tag | issue:352952814 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/PowerShell/PowerShell/7603/issue_layout |
| twitter:image | https://opengraph.githubassets.com/8388a71ab41aeb22a5af5ea1fef1120481b354223f5f8acc504c97d1132e7790/PowerShell/PowerShell/issues/7603 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/8388a71ab41aeb22a5af5ea1fef1120481b354223f5f8acc504c97d1132e7790/PowerShell/PowerShell/issues/7603 |
| og:image:alt | Steps to reproduce Importing a 150 MB CSV file with 50k rows, 700 columns and 5 +/- 3 characters per entry results in a memory consumption of 7GB of the PowerShell process. This is a ratio of aroun... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | bergmeister |
| hostname | github.com |
| expected-hostname | github.com |
| None | b9a586c06a05a7a86fc7e3f4dbd03e42f6869085879aa184aa6369456dbd50fb |
| turbo-cache-control | no-preview |
| go-import | github.com/PowerShell/PowerShell git https://github.com/PowerShell/PowerShell.git |
| octolytics-dimension-user_id | 11524380 |
| octolytics-dimension-user_login | PowerShell |
| octolytics-dimension-repository_id | 49609581 |
| octolytics-dimension-repository_nwo | PowerShell/PowerShell |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 49609581 |
| octolytics-dimension-repository_network_root_nwo | PowerShell/PowerShell |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 7aed05249554b889eb33d002851a973eebcc7e91 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width