Title: Addressing the Format challenges in PowerShell · Issue #10463 · PowerShell/PowerShell · GitHub
Open Graph Title: Addressing the Format challenges in PowerShell · Issue #10463 · PowerShell/PowerShell
X Title: Addressing the Format challenges in PowerShell · Issue #10463 · PowerShell/PowerShell
Description: Summary of the new feature/enhancement Since version 1, the implementation of the formatting layer in PowerShell has included many challenging limitations, some of which include the following: Format-* cmdlets output custom format data o...
Open Graph Description: Summary of the new feature/enhancement Since version 1, the implementation of the formatting layer in PowerShell has included many challenging limitations, some of which include the following: Form...
X Description: Summary of the new feature/enhancement Since version 1, the implementation of the formatting layer in PowerShell has included many challenging limitations, some of which include the following: Form...
Opengraph URL: https://github.com/PowerShell/PowerShell/issues/10463
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Addressing the Format challenges in PowerShell","articleBody":"# Summary of the new feature/enhancement\r\n\r\nSince version 1, the implementation of the formatting layer in PowerShell has included many challenging limitations, some of which include the following:\r\n\r\n* `Format-*` cmdlets output custom format data objects that are then rendered in the console once they are passed into `Out-Default` for processing. e.g. `Get-Process -Id $PID | Format-Table | Get-Member`.\r\n* Since these cmdlets convert PSObjects into format data objects, you cannot pipe their results to other commands to do something meaningful. e.g. `Get-Process -Id $PID | Format-Table | Stop-Process -WhatIf`\r\n* Other than `Format-List`, `Format-*` is not respected when you output heterogeneous types that are not compatible with one another. e.g. `& {Get-Process -Id $PID; Get-Item C:\\} | Format-Table`.\r\n* To create a command that produces objects that render in a specific format by default, you must define the format using format ps1xml files, and you must ensure that the PSTypeNames of objects returned from your command have a PSTypeName that matches the specific format you want. This would require a much more complicated example involving ps1xml format files, so I have left that out for now.\r\n\r\nI have attempted to eliminate many of these limitations in PowerShell 5.1 and earlier with my [FormatPx module](https://github.com/KirkMunro/FormatPx], and while that worked out quite well, I also discovered several issues with the FormatPx approach, namely:\r\n\r\n* Producing output formatted in a specific way from a command is easy with FormatPx, but capturing that output in such a way that it maintains the desired format when that output data may also be captured as reference and output via other means (e.g. `$Error` for errors) where you don't want the same format is a difficult problem to solve without some redesign.\r\n* Rendering formatted output when formatted output is not needed (e.g. when you are capturing output or piping a command to another command) is a processing expense that must be avoided until it is needed. Otherwise you may be spending CPU cycles formatting data, only to then have that format replaced with other formats.\r\n* When outputting some data with truly out of band data (e.g. errors, warnings, etc.) in the middle of the output, it is desirable to show that out of band data and then continue the output that you previously started rather than close off the output and then start output anew with headers, etc. being shown again.\r\n\r\nNow I have some users who would like to use `FormatPx` in PowerShell (Core) on Linux, and even though that only requires some relatively minor changes, there are issues I would like to deal with and I believe it would be much more valuable for the community if I take the knowledge gained through my `FormatPx` work and apply it natively to PowerShell 7, since it is open source after all.\r\n\r\n# Proposed technical implementation details\r\n\r\nThe specific problems I would like to solve in PowerShell 7 are as follows:\r\n\r\n1. Make it much easier for scripters to select the format that they want for data output from any command that they author.\r\n\r\n By \"much easier\", I mean without having to unnecessarily muck around with the PSTypeNames array on non-custom objects that are output, and being able to easily select existing, named format views or identify new, command-specific formats that should be used as the default format for the objects output by the command, all while still just outputting objects that can be captured, output, used in expressions, or piped to other commands.\r\n\r\n1. Fix output processing such that heterogeneous types can be output to `Format-Table`, `Format-Wide`, or `Format-Custom` (but that's hardly used) with PowerShell rendering that output in tabular, wide, or custom formats, respectively, without implicitly treating objects with different types as out of band objects.\r\n\r\n This differs from today's behavior, in which the Formatting engine identifies the current \"shape\" to use when rendering formatted data, and any object whose type does not match the type(s) used in formatting data in the current shape are marked as out of band and shown as if they were output with Format-List if they are non-value, non-string types or shown as their string representation if they are value types or strings.\r\n\r\n This is a breaking change, because commands that output multiple heterogeneous types would render differently if they are piped to `Format-Table`, `Format-Wide`, or `Format-Custom`, and scripts that capture that formatted output and do something with it may have issues; however, it is not very likely that there are many scripts out there that produce heterogeneous data types with different formats that then have their data captured in string format and processed. For this reason, I believe it is worth the break, because it corrects a long-standing issue in PowerShell and makes PowerShell `Format-*` cmdlets function in a WYSIWYG fashion, as they should.\r\n\r\nTo solve these problems, I would like to make the following changes:\r\n\r\n1. Extend the `OutputType` attribute such that it includes a new `Format` property.\r\n\r\n For example, consider this command:\r\n\r\n ```powershell\r\n function Get-ProcessByStartTime {\r\n [CmdletBinding()]\r\n [OutputType([System.Diagnostics.Process], Format='Table', FormatParameters=@{\r\n View = 'StartTime'\r\n })]\r\n Get-Process | Where-Object StartTime | Sort-Object StartTime\r\n }\r\n ```\r\n\r\n While that is a contrived example, it demonstrates how a command could be written to apply a specific, named view to the objects that it outputs.\r\n\r\n Now consider this example, which would likely be a much more common approach to specifying custom formatting to use:\r\n\r\n ```powershell\r\n function Get-Something {\r\n [CmdletBinding()]\r\n [OutputType([System.ServiceProcess.ServiceController], Format='Table', FormatParameters=@{\r\n Property = 'Name','DisplayName'\r\n GroupBy = 'Status'\r\n })]\r\n Get-Service | Sort-Object -Property Status,Name\r\n }\r\n ```\r\n\r\n That example shows how you can define a default format for a command without having to create a custom format data entry: you simply specify the desired default format using the `OutputType` attribute.\r\n\r\n Here's one more example to show how it could work with custom objects:\r\n\r\n ```powershell\r\n function Get-OperatingSystem {\r\n [CmdletBinding()]\r\n [OutputType('MyModule.OperatingSystem', Format='List')]\r\n $osData = Get-CimInstance -Class Win32_OperatingSystem\r\n [pscustomobject]@{\r\n PSTypeName = 'MyModule.OperatingSystem'\r\n Name = $osData.Caption\r\n Version = $osData.Version\r\n Architecture = $osData.OSArchitecture\r\n }\r\n }\r\n ```\r\n\r\n In all cases, what I envision would happen behind the scenes is that the returned objects would be wrapped in a lightweight `PSFormatObject` type that is derived from `PSObject`. This type would capture the desired format as specified in the `OutputType` attribute, but it wouldn't actually render the data in that format -- rendering would be deferred until `Out-Default`, allowing the command output to be piped to other commands like normal and saving the time required to render the data until it is needed. If you pipe the output from these commands to any `Format-*` command, the data would immediately be rendered in the desired format. If you show the data output from these commands without using a `Format-*` command, the data would render using the format information that was attached to the `PSFormatObject` objects.\r\n\r\n Note that commands that return multiple object types could define the desired format for each object type.\r\n\r\n1. Fix the heterogeneous output type problem.\r\n\r\n To solve this problem, the formatting engine would be updated such that a new \"shape\" doesn't result in out of band processing. Instead, if the object was not a scalar nor a string (and perhaps other simple types -- I would need to double-check the code to ensure I'm covering the proper types), and if the object was output on the standard output stream, it would result in the closing of the current group/shape, and a new format would be started with the changed, incompatible, heterogeneous type. Non-standard output data as well as scalars and strings would still result in out of band data rendering.\r\n\r\n# Alternate proposals and considerations\r\n\r\n## Alternate proposal: Change how `Format-*` cmdlets work, such that formatting is always deferred until `Out-Default`\r\n\r\nInstead of extending `OutputType`, we could update `Format-*` cmdlets to do the same thing (create a `PSFormatObject` that contains a shared format object reference for each object processed). This would allow piping beyond `Format-*`, and `Format-*` cmdlets would be used to define the format you want when the data is output rather than to convert the data to the desired format output.\r\n\r\nOn the plus side, this would allow scripters to just use `Format-*` where they want, even without dealing with custom type names.\r\n\r\nThere are several downsides to just changing `Format-*` behavior, including the following:\r\n* It would result in functions that return format data in downlevel versions, and object data in current + future versions, which would be confusing.\r\n* It would require users to use the same `Format-*` invocation anywhere that objects are returned from a function or script, which is more difficult than defining the returned format as part of `OutputType`.\r\n* It is less declarative than an `OutputType` format, and less discoverable (`OutputType` has the added benefit of being easily parsed and discovered via programmatic inspection).\r\n\r\nFor these reasons, I would stay away from this approach, and stick with the proposal of extending `OutputType`.\r\n\r\n## Alternate proposal: use something like @lzybkr's PSMore module\r\n\r\n@lzybkr started a [PSMore module](https://github.com/lzybkr/PSMore) a few years ago to address some of PowerShell's formatting issues. That module is a side project that has not seen much movement.\r\n\r\nRegardless of how that module proceeds going forward, I believe the proposal documented herein to be worth the investment because it is additive, and it solves some significant formatting issues without changing how formatting is defined in configuration and without changing how formatting is rendered in PowerShell today. I also don't believe that these changes get in the way of what could be done in the PSMore module going forward.\r\n\r\n## Consideration: Define a `Format-Default` cmdlet\r\n\r\nWith support for more easily customizable formatting without breaking the pipeline-ability of commands, I would also advocate for defining a `Format-Default` cmdlet. This cmdlet would simply replace any `PSFormatObject` objects that were passed into it with their `PSObject` counterparts that do not have format data associated with them, allowing objects to have custom formatting stripped from them.\r\n\r\n## Consideration: Deferring output may result in named formats being unavailable if they are defined in modules that are unloaded\r\n\r\nConsider this scenario:\r\n1. You have a module with a command that returns data configured to use a specific named format.\r\n1. The named format is loaded as part of the same module.\r\n1. You invoke that command and capture the results in a variable.\r\n1. You then unload that module.\r\n1. You then pass your captured data to `Out-Default`.\r\n\r\nBy the time the data reaches `Out-Default`, the desired format that was specified in the command will not be available in memory anymore because it will have unloaded with the module.\r\n\r\nIn this case, the data would fallback to the default output for that data, as if it did not have associated `PSFormatObject` information to work with.\r\n\r\n## Consideration: Use a new `System.Management.Automation.OutputAttribute` instead of extending `OutputType`\r\n\r\nThe name `OutputType` indicates that the attribute is used to define the output type. Instead of extending that to include formatting information as well, it may be better to define a new `Output` attribute (`System.Management.Automation.OutputAttribute`) that supercedes `OutputType`, and use that attribute to define the type of objects that are output, the format used to render the objects that are output, which parameter sets return those type/format details, and leave the door open for additional future properties that could be added to the `Output` attribute (e.g. `Contract=$true` to indicate that output is contractual in a function, which would result in errors if an object that does not match an output type was returned from that function).\r\n\r\nCreating a new attribute in place of extending the `OutputType` attribute has two benefits:\r\n\r\n1. Downlevel users will not be confused when they see `OutputType` used with different syntax and get errors. Instead they will see a new attribute that they haven't used before. They'll still get errors, but at least those errors won't be related to an attribute that is supported/documented in the version they are using.\r\n1. Since this makes output attribution about much more than a type, having a new attribute for that with a different name may bring more clarity to what is being done. The old attribute would still be supported, so backwards compatibility would persist, but the Parser should probably raise an error if both attributes are used in the same command (see additional comments about Parser improvements when it comes to attributes below).\r\n\r\n## Consideration: New attributes and attribute extensions would result in runtime errors in downlevel versions\r\n\r\nIt is worth pointing out that it doesn't matter if you extend an attribute with additional properties or if you define a new attribute -- both result in _runtime_ errors, not parse errors. I'm not surprised for the latter since attributes can be defined in modules, but I'm a little surprised for the former. I feel it would be better if attribute extensions via additional properties resulted in parse errors, but there may be a valid reason that they don't that I am not aware of. At any rate, I call this out as a consideration because scripts written with attribute changes will result in runtime errors, not parse errors, in downlevel versions of PowerShell.\r\n\r\nI think it is also worth updating the PowerShell parser to return parse errors if attributes are used that do not exist in a version of PowerShell, or if properties within those attributes that are used do not exist in a version of PowerShell. That is something that can be determined at parse time, and having a parser error would help guide users towards finding a version of PowerShell where those attributes and their properties are defined/supported. This work should be done as a separate PR (and a separate issue will be logged to track this need if we have consensus on this point).\r\n","author":{"url":"https://github.com/KirkMunro","@type":"Person","name":"KirkMunro"},"datePublished":"2019-08-29T20:37:32Z","url":"https://github.com/PowerShell/PowerShell/issues/10463"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:7c95a88f-1d98-13ac-869e-84f0e1a095a3 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C506:12AA8F:1029AC6:175011B:6AFDBAF5 |
| html-safe-nonce | f552045d7b7b62f1547df5715b445efc02a167163d206b9e5cef4b56df8ef535 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNTA2OjEyQUE4RjoxMDI5QUM2OjE3NTAxMUI6NkFGREJBRjUiLCJ2aXNpdG9yX2lkIjoiNzAxNzkzODkzMDk1NTU2NTgxMyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 7b636f5a9c48b1290a7b27851471e4e4f81b20535b91553ccceb97fbb031b49f |
| hovercard-subject-tag | issue:487164650 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/PowerShell/PowerShell/10463/issue_layout |
| twitter:image | https://opengraph.githubassets.com/a7665261e23ed6633a7ac6533e53d7ecfd28e028748241131aa35c802d320d98/PowerShell/PowerShell/issues/10463 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/a7665261e23ed6633a7ac6533e53d7ecfd28e028748241131aa35c802d320d98/PowerShell/PowerShell/issues/10463 |
| og:image:alt | Summary of the new feature/enhancement Since version 1, the implementation of the formatting layer in PowerShell has included many challenging limitations, some of which include the following: Form... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | KirkMunro |
| hostname | github.com |
| expected-hostname | github.com |
| None | 8309f07eb60a4237a657068c83d0bb57be4c7e02937f5e9933126f4bd596aac4 |
| turbo-cache-control | no-preview |
| go-import | github.com/PowerShell/PowerShell git https://github.com/PowerShell/PowerShell.git |
| octolytics-dimension-user_id | 11524380 |
| octolytics-dimension-user_login | PowerShell |
| octolytics-dimension-repository_id | 49609581 |
| octolytics-dimension-repository_nwo | PowerShell/PowerShell |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 49609581 |
| octolytics-dimension-repository_network_root_nwo | PowerShell/PowerShell |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2f7154d847f1ddc5baa83b109a3e1d924605e2a3 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width