René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Engine event processing bypasses \"ShouldQueueAndProcessInExecutionThread\" causing state corruption and crash due to Runspace affinity","articleBody":"This was found while investigating a PowerShell class static/instance method concurrency bug.\r\n\r\n## Background\r\n\r\nWhen invoking a script block using [`InvokeWithPipe`](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/lang/scriptblock.cs#L910), if the script block is bound to a different Runspace (e.g. created in a different Runspace), then the script block will be marshaled to that Runspace using the `EventManager` and is supposed to be executed by the main pipeline thread of that Runspace.\r\n\r\nThis is how it's done:\r\n1. The thread that is calling `InvokeWithPipe` finds that the script block is bound to another Runsapce and needs to run in that Runspace. (code [here](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/lang/scriptblock.cs#L961))\r\n2. The thread queues an event on the `EventManager` of the target Runspace, wishing the main pipeline thread of the target Runspace would pick up the event and run the script block. (code [here](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/lang/scriptblock.cs#L978))\r\n3. The thread waits for the main pipeline thread of the target Runspace to finish running the script block. (code [here](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/EventManager.cs#L984))\r\n\r\n## Issues\r\n\r\nThe problem is that, to avoid a possible hang, the thread, which waits for the main pipeline thread of the target Runspace, will pick up the event and execute it (not the required main pipeline thread) after waiting for 250 mSec (See the code [here](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/EventManager.cs#L998)). This will result in 2 threads running in the same Runspace and changing its state simultaneously.  This would cause:\r\n\r\n1. Deadlock.\r\n2. Runspace state corruption and process crash.\r\n\r\n## Repro Steps\r\n\r\n### Deadlock\r\n\r\nBoth arguments for '-sb' and '-arg' are script blocks that are created in the powershell console session. So when `bar` runs `$sb.InvokeReturnAsIs($arg)` in a new Runspace, it needs to marshal it back to the powershell console session. This is what happens:\r\n\r\n1. The new Runspace thread (**aka. requesting thread**) cannot execute the script block because it has to run in the powershell console session (**aka. target Runspace**) which is bound with the script block, so it queues an event for the pipeline thread of the target Runspace to run the script block;\r\n2. However, the target Runspace is completely unresponsive because it’s blocked on `$ps.Invoke()`;\r\n3. So, after 250ms, the requesting thread go ahead to process the event itself to run the script block. Note that – an event is now executing;\r\n4. Again, the requesting thread finds it cannot run `$arg` and goes back to step (1). However, this time when it comes to step (3), an event is already executing. So [`this.ProcessPendingActions()`](https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/EventManager.cs#L1000) will immediately return, and the requesting thread will be stuck in the while loop.\r\n\r\n```powershell\r\n## The deadlock happens on PowerShell Core RC builds\r\n$ps = [powershell]::Create()\r\n$ps.AddScript('function bar { param([scriptblock]$sb, [scriptblock]$arg) $sb.InvokeReturnAsIs($arg) }').Invoke()\r\n$ps.Commands.Clear()\r\n$ps.AddCommand(\"bar\").AddParameter('sb', {param([scriptblock] $s) $s.InvokeReturnAsIs()}).AddParameter('arg', {[Console]::WriteLine(\"blah\")}) \u003e $null\r\n$ps.Invoke()\r\n```\r\n\r\n### Runspace state corruption and process crash\r\n\r\n\u003e **Note: this doesn't repro on latest PowerShell Core anymore because we have fixed the PowerShell class static method to not route method execution to other Runspaces incorrectly. But the underlying problem in `EventManager` is still there. You can run this repro in Windows PowerShell v5.1 to see the results.**\r\n\r\nThis repro creates a script `DoInvokeInParallel.ps1` that dot-sources an `Invoker.ps1` file which defines a class with a static method.  Run `DoInvokeInParallel.ps1` in multiple `Runspaces` via a `RunspacePool` to use the class static method concurrently.\r\n\r\n`Invoker.ps1` file\r\n```powershell\r\nclass Invoker\r\n{\r\n    static [object[]] Invoke(\r\n        [scriptblock] $scriptToInvoke,\r\n        [object[]] $args)\r\n    {\r\n        return $scriptToInvoke.Invoke($args)\r\n    }\r\n}\r\n```\r\n\r\n`DoInvokeInParallel.ps1` file\r\n``` powershell\r\n$invokerPath = Join-Path $PSScriptRoot Invoker.ps1\r\n. $invokerPath\r\n\r\n$rsp = [runspacefactory]::CreateRunspacePool(1, 10, $host)\r\n$rsp.Open()\r\n\r\n$scriptTemplate = @'\r\n    . {0}\r\n    1..100 | foreach {{\r\n        $results = [Invoker]::Invoke({{ \"RS_{1} Loop $_ `r`n\" }}, $null)\r\n        Write-Output $results\r\n    }}\r\n'@\r\n\r\nclass Task\r\n{\r\n    [powershell] $powershell\r\n    [System.IAsyncResult] $Async\r\n}\r\n\r\n$tasks = @()\r\n\r\n1..10 | foreach {\r\n    $task = [Task]::new()\r\n    $script = $scriptTemplate -f $invokerPath,$_\r\n    $task.powershell = [powershell]::Create()\r\n    $null = $task.powershell.AddScript($script)\r\n    $task.powershell.RunspacePool = $rsp\r\n    $task.Async = $task.powershell.BeginInvoke()\r\n    $tasks += $task\r\n}\r\n\r\nforeach ($task in $tasks)\r\n{\r\n    $results = $task.powershell.EndInvoke($task.Async)\r\n    Write-Host $results\r\n    Write-Host $task.powershell.Streams.Error\r\n    $task.powershell.Dispose()\r\n}\r\n\r\n$rsp.Dispose()\r\n```\r\n**Run `DoInvokeInParallel.ps1`. The result is multiple \"invalid session state\" asserts if you are using a debug flavor Windows PowerShell. Eventually, the process will crash.**\r\n","author":{"url":"https://github.com/PaulHigin","@type":"Person","name":"PaulHigin"},"datePublished":"2017-06-13T21:14:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/4003/PowerShell/issues/4003"}