The PowerShell Problem
huettenhain opened this issue · comments
In fcc36d8, I added some extremely weak PowerShell support, this was improved slightly in 504f015. The main issues are the following:
Our current workaround is to:
- wrap all output as hex with a magic prefix when running under PowerShell
- unwrap all input when we are running under PowerShell and the magic prefix is detected
- simply accept the fact that PowerShell has no streaming stdin/stdout
If either of the above two PowerShell issues are ever resolved in a way that supports the binary refinery design, the workaround should be removed because it introduces an unnecessary encoding step.
You could add support for Use-RawPipeline in the meantime.
Hey @ImportTaste, the way I understand Use-RawPipeline
, it should already work. I have, however, not tried it out myself. If you already tested it, are there any issues to report?
Doing some extremely simple tests with Use-RawPipeline
, it does seem to work as expected in providing a pipeline for raw binary data.
Use-RawPipeline
provides the following main cmdlets:
Invoke-NativeCommand
, aliased torun
, for running a native command and treating its input and output as raw binary streams.Receive-RawPipeline
, aliased to2ps
, for retrieving the raw binary streams from a native command pipeline, and putting it back into a form that other Powershell cmdlets can consume (optionally specifying an encoding).
There are also cmdlets to read raw binary streams from file, and dump them again back to file.
PS> run emit M7EwMzVzBkI3IwNTczM3cyMg2wQA | run b64 | run zl | run hex | 2ps
Hello World
PS> run emit "Once upon a time, at the foot of a great mountain ..." | run aes "pbkdf2[32,s4lty]:swordfish" --iv md5:X -R | run ccp md5:X | run aes "pbkdf2[32,s4lty]:swordfish" --iv cut:0:16 | 2ps
Once upon a time, at the foot of a great mountain ...
PS> run emit .\CyberChef_v9.32.3.zip | run xtzip "["| run dump "cyberchef_extracted/{path}" "]" | 2ps
PS> Get-ChildItem .\cyberchef_extracted\ -Name
assets
images
modules
ChefWorker.js.LICENSE.txt
CyberChef_v9.32.3.html
DishWorker.js.LICENSE.txt
InputWorker.js.LICENSE.txt
LoaderWorker.js.LICENSE.txt
ZipWorker.js.LICENSE.txt
Of course, certain characters which Powershell uses as part of its shell language, such as []
, {}
, and ,
, must be quoted.
BTW, I think that my tests above bypass your PowerShell check in is_powershell_process
, as I am running PowerShell 7.1.3; all PowerShell versions after 6 have the process name pwsh.exe
, rather than powershell.exe
which is_powershell_process
checks for. Therefore, I think the tests above are not using the JSON serialization support, and it is Use-RawPipeline
actually doing the work of making the streams work properly. Trying to use refinery without Use-RawPipeline
definitely doesn't work at least, for me:
> emit M7EwMzVzBkI3IwNTczM3cyMg2wQA | b64 | zl | hex
(00:18:30) failure in zl: exception of type ValueError; could not detect any zlib stream.
Alright. Still not sure how to handle this in the most graceful manner. I can fix is_powershell_process
to include a check for pwsh.exe
, but what I would like best would be to disable the horrible JSON encoding alltogether, it only works inside a frame anyway. Then, log a warning when running inside PowerShell, unless the user is using Use-RawPipeline. However, I have no way of knowing whether that is the case as far as I can see.
As a side note, when I work with all-text input/output, I don't always have to quote [
and ]
and I don't think I have fully grokked when that is the case:
PS C:\> emit Foo Bar [| ccp x | sep ]
xFoo
xBar
PS C:\> emit Foo Bar [| ccp x ]]
usage: ccp [-h] [-L] [-Q] [-0] [-v] data
ccp: error: unrecognized arguments: ]
Yes, I don't think there's any way for binref units to inspect their environment in such a way that they can tell when they are being run under Use-RawPipeline
's Inspect-NativeCommand
.
Another way to handle this would be to display a very prominent warning as a post-install step on Windows systems spelling out the issues with piping binary data in Powershell, and either listing the workarounds (use cmd.exe
, use Use-RawPipeline
, or accept the current JSON-serialization-based Powershell support), or just linking users to this issue. Is it possible to do this from setup.py
?
Also, Powershell versions after 6 are now available on Mac and Linux, and officially supported. I think the number of people who actually use it as their day-to-day interactive shell on those platforms is extremely small, though, and the people who are doing that likely know the limitations.
Yes, I don't think there's any way for binref units to inspect their environment in such a way that they can tell when they are being run under
Use-RawPipeline
'sInspect-NativeCommand
.Another way to handle this would be to display a very prominent warning as a post-install step on Windows systems spelling out the issues with piping binary data in Powershell, and either listing the workarounds (use
cmd.exe
, useUse-RawPipeline
, or accept the current JSON-serialization-based Powershell support), or just linking users to this issue. Is it possible to do this fromsetup.py
?
It's easy enough to tell if the module is loaded through Get-Module. If it is loaded, it just needs to run different commands.
I don't think there is any way to run Get-Module
from inside Python, and I don't think there is any way for us to find out whether Use-RawPipeline
is being used or not. I don't think there is a lot of benefit in supporting it; PowerShell either supports binary pipelines, or it does not - currently, it does not.
Right now I am leaning heavily towards treating a PowerShell parent process as a critical error, i.e. abort execution with a short explanation of why that is currently not supported. There are just too many ways this can go wrong unless our band-aid is completely flawless, and I am not confident I can make it so.
Alright, I had another idea about a better band-aid yesterday, and implemented it in 03436e2. This works a lot better now, and there is a warning displayed by default which can be disabled through an environment variable:
Oh yes, I like that! Not sure why I had not thought of that before.. 😅
This was implemented in 504f015. I have changed the names of the variables a bit and added information to the warning message:
(venv) PS> run emit c0019718c4d4538452affb97c70d16b7af3e4816d059010c277c4e579075c944 `
>> |run perc SETTINGS [ `
>> | run put keylen cut::1 `
>> | run rc4 cut::keylen `
>> | run xtp socket ]`
>> |2ps
WARNING: PowerShell has no support for binary pipelines or streaming. Binary Refinery uses an unreliable and slow
workaround: It is strongly recommended to use the command processor instead. Proceed at your own peril!
- To silence this warning: $env:REFINERY_SILENCE_PS1_WARNING=1
- To disable the band-aid: $env:REFINERY_DISABLE_PS1_BANDAID=1
- To get more information: https://github.com/binref/refinery/issues/5
[BRPS1]:72656D6D2E6475636B646E732E6F72673A37303037
(venv) PS> $env:REFINERY_DISABLE_PS1_BANDAID=1
(venv) PS> run emit c0019718c4d4538452affb97c70d16b7af3e4816d059010c277c4e579075c944 `
>> |run perc SETTINGS [ `
>> | run put keylen cut::1 `
>> | run rc4 cut::keylen `
>> | run xtp socket ]`
>> |2ps
remm.duckdns.org:7007
(venv) PS>
I will push out a new release now.
There is a PR open in PowerShell that would (hopefully) fix this: PowerShell/PowerShell#17857
Alright! The PR that fixes this in PowerShell has been merged, the next PowerShell preview should support this. I am very excited to try that out.
Noting from PowerShell/PowerShell#1908 (comment) that the following command will be required to enable this experimental PowerShell feature:
Enable-ExperimentalFeature PSNativeCommandPreserveBytePipe
In PowerShell 7.4.0-preview.4, refinery now works as expected after running:
$env:REFINERY_DISABLE_PS1_BANDAID=1
Enable-ExperimentalFeature PSNativeCommandPreserveBytePipe
Since it will probably be quite a while before this is in an mainstream PowerShell release, I'm afraid we have to keep this issue open - but at least there is now a way to make it work inside some version of PowerShell.