Fail to RemoteForward a unix domain socket

Question

Fail to RemoteForward a unix domain socket

BusyJay opened this issue 4 years ago · comments

Jay commented 4 years ago

"OpenSSH for Windows" version

7.7.2.1

Server OperatingSystem

Linux

Client OperatingSystem

Windows 10 Pro

What is failing

I'm trying to forward my local gpg-agent to remote server. Same configuration works fine when both ends are Linux. When trying to forward from Windows to Linux, it doesn't work anymore.
There are three kind of errors.

When using configuration RemoteForward /run/user/1000/gnupg/S.gpg-agent ~/AppData/Roaming/gnupg/S.gpg-agent.extra, client reports forward success, but it won't work when actually testing it.

debug1: remote forward success for: listen /run/user/1000/gnupg/S.gpg-agent:-2, connect ~/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 3 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: /run/user/1000/gnupg/S.gpg-agent
debug2: fd 8 setting O_NONBLOCK
debug3: unable to connect to pipe ~/AppData/Roaming/gnupg/S.gpg-agent.extra, error: 3
debug1: connect_next: host ~/AppData/Roaming/gnupg/S.gpg-agent.extra ([unix]:~/AppData/Roaming/gnupg/S.gpg-agent.extra): No such file or directory
connect to ~/AppData/Roaming/gnupg/S.gpg-agent.extra port -2 failed: No such file or directory
debug1: failure forwarded-streamlocal@openssh.com

I suspect it's because CreateFileW can't expand charater ~ to home directory.

When configuring using RemoteForward /run/user/1000/gnupg/S.gpg-agent C:/Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra, client reports parse error: Bad forwarding specification.

I guess configuration parser just can't stand :.

When configuring using RemoteForward /run/user/1000/gnupg/S.gpg-agent /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra, and execute in disk C, client finally build up the channel successfully, but fails when handling connection.

debug1: remote forward success for: listen /run/user/1000/gnupg/S.gpg-agent:-2, connect /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 3 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: /run/user/1000/gnupg/S.gpg-agent
debug2: fd 8 setting O_NONBLOCK
debug1: connect_next: host /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra ([unix]:/Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra) in progress, fd=8
debug3: fd 8 is O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra port -2: failed.

That's because w32_getsockopt rejects work on a unix domain socket, it seems like a bug.

I guess there is no way to get around this now. Any suggestions?

Expected output
gpg-agent is forwarded successfully.

Actual output
It fails.

Editted:
For those who wants to forward gpg agent like me, I wrote a tool that built a bridge between GnuPG and openssh.

WSLUser · Answer 1 · Tue Mar 17 2020 01:47:51 GMT+0800 (China Standard Time)

Well at a first step, try upgrading to 8.1 and attempt to reproduce the issue. Submit updated logs if it still appears.

Jay · Answer 2 · Thu Apr 09 2020 17:15:24 GMT+0800 (China Standard Time)

Turns out even OpenSSH for Windows supports UDS, it's still not enough. Because GnuPG on windows does not use UDS at all. The file is a simulated UDS.

For those who wants to forward gpg agent like me, I wrote a tool that built a bridge between GnuPG and openssh. It has been working fine for a month now.

Peter Staev · Answer 3 · Sat Apr 11 2020 17:34:49 GMT+0800 (China Standard Time)

I just got the same problem. I managed to get the path working, but it fails with the same error as OP. This is using v8.1 on Windows 10 Pro (1909). Remote forward is set up like this in the config:

RemoteForward /Users/me/.gnupg/S.gpg-agent C\:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra

debug1: remote forward success for: listen /Users/me/.gnupg/S.gpg-agent:-2, connect C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra:-2
debug1: All remote forwarding requests processed

when trying to run gpg --no-autostart -K

debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
gpg: no gpg-agent running in this session
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
gpg: can't connect to the agent: End of file
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file

Danny Maertens · Answer 4 · Fri Aug 21 2020 06:57:23 GMT+0800 (China Standard Time)

@BusyJay , can you share your steps for this solution

Jay · Answer 5 · Fri Aug 21 2020 11:48:39 GMT+0800 (China Standard Time)

You mean how does the tool work around the problem? The simulated socket used by gnupg stores the port of its http service. So the tool just read the port and build a local http redirect service bridging between gnupg and ssh.

The steps about how to use the tool is written in README: https://github.com/BusyJay/gpg-bridge.

Idar Lund · Answer 6 · Fri Nov 06 2020 21:07:17 GMT+0800 (China Standard Time)

I'm so close in using Yubikey+gpg to admin linux servers over windows' built in openssh client! This agent forwarding is the only missing bit so that I can forward my gpg agent (and yubikey) to jump hosts to be able to manage other linux hosts.

Microsoft: Please fix this issue ❤️

c3c · Answer 7 · Sat Nov 28 2020 02:29:33 GMT+0800 (China Standard Time)

Running into the same issue. +1

Ilya Danilkin · Answer 8 · Thu Jan 28 2021 07:06:12 GMT+0800 (China Standard Time)

Few months later and I too got the same issue here, with the same conditions and the same setup.

OpenSSH version: OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

Regarding this:

I just got the same problem. I managed to get the path working, but it fails with the same error as OP. This is using v8.1 on Windows 10 Pro (1909). Remote forward is set up like this in the config:
RemoteForward /Users/me/.gnupg/S.gpg-agent C\:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra

you could just change the : to \: and leave the other slashes as is like this:

RemoteForward /run/user/1000/gnupg/S.gpg-agent C\:/Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra

Nagy-Egri Máté Ferenc · Answer 9 · Mon May 03 2021 16:10:32 GMT+0800 (China Standard Time)

I did the same hack to escape @nezorflame suggested and it seems that the agent is connected.

ssh -vvv my_remote_end
...
debug1: Remote connections from /run/user/1001/gnupg/S.gpg-agent:-2 forwarded to local address C:/Users/mate/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
...
debug1: remote forward success for: listen /run/user/1001/gnupg/S.gpg-agent:-2, connect C:/Users/mate/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
debug1: All remote forwarding requests processed
...

Also checking if the agent could be reconnected after SSH connection has been established is successful (as suggested here):

gpg-connect-agent /bye
echo $?
0

Also what @PeterStaev posted as being a problem also seems successful (at least to me):

gpg --no-autostart -K
/home/mate/.gnupg/pubring.kbx
-----------------------------
sec   rsa4096 2019-09-10 [SC]
      ...
uid           [ unknown] My Name <my@email.com>
ssb   rsa4096 2019-09-10 [E]

However, when I try to sign something, pinentry-qt doesn't pop up on my local box, but bails out with missing ioctl device.

echo "" | gpg --clearsign
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512


gpg: signing failed: Inappropriate ioctl for device
gpg: [stdin]: clear-sign failed: Inappropriate ioctl for device

I know that one solution is to att export GPG_TTY=$(tty) however that defeats the entire purpose of agent forwarding. I don't want to deploy my private keys on the remote end, neither do I want to be prompted on the command-line for passwords, because I want to use the Git GUI of VS Code from a remote session and I want the password prompt to pop up on my local machine running Windows. gpg.exe locally pops open pinentry-qt as expected when prompting for a password. How can I tell the remote gpg program to tunnel through the agent to hit my local pinentry program?

Lucas Savva · Answer 10 · Sun Nov 14 2021 23:24:37 GMT+0800 (China Standard Time)

Hey, would just like to bump this as Google is leading me here in my searches.

I have git-bash installed too (Not in WSL, just on Windows itself) which ships with OpenSSH 8.4. Using either a C\:/Users/... or /c/Users/... style path in my ssh config, I am able to get a 100% working agent forward set up to a remote Linux host with no real hassle. Using that binary is working in all the local shells I've tried too, those being Powershell, cmd.exe, and Git Bash.

For my use case, all I really wanted was working forwarding within VS Code Remote, so I changed remote.SSH.path to C:\Program Files\Git\usr\bin\ssh.exe and now I can sign things :) The down side is I must give up using the OpenSSH Auth Agent service which remembers my key passphrase, but that's a sacrifice I'm willing to make.

I'm still confused as to why it would work on the the basic OpenSSH implementation and not OpenSSH For Windows? Are they not the same, at least in regards to how forwarding is handled?

My versions:

PS C:\> ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
PS C:\Program Files\Git\usr\bin> .\ssh -V
OpenSSH_8.4p1, OpenSSL 1.1.1h  22 Sep 2020

Hoàng · Answer 11 · Sat May 28 2022 23:07:05 GMT+0800 (China Standard Time)

Is there any update on this?

I did some mix and match with Git for Windows's SSH and GPG¹, with Win32-OpenSSH and Gpg4win¹. Only Git for Windows bundle works, any other combination fails.

In the process I also noticed that StreamLocalBindUnlink does not work if socket forwarding fails for some reason.

I mean GPG extra socket ↩ ↩²

Giovanni Bassi · Answer 12 · Thu Oct 27 2022 08:17:39 GMT+0800 (China Standard Time)

I got this to work in WSL2, and I believe I have a clue of what is happening and why this does not work on Windows.
The $env:LOCALAPPDATA/gnupg/S.gpg-agent and $env:LOCALAPPDATA/gnupg/S.gpg-agent.extra files on Windows are not really Unix sockets. And ssh is able to forward only Unix sockets, AFAIK. So this configuration will not work with Windows SSH until Windows OpenSSH implementation supports Unix Sockets. Windows already supports them.

This might explain why some people have been able to make it work with git ssh, it is probably emulating Unix sockets somehow, or even using the new and now available AF Unix Socket support.

So, how do you get it to work with WSL2? I'm using wsl-relay: https://github.com/Lexicality/wsl-relay

All I need to do is (in WSL):

socat UNIX-LISTEN:$HOME/.gnupg/S.gpg-agent.extra,fork, EXEC:'wsl-relay.exe --input-closes --pipe-closes --gpg=S.gpg-agent.extra',nofork

And in the ssh config file (in WSL):

RemoteForward /run/user/1000/gnupg/S.gpg-agent /home/<user>/.gnupg/S.gpg-agent.extra

And it works.
See how I wired it automatically so I don't have to run socat manually here:
https://github.com/giggio/bashscripts/blob/8ba5ac02849904e787cb51b4c607e9858c12baf7/bashrc-wsl.bash#L109

Idar Lund · Answer 13 · Mon Oct 31 2022 18:16:41 GMT+0800 (China Standard Time)

WSL/WSL2 is not the issue. We want it to be supported in the native Windows OpenSSH client 🙂
Microsoft: Could you please prioritize this issue? ❤️

Giovanni Bassi · Answer 14 · Tue Nov 01 2022 02:21:59 GMT+0800 (China Standard Time)

WSL/WSL2 is not the issue. We want it to be supported in the native Windows OpenSSH client 🙂 Microsoft: Could you please prioritize this issue? ❤️

I'm not saying WSL2 is the issue, I'm just explaining the mechanism of why it does not work. It might help someone find a workaround until Microsoft fixes it.

Nagy-Egri Máté Ferenc · Answer 15 · Thu Nov 03 2022 20:38:13 GMT+0800 (China Standard Time)

In the context of GPG agent forwarding, I suggest checking out win-gpg-agent which bridges the lack of this feature, and also do note that upstream GnuPG has picked up Windows named pipe support and it's going into GnuPG 2.3.9 and gpg4win 4.0.5. So if you're looking for this feature to get GPG working, it might be worth to just wait it out.

Nagy-Egri Máté Ferenc · Answer 16 · Sun Jan 22 2023 01:41:32 GMT+0800 (China Standard Time)

@maertendMSFT The harder part of the problem has been solved now. Could some resources be put to handling the forward of unix sockets to named pipes? IIUC Windows devs still need an external tool that ties the knot on connecting the unix socket (already known to Windows and OpenSSH) to the named pipe gpg-agent is listening on. Either OpenSSH internalizes this, or external tools need to bridge the gap.

Giovanni Bassi · Answer 17 · Tue Jan 24 2023 05:44:59 GMT+0800 (China Standard Time)

In the context of GPG agent forwarding, I suggest checking out win-gpg-agent which bridges the lack of this feature, and also do note that upstream GnuPG has picked up Windows named pipe support and it's going into GnuPG 2.3.9 and gpg4win 4.0.5. So if you're looking for this feature to get GPG working, it might be worth to just wait it out.

Thanks, @MathiasMagnus, that worked, now I don't need wsl-ssh-pageant to bridge ssh from WSL to Windows, npiperelay is enough.

Fail to RemoteForward a unix domain socket

Footnotes