Proxy support issues since 8.0
BenCastricum opened this issue · comments
With 8.0 we are having connectivity issues making rpki-client not functional anymore. We are behind a company proxy with a dockerized version of rpki-client-portable 8.0. I tried to narrow it down a bit:
Some RPKI logging:
user@hostname:~/etc/rpki$ sudo docker exec -ti <containerid> sh
/ # rpki-client -v -t /etc/tals/afrinic.tal
rpki-client: ta/afrinic: pulling from https://rpki.afrinic.net/repository/AfriNIC.cer
rpki-client: ta/afrinic: loaded from network
rpki-client: https://rrdp.afrinic.net/notification.xml: pulling from network
rpki-client: https://rrdp.afrinic.net/notification.xml: connect: Connection refused
rpki-client: https://rrdp.afrinic.net/notification.xml: load from network failed, fallback to rsync
rpki-client: .rsync/rpki.afrinic.net/repository: pulling from rsync://rpki.afrinic.net/repository
ERROR: rejecting unrequested file-list name: 04E8B0D80F4D11E0B657D8931367AE7D
rsync error: protocol incompatibility (code 2) at flist.c(998) [Receiver=3.2.4]
rpki-client: rsync rsync://rpki.afrinic.net/repository failed
rpki-client: .rsync/rpki.afrinic.net/repository: load from network failed, fallback to cache
rpki-client: rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/62gPOPXWxxu0sQa4vQZYUBLaMbY.mft: no valid mft available
rpki-client: all files parsed: generating output
Processing time 2 seconds (0 seconds user, 0 seconds system)
Skiplist entries: 0
Route Origin Authorizations: 0 (0 failed parse, 0 invalid)
AS Provider Attestations: 0 (0 failed parse, 0 invalid)
BGPsec Router Certificates: 0
Certificates: 1 (0 invalid)
Trust Anchor Locators: 1 (0 invalid)
Manifests: 1 (1 failed parse, 0 stale)
Certificate revocation lists: 0
Ghostbuster records: 0
Repositories: 2
Cleanup: removed 0 files, 3 directories, 0 superfluous
VRP Entries: 0 (0 unique)
VAP Entries: 0 (0 unique)
/ # rpki-client -V
rpki-client-portable 8.0
With an intermediate proxy, we were able to get logging, this is from the same session as above:
CONNECT Oct 18 08:25:13.324 [1]: Connect (file descriptor 4): 127.0.0.1
CONNECT Oct 18 08:25:13.325 [1]: Request (file descriptor 4): CONNECT rpki.afrinic.net:443 HTTP/1.1
INFO Oct 18 08:25:13.325 [1]: Found upstream proxy http 172.16.154.2:8080 for rpki.afrinic.net
INFO Oct 18 08:25:13.325 [1]: opensock: opening connection to 172.16.154.2:8080
INFO Oct 18 08:25:13.325 [1]: opensock: getaddrinfo returned for 172.16.154.2:8080
CONNECT Oct 18 08:25:13.327 [1]: Established connection to upstream proxy "172.16.154.2" using file descriptor 5.
CONNECT Oct 18 08:25:14.151 [1]: Connect (file descriptor 6): 127.0.0.1
CONNECT Oct 18 08:25:14.151 [1]: Request (file descriptor 6): CONNECT rpki.afrinic.net:873 HTTP/1.0
INFO Oct 18 08:25:14.151 [1]: Found upstream proxy http 172.16.154.2:8080 for rpki.afrinic.net
INFO Oct 18 08:25:14.151 [1]: opensock: opening connection to 172.16.154.2:8080
INFO Oct 18 08:25:14.151 [1]: opensock: getaddrinfo returned for 172.16.154.2:8080
CONNECT Oct 18 08:25:14.154 [1]: Established connection to upstream proxy "172.16.154.2" using file descriptor 7.
INFO Oct 18 08:25:15.311 [1]: Closed connection between local client (fd:6) and remote client (fd:7)
INFO Oct 18 08:25:15.315 [1]: Closed connection between local client (fd:4) and remote client (fd:5)
As far as I can tell, the rrdp sessions or notification requests do not use the proxy anymore while TA seems to download fine. Rsync seems to fail as well but that seems to be a know bug of rsync. The combination of these bugs makes rpki-client-portable 8.0 unusable for us.
Looking at the rpki-client output:
rpki-client: https://rrdp.afrinic.net/notification.xml: connect: Connection refused
This error indicates that the connection to the proxy failed. What is the contents of your http_proxy env var?
The log from the proxy has no CONNECT rrdp.afrinic.net:443 HTTP/1.1 line. Which also indicates that the connection to the proxy failed.
Can you run rpki-client with something like strace and find the failing connect call?
I wonder why the 2nd connect failed.
Not much we can do about the issue in rsync, guess you need wait for a fixed rsync.
I have compiled it myself now so I can run it outside the docker container. First I noticed that rpki-client uses the http_proxy environment setting, instead of the https_proxy setting for https connections.
but setting them both to
http_proxy=http://127.0.0.1:8888/
https_proxy=http://127.0.0.1:8888/
and then strace it shows that the incorrect port is being used for connecting to the proxy:
...
31277 socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_TCP) = 7
31277 connect(7, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
31277 poll([{fd=5, events=POLLIN}, {fd=7, events=POLLOUT}], 2, 15000) = 1 ([{fd=7, revents=POLLOUT|POLLERR|POLLHUP}])
31277 getsockopt(7, SOL_SOCKET, SO_ERROR, [ECONNREFUSED], [4]) = 0
...
I hope this helps.
Can you give this patch a try? This should fix the problem with the proxyport settings.
Yes, this indeed fixes the proxy issue! Thanks! Looking forward to 8.1 :)
@cjeker, how far are we from 8.1? I'm also happy to simply add this fix to the container builds later (in case @BenCastricum refers to rpki/rpki-client:{latest,edge} on Docker Hub and Red Hat Quay).
I have no good answer right now. The fix is committed and should show up in rpki-client-openbsd soon.
So the latest -current builds should have the fix tomorrow. For a real release it may take longer.
No need to rush things for me, We won't be using the rpki-client:edge container, but prefer a official tagged release. The edge container also still contains the bugged rsync version,
I'm happy to push the fix also to rpki/rpki-client:8.0 which is currently rpki/rpki-client:latest, if that helps?
Then it would still not work 100% due to the rsync issues. We prefer an 8.1 container release with both issues fixed. That makes it clear to coworkers/staff that 8.0 is skipped due to bugs which are (hopefully) fixed in 8.1
I'm working at https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/40669 to get a fixed rsync 3.2.7 into Alpine Linux 3.16.
Edit: It got merged and https://gitlab.alpinelinux.org/alpine/aports/-/commit/a2b5319e9212ae4b1fe3e87298114bc483160895 should hopefully lead to updated rsync package soon.
As of writing, rpki/rpki-client:latest, currently evaluating to rpki/rpki-client:8.0, as well as rpki/rpki-client:edge both ship the rpki-client bugfix as well as rsync 3.2.7. I'm personally using podman locally, but commands should be substitutable with docker:
$ podman pull rpki/rpki-client:latest
$ podman inspect rpki/rpki-client:latest | jq -r '.[0].Config.Labels."org.opencontainers.image.revision"'
a1eed1d6b27e874506e310848ba8384f947f0c57