Cannot connect to GRR through HAProxy

[Trevor Reed]

When I try to connect a client to my new GRR install behind HAProxy I get the following in the client logs. Any ideas?

INFO:2020-11-20 09:57:34,139 comms:252] Could not connect to GRR servers ['https://server.domain/'], directly or through these proxies: [''].

backend grr-be
        redirect scheme https if !{ ssl_fc }
        server GRR01 172.24.226.101:8080
        http-response add-header X-FRAME-OPTIONS SAMEORIGIN
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        balance roundrobin
        description GRR

[Trevor Reed]

I can connect clients directly via the internal IP to GRR and can download the server.pem file via the Loadbalanced domain name. i.e

$ curl https://server.domain/server.pem
-----BEGIN CERTIFICATE-----

[mbushkov]

Hi Trevor! Thanks for reporting this. A few questions:

• What is the client/server version?
• How was the server installed? Double checking - you haven't enabled Fleetspeak in this install, correct?
• Which platform do the clients run on?
• Do you see anything if you run the clients in verbose mode? See "Run the client in verbose mode" here.

[Trevor Reed]

• Server Version: 3.4.2.4 installed via DEB package on Ubuntu.
• Fleetspeak is not enabled in the config:

Server.fleetspeak_enabled: false
Client.fleetspeak_enabled: false
ClientBuilder.fleetspeak_bundled: false
Server.fleetspeak_server: ''

• Connection issues have been noticed on Windows and MacOS

Verbose output:

$ sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --config=/usr/local/lib/grr/grr_3.4.2.4_amd64/grr.yaml --verbose
I1124 11:15:30.616528 4545949184 client_logging.py:117] Writing log file to /var/log/GRRlog.txt
INFO:2020-11-24 11:15:30,616 client_logging:117] Writing log file to /var/log/GRRlog.txt
INFO:2020-11-24 11:15:30,627 comms:1330] Starting client aff4:/C.ed6e3d3da57ed745
DEBUG:2020-11-24 11:15:30,633 admin:312] Sending startup information.
DEBUG:2020-11-24 11:15:30,734 connectionpool:943] Starting new HTTPS connection (1): server.domain:443
INFO:2020-11-24 11:15:30,736 comms:252] Could not connect to GRR servers ['https://server.domain/'], directly or through these proxies: [''].

and as stated in my initial report, I can download the server.pem with curl from the client

[mbushkov]

Hm, it might be that curl is doing something differently from how GRR client does it.

Please do the following on your client machine:

sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --break_on_start

This will give you a Python debugger prompt. In this prompt, please run the following lines of code:

import requests
requests.get("https://server.domain/server.pem")

Please post whichever output you get.

[Trevor Reed]

$ sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr break_on_start
Traceback (most recent call last):
  File "grr_response_client/client.py", line 107, in <module>
  File "absl/app.py", line 299, in run
  File "absl/app.py", line 250, in _run_main
  File "grr_response_client/client.py", line 73, in main
  File "grr_response_client/client_startup.py", line 25, in ClientInit
  File "grr_response_core/lib/config_lib.py", line 1736, in ParseConfigCommandLine
RuntimeError: A config file is not specified.
[10502] Failed to execute script client

$ sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --config=/usr/local/lib/grr/grr_3.4.2.4_amd64/grr.yaml break_on_start
I1125 14:58:53.150907 4473904640 client_logging.py:117] Writing log file to /var/log/GRRlog.txt

I never get a Python debugger prompt

[mbushkov]

--break_on_start has to be specified as a flag - notice the double dashes.

I.e. instead of

sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr break_on_start

please try

sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --break_on_start

[Trevor Reed]

I apologize, I misread that command on mobile. Here is the output.

$ sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --break_on_start
Password:

/Users/treed/grr_response_client/client.py(70)main()
(Pdb) import requests
(Pdb) requests.get("https://server.domain/server.pem")
*** requests.exceptions.SSLError: HTTPSConnectionPool(host='server.domain', port=443): Max retries exceeded with url: /server.pem (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

The Certificate is a Lets Encrypt certificate

[Trevor Reed]

After some quick GoogleFu, it appears that I need to use the Fullchain.pem on the server rather than the normal certificate due to trust issues.

[Trevor Reed]

I now get

<Response [200]>

But starting grr from the commandline

$ sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --config=/usr/local/lib/grr/grr_3.4.2.4_amd64/grr.yaml --verbose

still returns

$ sudo /usr/local/lib/grr/grr_3.4.2.4_amd64/grr --config=/usr/local/lib/grr/grr_3.4.2.4_amd64/grr.yaml --verbose
I1129 22:41:54.449746 4614868480 client_logging.py:117] Writing log file to /var/log/GRRlog.txt
INFO:2020-11-29 22:41:54,449 client_logging:117] Writing log file to /var/log/GRRlog.txt
INFO:2020-11-29 22:41:54,461 comms:1330] Starting client aff4:/C.ed6e3d3da57ed745
DEBUG:2020-11-29 22:41:54,467 admin:312] Sending startup information.
DEBUG:2020-11-29 22:41:54,505 connectionpool:943] Starting new HTTPS connection (1): server.domain:443
INFO:2020-11-29 22:41:54,507 comms:252] Could not connect to GRR servers ['https://server.domain/'], directly or through these proxies: [''].