python-hpilo can't connect anymore to an iLO2 server on python3.10
accessiblepixel opened this issue · comments
Hello.
I use Home Assistant with the python-hpilo functionality to monitor some old servers in a dashboard.
From what I have read, the supported versions of TLS have had the defaults changed to not allow anything "worse" than TLS1.2, so the old versions of SSLv3 that iLO2 uses doesn't work anymore in python 3.10
I tried changing some options in hpilo.py to enable SSLv3 but haven't been successful thus far and I am unsure how to proceed.
Home Assistant has recently bumped its python version to 3.10 and downgrading to an older version isn't really an option, because even if I did downgrade, it would only be a stop gap solution.
I'm not sure how to proceed and I've tried as much as I can to figure out the problem by myself.
A potential workaround I could see is running a proxy webserver to sit between the two to "upgrade" it's TLS version, but I'm not sure.
I would appreciate any help that you can provide.
Kind regards,
Jessica
Here are the attached parts of the log that might be useful
`
Logger: homeassistant.components.sensor
Source: components/hp_ilo/sensor.py:166
Integration: Sensor (documentation, issues)
First occurred: 06:22:44 (4 occurrences)
Last logged: 06:22:45
hp_ilo: Error on device update!
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 401, in _get_socket
return ssl.wrap_socket(sock, ssl_version=ssl.PROTOCOL_TLS)
File "/usr/local/lib/python3.10/ssl.py", line 1442, in wrap_socket
return context.wrap_socket(
File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 446, in _async_add_entity
await entity.async_device_update(warning=False)
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 702, in async_device_update
await task
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/hp_ilo/sensor.py", line 166, in update
ilo_data = getattr(self.hp_ilo_data.data, self._ilo_function)()
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 1035, in get_embedded_health
return self._info_tag('SERVER_INFO', 'GET_EMBEDDED_HEALTH', 'GET_EMBEDDED_HEALTH_DATA',
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 730, in _info_tag
header, message = self._request(root)
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 238, in _request
self._detect_protocol()
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 278, in _detect_protocol
header, data = self._communicate(b'', ILO_HTTP, save=False)
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 406, in _communicate
sock = self._get_socket()
File "/usr/local/lib/python3.10/site-packages/hpilo.py", line 403, in _get_socket
raise IloCommunicationError("Cannot establish ssl session with %s:%d: %s" % (self.hostname, self.port, str(exc)))
hpilo.IloCommunicationError: Cannot establish ssl session with ilo2.local.dns:443: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:997)
`
Nothing we can do on the python-hpilo side. You'll need to use a python version that supports older ssl versions or ilos that support newer ssl versions.
There's a flag that can be set in python to use a different encryption method, which I tried setting in the python-hpilo code, but because of the way the code is written it doesn't seem to pull that global configuration for python to override supported SSL versions.
I don't know much python to figure out every request and make it respect the allowed ciphers and python 3.10 settings, and I imagine that it's only going to get worse the more time that passes, unless a way to define "these ciphers are acceptable" is made available in the code.
There is a global method to add it, which I tried but it seems that python-hpilo doesn't respect that global, and defines it's own ssl parameters for each request and I'm not adept at python enough to change it myself.
Changing to a newer version of iLO is not possible on the legacy iLO hardware (especially hardware that runs iLO2) as there are no updates to it. iLO4 is still in production and receives updates from HP, but not often now that they've moved onto iLO5.
python-hpilo configures the ssl library to use any supported ssl version, including deprecated ones. However, more recent python/openssl versions have stopped shipping support for these versions completely, so no matter what you do in code: those versions cannot be used. The only option is to use older python and openssl versions, or newer ilo's. ilo/ilo2 is basically unsupportable with modern python, as they indeed don't have firmware available that supports newer ssl versions. That said, hardware with such old ilo's is well past its use-by date and should be replaced. HP hasn't supported it for years and the power and performance benefits of upgrading are huge and will offset the extra hardware cost.
That might be true, the hardware is a bit older, but it still is perfectly functional... Buying new hardware isn't an option for me, especially for those servers that I have that are running iLO2, because new servers are expensive and I can't afford new ones.
I'm an individual and I'm disabled, the only reason I have servers with iLO is in case something crashes, I can access the console without getting up and going to the physical console. I certainly wouldn't use it over the internet, and I monitor hardware temperatures via home assistant, or at least I did until it wasn't supported, when they bumped their python version.
I guess I'll try putting nginx between it and home assistant to handle the ssl as a workaround, since I should be able to configure nginx to proxy sslv3 and then provide the pages as TLS1.2.
My ilo4 server works fine, but that supports TLS1.2, but yeah... guess I'll figure something out, since using a different python version isn't possible as it's integrated into home assistant... so I guess i'll try nginx as a proxy.
If that works, that'd be a good addition to the python-hpilo docs. The ssl situation is painful for many people.
I tried all I could to get some kind of reverse proxy working for grabbing sensor data from the server.
I had some success with nginx and ha proxy for accessing the webpage configurations in firefox just fine (I imagine console wouldn't work, but I could see, set and change all the other things within the ILO2 interface)
Ultimately the way I figured to collect the sensor data again was to enable network IPMI within ILO2, and then create a bash script that pulls the data from each sensor, and publishes it to a MQTT server on home assistant. It's much faster than python-hpilo was for collecting all the details, but the disadvantage is you have to run the script somewhere that can access both ILO and the MQTT server. For now I'll still use the python-hpilo integration in home assistant for my iLO4 server (since it's still fairly well supported) but for iLO2 (and possibly 3) this is how I can monitor temperatures, fan speeds and current consumption.
My shell script is loosely based on the information I found on the Home Assistant Community Forums here https://community.home-assistant.io/t/ipmi-sensors/279248
Hopefully this will prove useful to someone who wants a fairly easy way to monitor sensors from an older iLO system. I really don't understand why they didn't have a way to disable SSl entirely, maybe not by default but... when protocols upgrade in the future, you can always add in support for them with a proxy, stripping them is a bit more difficult, but it's possible!
If you'd like to proxy iLO2 https connection and upgrade it to TLS1.2 or TLS1.3 with nginx, you can use this fairly basic configuration.
server {
listen 80;
server_name <dns-name-of-iloproxy>;
location / {
return 301 https://$server_name$request_uri;
}
}
server {
listen 443 ssl;
server_name <dns-name-of-ilo-proxy>;
ssl_certificate /path/to/fullchain.pem;
ssl_certificate_key /path/to/privkey.pem;
ssl_protocols TLSv1.2;
ssl_ciphers HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP;
#ssl_prefer_server_ciphers on;
proxy_ssl_ciphers HIGH:!aNULL:!MD5;
#proxy_ssl_verify on;
#proxy_ssl_verify_depth 2;
proxy_ssl_session_reuse on;
location / {
proxy_pass https://<ILO IP>;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
You also need to change some settings in openSSL's config to allow older versions of the protocols to be used. I changed the bottom entries in the /usr/lib/ssl/openssl.cnf file to:
[system_default_sect]
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=1
Of course you need to run this somewhere that can access ILO itself but the rest of the network can't (so firewall ILO or something, and only give iLO access to the iloproxy nginx server). I'm using Debian 11 as my base operating system, but I shouldn't imagine that the configuration would be any different for another, so long as openssl can be configured to allow those protocols.
I did also get a configuration working with HA proxy, that could verify the TLS1.0 SSL certificate with that openSSL configuration too. I don't have it handy right now but if that would be useful for someone, I can provide that.
I don't believe you'd be able to use the console, although I didn't try, I think it's hosted on a different port, and the XML status based logins I couldn't get working either, hence why I swapped to using the IPMI and MQTT solution above.
Hope this at least helped someone :)
Kind regards,
Jessica
P.S Not sure if github mangled any of my config/code files there...
Edit - I managed to pull up my HAProxy configuration, and I grabbed a screenshot of ilo2 access working in Firefox (I don't have TLS1.0 or TLS1.1 enabled in my config, just to show it's secure. I run my own trusted root ca here that has been added to the system's trust root store. so I don't get certificate warnings.
Here's a link to the screenshot: https://i.lowrex.com/i/BNPyd.png
And here's the full config file for HAProxy
Version Info - HA-Proxy version 2.2.9-2+deb11u3 2022/03/10 - (Status: long-term supported branch - will stop receiving fixes around Q2 2025.), which is just the standard version included in Debian 11.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
tune.bufsize 32768
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2
defaults
log global
mode http
option httplog
option dontlognull
option splice-auto
timeout connect 9999
timeout client 10000
timeout server 20000
frontend port_443
bind *:443 ssl crt /path/to/fullchain.pem
#You need to have your privkey either in this file, or you need it in a file called fullchain.pem.key in the same directory.
http-request add-header X-Forwarded-Proto https
mode http
default_backend port_443
backend port_443
mode http
server port_443 <dns-or-ip-of-ilo>:443 ssl verify required ca-file /path/to/root/that/signed/ilo/certificate/rootca.crt ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS:!CAMELLIA@SECLEVEL=1
I'm not sure, but I think you also need to modify the openssl.cnf as with the nginx version to allow older versions, and the same firewall/security advice applies. I'm not exactly sure how much of the configuration is required, and it's a little slow to connect sometimes, but it works pretty well. I'll be sticking with the HAProxy version, just because it also can verify the certificate on my ILO server, but if you don't need that, the nginx one is probably easier. Good luck! I don't think python-hpilo likes it, because I think it logs in with XML and now i've got the MQTT way of getting sensor information... I've put about all the effort into it I have time for :)
Final edit, to include screenshot of TLS information for the page in Firefox. https://i.lowrex.com/i/BNIlH.png and a copy of the shell script I put together for logging the sensors from iLO to MQTT (based on that linked home assistant article earlier)
#!/bin/bash
SHELL=/bin/sh PATH=/bin:/sbin:/usr/bin:/usr/sbin
MQTT_IP='127.0.0.1'
MQTT_USER='mqtt'
MQTT_PW="hackme"
IPMI_IP='127.0.0.2'
IPMI_USER='server'
IPMI_PW='hunter2'
NUMBER_OF_SENSORS='9'
CUT_COLUMN='5'
TOPIC[1]='server_fan_1_speed'
TOPIC[2]='server_fan_2_speed'
TOPIC[3]='server_fan_3_speed'
TOPIC[4]='server_ambient_inlet'
TOPIC[5]='server_power_supply_1'
TOPIC[6]='server_power_supply_2'
TOPIC[7]='server_total_power'
TOPIC[8]='server_power_supply_zone_temperature_1'
TOPIC[9]='server_power_supply_zone_temperature_2'
SDR[1]='7.1'
SDR[2]='7.3'
SDR[3]='7.4'
SDR[4]='39.1'
SDR[5]='10.1'
SDR[6]='10.2'
SDR[7]='7.7'
SDR[8]='10.4'
SDR[9]='10.5'
for i in $(eval echo "{1..$NUMBER_OF_SENSORS}")
do
PAYLOAD=$(ipmitool -I lanplus -H "$IPMI_IP" -U "$IPMI_USER" -P "$IPMI_PW" sdr entity "${SDR[$i]}"|cut -d '|' -f$CUT_COLUMN|awk '{print $1}')
mosquitto_pub -r -t "${TOPIC[$i]}" -m "$PAYLOAD" -h "$MQTT_IP" -u "$MQTT_USER" -P "$MQTT_PW"
done
Obviously change the IPs, passwords and names of sensors, but this is what I use on a DL360 G6, which I just run in Cron every minute with
*/1 * * * * cd /root/ && ./mqtt.sh>>ipmi-crontab.log
That's it, that's everything that I've done to make accessing my ILO2 instance a bit easier and do my logging. Might not be that reliable, but should be enough stuffs to get you started :)
@accessiblepixel Thanks for your nginx example. It pointed me in the right direction to reach our ILO3 servers.
However if you lower the global security level in /usr/lib/ssl/openssl.cnf it affects (and can break) other services on that host. Your changes also risks being overwritten in case of updates.
It's better to override ssl options for nginx only using environment in the service.
This worked for me on Ubuntu 22.04:
sudo vi /etc/nginx/old_cipher_openssl.cnf
openssl_conf = default_conf
[ default_conf ]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT:@SECLEVEL=0
sudo vi /usr/lib/systemd/system/nginx.service
[Service]
Environment="OPENSSL_CONF=/etc/nginx/old_cipher_openssl.cnf"
...
Edit:
It's also required to Enable "Enforce AES/3DES Encryption" under Administration - Security - Encryption.
It's better to override ssl options for nginx only using environment in the service.
@sorano Glad I could help. I didn't know there was a way to tell nginx to use a specific OpenSSL config file (and in my case it didn't matter since I was working in an LXC so it was the only thing running) but definitely good information to add to the repository!
It's also good to know that these bodges do have potential in ILO3 as well :)
Have a great day,
Jessica
I got hit with an updated nginx package that shipped an updated service file which caused my edited changes to get lost.
This is how to make service overrides that survives package updates:
sudo systemctl edit nginx.service
[Service]
Environment="OPENSSL_CONF=/etc/nginx/old_cipher_openssl.cnf"
Is also have proble to connect ilo3 to hp ilo integration on Home Assistant. Can someone post complete guide how to do it.
thx in advance
@lpt2007 Possibly, although maybe not directly from Home Assistant > iLO (if it uses old versions of TLS) since they bumped python versions and now disallow old ciphers, however if you enable the network IPMI within ILO you can probably do it with mqtt (with the Mosquitto broker addon in home assistant) and a variation of the script I added as a "Final edit" above in #274 (comment) which was based on information I found on the home assistant community (also linked within that comment).
I'd imagine it'd be fairly similar, but you'd have to query it with ipmi to see what information iLO3 gives out to determine what you will be able to log...
Running ipmitool -I lanplus -H <IP> -P <password> -U <user> sensor will give a list of what is available (after enabling network IPMI within ILO. Not sure where it is in iLO3, only personally used iLO2 and iLO4).
Good luck!
Thanks for tip 👍
When I run ipmitool -I lanplus -H <IP> -P <password> -U <user> sensor get this:
Unable to Get Channel Cipher Suites UID Light | 0x0 | discrete | 0x0080| na | na | na | na | na | na Sys. Health LED | 0x0 | discrete | 0x0080| na | na | na | na | na | na Power Supply 1 | 60 | Watts | ok | na | na | na | na | na | na Power Supply 2 | 45 | Watts | ok | na | na | na | na | na | na Power Supply 3 | 70 | Watts | ok | na | na | na | na | na | na Power Supply 4 | 70 | Watts | ok | na | na | na | na | na | na Power Supplies | 0x0 | discrete | 0x0180| na | na | na | na | na | na Fan 1 | 30.184 | percent | ok | na | na | na | na | na | na Fan 2 | 30.184 | percent | ok | na | na | na | na | na | na Fan 3 | 30.184 | percent | ok | na | na | na | na | na | na Fan 4 | 30.184 | percent | ok | na | na | na | na | na | na Fans | 0x0 | discrete | 0x0180| na | na | na | na | na | na Temp 1 | 29.000 | degrees C | ok | na | na | na | na | 41.000 | 45.000 Temp 2 | 43.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000 Temp 3 | 36.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000 Temp 4 | 48.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000 Temp 5 | 39.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000 Temp 6 | 50.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 7 | 44.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 8 | 44.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 9 | 46.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 10 | 47.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 11 | 46.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 12 | 46.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 13 | 46.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 14 | 46.000 | degrees C | ok | na | na | na | na | 95.000 | 127.000 Temp 15 | 41.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 16 | 43.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 17 | 42.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 18 | 43.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 19 | 41.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 20 | 41.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 21 | 42.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 22 | 44.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 23 | 53.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 24 | 44.000 | degrees C | ok | na | na | na | na | 60.000 | 65.000 Temp 25 | 46.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 26 | 46.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 Temp 27 | 47.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 28 | 58.000 | degrees C | ok | na | na | na | na | 92.000 | 96.000 Temp 29 | 52.000 | degrees C | ok | na | na | na | na | 92.000 | 96.000 Temp 30 | na | | na | na | na | na | na | 70.000 | 75.000 Temp 31 | na | | na | na | na | na | na | 70.000 | 75.000 Temp 32 | 39.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 33 | 49.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 34 | 48.000 | degrees C | ok | na | na | na | na | 126.000 | 127.000 Temp 35 | na | | na | na | na | na | na | 70.000 | 75.000 Temp 36 | na | | na | na | na | na | na | 70.000 | 75.000 Memory | 0x0 | discrete | 0x4080| na | na | na | na | na | na Power Meter | 472 | Watts | ok | na | na | na | na | na | na Cntlr 1 Bay 1 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 1 Bay 2 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 1 Bay 3 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 1 Bay 4 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 2 Bay 5 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 2 Bay 6 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 2 Bay 7 | 0x1 | discrete | 0x0180| na | na | na | na | na | na Cntlr 2 Bay 8 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
Can I get something else from ipmitool?
That's basically the information you can get, there might be different stuff available with other commands - one of them I use is sdr and then tell it which sensors I want then cut that stuff out with awk and what not (as above), but you'd have to read the manual. Not all HPE sensors are shown through ipmi, but that's basically the output you've got to strip and parse, and then ingest into ipmi