Telegraf as bonus for collecting system metrics
frennkie opened this issue · comments
Telegraf is a nice solution to gather system metrics that can be collected locally or remotely in a InfluxDB and this can then be displayed in Grafana dashboard. https://medium.com/@dorian599/iot-raspberry-pi-container-and-system-monitoring-with-influxdb-telegraf-and-grafana-a1767c38c109
sudo -i
curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
DISTRIB_ID=$(lsb_release -c -s)
echo "deb https://repos.influxdata.com/debian ${DISTRIB_ID} stable" | tee /etc/apt/sources.list.d/influxdb.list
apt-get update
apt-get install -y telegraf
sudo usermod -aG video telegraf
setcap 'cap_net_admin,cap_net_raw+ep' $(which ping)
Get a RaspiBlitz specific telegraf.conf from somewhere and put in /etc/telegraf/telegraf.conf
systemctl restart telegraf
Hi, I would like to contribute to this topic. How can I do so?
I had created a telegraf.conf for my own raspiblitz and a matching grafana dashboard
I'm quite keen with programming, etc... but haven't yet worked on a github project
so I might need some pointers there
kindest regards, patrick
@PatrickScheich In the next week we will produce some introduction tutorials on how to start contributing. But basically you can start small with the following:
- get it working on your own raspiblitz
- edit the
/mnt/hdd/app-data/custom-installs.shin a way that it contains all needed setup shell commands so that it will reinstall the service sucessfully if you start your RaspiBlitz with a fresh sd card (update/recovery) - check/home/admin/raspiblitz.recover.logfor debug output after recovery. - then you can can post that shell script code in this issue here .. we can point you to the next steps from there
Great, I will take a look into this.
Unfortunately telegraf needs a bunch of individual decisions on config settings like
- ip and port of the backend influxdb
- name of the target database
- credentials for the database
I will assume that proper entries will exists in raspiblitz.conf
Check if you can set those parameters on the install commands you can add to custom-installs.sh so that they dont need interaction ... in the next step when turning custom-installs.sh into an install script those parameters then get written into the raspiblitz.conf and the install script gets called using those config values during the recover process.
here we go, it took a bit of time...
attached you find the files that should go to /mnt/hdd/app-data.
plus a snippet for raspiblitz.conf
beside the telegraf part I added some code that autocopies my ssh-key into /root/.ssh/authorized_keys
app-data-for-custom-install.zip
The Grafana dashboards are a bit tricky
In my installation they are working fine, but they are not yet generic
(with proper variable setup for datasource name and hostname of the raspiblitz)
I surely can provide the JSON model for reference if helpful. Let me know.
Screenshots of Grafana dashboards
Quick Summary:
System and services uptime and connections/peers:
Blockchain and Mempool:
Basic Computer Metrics:
Coming along nicely... @PatrickScheich your plan is to run the influxdb and Grafana locally on the RaspiBlitz, correct?
I already have a online VPS running both and I also have more than one active Raspiblitz. So it would be great it you take this into consideration for the Grafana dashboard.. 😊
I surely can provide the JSON model for reference if helpful. Let me know.
Having a nice would definitely be great.
your plan is to run the influxdb and Grafana locally on the RaspiBlitz, correct?
@frennkie no, only telegraf will run on the raspiblitz for metric collection.
influxDB and Grafana will be located elsewhere
Can you share the telegraf config file and the Grafana JSON?
sure,
- the
telegraf.confis in the zip file - the grafana JSON is attached here grafana.json.txt
- You hardcoded your node name - this should be a variable (very easy to do - see below)
- You used a panel (in the Quick Summary) that's marked a deprecated in my Grafana installation - would make sense to use the current version.
Can you check this updated version on your Grafana: grafana.json.txt
I also added the telegraf user to the lndadmin group using:
sudo usermod telegraf -a -G lndadmin
and changed the lnddir here:
[[inputs.exec]]
interval = "60s"
commands = ["/usr/local/bin/lncli --lnddir=/mnt/hdd/app-data/lnd getinfo" ]
name_override = "ln-info"
data_format = "json"
The installation of telegraf that you put into custom-installs.sh would be best go into a bonus script in the ~/config.scripts directory. There are many examples (and now there is even a Youtube video explaining this) how to do it. Could you work on a PR?
Could you work on a PR?
Hmm, I had not worked with this feature of github yet.
I used to work locally with git and VSC but have no idea I how to collaborate
(and now there is even a Youtube video explaining this)
Maybe you could point me to the video, I will check it first
There are many examples
can you name some, I will look into them and try to adopt
thanks for the hint with Grafana variables
This was recorded this week.
Regarding git and GitHub ... There are many resources... Just try one or two tutorials and see what works best for you. Definitely a must have skill - IMHO.. 😉
@frennkie I opened the PR
OK I merged the PR into dev for upcomming v1.7 release :) @PatrickScheich & @frennkie I am a complete noob to telegraf .. so if one of you can test this on RC1 release (targeting this weekend) or even demo as part of the lightning hacksprint this weekend please feel invited - all info on the event: https://wiki.fulmo.org/wiki/Lightning_HackSprint_March_2021
The entrypoint to test this with the coming v1.7RC1 would be by config script:
/home/admin/config.scripts/bonus.telegraf.sh
OK .. so the install looks good. Closing issue.