[Q] The "horizon.test.pickle" test is getting an error.

[HoHo Kim]

I am trying to install the latest version of skyline, so after changing the installation script as shown below, I installed skyline on Centos8.

[root@localhost dawn]# vim skyline.dawn.sh
...
...
if [ ! -f "/tmp/skyline.dawn.skyline.${SKYLINE_RELEASE}.txt" ]; then
  echo "Checking out Skyline at $SKYLINE_RELEASE"
  sleep 1
  cd /opt/skyline/github/skyline || exit 1
  #git checkout $SKYLINE_RELEASE
  if [ $? -ne 0 ]; then
    echo "error :: failed to check out Skyline at $SKYLINE_RELEASE"
    exit 1
  fi
...
...
echo "Seeding Skyline with data"
sleep 2
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" || exit 1
source bin/activate
#bin/python${PYTHON_MAJOR_VERSION} /opt/skyline/github/skyline/utils/seed_data.py
deactivate
cd /tmp || exit
...
...

After installation, I tried to "Seeding Skyline with data" in the installation script and an error occurred.
I have tried it several times, but I always get the same error.

I did the test like below.

- I opened a terminal window to check if the connection to port 2024 was successful. (Connection success)
[root@localhost ~]# nc -vz 127.0.0.1 2024
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 127.0.0.1:2024.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

- I opened another terminal window and executed the command below, but a connection error occurred.
[root@localhost ~]# PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
[root@localhost ~]# PROJECT="skyline-py386"
[root@localhost ~]# cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
[root@localhost skyline-py386]# source bin/activate
(skyline-py386) [root@localhost skyline-py386]#
(skyline-py386) [root@localhost skyline-py386]# bin/python${PYTHON_MAJOR_VERSION} /opt/skyline/github/skyline/utils/seed_data.py
notice :: testing the Horizon parameters
info   :: settings.UDP_PORT :: 2025
info   :: settings.HORIZON_IP :: 127.0.0.1
info   :: connect_test_metric :: horizon.test.params
notice :: Horizon parameters OK
notice :: 1000 data points to push via a pickle to Horizon
info   :: for metric :: horizon.test.pickle
info   :: using end_timestamp 1616837281 and initial 1616836281
notice :: adding anomalous data point - [1616837266, 33672.0] - value was 11393 and was modified with + 22279
notice :: adding anomalous data point - [1616837267, 30720.0] - value was 11534 and was modified with + 19186
notice :: adding anomalous data point - [1616837268, 33696.0] - value was 11451 and was modified with + 22245
notice :: adding anomalous data point - [1616837269, 30183.0] - value was 11392 and was modified with + 18791
notice :: adding anomalous data point - [1616837270, 30392.0] - value was 11449 and was modified with + 18943
notice :: adding anomalous data point - [1616837271, 33254.0] - value was 11458 and was modified with + 21796
notice :: adding anomalous data point - [1616837272, 34505.0] - value was 11438 and was modified with + 23067
notice :: adding anomalous data point - [1616837273, 32679.0] - value was 11423 and was modified with + 21256
notice :: adding anomalous data point - [1616837274, 33418.0] - value was 11388 and was modified with + 22030
notice :: adding anomalous data point - [1616837275, 35372.0] - value was 11456 and was modified with + 23916
notice :: adding anomalous data point - [1616837276, 30460.0] - value was 11471 and was modified with + 18989
notice :: adding anomalous data point - [1616837277, 31075.0] - value was 11340 and was modified with + 19735
notice :: adding anomalous data point - [1616837278, 32567.0] - value was 11398 and was modified with + 21169
notice :: adding anomalous data point - [1616837279, 31994.0] - value was 11409 and was modified with + 20585
notice :: adding anomalous data point - [1616837280, 31082.0] - value was 11473 and was modified with + 19609
notice :: sending 1000 data points
sent 100 of 1000 data points to Horizon via pickle for horizon.test.pickle to 127.0.0.1:2024
sent 200 of 1000 data points to Horizon via pickle for horizon.test.pickle to 127.0.0.1:2024
sent 300 of 1000 data points to Horizon via pickle for horizon.test.pickle to 127.0.0.1:2024
sent 400 of 1000 data points to Horizon via pickle for horizon.test.pickle to 127.0.0.1:2024
Traceback (most recent call last):
  File "/opt/skyline/github/skyline/utils/seed_data.py", line 59, in pickle_data_to_horizon
    **sock.connect((ip, port))
TimeoutError: [Errno 110] Connection timed out**

error :: failed to send pickle data to Horizon
error :: failed to send 100 data points to Horizon via pickle for horizon.test.pickle

- There seems to be no special error log.
[root@localhost ~]# find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
> do
>   echo "#####
> # Checking for errors in $skyline_logfile"
>   cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
>   echo ""
>   echo ""
> done
#####
# Checking for errors in /var/log/skyline/webapp.access.log

#####
# Checking for errors in /var/log/skyline/boundary.log

#####
# Checking for errors in /var/log/skyline/ionosphere.log

#####
# Checking for errors in /var/log/skyline/luminosity.log

#####
# Checking for errors in /var/log/skyline/panorama.log

#####
# Checking for errors in /var/log/skyline/analyzer.log

#####
# Checking for errors in /var/log/skyline/horizon.log

#####
# Checking for errors in /var/log/skyline/webapp.log

- However, a connection error occur.
[root@localhost ~]# nc -vz 127.0.0.1 2024
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection timed out.

[earthgecko]

Hi @angry-tony the reason it is timing out is because carbon-relay is running as well. When Graphite is installed by the script carbon-relay connects to the socket to send data. The socket can only accept one connection at a time. So when the script runs it connects and at some point between submissions, carbon-relay connects and submits as well, when this happens the script cannot connect to the socket as carbon-relay keeps the connection to the socket open.

I will update the script and docs to state this.

However, if you installed telegraf and started submitting metrics to Graphite you should see telegraf data in Graphite and Skyline now, so you should not need to seed data.