Runlast fails every time since a few weeks ago
teo4all opened this issue · comments
Starting a few weeks ago, Runlast fails to start docker-compose every time on QNAP Nas.
I have set-up a daily on/off schedule for the NAS, it turns off at 1 am and turns back on at 7 am. I used Runlast to start docker-compse containers for three months without an issue, but in the last 3 to for weeks, it fails every morning and I have to manually start the containers, using the same script.
The weird thing is that in the last 4 weeks since it wasn't working, 1 or 2 days it worked...
This is the latest log, but they all look like this every day
`RunLast (20220426)
[Mon May 16 07:08:40 EEST 2022] begin "start" scripts ...
[Mon May 16 07:08:41 EEST 2022] -> execute: "'/share/CACHEDEV1_DATA/.qpkg/RunLast/scripts/start-docker-compose.sh'" ...
[Mon May 16 07:09:23 EEST 2022] => exitcode: (255)
[Mon May 16 07:09:23 EEST 2022] => stdout: ""
[Mon May 16 07:09:23 EEST 2022] => stderr: "[4222] Failed to execute script docker-compose
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 449, in send
File "urllib3/connectionpool.py", line 727, in urlopen
File "urllib3/util/retry.py", line 410, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/api/client.py", line 214, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 237, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 200, in perform_command
File "compose/cli/command.py", line 70, in project_from_options
File "compose/cli/command.py", line 153, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 197, in init
File "docker/api/client.py", line 222, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"
[Mon May 16 07:09:23 EEST 2022] end "start" scripts`
This is the start-docker-compose.sh script (and it works perfectly manually triggered every morning):
`#!/bin/bash
$(getcfg container-station Install_path -f /etc/config/qpkg.conf)/bin/docker-compose -f /share/Container/qnap-docker-compose/docker-compose.yml up -d`
Sounds like a race-condition. :(
I'm not much good troubleshooting containers (never did get into the whole container thing), so I can only guess as to what the cause is.
-
Have you tried introducing a delay in your script? If Container Station hasn't completed its startup at the time your script runs, this could cause a problem. Try adding a
sleep 300before running the rest of the script. -
Have you checked the environment variables available when your script executes? You might look at adding a line to your script to save the current variables to file:
printenv > /var/log/runlast.vars, then compare this against the variables available at the CLI when QTS has finished starting-up. If Container Station is using Python, then the$PYTHONPATHvariable might be relevant.
Adding a delay to the script fixed the issue. At the moment "sleep 300" works so I wont mess with it, but I have a feeling that lower numbers might also work. And I realize this is just a workaround and there still is an issue there.
Agree. I wish there was a way to check if Container Station had completed startup. Maybe via its Operating_Status line in /etc/init.d/qpkg.conf?
https://gist.github.com/OneCDOnly/cfbe704cad21f9185417f0d877dc4ddb