Travis often flaky running "apt-get install softhsm"
mhutchinson opened this issue · comments
I see this on a daily basis. Here's one example: https://travis-ci.org/google/trillian/jobs/586157938
Installing APT Packages
30.66s$ sudo -E apt-get -yq --no-install-suggests --no-install-recommends $(travis_apt_get_options) install softhsm
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libbotan-1.10-1 libsofthsm softhsm-common
The following NEW packages will be installed:
libbotan-1.10-1 libsofthsm softhsm softhsm-common
0 upgraded, 4 newly installed, 0 to remove and 189 not upgraded.
Need to get 1,039 kB of archives.
After this operation, 3,787 kB of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libbotan-1.10-1 amd64 1.10.12-1
Ign:2 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm-common amd64 1.3.7-4
Ign:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libsofthsm amd64 1.3.7-4
Ign:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm amd64 1.3.7-4
Ign:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libbotan-1.10-1 amd64 1.10.12-1
Ign:2 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm-common amd64 1.3.7-4
Ign:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libsofthsm amd64 1.3.7-4
Ign:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm amd64 1.3.7-4
Ign:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libbotan-1.10-1 amd64 1.10.12-1
Ign:2 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm-common amd64 1.3.7-4
Ign:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libsofthsm amd64 1.3.7-4
Ign:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm amd64 1.3.7-4
Ign:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libbotan-1.10-1 amd64 1.10.12-1
Ign:2 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm-common amd64 1.3.7-4
Ign:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libsofthsm amd64 1.3.7-4
Ign:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm amd64 1.3.7-4
Ign:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libbotan-1.10-1 amd64 1.10.12-1
Ign:2 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm-common amd64 1.3.7-4
Ign:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libsofthsm amd64 1.3.7-4
Ign:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm amd64 1.3.7-4
Err:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libbotan-1.10-1 amd64 1.10.12-1
Could not connect to apt.cache.travis-ci.com:80 (34.96.81.152), connection timed out
Err:2 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm-common amd64 1.3.7-4
Unable to connect to apt.cache.travis-ci.com:http:
Err:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libsofthsm amd64 1.3.7-4
Unable to connect to apt.cache.travis-ci.com:http:
Err:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 softhsm amd64 1.3.7-4
Unable to connect to apt.cache.travis-ci.com:http:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/b/botan1.10/libbotan-1.10-1_1.10.12-1_amd64.deb Could not connect to apt.cache.travis-ci.com:80 (34.96.81.152), connection timed out
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/s/softhsm/softhsm-common_1.3.7-4_amd64.deb Unable to connect to apt.cache.travis-ci.com:http:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/s/softhsm/libsofthsm_1.3.7-4_amd64.deb Unable to connect to apt.cache.travis-ci.com:http:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/s/softhsm/softhsm_1.3.7-4_amd64.deb Unable to connect to apt.cache.travis-ci.com:http:
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
apt-get.diagnostics
apt-get install failed
$ cat ${TRAVIS_HOME}/apt-get-update.log
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main Sources [193 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2,243 B]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [136 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [3,516 B]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [944 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [754 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [410 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:13 http://security.ubuntu.com/ubuntu xenial-security/restricted i386 Packages [12.7 kB]
Get:14 http://security.ubuntu.com/ubuntu xenial-security/restricted Translation-en [2,204 B]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [582 kB]
Get:16 http://security.ubuntu.com/ubuntu xenial-security/universe i386 Packages [501 kB]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [252 kB]
Get:18 http://archive.ubuntu.com/ubuntu xenial/main Sources [1,103 kB]
Get:19 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6,114 B]
Get:20 http://security.ubuntu.com/ubuntu xenial-security/multiverse i386 Packages [6,296 B]
Get:21 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [2,699 B]
Get:22 http://archive.ubuntu.com/ubuntu xenial/restricted Sources [5,179 B]
Get:23 http://archive.ubuntu.com/ubuntu xenial/universe Sources [9,802 kB]
Get:24 http://archive.ubuntu.com/ubuntu xenial/multiverse Sources [215 kB]
Get:25 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,558 kB]
Get:26 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages [1,552 kB]
Get:27 http://archive.ubuntu.com/ubuntu xenial/main Translation-en [799 kB]
Get:28 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:29 http://archive.ubuntu.com/ubuntu xenial/restricted i386 Packages [14.5 kB]
Get:30 http://archive.ubuntu.com/ubuntu xenial/restricted Translation-en [3,019 B]
Get:31 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9,827 kB]
Get:32 http://archive.ubuntu.com/ubuntu xenial/universe i386 Packages [9,804 kB]
Get:33 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [6,256 kB]
Get:34 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:35 http://archive.ubuntu.com/ubuntu xenial/multiverse i386 Packages [172 kB]
Get:36 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [131 kB]
Get:37 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [429 kB]
Get:38 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [2,696 B]
Get:39 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [328 kB]
Get:40 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [9,424 B]
Get:41 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1,322 kB]
Get:42 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [1,105 kB]
Get:43 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [567 kB]
Get:44 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:45 http://archive.ubuntu.com/ubuntu xenial-updates/restricted i386 Packages [13.1 kB]
Get:46 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2,337 B]
Get:47 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [986 kB]
Get:48 http://archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [897 kB]
Get:49 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [438 kB]
Get:50 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [19.1 kB]
Get:51 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse i386 Packages [17.9 kB]
Get:52 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8,978 B]
Get:53 http://archive.ubuntu.com/ubuntu xenial-backports/main Sources [5,073 B]
Get:54 http://archive.ubuntu.com/ubuntu xenial-backports/universe Sources [7,665 B]
Get:55 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7,942 B]
Get:56 http://archive.ubuntu.com/ubuntu xenial-backports/main i386 Packages [7,942 B]
Get:57 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [4,571 B]
Get:58 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8,807 B]
Get:59 http://archive.ubuntu.com/ubuntu xenial-backports/universe i386 Packages [8,450 B]
Get:60 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4,428 B]
Err:61 http://apt.postgresql.org/pub/repos/apt xenial-pgdg InRelease
Could not connect to apt.cache.travis-ci.com:80 (34.96.81.152), connection timed out
Fetched 52.0 MB in 30s (1,734 kB/s)
Reading package lists...
W: Failed to fetch http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/InRelease Could not connect to apt.cache.travis-ci.com:80 (34.96.81.152), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
The command "sudo -E apt-get -yq --no-install-suggests --no-install-recommends $(travis_apt_get_options) install softhsm" failed and exited with 100 during .
One way to mitigate this and speed up Travis would be to install softhsm only for the targets that actually need it. I think that's probably just the integration tests?
https://docs.travis-ci.com/user/installing-dependencies
When PR #1865 is merged we will only install softhsm for 1/14 targets. That's a 93% reduction in the number of installs, which makes the probability of encountering a flake here less likely. It's possible that the results are even better than that as the failed installs could be related to throttling from having so many concurrent requests.