Error while installing chasquid-1.6-1 on Arch Linux

Question

Error while installing chasquid-1.6-1 on Arch Linux

aellwein opened this issue 3 years ago · comments

I'm trying to install chasquid-1.6-1 on Arch Linux and i'm getting following error:

==> Starting check()...
?   	blitiri.com.ar/go/chasquid	[no test files]
?   	blitiri.com.ar/go/chasquid/cmd/chasquid-util	[no test files]
?   	blitiri.com.ar/go/chasquid/cmd/dovecot-auth-cli	[no test files]
?   	blitiri.com.ar/go/chasquid/cmd/mda-lmtp	[no test files]
?   	blitiri.com.ar/go/chasquid/cmd/smtp-check	[no test files]
?   	blitiri.com.ar/go/chasquid/cmd/spf-check	[no test files]
ok  	blitiri.com.ar/go/chasquid/internal/aliases	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/auth	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/config	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/courier	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/domaininfo	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/dovecot	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/envelope	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/expvarom	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/haproxy	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/maillog	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/normalize	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/protoio	(cached)
?   	blitiri.com.ar/go/chasquid/internal/protoio/testpb	[no test files]
ok  	blitiri.com.ar/go/chasquid/internal/queue	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/safeio	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/set	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/smtp	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/smtpsrv	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/sts	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/testlib	(cached)
ok  	blitiri.com.ar/go/chasquid/internal/tlsconst	(cached)
?   	blitiri.com.ar/go/chasquid/internal/trace	[no test files]
ok  	blitiri.com.ar/go/chasquid/internal/userdb	(cached)
?   	blitiri.com.ar/go/chasquid/test/util	[no test files]
t-01-simple_local ...
.msmtp-bin: cannot locate host testserver: Name or service not known
.msmtp-bin: could not send mail (account default from msmtprc)
==> ERROR: A failure occurred in check().
    Aborting...

So it looks to me like some test is not passing while installation and therefore whole procedure fails.

Max Mazurov · Answer 1 · Tue Jul 13 2021 03:36:39 GMT+0800 (China Standard Time)

Looks like the way to run tests has changed since I wrote the build script.
@albertito, is this possible to run integration tests without docker and without root access?

Max Mazurov · Answer 2 · Tue Jul 13 2021 03:37:18 GMT+0800 (China Standard Time)

P.S. Currently PKGBUILD does it like this

check() {
    cd "${srcdir}/${pkgname}-git"

    go test ${GOFLAGS} ./...

    # run.sh replaces binary in the repo root so back it up.
    mv chasquid chasquid.bkp

    setsid -w ./test/run.sh
    # https://blitiri.com.ar/git/r/chasquid/c/43573797379b6e9364e46bc01554698bdecd040d/
    #setsid -w ./test/stress.sh
    setsid -w ./cmd/chasquid-util/test.sh
    setsid -w ./cmd/mda-lmtp/test.sh
    setsid -w ./cmd/dovecot-auth-cli/test.sh

    mv chasquid.bkp chasquid
}

Max Mazurov · Answer 3 · Tue Jul 13 2021 03:58:14 GMT+0800 (China Standard Time)

@aellwein in meanwhile you can use makepkg --nocheck to disable check() function (consult manual if you use an AUR helper).

Alberto Bertogli · Answer 4 · Tue Jul 13 2021 04:28:33 GMT+0800 (China Standard Time)

Looks like the way to run tests has changed since I wrote the build script.
@albertito, is this possible to run integration tests without docker and without root access?

Thanks @aellwein for reporting this issue, and @foxcpp for taking a look!

The integration tests are runnable without root, and without Docker. But there are some portability compromises. Docker just ensures an environment that is viable for them. But I run them as a normal user and without Docker on my machine.
I would expect that if they worked in Arch before, they should continue to run fine now, nothing has drastically changed.

Are those tests running as root? If so, maybe the workaround to make msmtp g-s isn't not working, and thus causing the failure?

That's not something that changed in chasquid, it's been around for a while, but it could be that Arch' msmtp wasn't g+s before, or that the semantics of cp have changed and they're now preserving the sticky bit when copying as root?

I think the first step is figuring out what's causing the failure.

Are the tests running as root?
Can you ls -l test/util/.msmtp-bin to see what are the permissions?
Is msmtp built with non-glibc, or in any mode that would disable the $HOSTALIASES env variable?

Max Mazurov · Answer 5 · Tue Jul 13 2021 04:36:20 GMT+0800 (China Standard Time)

Tests are definitely not running as root.

Alberto Bertogli · Answer 6 · Tue Jul 13 2021 04:45:16 GMT+0800 (China Standard Time)

That's good, the tests are mainly made to run as not-root anyway, so that rules out a class of potential issues.

My next suspect is that the $HOSTALIASES mechanism isn't working correctly in this case.

Was msmtp built with a standard glibc, or is there anything peculiar about it?

You can try reproducing this by running:

( cd test/t-01-simple_local/; HOSTALIASES=hosts ../util/.msmtp-bin -C msmtprc someone@testserver < content )

On a system where the HOSTALIASES is working, you should get .msmtp-bin: cannot connect to testserver, port 1587: Connection refused or something very similar. What do you see if you try that?

Also, what version of msmtp are you using?

Max Mazurov · Answer 7 · Tue Jul 13 2021 05:02:29 GMT+0800 (China Standard Time)

I could not reproduce the issue using Arch-provided msmtp package (1.8.15). check() completes successfully and command you also outputs the expected output.

Alberto Bertogli · Answer 8 · Tue Jul 13 2021 05:59:57 GMT+0800 (China Standard Time)

Thank you @foxcpp!

@aellwein can you tell us more about your build environment, so we can debug this further?

Alex Ellwein · Answer 9 · Tue Jul 13 2021 14:37:56 GMT+0800 (China Standard Time)

@albertito Sure, i'm on Arch x86_64, keeping it latest and as of now

Linux 5.12.14-arch1-1 #1 SMP PREEMPT Thu, 01 Jul 2021 07:26:06 +0000 x86_64 GNU/Linux

Go has the version go version go1.16.5 linux/amd64.

The PKGBUILD revision in AUR tree is
* 0906963 - (HEAD -> master, origin/master, origin/HEAD) chasquid 1.6 (8 months ago) <fox.cpp>

The build dependencies installed by PKGBUILD are:

gsasl-1.10.0-1  libidn-1.37-1  msmtp-1.8.15-1

I use yay as my AUR helper.
The full output of the installation is attached.

Feel free to tell me, what else you need.

output.txt

Alberto Bertogli · Answer 10 · Wed Jul 14 2021 18:09:15 GMT+0800 (China Standard Time)

Can you run the following? It should check if python3 accepts $HOSTALIASES:

$ echo "testserver localhost" > /tmp/extra-hosts
$ HOSTALIASES=/tmp/extra-hosts python3 -c 'import socket; print(socket.gethostbyname("testserver"))'

If it works, it should print "127.0.0.1" or "::1", let us know what it shows.

If that is successful, then I think the next steps will involve downloading chasquid and running the check mentioned above to see if msmtp is honouring $HOSTALIASES.

I understand that at this point you were able to work around this issue using foxcpp's suggestion, right?

Thanks!

Alex Ellwein · Answer 11 · Wed Jul 14 2021 18:57:13 GMT+0800 (China Standard Time)

If it works, it should print "127.0.0.1" or "::1", let us know what it shows.

$ cat /tmp/extra-hosts
testserver localhost
$ HOSTALIASES=/tmp/extra-hosts python3 -c 'import socket; print(socket.gethostbyname("testserver"))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known
$ python3 --version
Python 3.9.6

I understand that at this point you were able to work around this issue using foxcpp's suggestion, right?

Yes, thanks. I even managed to compile it and install via Makefile as described in the documentation, but i would rather prefer the solution on the package level, to keep track on updates.

Max Mazurov · Answer 12 · Wed Jul 14 2021 21:20:01 GMT+0800 (China Standard Time)

If nscd is running, glibc will ignore HOSTALIASES environment variable on per-process level leading to chasquid integration tests failing.
systemctl status nscd should tell you if that's the case.

Alex Ellwein · Answer 13 · Wed Jul 14 2021 22:27:00 GMT+0800 (China Standard Time)

➜  ~ systemctl status nscd
○ nscd.service - Name Service Cache Daemon
     Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

➜  ~ cat /etc/nsswitch.conf
# Name Service Switch configuration file.
# See nsswitch.conf(5) for details.

passwd: files systemd
group: files [SUCCESS=merge] systemd
shadow: files

publickey: files

hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns
networks: files

protocols: files
services: files
ethers: files
rpc: files

@albertito
It seems not only Python doesn't recognize the HOSTALIASES environment variable:

➜  ~ env HOSTALIASES=/tmp/extra-hosts host testserver
Host testserver not found: 3(NXDOMAIN)

Alberto Bertogli · Answer 14 · Thu Jul 15 2021 05:05:11 GMT+0800 (China Standard Time)

Good call including the contents of nsswitch.conf!

Look at this line:

hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns

That means that systemd-resolved is in action, and intercepts all dns lookups (as it was designed). If it's up and responding, the normal dns lookups of libnss_dns won't take place, and that's the one which expands $HOSTALIASES.

So my suspicion is that systemd-resolved is intercepting the lookups, and not honouring $HOSTALIASES.

If it isn't too much hassle, can you change that line in /etc/nsswitch.conf to be:

hosts: files mymachines myhostname dns

and then run the python3 one-liner again, to see if it works when systemd-resolved is out of the path?

In any case, at this point I think I'm going to make the integration tests that rely on $HOSTALIASES skip if it's not working (instead of failing as they currently do).

Max Mazurov · Answer 15 · Thu Jul 15 2021 05:17:45 GMT+0800 (China Standard Time)

I just verified this on my system. I started systemd-resolved and indeed HOSTALIASES is no longer working.

Alberto Bertogli · Answer 16 · Thu Jul 15 2021 07:39:49 GMT+0800 (China Standard Time)

Thank you for checking! I wrote commit d78056a to skip the affected tests if $HOSTALIASES is not working. You're both mentioned in the commit message, please let me know if you prefer to be credit differently.

The patch is currently in the next branch; if there aren't any issues in the next couple of days, I'll merge it into master as usual.

Alex Ellwein · Answer 17 · Thu Jul 15 2021 15:43:01 GMT+0800 (China Standard Time)

I've also tried with adjusting nsswitch.conf and the python one-liner, looks good to me. 👍🏻
Thank you for the support! ❤️

Alberto Bertogli · Answer 18 · Wed Jul 21 2021 07:02:21 GMT+0800 (China Standard Time)

The fix for this is now in the master branch, and will be included in the next release.

Thanks a lot for your help and patience with resolving this!