FRR service not coming up on Ubuntu 16.04.7

[ChinmayaAgarwal]

Description

We have built FRR from source (following the link https://docs.frrouting.org/projects/dev-guide/en/latest/building-frr-for-ubuntu1604.html). We have modified the "configure" command to modify the "sysconfdir" parameter to "/etc/frr".

./configure \
    --prefix=/usr \
    --includedir=\${prefix}/include \
    --bindir=\${prefix}/bin \
    --sbindir=\${prefix}/lib/frr \
    --libdir=\${prefix}/lib/frr \
    --libexecdir=\${prefix}/lib/frr \
    **--sysconfdir=/etc/frr \**
    --localstatedir=/var \
    --with-moduledir=\${prefix}/lib/frr/modules \
    --enable-configfile-mask=0640 \
    --enable-logfile-mask=0640 \
    --enable-snmp=agentx \
    --enable-multipath=64 \
    --enable-user=frr \
    --enable-group=frr \
    --enable-vty-group=frrvty \
    --with-pkg-git-version \
    --with-pkg-extra-version=-MyOwnFRRVersion

The FRR builds successfully but when we try to start the service, I see below issue:-

May 21 05:48:32 mclican-frr-test-temp systemd[1]: Stopped FRRouting.
May 21 05:48:32 mclican-frr-test-temp systemd[1]: Starting FRRouting...
May 21 05:48:32 mclican-frr-test-temp watchfrr[8725]: [T83RR-8SM5G] watchfrr 8.0.1-MyOwnFRRVersion-gUNKNOWN starting: vty@0
May 21 05:48:32 mclican-frr-test-temp watchfrr[8725]: [ZCJ3S-SPH5S] zebra state -> down : initial connection attempt failed
May 21 05:48:32 mclican-frr-test-temp watchfrr[8725]: [ZCJ3S-SPH5S] bgpd state -> down : initial connection attempt failed
May 21 05:48:32 mclican-frr-test-temp watchfrr[8725]: [ZCJ3S-SPH5S] staticd state -> down : initial connection attempt failed
May 21 05:48:32 mclican-frr-test-temp watchfrr[8725]: [YFT0P-5Q5YX] Forked background command [pid 8726]: /usr/lib/frr/watchfrr.sh restart all
May 21 05:48:32 mclican-frr-test-temp watchfrr.sh[8735]: Cannot stop staticd: pid file not found

I saw another issue on github with same behaviour #3652 and following the resolution mentioned I deleted "daemons.conf" file and modified "daemons" file to have below option:-

watchfrr_options="-r '/usr/lib/frr/watchfrr.sh restart %s' -s '/usr/lib/frr/watchfrr.sh start %s' -k '/usr/lib/frr/watchfrr.sh stop %s'"

When I restarted frr again, I see a new issue:-

May 21 05:49:58 mclican-frr-test-temp watchfrr[8811]: [T83RR-8SM5G] watchfrr 8.0.1-MyOwnFRRVersion-gUNKNOWN starting: vty@0
May 21 05:49:58 mclican-frr-test-temp watchfrr[8811]: [ZCJ3S-SPH5S] zebra state -> down : initial connection attempt failed
May 21 05:49:58 mclican-frr-test-temp watchfrr[8811]: [ZCJ3S-SPH5S] bgpd state -> down : initial connection attempt failed
May 21 05:49:58 mclican-frr-test-temp watchfrr[8811]: [ZCJ3S-SPH5S] staticd state -> down : initial connection attempt failed
May 21 05:49:58 mclican-frr-test-temp watchfrr[8811]: [YFT0P-5Q5YX] Forked background command [pid 8812]: /usr/lib/frr/watchfrr.sh restart all
May 21 05:49:58 mclican-frr-test-temp watchfrr.sh[8825]: Cannot stop zebra: pid file not found
May 21 05:49:58 mclican-frr-test-temp watchfrr.sh[8829]: Failed to start zebra!
May 21 05:50:53 mclican-frr-test-temp frrinit.sh[8800]: watchfrr failed to start, exited 1
May 21 05:50:53 mclican-frr-test-temp frrinit.sh[8800]:  * Failed to start watchfrr!

What could be the reason for this issue? And how can we resolve it?

Version

OS Version: Ubuntu 16.04.7 LTS
Kernel: 4.4.0-210-generic
FRR Version: FRRouting 8.1

How to reproduce

Build FRR from tag frr-8.0.1 and try restarting the service on Ubuntu 16.04.7 LTS

Expected behavior

FRR service should have run successfully.

Actual behavior

FRR service not coming up.

Additional context

No response

Checklist

• I have searched the open issues for this bug.
• I have not included sensitive information in this report.

[Donald Sharp]

What happens when you run sudo /usr/lib/frr/zebra --log stdout --log-level debug?