watchman crashing on ubuntu 22.04 docker container start

[Michael James]

Hello. My company uses the Metro module bundler for bundling our javascript code. After updating the docker container base image from Ubuntu 18 to 22.04 we're experiencing the startup error below causing Watchman to crash.

2023-10-10T19:12:02,546: [cli] failed to identify PDU: fill_buffer: Connection reset by peer
2023-10-10T19:12:02,546: [cli] unable to talk to your watchman on /usr/local/var/run/watchman/pod_user-state/sock! (Connection reset by peer)

Watchman:  watchman --no-pretty get-sockname returned with exit code=1, signal=null, stderr= 2023-10-10T19:12:02,546: [cli] failed to identify PDU: fill_buffer: Connection reset by peer
2023-10-10T19:12:02,546: [cli] unable to talk to your watchman on /usr/local/var/run/watchman/pod_user-state/sock! (Connection reset by peer)

The odd thing is that the unix socket seems to be fine and manually restarting watchman (or the node.js process which calls into watchman) fixes the issue:

# unix socket exists
ls -l /usr/local/var/run/watchman/pod_user-state/sock
srw------- 1 pod_user pod_user 0 Oct 12 16:40 /usr/local/var/run/watchman/pod_user-state/sock

var watchman = require('fb-watchman');
var util = require('util')

var client = new watchman.Client();
client.command(['list-capabilities']); // successfully starts watchman daemon and command exits successfully

Relevant tooling versions

• fb-watchman node package -- fb-watchman@2.0.1
• watchman daemon -- 20231008.002904.0 (installed via the .deb package distributed here)
• node.js -- v18.16.1

Watchman log file

The watchman daemon log file seems to indicate that it fails to decode a message sent to it via the BSER protocol but I'm not sure whether that's a red herring:

cat /usr/local/var/run/watchman/pod_user-state/log
2023-10-12T16:40:55,221: [listener] Watchman 20230430.125247.0 bfe098858e78367191e6eb1d2ba863295ebbf168 starting up on airmetro-michael-james-gd-74f647cb54-2kjh4
2023-10-12T16:40:55,223: [sanitychecks] starting sanityCheckThread
2023-10-12T16:40:59,518: [client=3:stm=140064856214848:pid=32] path /service is on filesystem type overlay
2023-10-12T16:40:59,518: [client=3:stm=140064856214848:pid=32] failed to use watcher eden: overlay is not a FUSE file system.
2023-10-12T16:40:59,519: [client=3:stm=140064856214848:pid=32] root /service using watcher mechanism inotify (auto was requested)
2023-10-12T16:41:06,184: [io 140064520675616 /service] PERF: {"user_time": 1.3789940000000001, "system_time": 5.2717010000000002, "start_time": 1697128859.5203249, "elapsed_time": 6.664568, "description": "full-crawl", "buildinfo": "bfe098858e78367191e6eb1d2ba863295ebbf168", "pid": 62, "meta": {"root": {"watcher": "inotify", "path": "/service", "recrawl_count": 0, "case_sensitive": true}}, "version": "20230430.125247.0"}
2023-10-12T16:41:06,184: [io 140064520675616 /service] crawl complete
2023-10-12T16:41:06,185: [client=3:stm=140064856214848:pid=32] PERF: {"user_time": 1.3807769999999999, "system_time": 5.2717780000000003, "start_time": 1697128859.5183339, "elapsed_time": 6.6667059999999996, "description": "dispatch_command:watch-project", "buildinfo": "bfe098858e78367191e6eb1d2ba863295ebbf168", "pid": 62, "meta": {"args": ["watch-project", "/service"], "client": {"pid": 32}, "root": {"watcher": "inotify", "path": "/service", "recrawl_count": 0, "case_sensitive": true}}, "version": "20230430.125247.0"}
2023-10-12T16:41:59,573: [sanitychecks] Failed to decode get-pid response: fill_buffer: Connection reset by peer Connection reset by peer
Fatal error detected at:
watchman(+0x6e75c2)[0x55ef2c2ce5c2]
watchman(+0x6675fd)[0x55ef2c24e5fd]
watchman(+0x21ae55)[0x55ef2be01e55]
watchman(+0x216210)[0x55ef2bdfd210]
watchman(+0x14add5)[0x55ef2bd31dd5]
watchman(+0x14a64c)[0x55ef2bd3164c]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc2b2)[0x7f63b6c522b2]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b42)[0x7f63b68d9b42]
/lib/x86_64-linux-gnu/libc.so.6(+0x1269ff)[0x7f63b696b9ff]
(safe mode, symbolizer not available)

Does anyone know what could be going wrong here?

[Michael James]

We were able to work around the issue by installing Watchman using the pre-built binaries (instead of using the .deb package)

$ unzip watchman-*-linux.zip
$ cd watchman-vYYYY.MM.DD.00-linux
$ sudo mkdir -p /usr/local/{bin,lib} /usr/local/var/run/watchman
$ sudo cp bin/* /usr/local/bin
$ sudo cp lib/* /usr/local/lib
$ sudo chmod 755 /usr/local/bin/watchman
$ sudo chmod 2777 /usr/local/var/run/watchman

We also had to install the libssl1.1 package as well: https://gist.github.com/joulgs/c8a85bb462f48ffc2044dd878ecaa786

[R70YNS]

Installed libssl1.1, removed watchman and reinstalled (latest version) via apt-get and it's back working.