Could not run as a plugin, but works perfectly as a standalone service

Question

Could not run as a plugin, but works perfectly as a standalone service

sethforprivacy opened this issue 2 years ago · comments

Hi there, just finished building out a Core LN + c-lightning-REST + RTL setup, and ran into this error when trying to use c-lightning-REST as a plugin for lightningd:

plugin-plugin.js: Killing plugin: exited before replying to getmanifest

The plugin works perfectly when run as it's own service (systemd script here, config file here), but no matter what I tried lightningd would crash instantly on trying to load the plugin.

Here are the commands I ran before adding it to the lightningd config file:

    5  wget https://github.com/Ride-The-Lightning/c-lightning-REST/releases/download/v0.7.2/v0.7.2.tar.gz.asc
    6  tar -xvf v0.7.2.tar.gz
    7  cd c-lightning-REST-0.7.2
    8  npm install --only=prod

Then I added via full path as specified here: https://github.com/Ride-The-Lightning/c-lightning-REST#option-2-run-as-c-lightning-plugin

Not sure how to get better logs, but happy to collect what I can for you!

Suheb · Answer 1 · Fri May 20 2022 23:53:48 GMT+0800 (China Standard Time)

Hi @sethforprivacy is it possible for you to share your core lightning config settings? It should be in the .lightning folder.

Seth For Privacy · Answer 2 · Sat May 21 2022 00:04:45 GMT+0800 (China Standard Time)

Hi @sethforprivacy is it possible for you to share your core lightning config settings? It should be in the .lightning folder.

Here it is:

https://paste.sethforprivacy.com/?3e2080ae7cbf5ad0#BMcHpNDXn4rtv8SjQ8sd2HNTxxjMbqkYyLisHGdYgEU

When I was trying c-lightning-rest as a plugin I called it via full-path, i.e.:

important-plugin=/home/lightningd/c-lightning-REST-0.7.2/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https

My best guess would be it being related to lightningd not properly recognizing the location of the node binary, but nothing I could find to resolve that worked.

Suheb · Answer 3 · Sat May 21 2022 06:08:10 GMT+0800 (China Standard Time)

what all do you have in the lightningd folder?
The only difference that I could find in my settings is that I am keeping the c-lightning-REST folder separate from all of core lightning's and plugin folders.

Keep in mind that C-Lightning-REST is not an "official" plugin, so not sure if there is manifest where it needs to be registered. But that's not required for it to run as a plugin.

May be, I need to get it registered as an official plugin (not sure if that's a thing either) 😄

devast · Answer 4 · Sat May 21 2022 17:33:38 GMT+0800 (China Standard Time)

Adding my 2c, maybe it helps:
C-ln tries to load every executable it finds in ~/.lightning/plugins/* .
So having plugin-plugin.js: Killing plugin: exited before replying to getmanifest is expected, if you have any executable in that path, that is not a c-ln plugin. I think loading the plugin might fail, cause it's not an executable.
Could you check what permissions you have on plugin.js ? If it does not have executable flag on the file, you need to chmod +x plugin.js

Justin Litchfield · Answer 5 · Sun May 22 2022 03:48:40 GMT+0800 (China Standard Time)

I just got bit by the same exact issue - where the plugin config in the .lightning/config seems incorrect.

May 21 19:45:22  lightningd[871350]: 2022-05-21T19:45:22.631Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
May 21 19:45:22  lightningd[871350]: lightningd: /home/litch/.lightning/config line 21: rest-port=3002: unrecognized option

Suheb · Answer 6 · Sun May 22 2022 21:55:02 GMT+0800 (China Standard Time)

I just got bit by the same exact issue - where the plugin config in the .lightning/config seems incorrect.

May 21 19:45:22  lightningd[871350]: 2022-05-21T19:45:22.631Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
May 21 19:45:22  lightningd[871350]: lightningd: /home/litch/.lightning/config line 21: rest-port=3002: unrecognized option

Hi @litch can you share your config file settings?

Justin Litchfield · Answer 7 · Sun May 22 2022 23:21:04 GMT+0800 (China Standard Time)

The config file sections were exactly as in the readme:

mainnet
bitcoin-rpcuser=<>
bitcoin-rpcpassword=<>
bitcoin-rpcconnect=<>
bitcoin-rpcport=8332

proxy=127.0.0.1:9050
announce-addr=<>:9735

large-channels
experimental-dual-fund
funder-policy=match
funder-policy-mod=100
lease-fee-base-msat=500sat
lease-fee-basis=52
channel-fee-max-base-msat=5sat
channel-fee-max-proportional-thousandths=1


clboss-auto-close=true
clboss-min-onchain=10000000

plugin=/home/<>/c-lightning-REST/plugin.js
rest-port=3002
rest-docport=4001
rest-protocol=https

Similarly to @sethforprivacy - it works swimmingly if I run it otherwise.

Seth For Privacy · Answer 8 · Mon May 23 2022 20:26:45 GMT+0800 (China Standard Time)

@saubyk - I don't have any plugins in there, just this structure:

tiny in tiny in ~ on  master [!?] took 5s ❯ sudo tree /etc/lightningd
/etc/lightningd
├── lightningd.conf
├── lightningd.log
└── lightning-rpc

0 directories, 3 files
tiny in tiny in ~ on  master [!?] ❯ sudo tree /home/lightningd/.lightning
/home/lightningd/.lightning
└── bitcoin
    ├── backup.lock
    ├── crash.log.20220517195817
    ├── crash.log.20220522165328
    ├── data.clboss
    ├── gossip_store
    ├── hsm_secret
    ├── keys.clboss
    ├── lightningd.sqlite3
    └── lightning-rpc

1 directory, 9 files

I was loading the c-lightning-REST plugin from a directory in /home/lightningd (where it's running from now as a standalone app without issues under the same user as lightningd).

@devastgh - plugin.js is properly permissioned for the user, and has the execute bit set:

-rwxrwxr-x   1 lightningd lightningd   1698 May  9 22:31 plugin.js

Happy to help however I can!

Seth For Privacy · Answer 9 · Mon Aug 15 2022 20:39:57 GMT+0800 (China Standard Time)

Still the same issue with v0.8.0:

2022-08-15T12:34:26.952Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:34:27.319Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:34:27.319Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!

Config tested:

# c-lightning-REST settings
important-plugin=/home/lightningd/c-lightning-REST-0.8.0/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=http
rest-domain=cln.sethforprivacy.com

Folder structure for cln-REST:

sethforprivacy in apps in ~ on  master [!?] via  v16.16.0 ❯ sudo tree -L 1 /home/lightningd/c-lightning-REST-0.8.0/
/home/lightningd/c-lightning-REST-0.8.0/
├── amd64.Dockerfile
├── app.js
├── arm32v7.Dockerfile
├── arm64v8.Dockerfile
├── certs
├── cl-rest-config.json
├── cl-rest.js
├── controllers
├── docapp.js
├── docker-entrypoint.sh
├── docs
├── LICENSE
├── lightning-client-js.js
├── methods.json
├── node_modules
├── package.json
├── package-lock.json
├── plugin.js
├── README.md
├── routes
├── sample-cl-rest-config.json
├── screenshots
├── test
└── utils

devast · Answer 10 · Mon Aug 15 2022 20:47:34 GMT+0800 (China Standard Time)

Could you just check it without any options specified in c-lightning config?
If it works, set the options one by one, to check which one doesn't work.
I assure you it loads fine as a plugin for me without any options specified.

Seth For Privacy · Answer 11 · Mon Aug 15 2022 20:57:46 GMT+0800 (China Standard Time)

Could you just check it without any options specified in c-lightning config? If it works, set the options one by one, to check which one doesn't work. I assure you it loads fine as a plugin for me without any options specified.

Same issue, doesn't help:

# c-lightning-REST settings
important-plugin=/home/lightningd/c-lightning-REST-0.8.0/plugin.js
#rest-port=3001
#rest-docport=4001
#rest-protocol=http
#rest-domain=cln.sethforprivacy.com

sethforprivacy in apps in ~ on  master [!?] via  v16.16.0 ❯ tail -f /etc/lightningd/lightningd.log
2022-08-15T12:56:55.282Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:56:55.427Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:56:55.427Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!




2022-08-15T12:56:55.747Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:56:55.893Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:56:55.893Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!

devast · Answer 12 · Tue Aug 16 2022 17:17:23 GMT+0800 (China Standard Time)

Maybe try it with just plugin= not important.
Also, try to load it while the file cl-rest-config.json is not present in c-lightning-REST directory.
If it still fails to load, then i'm out of ideas.

jochemin · Answer 13 · Wed Nov 02 2022 16:01:52 GMT+0800 (China Standard Time)

Same error on fresh install, it works when run as it's own service but fails trying to run it as plugin.

ShahanaFarooqui · Answer 14 · Sat Nov 05 2022 00:07:11 GMT+0800 (China Standard Time)

Could you run the plugin after adding rest-execmode=test in the config. It might provide us better insights on the failure reason.

I tried all above mentioned scenarios on my local Ubuntu 22 and they all are working successfully. It seems like some local setting issue.

Seth For Privacy · Answer 15 · Sat Nov 05 2022 00:08:19 GMT+0800 (China Standard Time)

I'll test again ASAP.

Seth For Privacy · Answer 16 · Sat Nov 05 2022 00:16:11 GMT+0800 (China Standard Time)

@ShahanaFarooqui:

Nov 04 16:15:22 lightningd[2517231]: # Fatal error in , line 0
Nov 04 16:15:22 lightningd[2517231]: # Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).

Seth For Privacy · Answer 17 · Sat Nov 05 2022 00:16:37 GMT+0800 (China Standard Time)

Still fails to start the plugin, above seems the error returned with execmode=test set.

Seth For Privacy · Answer 18 · Sat Nov 05 2022 00:17:56 GMT+0800 (China Standard Time)

The only thing I can think of that may be causing this is the recommended hardening in the lightningd systemd script:

# Hardening measures
####################

# Provide a private /tmp and /var/tmp.
PrivateTmp=true

# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

Testing without it now.

Seth For Privacy · Answer 19 · Sat Nov 05 2022 00:19:46 GMT+0800 (China Standard Time)

Disabling those protections resolved the issue, will nail down which one is to blame shortly.

Seth For Privacy · Answer 20 · Sat Nov 05 2022 00:23:20 GMT+0800 (China Standard Time)

Narrowed down to this:

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

If the user has this hardening measure enabled in their systemd script, c-lightning-REST cannot function as a plugin.

Disabling the hardening has everything working perfectly!

Ideally there is some way you can have both this hardening and the plugin, but that's far above my pay grade.

ShahanaFarooqui · Answer 21 · Sat Nov 05 2022 00:30:07 GMT+0800 (China Standard Time)

@sethforprivacy Thank you for the quick testing and solution.

@jochemin & @litch Could you also verify your permissions or add rest-execmode=test to get some failure insights. I am keeping this issue open till then.

Justin Litchfield · Answer 22 · Sat Nov 05 2022 00:34:00 GMT+0800 (China Standard Time)

I was also running the "hardening" systemd script, and that makes perfect sense as to the cause.

I echo the, "I would prefer to run this and have the thing hardened" but likewise don't have enough insight into the changes required to have a strong opinion about whether that would be better than, perhaps a log message that detects that flag and tells the user that that is the cause of the error.

jochemin · Answer 23 · Sat Nov 05 2022 02:24:21 GMT+0800 (China Standard Time)

No luck for me. This is my systemd service:

`# /etc/systemd/system/lightningd.service

[Unit]
Description=Core Lightning daemon
Requires=bitcoind.service
After=bitcoind.service
Wants=network.target
After=network.target

[Service]
ExecStart=/home/jochemin/clightning/usr/bin/lightningd --daemon
ExecStop=/bin/sh -c '/home/jochemin/clightning/usr/bin/lightning-cli stop'

`# Process management
####################
Type=forking
Restart=on-failure
TimeoutSec=300
RestartSec=30

`# Directory creation and permissions
####################################
User=jochemin
Group=adm
Environment=PATH=/home/jochemin/clightning/usr/bin:/home/jochemin/bitcoin/bin

# Hardening measures #################### # Provide a private /tmp and /var/tmp.
PrivateTmp=true

`# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full

# Disallow the process and all of its children to gain # new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices # such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings. #MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target`

I've commented MemoryDenyWriteExecute but same error:

2022-11-04T18:21:08.430Z INFO plugin-plugin.js: Killing plugin: exited before replying to getmanifest

Seth For Privacy · Answer 24 · Sat Nov 05 2022 02:27:16 GMT+0800 (China Standard Time)

Try also commenting out this line, I realized I had it commented out already for other issues:

ProtectSystem=full

ShahanaFarooqui · Answer 25 · Sat Nov 05 2022 02:29:46 GMT+0800 (China Standard Time)

@jochemin Also add rest-execmode=test in your lightning config file to see the failure log from this plugin.

jochemin · Answer 26 · Sat Nov 05 2022 02:33:42 GMT+0800 (China Standard Time)

nov 04 19:32:02 jbtc lightningd[537230]: /usr/bin/env: 'node': No such file or directory
nov 04 19:32:02 jbtc lightningd[537213]: lightningd: /home/jochemin/.lightning/config line 16: rest-execmode=test: unrecognized option

It seems a problem with node executable.

I've checked and I have node executable in /user/bin

Out of ideas

jochemin · Answer 27 · Sat Nov 05 2022 02:38:52 GMT+0800 (China Standard Time)

Try also commenting out this line, I realized I had it commented out already for other issues:

ProtectSystem=full

Nope, same error.

ShahanaFarooqui · Answer 28 · Sat Nov 05 2022 03:45:20 GMT+0800 (China Standard Time)

@jochemin It either looks like an incorrect naming issue which can be resolved via symlink OR nodejs-legacy installation might work.

Below are some links for your reference related to these solutions:
nodejs/node-v0.x-archive#3911 (comment)
nodejs/node-v0.x-archive#3911 (comment)
https://askubuntu.com/questions/1044692/usr-bin-env-node-no-such-file-or-directory-even-though-it-is-in-path
https://stackoverflow.com/questions/30281057/node-forever-usr-bin-env-node-no-such-file-or-directory

jochemin · Answer 29 · Mon Nov 07 2022 15:00:37 GMT+0800 (China Standard Time)

/usr/bin/env: 'node': No such file or directory

Thanks @ShahanaFarooqui , no luck for me. My executable name is node so no naming isuue, I installed nodejs from Ubuntu repositories with sudo apt-install nodejs.

I've tried to install last LTS version but the error continues.

Mi SO is Ubuntu Server 22.04 and node version is 18.12.1

I'll keep trying.

ShahanaFarooqui · Answer 30 · Mon Nov 07 2022 23:43:11 GMT+0800 (China Standard Time)

@jochemin Thank you for the update. Please post your solution, if you find any, as it might help other users.

I am closing this issue now as there is no action required from us anymore.

jochemin · Answer 31 · Wed Nov 09 2022 15:45:28 GMT+0800 (China Standard Time)

I've solved it. The problem was I had this line in the systemd service:

Environment=PATH=/home/jochemin/clightning/usr/bin:/home/jochemin/bitcoin/bin

But node executable is in /usr/bin so it was not accesible. I've added that route and now it works perfect!