Plume crashes and I don't know how to look in logs

Question

Plume crashes and I don't know how to look in logs

marcelcosta opened this issue 4 years ago · comments

We have recently installed plume in our server. Sometimes (three times a week) the service has gone down and I had to restart it. I don't know how to check whats causing it to crash, as I haven't find any log.

Thank you,
Marcel

Plume version: 0.4.0
Operating system: Ubuntu 18.04 (server side)

Ana Gelez · Answer 1 · Mon Nov 02 2020 15:51:45 GMT+0800 (China Standard Time)

What do you mean by "crash", what is the actual error? A 504 error page? A blank page? Something else?

Marcel Costa · Answer 2 · Mon Nov 02 2020 16:03:18 GMT+0800 (China Standard Time)

A 504 error page. Restarting plume.service solves it (until next time).

Marcel Costa · Answer 3 · Mon Nov 02 2020 16:26:03 GMT+0800 (China Standard Time)

I have thought in looking at journalctl register. I don't know if it helps, but it shows this errors from midnight:

plume[19415]: thread '<unnamed>' panicked at 'instance::shared_inbox: user error: (Some(Object({"error": String("Gone")})), Inbox(InvalidObject(None)))', src/inbox.rs:30:9                                        
plume[19415]: stack backtrace:
plume[19415]:    0:     0x55c63d785324 - <unknown>
plume[19415]:    1:     0x55c63d7a805c - core::fmt::write::hbf789d5b9c29db93
plume[19415]:    2:     0x55c63d77e617 - <unknown>
plume[19415]:    3:     0x55c63d787b95 - <unknown>
plume[19415]:    4:     0x55c63d787881 - <unknown>
plume[19415]:    5:     0x55c63d7881fa - std::panicking::rust_panic_with_hook::h6f42bc3a3471b522
plume[19415]:    6:     0x55c63d787de0 - rust_begin_unwind
plume[19415]:    7:     0x55c63d7a66d1 - core::panicking::panic_fmt::h4ebb133dcc5cc5ac
plume[19415]:    8:     0x55c63d7a64f3 - core::result::unwrap_failed::h2bcdded874600e35
plume[19415]:    9:     0x55c63cdb2ced - <unknown>
plume[19415]:   10:     0x55c63cdc123b - <unknown>
plume[19415]:   11:     0x55c63d6ebaea - <unknown>
plume[19415]:   12:     0x55c63d6f6523 - <unknown>
plume[19415]:   13:     0x55c63d6f4355 - <rocket::rocket::Rocket as hyper::server::Handler>::handle::hf70571b08c91baa1
plume[19415]:   14:     0x55c63d6e2646 - <unknown>
plume[19415]:   15:     0x55c63d6f217c - <unknown>
plume[19415]:   16:     0x55c63d6eebb7 - <unknown>
plume[19415]:   17:     0x55c63d704ed7 - <unknown>
plume[19415]:   18:     0x55c63d78e307 - __rust_maybe_catch_panic
plume[19415]:   19:     0x55c63d6c644e - <unknown>
plume[19415]:   20:     0x55c63d776d1f - <unknown>
plume[19415]:   21:     0x55c63d78d460 - <unknown>
plume[19415]:   22:     0x7fd5fbc346db - start_thread
plume[19415]:   23:     0x7fd5fb745a3f - __clone
plume[19415]:   24:                0x0 - <unknown>
plume[19415]: Error: No matching routes for GET /poco.                                                                                                                                                             
plume[19415]: Warning: Responding with 404 Not Found catcher.
plume[19415]: Error: No matching routes for GET /api/v1/directory?limit=1.
plume[19415]: Warning: Responding with 404 Not Found catcher.
plume[19415]: Error: No matching routes for GET /poco. 
plume[19415]: Warning: Responding with 404 Not Found catcher.
plume[19415]: Error: No matching routes for GET /api/v1/directory?limit=1.
plume[19415]: Warning: Responding with 404 Not Found catcher.

trinity-1686a · Answer 4 · Mon Nov 02 2020 17:00:16 GMT+0800 (China Standard Time)

That bit of log shows someone send an Activity and their account was deleted. I'm not sure what /poco is supposed to be, /api/v1/directory?limit=1 is a Mastodon Api endpoint, so it's definitely missing from Plume. I don't think it's what is making Plume unresponsive

Ana Gelez · Answer 5 · Tue Nov 03 2020 16:46:48 GMT+0800 (China Standard Time)

I think this bug is caused by rocket, and restarting plume every 2 hours or so with a cron job or a systemd timer is the best workaround we've found so far. Upgrading to rocket 0.5 should fix it.

Marcel Costa · Answer 6 · Tue Nov 03 2020 21:02:03 GMT+0800 (China Standard Time)

Is rocket 0.5 a non stable version? In their web they announce 0.4. If this is the case, I think I would wait to upgrade and go for the cron job.

Ana Gelez · Answer 7 · Tue Nov 03 2020 21:19:08 GMT+0800 (China Standard Time)

It is unstable yes, and we had to plan to update, but it requires a lot of changes unfortunately…

Freyja Wildes · Answer 8 · Fri Dec 11 2020 21:30:57 GMT+0800 (China Standard Time)

Hello,
Coming back to this. I have a bit of the same issue.
My Plume instance is running pre-build binaries, on a debian 10 VM. That VM is on a dedicated server also running a debian 10 and kvm virtualisation.
The setup is:

A front nginx server
An nginx server on the Plume instance
The Plume service running on the Plume instance.

The Nginx running on the Plume instance was complaining about connection refused from the Rocket server. Looking at journalctl, I get:
`Dec 11 13:15:01 twili-services CRON[21956]: pam_unix(cron:session): session closed for user root

Dec 11 13:15:13 twili-services plume[16611]: Warning: Responding with 503 Service Unavailable catcher.

Dec 11 13:15:13 twili-services plume[16611]: Error: No matching routes for GET /favicon.ico image/webp.

Dec 11 13:15:13 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.

Dec 11 13:15:13 twili-services plume[16611]: Error: Failed to write response: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }.

Dec 11 13:15:43 twili-services plume[16611]: Warning: Responding with 503 Service Unavailable catcher.

Dec 11 13:15:44 twili-services plume[16611]: Error: No matching routes for GET /favicon.ico image/webp.

Dec 11 13:15:44 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.

Dec 11 13:17:01 twili-services CRON[21963]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 11 13:17:01 twili-services CRON[21964]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Dec 11 13:17:01 twili-services CRON[21963]: pam_unix(cron:session): session closed for user root

Dec 11 13:19:32 twili-services systemd[1]: Starting Cleanup of Temporary Directories...

Dec 11 13:19:33 twili-services systemd[1]: systemd-tmpfiles-clean.service: Succeeded.

Dec 11 13:19:33 twili-services systemd[1]: Started Cleanup of Temporary Directories.

Dec 11 13:25:01 twili-services CRON[21987]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 11 13:25:01 twili-services CRON[21988]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Dec 11 13:25:01 twili-services CRON[21987]: pam_unix(cron:session): session closed for user root

Dec 11 13:35:01 twili-services CRON[22013]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 11 13:35:01 twili-services CRON[22014]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Dec 11 13:35:01 twili-services CRON[22013]: pam_unix(cron:session): session closed for user root

Dec 11 13:40:32 twili-services plume[16611]: Error: Failed to write response: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }.

Dec 11 13:40:32 twili-services plume[16611]: Error: No matching routes for GET /favicon.ico image/webp.

Dec 11 13:40:32 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.

Dec 11 13:40:33 twili-services plume[16611]: Error: No matching routes for GET /api/v1/instance.

Dec 11 13:40:33 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.

Dec 11 13:41:02 twili-services plume[16611]: Error: No matching routes for POST /api/meta application/octet-stream.

Dec 11 13:41:02 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.

Dec 11 13:41:32 twili-services plume[16611]: thread '' panicked at 'Expected a successful outcome!', <::std::macros::panic macros>:2:4

Dec 11 13:45:01 twili-services CRON[22041]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 11 13:45:01 twili-services CRON[22042]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Dec 11 13:45:01 twili-services CRON[22041]: pam_unix(cron:session): session closed for user root

Dec 11 13:45:03 twili-services plume[16611]: Error: No matching routes for GET /api/v1/instance.

Dec 11 13:45:03 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.

Dec 11 13:55:01 twili-services CRON[22068]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 11 13:55:01 twili-services CRON[22069]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Dec 11 13:55:01 twili-services CRON[22068]: pam_unix(cron:session): session closed for user root

Dec 11 14:05:01 twili-services CRON[22094]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 11 14:05:01 twili-services CRON[22095]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Dec 11 14:05:01 twili-services CRON[22094]: pam_unix(cron:session): session closed for user root

Dec 11 14:06:17 twili-services plume[16611]: Warning: Responding with 503 Service Unavailable catcher.
`

Notice the broken pipe error for Plume and the 503 response. No idea why this is happening. I am willing to dig into it with some help

Cameron · Answer 9 · Sun Apr 25 2021 14:49:30 GMT+0800 (China Standard Time)

FWIW I'm having this issue on 0.6.0. Sure, I set a crontab to restart it every 2 hours, but that's a pretty big hack.