Plume crashes and I don't know how to look in logs
marcelcosta opened this issue · comments
We have recently installed plume in our server. Sometimes (three times a week) the service has gone down and I had to restart it. I don't know how to check whats causing it to crash, as I haven't find any log.
Thank you,
Marcel
- Plume version: 0.4.0
- Operating system: Ubuntu 18.04 (server side)
What do you mean by "crash", what is the actual error? A 504 error page? A blank page? Something else?
A 504 error page. Restarting plume.service solves it (until next time).
I have thought in looking at journalctl register. I don't know if it helps, but it shows this errors from midnight:
plume[19415]: thread '<unnamed>' panicked at 'instance::shared_inbox: user error: (Some(Object({"error": String("Gone")})), Inbox(InvalidObject(None)))', src/inbox.rs:30:9
plume[19415]: stack backtrace:
plume[19415]: 0: 0x55c63d785324 - <unknown>
plume[19415]: 1: 0x55c63d7a805c - core::fmt::write::hbf789d5b9c29db93
plume[19415]: 2: 0x55c63d77e617 - <unknown>
plume[19415]: 3: 0x55c63d787b95 - <unknown>
plume[19415]: 4: 0x55c63d787881 - <unknown>
plume[19415]: 5: 0x55c63d7881fa - std::panicking::rust_panic_with_hook::h6f42bc3a3471b522
plume[19415]: 6: 0x55c63d787de0 - rust_begin_unwind
plume[19415]: 7: 0x55c63d7a66d1 - core::panicking::panic_fmt::h4ebb133dcc5cc5ac
plume[19415]: 8: 0x55c63d7a64f3 - core::result::unwrap_failed::h2bcdded874600e35
plume[19415]: 9: 0x55c63cdb2ced - <unknown>
plume[19415]: 10: 0x55c63cdc123b - <unknown>
plume[19415]: 11: 0x55c63d6ebaea - <unknown>
plume[19415]: 12: 0x55c63d6f6523 - <unknown>
plume[19415]: 13: 0x55c63d6f4355 - <rocket::rocket::Rocket as hyper::server::Handler>::handle::hf70571b08c91baa1
plume[19415]: 14: 0x55c63d6e2646 - <unknown>
plume[19415]: 15: 0x55c63d6f217c - <unknown>
plume[19415]: 16: 0x55c63d6eebb7 - <unknown>
plume[19415]: 17: 0x55c63d704ed7 - <unknown>
plume[19415]: 18: 0x55c63d78e307 - __rust_maybe_catch_panic
plume[19415]: 19: 0x55c63d6c644e - <unknown>
plume[19415]: 20: 0x55c63d776d1f - <unknown>
plume[19415]: 21: 0x55c63d78d460 - <unknown>
plume[19415]: 22: 0x7fd5fbc346db - start_thread
plume[19415]: 23: 0x7fd5fb745a3f - __clone
plume[19415]: 24: 0x0 - <unknown>
plume[19415]: Error: No matching routes for GET /poco.
plume[19415]: Warning: Responding with 404 Not Found catcher.
plume[19415]: Error: No matching routes for GET /api/v1/directory?limit=1.
plume[19415]: Warning: Responding with 404 Not Found catcher.
plume[19415]: Error: No matching routes for GET /poco.
plume[19415]: Warning: Responding with 404 Not Found catcher.
plume[19415]: Error: No matching routes for GET /api/v1/directory?limit=1.
plume[19415]: Warning: Responding with 404 Not Found catcher.
That bit of log shows someone send an Activity and their account was deleted. I'm not sure what /poco
is supposed to be, /api/v1/directory?limit=1
is a Mastodon Api endpoint, so it's definitely missing from Plume. I don't think it's what is making Plume unresponsive
I think this bug is caused by rocket, and restarting plume every 2 hours or so with a cron job or a systemd timer is the best workaround we've found so far. Upgrading to rocket 0.5 should fix it.
Is rocket 0.5 a non stable version? In their web they announce 0.4. If this is the case, I think I would wait to upgrade and go for the cron job.
It is unstable yes, and we had to plan to update, but it requires a lot of changes unfortunately…
Hello,
Coming back to this. I have a bit of the same issue.
My Plume instance is running pre-build binaries, on a debian 10 VM. That VM is on a dedicated server also running a debian 10 and kvm virtualisation.
The setup is:
- A front nginx server
- An nginx server on the Plume instance
- The Plume service running on the Plume instance.
The Nginx running on the Plume instance was complaining about connection refused from the Rocket server. Looking at journalctl, I get:
`Dec 11 13:15:01 twili-services CRON[21956]: pam_unix(cron:session): session closed for user root
Dec 11 13:15:13 twili-services plume[16611]: Warning: Responding with 503 Service Unavailable catcher.
Dec 11 13:15:13 twili-services plume[16611]: Error: No matching routes for GET /favicon.ico image/webp.
Dec 11 13:15:13 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.
Dec 11 13:15:13 twili-services plume[16611]: Error: Failed to write response: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }.
Dec 11 13:15:43 twili-services plume[16611]: Warning: Responding with 503 Service Unavailable catcher.
Dec 11 13:15:44 twili-services plume[16611]: Error: No matching routes for GET /favicon.ico image/webp.
Dec 11 13:15:44 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.
Dec 11 13:17:01 twili-services CRON[21963]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 11 13:17:01 twili-services CRON[21964]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 11 13:17:01 twili-services CRON[21963]: pam_unix(cron:session): session closed for user root
Dec 11 13:19:32 twili-services systemd[1]: Starting Cleanup of Temporary Directories...
Dec 11 13:19:33 twili-services systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Dec 11 13:19:33 twili-services systemd[1]: Started Cleanup of Temporary Directories.
Dec 11 13:25:01 twili-services CRON[21987]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 11 13:25:01 twili-services CRON[21988]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 13:25:01 twili-services CRON[21987]: pam_unix(cron:session): session closed for user root
Dec 11 13:35:01 twili-services CRON[22013]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 11 13:35:01 twili-services CRON[22014]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 13:35:01 twili-services CRON[22013]: pam_unix(cron:session): session closed for user root
Dec 11 13:40:32 twili-services plume[16611]: Error: Failed to write response: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }.
Dec 11 13:40:32 twili-services plume[16611]: Error: No matching routes for GET /favicon.ico image/webp.
Dec 11 13:40:32 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.
Dec 11 13:40:33 twili-services plume[16611]: Error: No matching routes for GET /api/v1/instance.
Dec 11 13:40:33 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.
Dec 11 13:41:02 twili-services plume[16611]: Error: No matching routes for POST /api/meta application/octet-stream.
Dec 11 13:41:02 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.
Dec 11 13:41:32 twili-services plume[16611]: thread '' panicked at 'Expected a successful outcome!', <::std::macros::panic macros>:2:4
Dec 11 13:45:01 twili-services CRON[22041]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 11 13:45:01 twili-services CRON[22042]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 13:45:01 twili-services CRON[22041]: pam_unix(cron:session): session closed for user root
Dec 11 13:45:03 twili-services plume[16611]: Error: No matching routes for GET /api/v1/instance.
Dec 11 13:45:03 twili-services plume[16611]: Warning: Responding with 404 Not Found catcher.
Dec 11 13:55:01 twili-services CRON[22068]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 11 13:55:01 twili-services CRON[22069]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 13:55:01 twili-services CRON[22068]: pam_unix(cron:session): session closed for user root
Dec 11 14:05:01 twili-services CRON[22094]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 11 14:05:01 twili-services CRON[22095]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 11 14:05:01 twili-services CRON[22094]: pam_unix(cron:session): session closed for user root
Dec 11 14:06:17 twili-services plume[16611]: Warning: Responding with 503 Service Unavailable catcher.
`
Notice the broken pipe error for Plume and the 503 response. No idea why this is happening. I am willing to dig into it with some help
FWIW I'm having this issue on 0.6.0. Sure, I set a crontab to restart it every 2 hours, but that's a pretty big hack.