Giters

hedhyw / json-log-viewer

Interactive viewer for JSON logs.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

It froze our production server while reading a 288MB log file

FKPSC opened this issue · comments

commented

I installed this log viewer on our production server after switching to pino for logging.

It took a moment to to load but it eventually loaded. I checked a couple of entries and pressed up to reload the file.

I was waiting for the file to load but I got a notification from our monitoring service that the production api server was down. I logged in to AWS to reboot the server from their EC2 dashboard. We were back online after 12 min.

300 MB is not huge for a log file. Even if it cannot handle big files, it might be good to have some check/limits and crash with a message rather than clog the whole system.

commented

Ohhh, I'm sorry to hear. I hope this didn't have any big consequences.

I will add some reasonable limit until stream reading is done: #55

commented

Big memory & CPU optimization included in https://github.com/hedhyw/json-log-viewer/releases/tag/v0.4.0

What changed:

  1. We still load the whole file (because we need to filter), but JSON parsed lazily (100 rows from config).
  2. Threshold between reloads (1 second from config).
  3. Reloading indicator.
  4. Limited the maximum size of the file (1 Gb from config).
  5. I've tested it on 600Mb file. It flies, but changes are significant, so we need to use more.

Please confirm that v0.4.0 works for you as expected!