Crash on startup with latest Docker (master)

Question

Crash on startup with latest Docker (master)

Elegant996 opened this issue 10 months ago · comments

Elegant commented 10 months ago

Type: Bug Report

Try to follow the update procedure described in the README and try again before opening this issue.

Your Environment

Version used:
jesec/flood:master-distroless from Docker as of December 7th 2023.

Summary

Docker image fails to launch with the following: TypeError: Cannot read properties of undefined (reading 'push')

Expected Behavior

Flood should start.

Current Behavior

Flood does not start.

Possible Solution

Steps to Reproduce

Use jesec/flood:master-distroless from Docker
Access web UI
Observe container crashes

Context

Use Flood.

Trim21 · Answer 1 · Fri Dec 08 2023 17:36:55 GMT+0800 (China Standard Time)

can you try latest image and paste full stack trace here？I can't re-produce.

Elegant · Answer 2 · Fri Dec 08 2023 18:15:29 GMT+0800 (China Standard Time)

Perhaps its because I'm not serving assets? I'm running a web server to serve the assets which has been working great until this last push. That error is literally all that appears before the container crashes:

Flood server 0.0.0-master.fc1e90d starting on https://10.244.5.200:3000



Starting without builtin authentication



Static assets not served



FATAL internal error. Please open an issue.

Uncaught exception: TypeError: Cannot read properties of undefined (reading 'push')

Please note that I have pulled the latest static assets from the latest docker image. I mount the assets directory from the image into the web server container. Same image ID.

My setup always pulls the latest image on startup.

Thanks!

Trim21 · Answer 3 · Fri Dec 08 2023 18:17:48 GMT+0800 (China Standard Time)

can you paste your full docker run command or compose config file?

Elegant · Answer 4 · Fri Dec 08 2023 18:20:31 GMT+0800 (China Standard Time)

Not running with any extra Docker commands (using K8s) these are the env's though:

FLOOD_OPTION_ALLOWEDPATH: "/mnt"
FLOOD_OPTION_ASSETS: "false"
FLOOD_OPTION_AUTH: "none"
FLOOD_OPTION_BASEURI: "/"
FLOOD_OPTION_HOST: *uses container IP*
FLOOD_OPTION_RTSOCKET: "/run/rtorrent/rtorrent.sock"
FLOOD_OPTION_RUNDIR: "/flood/data"
FLOOD_OPTION_SSL: "true"
FLOOD_OPTION_SSLCERT: "/tls/tls.crt"
FLOOD_OPTION_SSLKEY: "/tls/tls.key"

Trim21 · Answer 5 · Fri Dec 08 2023 18:23:00 GMT+0800 (China Standard Time)

Ok, I'll take a look

Trim21 · Answer 6 · Fri Dec 08 2023 18:33:05 GMT+0800 (China Standard Time)

I still can not re-produce...

Trim21 · Answer 7 · Fri Dec 08 2023 18:37:37 GMT+0800 (China Standard Time)

can you try latest image which will print error's stack trace?

Elegant · Answer 8 · Sat Dec 09 2023 10:32:57 GMT+0800 (China Standard Time)

Flood server 0.0.0-master.f42009f starting on https://10.244.5.250:3000

Starting without builtin authentication

Static assets not served

FATAL internal error. Please open an issue.
Uncaught exception: TypeError: Cannot read properties of undefined (reading 'push')
stack trace: TypeError: Cannot read properties of undefined (reading 'push')
    at ServerResponse._writeRaw (node:_http_outgoing:376:19)
    at ServerResponse._send (node:_http_outgoing:349:15)
    at ServerResponse.end (node:_http_outgoing:961:10)
    at /snapshot/flood/dist/index.js
    at g (/snapshot/flood/dist/index.js)
    at handleError (/snapshot/flood/dist/index.js)
    at X (/snapshot/flood/dist/index.js)
    at r.send (/snapshot/flood/dist/index.js)
    at ee (/snapshot/flood/dist/index.js)
    at s (/snapshot/flood/dist/index.js)
Stream closed EOF for media/rtorrent-0 (flood)

Sorry, stepped out for the evening. Double-checked asset locations for webserver, still correct and using this same image for those assets.

Trim21 · Answer 9 · Sat Dec 09 2023 12:24:39 GMT+0800 (China Standard Time)

I still can't find out reason…

does flood without any option works？

Trim21 · Answer 10 · Sat Dec 09 2023 13:51:00 GMT+0800 (China Standard Time)

please try latest image and paste stack trace again

Elegant · Answer 11 · Sun Dec 10 2023 00:10:07 GMT+0800 (China Standard Time)

No log; just fails to start the container now. Instant crash.

Trim21 · Answer 12 · Sun Dec 10 2023 00:25:28 GMT+0800 (China Standard Time)

zero log at all?

Trim21 · Answer 13 · Sun Dec 10 2023 00:28:32 GMT+0800 (China Standard Time)

sorry, I still can't re-produce this, I'm using docker-compose.

Elegant · Answer 14 · Sun Dec 10 2023 01:22:45 GMT+0800 (China Standard Time)

Interesting! I had a TCP health-check on port 3000 and the recent changes for testing somehow delayed the TCP health-check longer than my threshold permitted so K8s decided to kill the pod. I've disabled health-checks and got the log:

Flood server 0.0.0-master.be22e24 starting on https://10.244.5.187:3000

Starting without builtin authentication

Static assets not served

FATAL internal error. Please open an issue.
Uncaught exception: TypeError: Cannot read properties of undefined (reading 'push')
stack trace: TypeError: Cannot read properties of undefined (reading 'push')
    at ServerResponse._writeRaw (node:_http_outgoing:376:19)
    at ServerResponse._send (node:_http_outgoing:349:15)
    at ServerResponse.end (node:_http_outgoing:961:10)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
Stream closed EOF for media/rtorrent-0 (flood)

Trim21 · Answer 15 · Sun Dec 10 2023 02:08:32 GMT+0800 (China Standard Time)

can you try jesec/flood:master-dbg image？stack trace from jesec/flood:master-distroless is unexpectedly useless…

Elegant · Answer 16 · Mon Dec 11 2023 12:22:32 GMT+0800 (China Standard Time)

Now we're getting somewhere:

> flood@0.0.0-master.eaefd4c start:development:server
> NODE_ENV=development tsx watch server/bin/start.ts --host=0.0.0.0

npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /.npm
npm ERR! errno -13
npm ERR! 
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.
npm ERR! 
npm ERR! To permanently fix this problem, please run:
npm ERR!   sudo chown -R 201615:201615 "/.npm"

npm ERR! Log files were not written due to an error writing to the directory: /.npm/_logs
npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal
Stream closed EOF for media/rtorrent-0 (flood)

Try running as non-root? 😄 201615 is the user and group since this uses sys for the NFS mounts and that ID has sufficient access.

Trim21 · Answer 17 · Mon Dec 11 2023 15:03:17 GMT+0800 (China Standard Time)

I'm guessing this is a different problem, dbg image run flood with npm and run process with different user.

How did you mount volume in your container?

Elegant · Answer 18 · Mon Dec 11 2023 21:06:21 GMT+0800 (China Standard Time)

Mounting an emptyDir for the flood rundir and another one for the rtsocket. Regular directories for rtorrent are under /mnt which should not impact anything.

Trim21 · Answer 19 · Mon Dec 11 2023 21:27:48 GMT+0800 (China Standard Time)

Then I have no idea...

Elegant · Answer 20 · Mon Dec 11 2023 22:08:09 GMT+0800 (China Standard Time)

You are trying this as well where assets are not served and a webserver is being run? This worked fine until the other day... It's definitely related to the latest commits...

Trim21 · Answer 21 · Mon Dec 11 2023 22:09:54 GMT+0800 (China Standard Time)

Yes, I use same options with docker-compose and still can't re-produce it.

I guess this is caused by #661, but you can't run start dbg image as expected so I can't debug it...

Elegant · Answer 22 · Mon Dec 11 2023 23:23:57 GMT+0800 (China Standard Time)

Ah, well then let's try as root :)


> flood@0.0.0-master.eaefd4c start:development:server
> NODE_ENV=development tsx watch server/bin/start.ts --host=0.0.0.0

(node:36) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("file%3A///usr/src/app/node_modules/tsx/dist/loader.js", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
Stream closed EOF for media/rtorrent-0 (flood)

Trim21 · Answer 23 · Mon Dec 11 2023 23:25:00 GMT+0800 (China Standard Time)

Ah, well then let's try as root :)


> flood@0.0.0-master.eaefd4c start:development:server
> NODE_ENV=development tsx watch server/bin/start.ts --host=0.0.0.0

(node:36) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("file%3A///usr/src/app/node_modules/tsx/dist/loader.js", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
Stream closed EOF for media/rtorrent-0 (flood)

Looks like this container doesn't exit?

Elegant · Answer 24 · Tue Dec 12 2023 00:59:49 GMT+0800 (China Standard Time)

Yeah, I didn't edit my config properly so another probe killed it early. The master-dbg appears to run without issue if I run it as root. I tried running distroless as root but the issue remained. Trying to see if there's another way to uncover what is happening...

Trim21 · Answer 25 · Tue Dec 12 2023 01:01:57 GMT+0800 (China Standard Time)

I'll try fix source map in distroless image

Trim21 · Answer 26 · Tue Dec 12 2023 01:15:44 GMT+0800 (China Standard Time)

please try latest distroless image again, stack trace should work now.

Elegant · Answer 27 · Tue Dec 12 2023 01:55:08 GMT+0800 (China Standard Time)

Same errors but I realized something. If dbg works and the errors don't occur with npm, there's a good chance that the issue is actually related to the env's being used. After some deeper diving, these caused the issue:

FLOOD_OPTION_SSL: "true"
FLOOD_OPTION_SSLCERT: "/tls/tls.crt"
FLOOD_OPTION_SSLKEY: "/tls/tls.key"

Perhaps it's how those settings are being pushed?

Trim21 · Answer 28 · Tue Dec 12 2023 02:01:38 GMT+0800 (China Standard Time)

please paste stack trace…

Trim21 · Answer 29 · Tue Dec 12 2023 02:02:34 GMT+0800 (China Standard Time)

Same errors but I realized something. If dbg works and the errors don't occur with npm, there's a good chance that the issue is actually related to the env's being used. After some deeper diving, these caused the issue:
FLOOD_OPTION_SSL: "true"
FLOOD_OPTION_SSLCERT: "/tls/tls.crt"
FLOOD_OPTION_SSLKEY: "/tls/tls.key"
Perhaps it's how those settings are being pushed?

can you share your key and crt file？I have a ssl cert but didn't reproduce this issue.

Elegant · Answer 30 · Tue Dec 12 2023 02:05:11 GMT+0800 (China Standard Time)

Yep, it's regenerated on container creation though. I'm not overly familiar with js but I was wondering if this line is somehow forcing SSL to be false?

Trim21 · Answer 31 · Tue Dec 12 2023 02:06:29 GMT+0800 (China Standard Time)

It mean ssl is disabled by defualt

Trim21 · Answer 32 · Tue Dec 12 2023 02:08:54 GMT+0800 (China Standard Time)

OK, I can reproduce now, it need a http request to trigger this issue.

Trim21 · Answer 33 · Tue Dec 12 2023 02:23:06 GMT+0800 (China Standard Time)

OK, this bug is caused by @fastify/express not supporting http2, I would just disable http2 and everything should works fine