Greenlight container not listening on 3000
AdamZWinter opened this issue · comments
root@ip-172-31-6-35:/home/ubuntu# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
69c9809500ec bigbluebutton/greenlight:v3 "bin/start" About an hour ago Up About an hour 127.0.0.1:5050->3000/tcp greenlight-v3
b57e2995aae1 redis:6.2-alpine3.17 "docker-entrypoint.s…" About an hour ago Up About an hour 6379/tcp redis
83f57dab24f4 postgres:14.6-alpine3.17 "docker-entrypoint.s…" About an hour ago Up About an hour 5432/tcp postgres
root@ip-172-31-6-35:/home/ubuntu# curl http://127.0.0.1:5050
curl: (56) Recv failure: Connection reset by peer
root@ip-172-31-6-35:/home/ubuntu# docker exec -it 69c980 /bin/bash
69c9809500ec:/usr/src/app# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.11:36265 0.0.0.0:* LISTEN
udp 0 0 127.0.0.11:48184 0.0.0.0:*
Can I get some guidance here, please? Anyone? Thank you.
Hi Adam,
I just tried a new install and can see GreenLight properly listening
# netstat -antp | grep 5050
tcp 0 0 127.0.0.1:5050 0.0.0.0:* LISTEN 71134/docker-proxy
What version of BigBlueButton are you installing? Did you get any errors on the install? Is this a new install or upgrade? Did you make any custom changes after you installed?
Hi. Thank you for responding. The server is listening on 5050, but that port is mapped to port 3000 in the container. You can see this by running "docker ps" : 127.0.0.1:5050->3000/tcp
You can also see this in the Dockerfile, where it exposes port 3000: https://github.com/bigbluebutton/greenlight/blob/master/Dockerfile
When I bash into the container and run netstat, I don't see it listening on 3000. Naturally, I'm not getting any http response either.
# netstat -antp
tcp 0 0 127.0.0.11:36265 0.0.0.0:* LISTEN -
tcp 0 1 172.18.0.4:46009 172.18.0.2:5432 SYN_SENT 237425/nc
I've now looked at the docker logs:
docker logs 69c98 | wc -l
docker logs 69c98 | more
docker logs 69c98 | tail -n 50
and there is 100k lines of this:
Waiting for postgres to start up ...
Waiting for postgres to start up ...
Waiting for postgres to start up ...
Waiting for postgres to start up ...
Waiting for postgres to start up ...
Waiting for postgres to start up ...
Waiting for postgres to start up ...
Waiting for postgres to start up ...
which appears to come from the entry point for the container: bin/start
and is stuck in that loop.
This is a brand-new server. I followed the instructions on this page exactly: https://docs.bigbluebutton.org/administration/install
Please advise. Thank you.
Running docker-compose up I get this:
root@ip-172-31-6-35:~/greenlight-v3# docker-compose up
Starting postgres ... done
Starting redis ... done
Starting greenlight-v3 ... done
Attaching to redis, postgres, greenlight-v3
redis | 1:C 23 Oct 2023 18:12:30.744 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis | 1:C 23 Oct 2023 18:12:30.744 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=1, just started
redis | 1:C 23 Oct 2023 18:12:30.744 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis | 1:M 23 Oct 2023 18:12:30.745 * monotonic clock: POSIX clock_gettime
postgres |
postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres |
redis | 1:M 23 Oct 2023 18:12:30.745 * Running mode=standalone, port=6379.
redis | 1:M 23 Oct 2023 18:12:30.745 # Server initialized
postgres | 2023-10-23 18:12:30.802 UTC [1] LOG: starting PostgreSQL 14.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
greenlight-v3 | Greenlight-v3 starting on port: 3000
greenlight-v3 | Postgres host: postgres
greenlight-v3 | Postgres port: 5432
greenlight-v3 | Redis host: redis
greenlight-v3 | Redis port: 6379
postgres | 2023-10-23 18:12:30.802 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres | 2023-10-23 18:12:30.802 UTC [1] LOG: listening on IPv6 address "::", port 5432
redis | 1:M 23 Oct 2023 18:12:30.745 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
postgres | 2023-10-23 18:12:30.811 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres | 2023-10-23 18:12:30.823 UTC [22] LOG: database system was shut down at 2023-10-23 18:02:16 UTC
postgres | 2023-10-23 18:12:30.832 UTC [1] LOG: database system is ready to accept connections
redis | 1:M 23 Oct 2023 18:12:30.746 * Loading RDB produced by version 6.2.12
redis | 1:M 23 Oct 2023 18:12:30.746 * RDB age 624 seconds
redis | 1:M 23 Oct 2023 18:12:30.746 * RDB memory usage when created 0.77 Mb
redis | 1:M 23 Oct 2023 18:12:30.746 # Done loading RDB, keys loaded: 0, keys expired: 0.
redis | 1:M 23 Oct 2023 18:12:30.746 * DB loaded from disk: 0.000 seconds
redis | 1:M 23 Oct 2023 18:12:30.746 * Ready to accept connections
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
greenlight-v3 | Waiting for postgres to start up ...
Running the commands in the start script manually, inside the container, there is no output from nc -zw3 $PGHOST $PGPORT
69c9809500ec:/usr/src/app# source bin/config.env
69c9809500ec:/usr/src/app# echo "Postgres host: $PGHOST"
Postgres host: postgres
69c9809500ec:/usr/src/app# echo $RAILS_ENV
production
69c9809500ec:/usr/src/app# nc -zw3 $PGHOST $PGPORT
69c9809500ec:/usr/src/app# which nc
/usr/bin/nc
I've narrowed this down to an issue with containers not being able to connect to each other. (I cannot ping between containers). Whatever ends up being the root problem, it should probably be addressed in these installation instructions: https://docs.bigbluebutton.org/administration/install
...since I've followed those directions exactly; using AWS c5a.2xlarge instance, and the standard Ubuntu 20.04 LTS that they provide as a common/default choice (not the one with SQL server).
Turns out that using Snap to install Docker in Ubuntu doesn't work, as the containers are not able to talk to each other. Uninstalled Docker and reinstalled with apt.
Found the solution here: https://stackoverflow.com/a/76323890/10664600 (though simply restarting was not the solution)
closing
Hi @AdamZWinter, thanks for sharing this. You will have definitely helped someone else who encounters the same issue.