Unable to get worker running remotely
AshifulBhuiyan opened this issue · comments
I am not able to get worker running (waiting for worker permanently) on my personal website. But it works when I run locally (0.0.0.0). This is the log I get repeating when I run mercury run --verbose
Just spams
log not found : /ws/client/id
Could it be my nginx config? I didn't do anything special to it except for a proxypass to the port
EDIT
another note, when I use pip install mljar-mercury
instead of just mercury
. It just all works fine.
Hi @AshifulBhuiyan,
Hard to say ... could you please send your nginx config?
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
map $http_upgrade $connection_upgrade { default upgrade; '' close; }
sendfile on;
keepalive_timeout 65;
server {
server name website.org;
location / {
proxy_set_headers Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forawareded_for;
aio threads;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://0.0.0.0:8000;
}
}
I managed to replicate the error on https://huggingface.co/spaces/ConquestAce/mathcodes
so most likely not the case of nginx
Is you website public? What is the address? Can I look into web console logs? Hard to say where is the issue ...
The mercury
package is the new version (V2) of Mercury, the mljar-mercury
is old version (V1). Please use V2.
I am using v2, also pages.mathcodes.org
In HF Spaces Mercury cant connect to websocket (two red icons) - I was trying to debug the issue, but without any success ... It must be something with HF cloud settings.
I understand that in your case you have only one red icon?
OK, I see that you have two red icons as well ...
WebSocket:Disconnected and
Worker Unknown
Session ID: id
@AshifulBhuiyan yes, similar problem as in HF ...
To summarize:
- when you run docker-compose locally it is all working
- when you deploy to cloud then it can't connect ...
I understand that you followed the docs to setup docker-compose https://runmercury.com/docs/docker-compose/
Any ideas what might be the issue?
I am not using docker right now. I haven't been able to get it to run locally whatsoever.
Only thing I did was:
pip install mercury
mercury run 0.0.0.0:8000
and have nginx reverse proxy that port,
It works perfectly fine on my machine locally, but for remote connections, the worker does not run.
I am going to take a guess and say it has to do with my nginx config not forwarding /ws/client/id
(but mljar-mercury
works as intended with all the same settings, in huggingface and in my remote site)
Fixed.
Issue was with nginx and the websocket not being tunneled properly before.
Needed to config nginx as such:
location /wsapp/ {
proxy_pass http://wsbackend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
}