how to serve a Pyro4 daemon on a server?

[Deleted user]

Hello!

What I'm about to ask is not an issue but more so a question.
I wasn't sure what would be the appropriate channel to ask questions regarding pyro4.

First of all, I'd like to thank you for making a great library to work with.
So far I found this library to be most intuitive.
So, thank you.

Anyway, to jump right in;
I guess the best way to put it simply is "how to serve a Pyro4 daemon on a server".
I'm currently working on a web server using FastAPI (it's much like Flask) to serve machine learning model(s).
A model made as per the client's specifications, which are then saved as a file. The saved model is called and loaded every time a prediction is made.
Having to read the model file every time prediction is made slows down the overall speed,
so I'm trying to approach this problem with the singleton design,
where the model's instance is kept alive and used for predictions.
I thought the best way to do that is to deploy the model as a Daemon.

Before:

1. client makes request to server for a model to be made
2. a model is created and then saved as a file
3. client makes request to server for a prediction to be made
4. the model file is loaded, then a prediction is made

After:

1. client makes request to server for a model to be made
2. a model is created, then a daemon with the model is launched
3. client makes request to server for a prediction to be made
4. server asks the model daemon for prediction

I've followed the tutorial on the warehouse/visit,
so I've been able to run a daemon with a model.
But I'm not sure how to launch the daemon from a server like FastAPI/Flask,
or if it is even doable.

If you would please, guide me to the right direction on how to achieve this?

[Irmen de Jong]

Why do you need to actually launch the (pyro) daemon from Flask? I would argue to make it fully decoupled and make sure it is started separately, from whatever mechanism you use that starts the Flask web server. On linux I assume you could easily add it as a service into systemd or the init scripts, i'm assuming flask is started this way as well.

Then you only have to make sure your flask request handler makes a Pyro call into the Pyro server to get/process the model.

Maybe you can move the creation of the machine learning model into the pyro server as well.

Also I believe people are generally using Celery to add async task queue to Flask, this may be worthwhile to investigate as well if the requests take too long.

[Deleted user]

Actually, Celery might be just what I'm looking for, seeing as how others are using Celery to deploy ml models. Thanks for your kind suggestion.

I'm still curious tho, I'd like to figure out how to create a custom pyro daemon on top of a web-server. Like, imagine a web-server that makes a Pyro daemon as per the user's request. A user makes request to the web-server to make a daemon that does n+3, daemon is created and then the URI is returned. The user can use the daemon by making request to the web-server with the URI with a number. The web-server uses the URI to find the daemon, throws the number at it, then returns to the user the result, which is the number + 3. Another user could do the same thing, except with a different condition like n x 8. Then there would be two daemons active, one that does n+3 and another that does n x 8. Is that too far-fetched, or outside the Pyro's scope?

[Irmen de Jong]

Pyro has been designed around services that have a fixed exposed interface. So adding different behavior at runtime kinda is working around that. Especially if you're leaning towards an execution model where the clients provide the code that the server should run -- this is something you could make (see the Flame functionality in Pyro4 as well) but it is something that is not supported at all anymore in Pyro5 due to the massive security implications.
More importantly perhaps though is that spawning daemons on demand, especialy depending on client requests, is a bad idea because clients could easily DOS a server by spawning large numbers of daemons

[Irmen de Jong]

Have a look at libraries such as DASK, I think their execution model is closer to what you're looking for