Harden core.blockstack.org service
jackzampolin opened this issue · comments
Discuss updating the core.blockstack.org
service to run effectively on multiple hosts
Just to clarify -- I think the biggest issue here isn't documentation or cleaning up services that aren't running -- it's improving our current infrastructure. The core and node services both need some improvements, as performance can (and does) deteriorate on them with loads they should be capable of handling.
I think replacing the node.blockstack.org
service should be pretty straight forward, however hardening the core.blockstack.org
service will be a bit more complicated.
Currently the service is tightly coupled to the VM that is running it and consists of a couple of different parts. I think we would require a couple of changes to make it more scalable:
- Make the long running jobs update the database (
mongodb
), and not dump files on disk. This would allow them to run as a continuous process and to run in a separate VM. - Decouple the flask app. Currently the flask app is tightly coupled to
uwsgi
and the nginx instance. If we could configure the flask app with amongodb_connection
and ablockstack_api_connection
then this could run in a separate VM and just expose a port.
If those two changes we made we could compose core.blockstack.org
of the following services:
blockstack-core
: Either rely onnode.blockstack.org
or start a new array ofblockstack-core
nodes for this service to rely onblockstack api
: These could connect tonode.blockstack.org
or share a VM/host with theblockstack-core
nodes that support this serviceindexer
(?): This would be a python process that runs continuously, updating themongodb
instance that the flask app draws fromflask app
: This app stitches together data fromblockstack api
andmongodb
and responds to user requests.mongodb
ngnix
These small changes would allow for a modular deployment where we could independently scale each of the components to deal with load. What do you think @kantai?
Decouple the flask app. Currently the flask app is tightly coupled to uwsgi and the nginx instance. If we could configure the flask app with a mongodb_connection and a blockstack_api_connection then this could run in a separate VM and just expose a port.
The flask app isn't coupled to uwsgi -- it could be run by any other python application server, but uwsgi is pretty easy to get running, and by using the uwsgi bindings in nginx, we get a little bit of a performance boost, though just running it in HTTP mode is fine too.
The way it is coupled right now, is we have
nginx -> flask app (running in uwsgi) -> api (6270) -> blockstackd
\-> mongodb
If we just configure the client.ini
on the api node to point at node.blockstack.org
, that decouples the blockstackd
component. Decoupling the flask app from the api service is doable, but I'm not sure there's much benefit to be reaped there -- that api service doesn't do much CPU-wise and it's threaded.
Decoupling it from the mongodb instance is a great idea -- we could add an environment variable there to support this.
Have a service running at core.technofractal.com
I am testing for the replacement.
This is complete and in an operational state: https://core.technofractal.com
All that needs to be done now is to get some certs and switch the DNS!
Full details and troubleshooting info -> https://github.com/blockstackinc/devops/tree/master/core.blockstack.org/docker
DNS has been switched. Closing this issue.