Discussion: Handling multiple processes within Embark predictably

Question

Discussion: Handling multiple processes within Embark predictably

0x-r4bbit opened this issue 5 years ago · comments

While I was working on the issue that Embark's service [SERVICE] on/off commands aren't working anymore, I ran into a couple of things that raised some questions I'd like to discuss in this issue.

Why is this command non-functional on latest master?

First and foremost it's important to know why the service command isn't working anymore. Just to recap what this command does, when running either $ embark run or embark console (and therefore, the same applies to Cockpit), we used to have a command

> service [SERVICE] on/off

that can be executed in any of Embark's interactive consoles and enables users to turn services inside Embark on and off respectively. Originally, there used to be dedicated commands for specific services, such as > webserver start/stop or > ipfs start/stop.

This has been more generalised in this PR and documented here so that any process can be started and stopped with a single console command.

As part of the bigger refactor in v5 a few months ago, this command has been entirely disabled to ensure better decoupling of modules inside Embark (notice how the comment points out that ProcessManager shouldn't know anything about deployment). This was done as a TODO so I assume this was actually a temporary step that was intended to be properly handled later on.

The reason this service command was registered as a deployment:deploy:beforeAll hook, was most likely because there is no other "signal" within Embark at which point we know that all necessary processes have actually been registered (and launched).

This can be easily verified by reintroducing the registration of the service commands without depending on the deployment:beforeAll event - by the time the command is registered, availableProcesses is essentially empty because none have registered yet.

Other issues with the service command

If we forget for a second the fact above and hijack the deployment:deploy:beforeAll event to get notified that all processes have been registered, there are still some things in flux with how we handle these services/processes.

After the refactor, we have introduced several stack components and layers to handle more concrete plugins. For example there's a stack component for Storage which, in practice will be used by plugins for ipfs and swarm. Same goes for Communication which at this point is primarily consumed by whisper.

The former results in no ipfs service being registered as such. Instead, there's now a storage service (as opposed to what our documentation says https://embark.status.im/docs/using_the_console.html#Enabling-and-disabling-processes). In addition, we explicitly document that whisper as a service can't be started/stopped as it's tied to the spinned up blockchain process and requires Embark to be restarted.

This is no longer true since we switched to spinning up a separate geth node for whisper and therefore should be able to start/stopping that process without affecting the blockchain process.

These things raise some questions

Above I've described different things that in my opinion need fixing, but they are also independent issues. I'd like some thoughts on the following:

1. How to Ensure a predictable registration lifecycle for processes

Assuming that we want to keep the service commands, we need to come up with a new way to ensure the registration of this command happens at a time all initial core process have been registered. Relying on deployment:deploy:beforeAll is not an options as we want to preserve semantic decoupling of the process manager.

I think what we need is, similar to the module registration/loading approaches discussed in the past, a registration/initialization phase for processes that enables us to signal when this is "done" so we're save to register a service command that needs to retrieve all registered processes.

We most likely still need a second phase that can be called #n times at runtime due to Embark's capability of extending Embark itself using plugins. Although, if we know for sure that processes are exclusively things that are stack components, we might not need such a routine as we dictate what stack components exists.

2. How to handle stack "processes" vs plugin "processes"

Due to the stack component vs plugin component architecture, we need to decide how we want to handle cases in which for example multiple processes for a certain stack process have been registered and launched.

To give an example, theoretically it's possible to have ipfs and swarm enabled within Embark. The process available inside the console is only Storage though. Turning that one on and off is broken at the moment, but I'm also wondering, should we still allow to turn of individual plugin processes?

Jonathan Rainville · Answer 1 · Tue Oct 29 2019 20:34:15 GMT+0800 (China Standard Time)

Good write up.

If I understand correctly, right now, the process manager is a stack component, but the actual processes are storage, communication, etc. and they are stack components too so we cannot really register them in the manager.

How about we move the process manager to be a core module? I don't think it's too farfetched, considering processes are core to Embark anyway.
That way, both stack components and plugins can register as processes and have the same type of handling.

r4bbit · Answer 2 · Tue Oct 29 2019 20:42:29 GMT+0800 (China Standard Time)

Thanks for your feedback @jrainville !

If I understand correctly, right now, the process manager is a stack component,

So, the process manager is already a core component as you can see here: https://github.com/embark-framework/embark/blob/master/packages/core/core/src/processes/processManager.js

so we cannot really register them in the manager.

Actually the registering process in itself is not really the issue here. Processes are being registered in several places throughout Embark.

The issue I've described above is that we don't have a notion of "registering processes is done". IOW: there's no explicit phase or signal that tells us that Embark is actually done registering (and launching processes).

This in fact goes hand in hand with having a register + init phase for modules as well. Most likely, registering processes within a module is going to be part of the init process of a module (among other things).

So I guess what I'm trying to explore here is that we need an API for that and ideally we discuss together what it should look like and how it should work.

Does that make sense?

Jonathan Rainville · Answer 3 · Tue Oct 29 2019 21:09:59 GMT+0800 (China Standard Time)

Ah ok, yes makes sense.

I checked the processManager again and I see that the only reason we wait for all processes to be registered (not even started, it doesn't care) is so that it can list correctly all processes here: https://github.com/embark-framework/embark/blob/master/packages/core/core/src/processes/processManager.js#L76

Basically, we could call _registerCommands straight away, but it wouldn't have the name of the processes, which sucks.

Now, like you said, having a way to wait for all processes to be registered is hard because the manager is a core component, so it's not supposed to know about if and when stack components and plugins are going to register to it.

So, a real solution would be to not have to wait at all. By that, I mean that the manager should just have to modify the registerConsoleCommands and change the labels to add the new process every time a new one registers. However, that is not doable (I think) with the current API, because I think it would just add a new console command and not replace it.

Another solution would be to not wait at all again and also not show the labels. We would instead add a new command called something like service list that prints the current list of registered services (plugins).

The last solution I can think of is to continue as it was before and hijack a random event to know when stuff should be ready. But it's really not ideal because, like this, the commands are not available at the start and it's a bit arbitrary.

In conclusion (that was a big essay wasn't it? haha), I don't think waiting for "all" the processes to be registered is doable, nor really useful, because we'll just end up with race conditions and no one likes that.
I then propose to go with the second option, the one where we add a command to show the registered processes, because it is quite easy, it removes the arbitrary wait and also is relatively intuitive (list commands are quite widespread)

r4bbit · Answer 4 · Tue Oct 29 2019 21:30:02 GMT+0800 (China Standard Time)

Appreciate your input. Let me get into your points.

I see that the only reason we wait for all processes to be registered (not even started, it doesn't care) is so that it can list correctly all processes here

Unfortunately, that's not the only case. We rely on availableProcesses here to determine whether or not a given command matches this command. In other words, if availableProcesses is empty, none of the service [SERVICE] on/off commands will be recognized.

Now, like you said, having a way to wait for all processes to be registered is hard because the manager is a core component, so it's not supposed to know about if and when stack components and plugins are going to register to it.

One way I can see this working if we had an actual initalization phase within Embark that either signals/broadcasts to several components that it's done initializing (this includes at least processes registered and loaded during bootstrap), or it comes with a built-in "after init" that will execute other lifecycle hooks, such as registering API calls etc. (which by the way, is an API we'd need anyways if we want a way to ensure any REST APIs are created at a time when it's valid to do so).

So, a real solution would be to not have to wait at all. By that, I mean that the manager should just have to modify the registerConsoleCommands and change the labels to add the new process every time a new one registers.

As mentioned above, I actually think it's perfectly fine for this to be asynchronous. We just need to define a predictable point in time at which things like API regsitration should happen.

Just to throw out ideas here (and yes it's very similar to what I've proposed about a year ago), Embark could have an init and registerAPICommands (and others) encoded in its protocol like this:

init() {
  return Promise.all(this.registeredModules.map(module => module.init())).
    then(modules => Promise.all(modules.map(module => module.registerAPICall()));
}

^ Obviously this is hyper simplified, but just to convey the concept. Notice that things like registering processes would be done in a module.init() function (which can and should be asynchronous by default).

I don't think waiting for "all" the processes to be registered is doable, nor really useful, because we'll just end up with race conditions and no one likes that.

I'm not sure what you mean by that. Why exactly is that not doable? Also if we have a predictable API that tells us when something should be done/finished, race conditions should be exactly what we avoid.

On whether this is useful or not... well this whole discussion is around the fact that we don't have a proper way to handle this, so I think it's not just useful but essential to get this right.

Jonathan Rainville · Answer 5 · Tue Oct 29 2019 21:41:02 GMT+0800 (China Standard Time)

Unfortunately, that's not the only case. We rely on availableProcesses here to determine whether or not a given command matches this command

That's true, but notice that this line is only called when the user inputs the command. That means that it's the user's responsibility to wait for the service to be registered (he can usually monitor that in the dashboard or cockpit).

One way I can see this working if we had an actual initalization phase within Embark that either signals/broadcasts to several components that it's done initializing (this includes at least processes registered and loaded during bootstrap)

Yeah, I guess doing an action called time to register processes (name pending) would work in that case, though who calls that? The cmd_controller?

an API we'd need anyways if we want a way to ensure any REST APIs are created at a time when it's valid to do so

I don't think that's true. The APIs can be registered from the get-go, because they either just list or call the appropriate function on the process if it is in the list (the list can grow in the meantime obviously).
Also, other APIs should be handled by the modules themselves when they are ready.
Unless you were talking about another API in another module that would need everything to be ready?

So, in the end, I'm not against a way to have all registrations captured and have events for when everything is started, but with the current commands and APIs, I don't think it's necessary

r4bbit · Answer 6 · Tue Oct 29 2019 22:11:07 GMT+0800 (China Standard Time)

That means that it's the user's responsibility to wait for the service to be registered

I think this is a question of good UX. I would say, as a user, I don't want to worry about whether Embark has loaded and initialized all of its internals, so I can use a specific command in the console. Embark should ensure that the commands I run are valid. Registering/exposing a command that doesn't work (yet) because some thing isn't finished initializing sounds like a bug to me.

Obviously, this is a different case when we're dealing with cases where we don't control when something is loaded/initialized. For example when someone loads a plugin/module at run-time after Embark has been bootstrap. That case needs to be handled separately and applies to module loading/initialization the same way as discussed above.

Yeah, I guess doing an action called time to register processes (name pending) would work in that case, though who calls that? The cmd_controller?

So the way I imagine this, as shown above in the code snippet, it would be something the command controller would not need to worry about. Embark's init() function (or whatever we want to call it) will take care of calling init() for every registered module, which may or may not cause modules to register processes. Once that is done, Embark knows it can run other APIs on every module, such as an API to register API calls, or register console commands or pretty much anything else that we allow modules to do.

To give a concrete example, ProcessManager._registerCommands() will be called by Embark once it knows it can actually do so. Notice that this requires ProcessManager to implement the same API (one for registering console commands) for this to work.

Also, other APIs should be handled by the modules themselves when they are ready.
Unless you were talking about another API in another module that would need everything to be ready?

Yes. I was generally referring to the idea that we probably should only do certain things like registering API calls, or console commands, when we know for sure that these are valid things to do. For example, we probably don't want to allow modules to create anything (such as REST APIs) on top of processes that don't exist yet.

The bottom line is that it would be great if we can say, inside and outside Embark, when we run this code we know that the things you need are available, because Embark ensured to take care of it.

but with the current commands and APIs, I don't think it's necessary

I think, in order to give certain guarantees about Embark's lifecycle (and its modules) we will have to iron this out sooner or later.

r4bbit · Answer 7 · Tue Nov 05 2019 16:15:49 GMT+0800 (China Standard Time)

As discussed offline, we're considering "removing" this command for the time-being as the root of this problem requires a little bit more effort. The functionality is already gone. In #2023 we're removing the corresponding docs for now as well.

@michaelsbradleyjr @iurimatias I think you both still wanted to leave some thoughts on the discussion above here?

Michael Bradley · Answer 8 · Wed Nov 06 2019 10:54:37 GMT+0800 (China Standard Time)

Everything that follows is very much a rough sketch, so my apologies in advance if anything's unclear. When giving examples of signals, I'm deliberately leaving out some object properties to convey the ideas more succinctly, though the structure of signals is definitely something that would require careful thought. The term "project" is used to refer to the user's own codebase, i.e. his/her dApp.

I think what we need is ... a registration/initialization phase for processes

I believe it's useful to think of the heart of Embark being:

a service supervisor - start, monitor, restart, stop
a bus for services to interact/react

Embark is almost a mini OS, in some sense. I've been thinking for awhile about our terminology/concepts with respect to Embark components and plugins and how they might evolve:

library
bundle
component
- application (or app)
- plugin

I'll attempt to explain what I mean by these terms and connect it back to "phase for processes"...

An Embark library is a package that is used as a building block for Embark components. It could be installed in a project with npm/yarn but probably wouldn't be useful that way because it's designed to be part of an Embark component.

An Embark bundle is a package that provides multiple Embark components. A bundle is installed into a project with npm/yarn. For example, embark init (default template) might add the @embark-framework/standard-bundle package to a project's package.json.

An Embark component is a package that is either an application or plugin. It is built from Embark libraries and other packages. A component is installed into a project with npm/yarn, as a dependency of a bundle or individually.

An application is a component that involves one/more processes, could be long-running or not. Examples in the current codebase would be blockchain (long-running) and graph (not long-running). Some applications behave as composites of other applications, the prime example being run. An app may provide one or more services that react to one another and/or interact with things external to Embark such as a web browser.

A plugin is a component that hooks into one or more applications. Like apps, plugins may provide services.

So in these terms Embark's CLI can be understood as a way to launch an application and hook up the applicable plugins.

if we know for sure that processes are exclusively things that are stack components, we might not need such a routine as we dictate what stack components exists

I feel like this might make Embark too tight, when, ideally, we want it to be even more flexible than it is now.

Keeping in mind what I wrote above, I'd like to do a thought experiment regarding what could be involved in Embark app bootstrapping and how that relates to phases of init, registration, etc.

Compared to how packages/embark/src/cmd/cmd.js and cmd_controller.js work at present, I'm aiming for something much more dynamic, flexible, and lightweight:

$ cd ~/dapps/foo
$ embark  # or `npx embark` if installed locally

No app was specified so we need to build the top-level --help output.

What Embark components are available?
1. for each dev/Dep in the project's package.json look in the corresponding node_modules/[pkg]/ directory for an embark.json file.
2. if embark.json is found check its type: key.
  - if the type is "bundle", then check the components: key for a list of package names and check each of those with these same steps.
  - if the type is "application" add its name and the JSON contents to an object that tracks available apps.
  - if the type is "plugin" add its name and the JSON contents to an object that tracks available plugins.
Convert tracked apps and plugins objects into text output that lists the available apps and cli-options, including any plugin cli-options. The output would look very similar to what embark --help outputs currently.

Notes

For (2) the idea is that an app declares the commands and cli-options it supports. Also, a plugin should declare what app/s it applies to (possibly "*") as well any cli-options it supports. This allows Embark's CLI and its help screens to be highly dynamic. Plugin cli-options would automatically include the plugin name after the double-dash, e.g. --snark-some-option.

$ embark [app] --help

An app was specified but with the help flag so we need to build specific --help output.

Steps (1) and (2) as before, but in (2) everything is ignored except the specified app and any plugins that apply to it.

$ embark [app] --[option] ... --[plugin]-[option] ...

An app and options were specified so we need to launch it!

Step (1) as before, then:

Check the project's embark.json for plugins: and match up with tracked plugins object from Step (1). Plugin options specified in the project's embark.json will be combined with any plugin cli-options that were given by the user. Likewise, project config options will be combined with any app cli-options that were given by the user.
Instantiate the specified app with the embark instance and app options. If any options weren't valid, there will be an exception and Embark should exit with an error message.
Instantiate each plugin with the embark instance, the app instance, and plugin-specific options. If any options weren't valid, there will be an exception and Embark should print an error message that the plugin could not be activated.
The embark instance signals START.
The app and plugins receive signals via an Rx Subject (inbound$) that's a property on the embark instance. The app and plugins send signals via another Rx Subject (outbound$) that's a property on the embark instance. Embark may react to those signals itself, e.g. {source: 'some-app', type: SERVICE_REGISTRATION, value: 'some-service-name'} could register a service as available and the source will subsequently signal {type: SERVICE_STATE, value: STARTED}. The embark instance can track that state however seems best. The same principle applies to activation of plugins from the console during runtime. In any case, whether or not Embark (the supervisory aspect) reacts to app/plugin signals, they will be looped back through inbound$. By sending through outbound$ and observing inbound$, app and plugin services can react to one another. What exactly that entails (e.g. "should X wait on Y to have started?") should be internal to the app/plugin, i.e. the supervisory aspect of Embark doesn't need to know about it.
If the app signals with {type: APP_STATE, value: DONE} then embark will wait for the plugin instances to react with {type: PLUGIN_STATE, value: STOPPED} signals and then the top-level process will exit. Similarly, if the user signals for embark to quit, then it will signal STOP, wait for signals from the app and plugins, and then exit.

Notes

The default export of each component (both apps and plugins) should be a constructor that causes no side-effects when instantiated.

Each component instance should have a validateOptions method that's called by its constructor before setting up instance properties based on the options.

By "embark instance" I mean the same kind of object we currently pass to constructors during bootstrapping. However, it can be streamlined quite a bit since much responsibility is shifted to the inner workings of apps and plugins, which Embark (the supervisory aspect) shouldn't need to know about.

Much of those inner workings can, though, be implemented in Embark libraries that are reused across apps and plugins, i.e. apps written by the Embark team or 3rd parties. But note that Embark libraries don't have to be used to implement an Embark app or plugin. So long as the app/plugin instance exposes the expected methods and properties (should be a small number of them!) it should be possible to implement it any way desired, e.g. to experiment with a different design for app-internals or to implement an app that's so different from the standard ones it needs unique internals.

To summarize everything: I believe the core supervisory aspect of Embark can be comprised of a relatively small amount of code that has relatively few responsibilities. This will allow for apps and plugins that can do just about anything and in any way that they want/need. We can still have very reusable code in the form of libraries that are used across apps and plugins, but by imposing very few requirements on apps and plugins, it's possible for us and our users to easily go off in new directions. This "core supervisory aspect" would embody the package you get when you do npm i [-g] embark, i.e. it could be installed locally or globally — it's small, has few dependencies, and at runtime hooks to code that is always installed locally in a project, e.g. components supplied by the hypothetical @embark-framework/standard-bundle mentioned above.

Imagine a scenario like this: Bob would like to use Embark to build Tezos dapps. Frank wants to use Embark to experiment with Cosmos. In both of their cases, some of the existing Embark library packages for blockchain, etc. don't do what they need, while others can be reused as-is. Each builds an Embark application modeled after the @embark-framework/run application along with some helper libraries. Those apps hook to the embark instance via inbound$ and outbound$ as described above, and they're ready to go! They create bundles and templates for use with embark init so it's easy to setup projects for use with their apps. Later, the Embark team learns of their projects and by studying the helper libraries that had to be created it's realized how the @embark-framework library packages can (or can't) be better generalized to improve their reusability, making it even easier in the future to create Embark apps and plugins for new use cases.

r4bbit · Answer 9 · Thu Nov 07 2019 17:39:09 GMT+0800 (China Standard Time)

@michaelsbradleyjr thanks for your thorough thoughts here!

I read your post three times to fully follow what you're aiming for. Just to be sure, here's what I understood from this:

Generally you're aiming for Embark kind of composing itself at run-time by reading some configuration which declares which modules and plugins are desired.

I think this is an interesting idea and certainly would provide super high flexibility if done done right. Also, I like the mindset of Embark being a mini OS. This is probably not something we can pursue in the near future, but either way, at the core of this (just to shift focus back to the original question in this issue): we need a registration + bootstrap phase at which point we know Embark is done bootstrapping (I guess this is what you were trying to say with the $inbound and $outbound stuff, which honestly, I didn't fully get).

So I'm fine with Embark becoming a highly composible thing that puts itself together by reading declarative config files at some point at run-time, but even without that, we need those core APIs that describe this bootstrap phase.

I'd propose we agree on such an API first (it has to work one way or the other) and then we can see how this can be done in a way that it's dynamically composable, which I think isn't too hard. In away we just need to determine the plugins/module we need and do something like engine.registerModule() in a dynamic fashion. After that, like I've mentioned above, we'll need this init or bootstrap phase.

Michael Bradley · Answer 10 · Thu Nov 07 2019 19:11:07 GMT+0800 (China Standard Time)

Thanks for the feedback @PascalPrecht!

Also, I like the mindset of Embark being a mini OS. This is probably not something we can pursue in the near future...

Does "in the near future" pertain to the dynamic configuration proposal or to Embark being a mini OS? If the latter, just to clarify, I meant that it already is a "mini OS" and wasn't proposing a direction of development; Iuri's describing Embark as being like a pluggable "game engine" is probably more accurate than my calling it a "mini OS".

we need those core APIs that describe this bootstrap phase

Rereading my own post, some of the consequences of the changes I proposed were implicit, when I should have been more explicit.

Basically, I'm proposing the supervisory aspect of Embark shouldn't worry about bootstrapping beyond instantiation and START. The can gets kicked to apps, and they could take any approach they wish. In practice, there would be a set of APIs and a general approach that we reuse across apps, but apps aren't constrained to them. This would allow more freely iterating on different/refactored designs for app-internals (e.g. run3, run4, run5, ...) and allow for 3rd party apps that work very different from the standard run. You can sort of do that now, by jamming an alternative run into cmd.js and cmd_controller.js (we did that during a refactor with run2), but it's pretty hacky.

What follows is basically me thinking out loud. I'm not sure that what I'm going to describe is achievable, I think it is, but I'm definitely painting in broad strokes...

I have a growing feeling that our overall architecture right now — the core Engine, etc. — is too inflexible, and that the wild spread of command handlers and events/requests all over the place is problematic. I don't know "the answer" (yet) to a better architecture, but I'm thinking about a services+agents architecture that gives more structure to how Embark components interact and react.

Services would follow a request/response model that's an abstraction on top of the async signals moving through inbound$ and outbound$, and they could also spontaneously emit signals regarding their own state changes. However, services wouldn't generally react to or interact with each other directly; a service's job would be to respond to requests, and the responsibilities of any one service should be limited and very well defined.

One or more agents would be responsible for creating the workflow between services, and their reactions to signals and responses should pluggable, i.e. so plugins can register actions for the agent to take at different points in its workflow. That would still leave us with the priority/ordering problem, but that's probably unavoidable. An agent could be implemented any number of ways, but redux-observable and xstate seem like natural choices.

So the brain of any Embark app (embark [app]) would be one/more agents. Its workflow and what parts of it are pluggable would be things we document. Besides registering actions with an app's agent/s, a plugin could send requests to an app's services directly or even implement its own agent/s. For major changes in setup/behavior, beyond the scope of a plugin, a developer could make their own app — that would involve reusing existing Embark services and/or creating new ones, and implementing an agent/s for the desired workflow.

If services are independent, i.e. self-contained and don't interact/react directly, I think the init and boostrapping problem becomes simpler or at least better contained. The complexities of waiting for services to be READY or to finish RESTARTING, not making another request to Service A until it has gotten a response from A and finished its workflow dealing with that response, etc. — all of that logic is inside the app's agent/s and not spread through the codebase.