[Workflow] How to run the workflows "distributed"?
paule96 opened this issue · comments
Hi Dapr team,
I lately take a look into the new workflow feature of dapr.
And as far as I understand is an activity in an workflow not more then an actor.
And a workflow itself is like a component around the activities.
So for me every call of CallChildWorkflowAsync
or CallActivityAsync
goes to the sidecare. This looks into the placement lookup table (or updates his own copy) and send this request to the correct instance / node that hosts this Activity
or workflow
to run it.
It's also written in the CallChildWorkflowAsync
documentation the following documentation:
// • You can distribute workflow logic across multiple compute nodes concurrently,
// which is useful if your workflow logic otherwise needs to coordinate a lot of
// tasks.
To make my issue easy to understand I created an example repo.
The example repo contains basicall 3 applications:
daprWorkflowDistributedIssue.ApiService
- is exposed in dapr as
WeatherApi
- provides the Microsoft most common
WeatherAPI
sample with one single easy endpoint/weatherforecast
- is exposed in dapr as
daprWorkflowDistributedIssue.DaprWorkflow1
- a simple console app that starts an apphost and register some dapr things
- a simple daprworkflow that contains just one activity called
CallWeatherWorkflow
- the activity will call the
WeatherApi
- a simple daprworkflow that contains just one activity called
- a simple console app that starts an apphost and register some dapr things
daprWorkflowDistributedIssue.DaprWorkflow2
- a simple console app that starts an apphost and register some dapr things
- a simple daprworkflow that will call the workflow
CallWeatherWorkflow
- a simple daprworkflow that will call the workflow
- a simple console app that starts an apphost and register some dapr things
The problem is now that the workflow CallWeatherWorkflow
works just okay. But when the workflow GetWeatherAndMood
tries to call CallWeatherWorkflow
you will get the following response:
{
"instanceID": "fb8c3106-a8a0-4474-a3e2-832a7e5374ce",
"workflowName": "CallWeatherWorkflow",
"createdAt": "2024-02-23T13:01:49.870327683Z",
"lastUpdatedAt": "2024-02-23T13:01:49.919064012Z",
"runtimeStatus": "FAILED",
"properties": {
"dapr.workflow.custom_status": "",
"dapr.workflow.failure.error_message": "No orchestrator task named 'CallWeatherWorkflow' was found.",
"dapr.workflow.failure.error_type": "OrchestratorTaskNotFound",
"dapr.workflow.input": ""
}
}
It looks for me that I can only interact with workflows / activities that are hosted in the same instance of the application. But I can't call to other workflows hosted somewhere else in my dapr application.
This creates for me also the question how to scale these workloads or even better how to scale the activities. In special in the Fan-Out
pattern.
Hope someone can help me to understand, what i miss here :)
It looks for me that I can only interact with workflows / activities that are hosted in the same instance of the application. But I can't call to other workflows hosted somewhere else in my dapr application.
@paule96 This is true, only the application that hosts a workflow can start one (and use only activities that have also been registered in that application). To have other Dapr applications interact with them you must expose a new endpoint that can be called via, for example, Dapr service invocation. That endpoint can then start/stop/manage the workflow.
With respect to scaling, I believe the note you mention refers to spreading execution across multiple replicas of the service (that registers the workflow). That is, there is no affinity for running a workflow's activities to the same node that orchestrates the workflow itself.
Hi @philliphoff,
thanks for the explanation. So the actual activities
will be run in the same application or they will be run "somewhere" inside of my dapr "network"?
Or does it just mean you have your sidecar for your application and you scale the application for example to 10 instances of your and it will spread the work to these 10 instances?
The activities will only run on the application (instances) in which they've been registered (along with the workflows themselves), but they will (should?) spread across those instances.
@philliphoff This makes sense. Thanks for your explanation and clarification 👍