senthilnathansk / durabletask-extensions

Extensions to Durable Task Framework

Durable Task Extensions

NOTE: WORK IN PROGRESS, NOT PRODUCTION READY.

Introduction

This project aims to extend Durable Task Framework with more features and make it easier to use.

Scope:

• More interfaces defining storage features
• Dependency injection integration
• EF Core MySql/PostgreSQL/SqlServer storages
• Distributed workers: Allows splitting orchestrations/activities implementation in multiple workers
• Indirect storage access via GRPC protocol: Centralize your storage choice and configuration in a single component.
• UI
• BPMN runner (not sure yet)

Motivation

Durable Task Framework is an open source framework that provides a foundation for workflow as code in .NET platform.

Azure Durable Functions connects Durable Task Framework to Azure serverless platform, making it simpler to create workflows as code.

The concepts of Durable Functions led to the development of Cadence. A platform that brings Durable Functions to other programming languages and extends it with concepts for better microservices orchestration, like tasks lists and distributed workers.

Because of the bad integration of Cadence with .NET platform, I decided to try to add to Durable Task Framework the features I like from Cadence.

NOTE: Cadence was recently forked by one of it's creators and Temporal was created, backed by a company focused on evolving the platform. That might change this landscape in a short term.

Components

LLL.DurableTask.Core

Extends Durable Task Core with more interfaces and features:

• IExtendedOrchestrationService:
  • Lock and execute specific orchestrations
  • Lock and execute specific activities
• IExtendedOrchestrationServiceClient:
  • Search orchestrations
  • Purge orchestration instance history

LLL.DurableTask.AzureStorage

Extends Durable Task Azure storage with:

• Dependency Injection
• Adapter implementing IExtendedOrchestrationServiceClient interface

Supported features

• UI: yes
• Distributed workers: no
• Storing activity input: no
• Failures rewind: yes
• Tags: no
• State per execution: no

Configuration

services.AddDurableTaskAzureStorage(options =>
{
    options.TaskHubName = "Test";
    options.StorageConnectionString = "UseDevelopmentStorage=true";
});

LLL.DurableTask.Emulator

Extends Durable Task Emulator storage with:

• Dependency Injection

Supported features

• UI: no
• Distributed workers: no
• Storing activity input: no
• Failures rewind: no
• Tags: no
• State per execution: no

Configuration

services.AddDurableTaskEmulatorStorage();

LLL.DurableTask.EFCore

Implements relational relational database storage using EFCore.

The implementation uses a combination of row locking, skip locked and polling to implement queues.

Supported features

• UI: yes
• Distributed workers: yes
• Storing activity input: yes
• Failures rewind: yes
• Tags: yes
• State per execution: yes

LLL.DurableTask.EFCore.InMemory

Extension to EFCore storage with queries specific to InMemory database.

Configuration

services.AddDurableTaskEFCoreStorage()
    .UseInMemoryDatabase("DatabaseName");

LLL.DurableTask.EFCore.MySql

Extension to EFCore storage with migrations and queries specific to MySql.

Configuration

services.AddDurableTaskEFCoreStorage()
    .UseMySql("YOUR_CONNECTION_STRING");

LLL.DurableTask.EFCore.PostgreSQL

Extension to EFCore storage with migrations and queries specific to PostgreSQL.

Configuration

services.AddDurableTaskEFCoreStorage()
    .UseNpgsql("YOUR_CONNECTION_STRING");

LLL.DurableTask.EFCore.SqlServer

Extension to EFCore storage with migrations and queries specific to Sql Server.

Configuration

services.AddDurableTaskEFCoreStorage()
    .UseSqlServer("YOUR_CONNECTION_STRING");

LLL.DurableTask.Client

Dependency injection extensions to configure TaskHubClient.

Allows management of orchestrations via code.

Depends on

• Storage

Configuration

services.AddDurableTaskClient();

Usage

public IActionResult BookPackage([FromService] TaskHubClient taskHubClient) {
    await taskHubClient.CreateOrchestrationInstanceAsync("BookParallel", "v1", new {
        bookFlight: true,
        bookHotel: true,
        bookCar: true
    });
    ...
}

LLL.DurableTask.Worker

Dependency injection extensions to configure TaskHubWorker.

Allows execution of orchestration/activity tasks.

A service scope is created for each orchestration and activity execution.

Orchestrations/activities/middlewares supports dependency injection.

Depends on

• Storage

Configuration

services.AddDurableTaskWorker(builder =>
{
    // Add orchestration with default name and version
    builder.AddOrchestration<BookParallel>();

    // Add orchestration with specific name and version
    builder.AddOrchestration<BookParallel>("BookParallel", "v1");

    // Add activity with default name and version
    builder.AddActivity<BookHotelActivity>();

    // Add activity with specific name and version
    builder.AddActivity<BookHotelActivity>("BookHotel", "v1");
});

Or you can also scan an assembly to add all orchestrations and/or activities marked with attributes OrchestrationAttribute or ActivityAttribute:

services.AddDurableTaskWorker(builder =>
{
    // Adds all orchestrations and activities from assembly
    builder.AddFromAssembly(typeof(Startup).Assembly);

    // Add only orchestrations from assembly
    builder.AddOrchestrationsFromAssembly(typeof(Startup).Assembly);

    // Add only activities from assembly
    builder.AddActivitiesFromAssembly(typeof(Startup).Assembly);
});

NOTE: When using storages that doesn't support distributed workers, make sure all your orchestrations and activities are implemented in the same worker and add the following lines to your worker configuration:

services.AddDurableTaskWorker(builder =>
{
    ...
    builder.HasAllOrchestrations = true;
    builder.HasAllActivities = true;
});

LLL.DurableTask.Server

Expose any storage implementation as API.

Allow microservices to connect to an API instead of directly to storage.

Depends on

• Storage

LLL.DurableTask.Server.Grpc

GRPC endpoints for server.

The chatty orchestration execution communication is done with bidirectional streaming, maintaining the orchestration session alive in the server side.

Activity execution and all remaining communication is done with non streamed rpc.

Configuration

services.AddDurableTaskServer(builder =>
{
    builder.AddGrpcEndpoints();
});
...
app.UseEndpoints(endpoints =>
{
    endpoints.MapDurableTaskServerGrpcService();
});

LLL.DurableTask.Server.Grpc.Client

Durable Task storage implementation using server GRPC endpoints.

Supports same features as the storage configured in the server.

Configuration

services.AddDurableTaskServerGrpcStorage(options =>
{
    options.BaseAddress = new Uri("YOUR_SERVER_ADDRESS");
});

LLL.DurableTask.Api

Exposes orchestration management operations in a REST API.

Depends on

• Storage
• Client

Configuration

// Add Durable Task Api services
services.AddDurableTaskApi();
...
app.UseEndpoints(endpoints =>
{
    // Map Durable Task Api endpoints under /api prefix
    // Example of endpoint path: /api/v1/orchestrations
    endpoints.MapDurableTaskApi();
});

Alternatively you can define your own prefix:

app.UseEndpoints(endpoints =>
{
    // Map Durable Task Api endpoints under /tasks-api prefix
    // Example of endpoint path: /tasks-api/v1/orchestrations
    endpoints.MapDurableTaskApi("/tasks-api");
});

The API is integrated by default with ASP.NET Core Authorization Policies. You must configure all Durable task policies and their requirements, like the example below:

services.AddAuthorization(c =>
{
    c.AddPolicy(DurableTaskPolicy.Entrypoint, p => p.RequireAssertion(x => true));
    c.AddPolicy(DurableTaskPolicy.Read, p => p.RequireRole("Reader"));
    c.AddPolicy(DurableTaskPolicy.ReadHistory, p => p.RequireRole("Reader"));
    c.AddPolicy(DurableTaskPolicy.Create, p => p.RequireRole("Administrator"));
    c.AddPolicy(DurableTaskPolicy.Terminate, p => p.RequireRole("Administrator"));
    c.AddPolicy(DurableTaskPolicy.RaiseEvent, p => p.RequireRole("Administrator"));
    c.AddPolicy(DurableTaskPolicy.Purge, p => p.RequireRole("Administrator"));
});

Alternatively, you can disable authorization integration on non production environments:

services.AddDurableTaskApi(options =>
{
    options.DisableAuthorization = true;
});

Cross-Origin Requests (CORS)

CORS configuration is required if you run Durable Task API and Durable Task UI from different domains.

To configure CORS, please follow Enable Cross-Origin Requests (CORS) in ASP.NET Core .

Durable Task API requires http methods: GET, POST, DELETE.

LLL.DurableTask.Ui

Beautifull UI to manage orchestrations built with React + Material UI.

Take a look in the screenshots. History visualization is my favorite :-)

Configuration

services.AddDurableTaskUi(options =>
{
    // Configure Durable Task UI
});
...
// Serve Durable Task Ui files under root path
app.UseDurableTaskUi();

Alternatively, you can define a path to serve the Ui from:

// Serve Durable Task Ui files under path /tasks
app.UseDurableTaskUi("/tasks");

You can configure Durable Task Ui with the following options:

Option	Default value	Description
ApiBaseUrl	"/api"	The base url of Durable Task Api
UserNameClaims	"preferred_username", "name", "sub"	Prioritized claims used to refer to the logged in user
Oidc	null	Object with OIDC integration configuration. OIDC is disabled when null

OIDC/OAuth2 integration

You can enable OIDC integration by configuring OIDC options:

Option	Default value	Description
Authority	null	The URL of the OIDC/OAuth2 provider
ClientId	null	Your client application's identifier as registered with the OIDC/OAuth2 provider
ResponseType	"id_token"	The type of response desired from the OIDC/OAuth2 provider
Scope	"openid"	The scope being requested from the OIDC/OAuth2 provider
Prompt	null	Information sent to IDP during OIDC authorization
Display	null	Information sent to IDP during OIDC authorization
LoadUserInfo	null	Flag to control if additional identity data is loaded from the user info endpoint in order to populate the user's profile.

The redirect_url and post_logout_redirect_uri values are computed automatically from the url used to access Durable Task Ui. You should configure both redirect urls on your OIDC server with the same url you use to access Durable Task Ui.

Compose components to build your own architecture

Microservices with server

Microservices with direct storage connection

Single service

UI for Durable Functions

Build requirements

• .NET 5 SDK
• .NET 3.1 Runtime

Sample

Inside the sample folder you will find an implementation of the classic book Flight, Car, Hotel with compensation problem.

The sample was built to demonstrate a microservices architecture with the following components:

• Server: Connects to storage and exposes it as GRPC endpoints.
• Api: Exposes REST API to manage orchestrations.
• UI: Exposes UI to manage orchestrations.
• OrchestrationWorker: Implements BookParallel and BookSquential orchestrations for the given problem.
• FlightWorker: Implements BookFlight and CancelFlight activities.
• CarWorker: Implements BookCar and CancelCar activities.
• HotelWorker: Implements BookHotel and CancelHotel activities.
• BPMNWorker: An experimental BPMN runner built on top of Durable Tasks. There are also BookParallel and BookSequential BPMN workflows for the given problem.

Runinng the sample

1. Configure a EFCore storage at the server
2. Simultaneously run all the projects listed above
3. Open the UI at https://localhost:5002/
4. Create the following test orchestrations and watch them be executed

   Name	Version	InstanceId	Input
   BookParallel	v1	(Empty)	(Empty)
   BookSequential	v1	(Empty)	(Empty)
   BPMN	(Empty)	(Empty)	{ "name": "BookParallel" }
   BPMN	(Empty)	(Empty)	{ "name": "BookSequential" }
   BPMN	(Empty)	(Empty)	{ "name": "Bonus" }