[Feature Request] Change Ref approach to expression-based

[Chad Retz]

Describe the solution you'd like

Akin to Hangfire and others, use LINQ expressions to let calls be made. This is more inline with how other .NET libraries work. We need to make sure status still work and we can extract return types in Task vs non-Task. This will also require updating docs and the blog post.

Also, as part of this effort, we need to modernize registration of activities and workflows. We probably need to:

• Make definitions the only collections on worker options instead of the "additional"
• Allow activities to also be registered by entire type/class that crawls looking for activity attributed
• Abstract activity definition a tad to accept a form with arg types, return type, and a Func<object?, Task<object?[]?>> invoker. Put an InvokeAsync on it I guess. This is needed for dependency injection support. Would accept alternative activity class factory approach if we can think of it, though this is probably good enough. Can expose this all the way out to the AddActivity overload in the worker options.

[Chris Gillum]

Make definitions the only collections on worker options instead of the "additional"

Would there still be a way to register dynamic activities - i.e. without requiring an attribute?

[Chad Retz]

Yes. That already exists today with ActivityDefinition.CreateWithoutAttribute. I will continue to support that, but probably not explicitly via a direct overload on worker options because I don't want to encourage it. I'm thinking on worker options, my adders are:

• AddActivityType<TActivityClass>()
• AddActivityType(Type)
• AddActivity(Delegate)
• AddActivity(MemberExpression) (will have to investigate this, may not be right)
• AddActivity(ActivityDefinition)

And basically all end up calling that last one for definition. And on ActivityDefinition I'll have:

• CreateAllFromType(Type)
• CreateFromDelegate(Delegate)
• CreateWithoutAttribute(string, Delegate)
• CreateWithoutAttribute(string name, Type[] argTypes, Type returnType, Func<object?, object?[]?> invoke)

And basically all things end up calling that last one. May be another in here for expression-based. I may just name these CreateAll and Create with overloads, I just have to decide if I want these unwieldy names on purpose to discourage this approach of activity definition creation.

[Chad Retz]

Posting what I posted in #dotnet-sdk on Slack:

I am struggling with type inference and expression trees. With the current approach you can have something like this:

var temp1 = await Workflow.ExecuteActivityAsync(Foo.StaticWithArgAndResult, 123);

var temp2 = await Workflow.ExecuteActivityAsync(Foo.Ref.InstanceWithArgAndResult, 123);

Nice and simple for static and non-static, but with the expression approach, you cannot have:

var temp2 = await Workflow.ExecuteActivityAsync<Foo>(foo => foo.InstanceWithArgAndResult(123));

Because I need the return type and C# doesn't allow partial generic inference. I can either do:

var temp2 = await Workflow.ExecuteActivityAsync<Foo, string>(foo => foo.InstanceWithArgAndResult(123));

Or

var temp2 = await Workflow.ExecuteActivityAsync((Foo foo) => foo.InstanceWithArgAndResult(123));

Both of which are quite ugly compared to the original. I could also break it into two like:

var temp2 = await Workflow.ActivityReference<Foo>().ExecuteAsync(foo => foo.InstanceWithArgAndResult(123));

But that's also ugly (and it makes static calls be way different than non-static). Does anyone have any better ideas or better preference? Are there any frameworks that deal with return types from the lambdas in expression trees that I can look to for inspiration? Do they also allow static and non-static? Do they also split into two steps?

As I work with it, it becomes fairly obvious that expression trees result in harder to read code. I am considering supporting them but leaving the "Ref" pattern around/recommended for this reason. But would love to find a way not to have two separate approaches.

[Chad Retz]

Ok, did some discussion with team. I am going to open a PR all-in on the expression approach (assuming I don't hit snags with inference). We can evaluate that then. That means you actually can't use the delegate/ref pattern if you wanted to (sorry!). So basically:

var temp1 = await Workflow.ExecuteActivityAsync(Foo.StaticWithArgAndResult, 123);

var temp2 = await Workflow.ExecuteActivityAsync(Foo.Ref.InstanceWithArgAndResult, 123);

Becomes:

var temp1 = await Workflow.ExecuteActivityAsync(() => Foo.StaticWithArgAndResult(123));

var temp2 = await Workflow.ExecuteActivityAsync((Foo foo) => foo.InstanceWithArgAndResult(123));

With the removal of the dependency, familiarity to .NET devs, and that we (really really) hate two ways of doing things, this just won out. But we'll reevaluate during PR time if I hit any snags.

[Chad Retz]

Ok, could use some more feedback here (x-posted to #dotnet-sdk on Slack). If I have:

await Workflow.ExecuteActivityAsync((Foo foo) => foo.SomeMethod(123 + 456));

I need to evaluate that argument. From what I read, that means I have to use an approach like https://stackoverflow.com/a/60002882 (basically create a lambda out of each argument and compile at call time) which is really slow. Even Compile(true) available in all but our oldest supported version (framework 4.6) is slow. If you had passed:

await Workflow.ExecuteActivityAsync((Foo foo) => foo.SomeMethod(579));

Then I could just use ConstantExpression.Value, but that is a non-obvious optimization to users to make sure they create their parameter outside the closure. And .NET can't infer params for something like:

await Workflow.ExecuteActivityAsync((Foo foo) => foo.SomeMethod, 123 + 456);

Looking at hangfire and others, they have elaborate evaluators and will cache some known expression tree patterns resorting to Compile() as needed.

Options (not all are mutually exclusive):

1. Abandon expression trees - pro: good performance, con: not as common in ecosystem
2. Accept the performance hit of compiling non-constant expressions for arguments of workflows/activities on each invocation - pro: easy to read, con: bad performance
3. Put in the effort for matching some common pattern types - pro: better performance in some cases, con: lots of code and probably won't support common use case of record instantiation (also https://learn.microsoft.com/en-us/dotnet/csharp/advanced-topics/expression-trees/expression-trees-execution#execution-and-lifetimes discourages caching saying equality/identity checks are more expensive than compile)
4. Disallow in-lambda evaluation (i.e. require all arguments be ConstantExpression provided outside the lambda) - pro: good performance, con: runtime only and encourages potentially uglier code
5. Warn on in-lambda evaluation - pro: encourages good performance, con: noisy, non-obvious, encourages potentially uglier code
6. Use something like https://github.com/dadhi/FastExpressionCompiler - pro: better performance, con: new dependency, some expressions don't work
7. Another option I am unaware of?

Thoughts? I didn't put "allow expression trees and ref pattern" above because we just can't reasonably have two approaches. I may do option 2 for now while implementing, but option 1 sure looks tempting and I am hoping I don't have to maintain two approaches just so I can benchmark differences.