LINQ Enumerable Extensions Do Not Work With ExpandoObject Collection Properties

Question

LINQ Enumerable Extensions Do Not Work With ExpandoObject Collection Properties

RonAmihai opened this issue 8 months ago · comments

LINQ extensions within expressions do not work when applied to properties of ExpandoObject.
(For comparison - ExpressionEvaluator does support that with OptionInstanceMethodsCallActive)

Given the following code:

dynamic dynamicData = new ExpandoObject();
dynamicData.some = new List<string> { "one", "two", "two", "three" };

var interpreter = new Interpreter(InterpreterOptions.LambdaExpressions).Reference(typeof(Enumerable));
interpreter.SetVariable("data", dynamicData);

// Works since 'Contains' is a method of List
var nonLinqMethodResult = interpreter.Eval("data.some.Contains(\"two\")");

// Throws since 'Any' is an extension method of Enumerable
var linqMethodResult = interpreter.Eval("data.some.Any(x => x == \"two\")");

The following exception is thrown:

Unhandled exception. DynamicExpresso.Exceptions.ParseException: Invalid Operation (at index 30).
 ---> System.InvalidOperationException: Extension node must override the property Expression.NodeType.
   at System.Linq.Expressions.Expression.get_NodeType()
   at System.Dynamic.Utils.ExpressionUtils.RequiresCanRead(Expression expression, String paramName, Int32 idx)
   at System.Linq.Expressions.ExpressionExtension.ValidateDynamicArgument(Expression arg, String paramName, Int32 index)
   at System.Linq.Expressions.ExpressionExtension.MakeDynamic(CallSiteBinder binder, Type returnType, ReadOnlyCollection`1 arguments)
   at DynamicExpresso.Parsing.Parser.ParseDynamicMethodInvocation(Type type, Expression instance, String methodName, Expression[] args)
   at DynamicExpresso.Parsing.Parser.ParseMethodInvocation(Type type, Expression instance, Int32 errorPos, String methodName, TokenId open, String openExpected, TokenId close, String closeExpected)
   at DynamicExpresso.Parsing.Parser.ParseMethodInvocation(Type type, Expression instance, Int32 errorPos, String methodName)
   at DynamicExpresso.Parsing.Parser.ParseMemberAccess(Type type, Expression instance)
   at DynamicExpresso.Parsing.Parser.ParseMemberAccess(Expression instance)
   at DynamicExpresso.Parsing.Parser.ParsePrimary()
   at DynamicExpresso.Parsing.Parser.ParseUnary()
   at DynamicExpresso.Parsing.Parser.ParseMultiplicative()
   at DynamicExpresso.Parsing.Parser.ParseAdditive()
   at DynamicExpresso.Parsing.Parser.ParseShift()
   at DynamicExpresso.Parsing.Parser.ParseTypeTesting()
   at DynamicExpresso.Parsing.Parser.ParseComparison()
   at DynamicExpresso.Parsing.Parser.ParseLogicalAnd()
   at DynamicExpresso.Parsing.Parser.ParseLogicalXor()
   at DynamicExpresso.Parsing.Parser.ParseLogicalOr()
   at DynamicExpresso.Parsing.Parser.ParseConditionalAnd()
   at DynamicExpresso.Parsing.Parser.ParseConditionalOr()
   at DynamicExpresso.Parsing.Parser.ParseConditional()
   at DynamicExpresso.Parsing.Parser.ParseAssignment()
   at DynamicExpresso.Parsing.Parser.ParseExpressionSegment()
   --- End of inner exception stack trace ---
   at DynamicExpresso.Parsing.Parser.ParseExpressionSegment()
   at DynamicExpresso.Parsing.Parser.ParseExpressionSegment(Type returnType)
   at DynamicExpresso.Parsing.Parser.Parse()
   at DynamicExpresso.Parsing.Parser.Parse(ParserArguments arguments)
   at DynamicExpresso.Interpreter.ParseAsLambda(String expressionText, Type expressionType, Parameter[] parameters)
   at DynamicExpresso.Interpreter.Parse(String expressionText, Type expressionType, Parameter[] parameters)
   at DynamicExpresso.Interpreter.Eval(String expressionText, Type expressionType, Parameter[] parameters)
   at DynamicExpresso.Interpreter.Eval(String expressionText, Parameter[] parameters)
   at Program.<Main>$(String[] args) in /Users/ronam/Projects/ConsoleApp1/ConsoleApp1/Program.cs:line 16

Ron Amihai · Answer 1 · Sun Feb 11 2024 01:29:59 GMT+0800 (China Standard Time)

Currently, as a temporary solution (until implementing proper dynamic LINQ support), I've solved that using LINQ extension methods for object.

Then, by using the extensions below, one can define the interpreter as follows to enable dynamic LINQ functionality:

var interpreter = new Interpreter(InterpreterOptions.LateBindObject | InterpreterOptions.LambdaExpressions)
    .Reference(typeof(DynamicLinqExtensions));

Notes:

LateBindObject is required for nested dynamic fields support, not for the LINQ functionality.
I've not implemented Cast / OfType extensions (they are not mandatory for basic functionality since every operation performs cast to IEnumerable<object?> internally).
Although I've implemented ThenBy / ThenByDescending extensions, they are not mandatory. I've added them to avoid needing an additional typeof(Enumerable) interpreter reference.
Scenarios like data.nested.list.Where(...).ToHashSet().SetEquals(new T[] { ... } ) will require adding .AsEnumerable() to the SetEquals method parameter initialization (SetEquals in that example is an instance method, not an extension method, and we cannot re-define it's behavior)

Implementation:

DynamicLinqExtensions

public static class DynamicLinqExtensions
{
  private static IEnumerable<object?> AsEnumerable(this object? source) => source is IEnumerable enumerable
      ? enumerable.Cast<object?>()
      : throw new InvalidCastException($"Type '{source?.GetType().ToString() ?? "null"}' is not enumerable");

  private static object? Aggregate(this object source, Func<object?, object?, object?> func) =>
      Enumerable.Aggregate(source.AsEnumerable(), func);

  private static object? Aggregate(this object source, object? seed, Func<object?, object?, object?> func,
      Func<object?, object?>? resultSelector = null) =>
      resultSelector is null
          ? Enumerable.Aggregate(source.AsEnumerable(), seed, func)
          : Enumerable.Aggregate(source.AsEnumerable(), seed, func, resultSelector);

  private static bool Any(this object source, Func<object?, bool>? predicate = null) =>
      predicate is null
          ? Enumerable.Any(source.AsEnumerable())
          : Enumerable.Any(source.AsEnumerable(), predicate);

  private static bool All(this object source, Func<object?, bool> predicate) =>
      Enumerable.All(source.AsEnumerable(), predicate);

  private static IEnumerable<object?> Append(this object source, object? element) =>
      Enumerable.Append(source.AsEnumerable(), element);

  private static IEnumerable<object?> Prepend(this object source, object? element) =>
      Enumerable.Prepend(source.AsEnumerable(), element);

  private static double? Average(this object source, Func<object?, double?>? selector = null) =>
      selector is null
          ? Enumerable.Average(source.AsEnumerable(), AsNumeric)
          : Enumerable.Average(source.AsEnumerable(), selector);

  private static IEnumerable<object?[]> Chunk(this object source, int size) =>
      Enumerable.Chunk(source.AsEnumerable(), size);

  private static IEnumerable<object?> Concat(this object first, object? second) =>
      Enumerable.Concat(first.AsEnumerable(), second.AsEnumerable());

  private static bool Contains(this object source, object? value) =>
      Enumerable.Contains(source.AsEnumerable(), value);

  private static int Count(this object source, Func<object?, bool>? predicate = null) =>
      predicate is null
          ? Enumerable.Count(source.AsEnumerable())
          : Enumerable.Count(source.AsEnumerable(), predicate);

  private static long LongCount(this object source, Func<object?, bool>? predicate = null) =>
      predicate is null
          ? Enumerable.LongCount(source.AsEnumerable())
          : Enumerable.LongCount(source.AsEnumerable(), predicate);

  private static bool TryGetNonEnumeratedCount(this object source, out int count) =>
      Enumerable.TryGetNonEnumeratedCount(source.AsEnumerable(), out count);

  private static IEnumerable<object?> DefaultIfEmpty(this object source, object? defaultValue = null) =>
      Enumerable.DefaultIfEmpty(source.AsEnumerable(), defaultValue);

  private static IEnumerable<object?> Distinct(this object source) =>
      Enumerable.Distinct(source.AsEnumerable());

  private static IEnumerable<object?> DistinctBy(this object source, Func<object?, object?> keySelector) =>
      Enumerable.DistinctBy(source.AsEnumerable(), keySelector);

  private static object? ElementAt(this object source, int index) =>
      Enumerable.ElementAt(source.AsEnumerable(), index);

  private static object? ElementAtOrDefault(this object source, int index) =>
      Enumerable.ElementAtOrDefault(source.AsEnumerable(), index);

  private static IEnumerable<object?> Except(this object first, object second) =>
      Enumerable.Except(first.AsEnumerable(), second.AsEnumerable());

  private static IEnumerable<object?> ExceptBy(this object first, object second, Func<object?, object?> keySelector) =>
      Enumerable.ExceptBy(first.AsEnumerable(), second.AsEnumerable(), keySelector);

  private static object? First(this object source, Func<object?, bool>? predicate = null) =>
      predicate is null
          ? Enumerable.First(source.AsEnumerable())
          : Enumerable.First(source.AsEnumerable(), predicate);

  private static object? FirstOrDefault(this object source, object? defaultValue) =>
      Enumerable.FirstOrDefault(source.AsEnumerable(), defaultValue);

  private static object? FirstOrDefault(this object source, Func<object?, bool>? predicate = null, object? defaultValue = null) =>
      predicate is null
          ? Enumerable.FirstOrDefault(source.AsEnumerable(), defaultValue)
          : Enumerable.FirstOrDefault(source.AsEnumerable(), predicate, defaultValue);

  private static IEnumerable<IGrouping<object?, object?>> GroupBy(this object source,
      Func<object?, object?> keySelector, Func<object?, object?>? elementSelector = null) =>
      elementSelector is null
          ? Enumerable.GroupBy(source.AsEnumerable(), keySelector)
          : Enumerable.GroupBy(source.AsEnumerable(), keySelector, elementSelector);

  private static IEnumerable<object?> GroupBy(this object source,
      Func<object?, object?> keySelector, Func<object?, IEnumerable<object?>, object?> resultSelector) =>
      Enumerable.GroupBy(source.AsEnumerable(), keySelector, resultSelector);

  private static IEnumerable<object?> GroupBy(this object source, Func<object?, object?> keySelector, Func<object?,
      object?> elementSelector, Func<object?, IEnumerable<object?>, object?> resultSelector) =>
      Enumerable.GroupBy(source.AsEnumerable(), keySelector, elementSelector, resultSelector);

  private static IEnumerable<object?> GroupJoin(this object outer, object inner, Func<object?, object?> outerKeySelector,
      Func<object?, object?> innerKeySelector, Func<object?, IEnumerable<object?>, object?> resultSelector) =>
      Enumerable.GroupJoin(outer.AsEnumerable(), inner.AsEnumerable(), outerKeySelector, innerKeySelector, resultSelector);

  private static IEnumerable<object?> Intersect(this object first, object second) =>
      Enumerable.Intersect(first.AsEnumerable(), second.AsEnumerable());

  private static IEnumerable<object?> IntersectBy(this object first, object second, Func<object?, object?> keySelector) =>
      Enumerable.IntersectBy(first.AsEnumerable(), second.AsEnumerable(), keySelector);

  private static IEnumerable<object?> Join(this object outer, object inner, Func<object?, object?> outerKeySelector,
      Func<object?, object?> innerKeySelector, Func<object?, object?, object?> resultSelector) =>
      Enumerable.Join(outer.AsEnumerable(), inner.AsEnumerable(), outerKeySelector, innerKeySelector, resultSelector);

  private static object? Last(this object source, Func<object?, bool>? predicate = null) =>
      predicate is null
          ? Enumerable.Last(source.AsEnumerable())
          : Enumerable.Last(source.AsEnumerable(), predicate);

  private static object? LastOrDefault(this object source, object? defaultValue) =>
      Enumerable.LastOrDefault(source.AsEnumerable(), defaultValue);

  private static object? LastOrDefault(this object source, Func<object?, bool>? predicate = null, object? defaultValue = null) =>
      predicate is null
          ? Enumerable.LastOrDefault(source.AsEnumerable(), defaultValue)
          : Enumerable.LastOrDefault(source.AsEnumerable(), predicate, defaultValue);

  private static ILookup<object, object?> ToLookup(this object source,
      Func<object?, object> keySelector, Func<object?, object?>? elementSelector = null) =>
      elementSelector is null
          ? Enumerable.ToLookup(source.AsEnumerable(), keySelector)
          : Enumerable.ToLookup(source.AsEnumerable(), keySelector, elementSelector);

  private static double? Max(this object source, Func<object?, double?>? selector = null) =>
      selector is null
          ? Enumerable.Max(source.AsEnumerable(), AsNumeric)
          : Enumerable.Max(source.AsEnumerable(), selector);

  private static object? MaxBy(this object source, Func<object?, double?> keySelector) =>
      Enumerable.MaxBy(source.AsEnumerable(), keySelector);

  private static double? Min(this object source, Func<object?, double?>? selector = null) =>
      selector is null
          ? Enumerable.Min(source.AsEnumerable(), AsNumeric)
          : Enumerable.Min(source.AsEnumerable(), selector);

  private static object? MinBy(this object source, Func<object?, double?> keySelector) =>
      Enumerable.MinBy(source.AsEnumerable(), keySelector);

  private static IOrderedEnumerable<object?> Order(this object source) =>
      Enumerable.Order(source.AsEnumerable());

  private static IOrderedEnumerable<object?> OrderBy(this object source, Func<object?, object?> keySelector) =>
      Enumerable.OrderBy(source.AsEnumerable(), keySelector);

  private static IOrderedEnumerable<object?> OrderDescending(this object source) =>
      Enumerable.OrderDescending(source.AsEnumerable());

  private static IOrderedEnumerable<object?> OrderByDescending(this object source, Func<object?, object?> keySelector) =>
      Enumerable.OrderByDescending(source.AsEnumerable(), keySelector);

  private static IOrderedEnumerable<object?> ThenBy(this IOrderedEnumerable<object?> source, Func<object?, object?> keySelector) =>
      Enumerable.ThenBy(source, keySelector);

  private static IOrderedEnumerable<object?> ThenByDescending(this IOrderedEnumerable<object?> source,
      Func<object?, object?> keySelector) =>
      Enumerable.ThenByDescending(source, keySelector);

  private static IEnumerable<object?> Reverse(this object source) =>
      Enumerable.Reverse(source.AsEnumerable());

  private static IEnumerable<object?> Select(this object source, Func<object?, object?> selector) =>
      Enumerable.Select(source.AsEnumerable(), selector);

  private static IEnumerable<object?> Select(this object source, Func<object?, int, object?> selector) =>
      Enumerable.Select(source.AsEnumerable(), selector);

  private static IEnumerable<object?> SelectMany(this object source, Func<object?, IEnumerable<object?>> selector) =>
      Enumerable.SelectMany(source.AsEnumerable(), selector);

  private static IEnumerable<object?> SelectMany(this object source, Func<object?, int, IEnumerable<object?>> selector) =>
      Enumerable.SelectMany(source.AsEnumerable(), selector);

  private static IEnumerable<object?> SelectMany(this object source,
      Func<object?, IEnumerable<object?>> collectionSelector, Func<object?, object?, object?> resultSelector) =>
      Enumerable.SelectMany(source.AsEnumerable(), collectionSelector, resultSelector);

  private static IEnumerable<object?> SelectMany(this object source,
      Func<object?, int, IEnumerable<object?>> collectionSelector, Func<object?, object?, object?> resultSelector) =>
      Enumerable.SelectMany(source.AsEnumerable(), collectionSelector, resultSelector);

  private static bool SequenceEqual(this object first, object second) =>
      Enumerable.SequenceEqual(first.AsEnumerable(), second.AsEnumerable());

  private static object? Single(this object source, Func<object?, bool>? predicate = null) =>
      predicate is null
          ? Enumerable.Single(source.AsEnumerable())
          : Enumerable.Single(source.AsEnumerable(), predicate);

  private static object? SingleOrDefault(this object source, object? defaultValue) =>
      Enumerable.SingleOrDefault(source.AsEnumerable(), defaultValue);

  private static object? SingleOrDefault(this object source, Func<object?, bool>? predicate = null, object? defaultValue = null) =>
      predicate is null
          ? Enumerable.SingleOrDefault(source.AsEnumerable(), defaultValue)
          : Enumerable.SingleOrDefault(source.AsEnumerable(), predicate, defaultValue);

  private static IEnumerable<object?> Skip(this object source, int count) =>
      Enumerable.Skip(source.AsEnumerable(), count);

  private static IEnumerable<object?> SkipWhile(this object source, Func<object?, int, bool> predicate) =>
      Enumerable.SkipWhile(source.AsEnumerable(), predicate);

  private static IEnumerable<object?> SkipWhile(this object source, Func<object?, bool> predicate) =>
      Enumerable.SkipWhile(source.AsEnumerable(), predicate);

  private static IEnumerable<object?> SkipLast(this object source, int count) =>
      Enumerable.SkipLast(source.AsEnumerable(), count);

  private static double? Sum(this object source, Func<object?, double?>? selector = null) =>
      selector is null
          ? Enumerable.Sum(source.AsEnumerable(), AsNumeric)
          : Enumerable.Sum(source.AsEnumerable(), selector);

  private static IEnumerable<object?> Take(this object source, int count) =>
      Enumerable.Take(source.AsEnumerable(), count);

  private static IEnumerable<object?> TakeLast(this object source, int count) =>
      Enumerable.TakeLast(source.AsEnumerable(), count);

  private static IEnumerable<object?> TakeWhile(this object source, Func<object?, bool> predicate) =>
      Enumerable.TakeWhile(source.AsEnumerable(), predicate);

  private static IEnumerable<object?> TakeWhile(this object source, Func<object?, int, bool> predicate) =>
      Enumerable.TakeWhile(source.AsEnumerable(), predicate);

  private static object?[] ToArray(this object source) =>
      Enumerable.ToArray(source.AsEnumerable());

  private static List<object?> ToList(this object source) =>
      Enumerable.ToList(source.AsEnumerable());

  private static Dictionary<object, object?> ToDictionary(this object source, Func<object?, object> keySelector,
      Func<object?, object?>? elementSelector = null) =>
      elementSelector is null
          ? Enumerable.ToDictionary(source.AsEnumerable(), keySelector)
          : Enumerable.ToDictionary(source.AsEnumerable(), keySelector, elementSelector);

  private static HashSet<object?> ToHashSet(this object source) =>
      Enumerable.ToHashSet(source.AsEnumerable());

  private static IEnumerable<object?> Union(this object first, object second) =>
      Enumerable.Union(first.AsEnumerable(), second.AsEnumerable());

  private static IEnumerable<object?> UnionBy(this object first, object second, Func<object?, object?> keySelector) =>
      Enumerable.UnionBy(first.AsEnumerable(), second.AsEnumerable(), keySelector);

  private static IEnumerable<object?> Where(this object source, Func<object?, bool> predicate) =>
      Enumerable.Where(source.AsEnumerable(), predicate);

  private static IEnumerable<object?> Where(this object source, Func<object?, int, bool> predicate) =>
      Enumerable.Where(source.AsEnumerable(), predicate);

  private static IEnumerable<object?> Zip(this object first, object second, Func<object?, object?, object?> resultSelector) =>
      Enumerable.Zip(first.AsEnumerable(), second.AsEnumerable(), resultSelector);

  private static IEnumerable<(object? First, object? Second)> Zip(this object first, object second) =>
      Enumerable.Zip(first.AsEnumerable(), second.AsEnumerable());

  private static IEnumerable<(object? First, object? Second, object? Third)> Zip(this object first, object second, object third) =>
      Enumerable.Zip(first.AsEnumerable(), second.AsEnumerable(), third.AsEnumerable());

  private static double? AsNumeric(object? item) => item is null ? null : Convert.ToDouble(item);
}

Christophe PLAT · Answer 2 · Sun Aug 18 2024 16:22:36 GMT+0800 (China Standard Time)

It's worth noting that the C# compiler doesn't allow it either:

dynamic dynamicData = new ExpandoObject();
dynamicData.Some = new List<string> { "one", "two", "two", "three" };

var result = dynamicData.Some.Any(x => x == "two");

raises compiler error

error CS1977: Cannot use a lambda expression as an argument to a dynamically dispatched operation
 without first casting it to a delegate or expression tree type.

Davide Icardi · Answer 3 · Mon Aug 19 2024 17:16:57 GMT+0800 (China Standard Time)

If C# compiler doesn't allow this scenario, I think we can ignore it.
@RonAmihai using ExpandoObject really necessary in your scenario? Maybe you can create some custom type. This will also improve performance and type safety.

Ron Amihai · Answer 4 · Tue Aug 20 2024 18:48:54 GMT+0800 (China Standard Time)

@davideicardi I've ended up implementing an optimized custom expression tree compiler for my exact scenario (object with nested Dictionary<string, object>).

I agree that ExpandoObject should be avoided in general. However, in cases where it can't be avoided, LINQ support can be an optional feature—not that mandatory, though.