OData + EfCore Filter dynamic properties
Thomas5x01 opened this issue · comments
Versions used
- "Microsoft.AspNetCore.OData" Version="8.2.5"
- "Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.4"
You can find a git repo at:
https://github.com/Thomas5x01/ODataEfCoreFilterDynamicProps
Describe the issue / bug
Applying a filter to a dynamic property of an open type causes an exception. E.g. https://localhost:7054/persondtos?$filter=LikesFootball eq false
causes
System.ArgumentException
HResult=0x80070057
Message = Method 'System.Object get_Item(System.String)' declared on type System.Collections.Generic.Dictionary2[System.String,System.Object]'
cannot be called with instance of type 'System.Object'
Source = System.Linq.Expressions
Stack:
at System.Linq.Expressions.Expression.ValidateCallInstanceType(Type instanceType, MethodInfo method)
(long StackTrace is at the bottom)
Reproduce steps
- Clone the repo
- Change Database Connection string in
Program.cs:21 - Run the app
- Request PersonDtos with filter:
GET https://localhost:7054/persondtos?$filter=LikesFootball eq false
Data Model
PersonEntity is used as an Entity for EfCore
public class PersonEntity
{
public long Id { get; set; }
public string Name { get; set; }
// EfCore expects a type that provides an indexer property
// https://learn.microsoft.com/en-Us/ef/core/modeling/shadow-properties#configuring-indexer-properties
// OData does not recognize PersonEntity as an open type
private readonly Dictionary<string, object> _data = new Dictionary<string, object>();
public object this[string key]
{
get => _data[key];
get => _data[key] = value;
}
}PersonDto is exposed as an EntityType for OData.
While processing a request, PersonEntity is projected into PersonDto.
public class PersonDto
{
public long Id { get; set; }
public string Name { get; set; }
// OData recognizes PersonDto as an open type because it owns Dictionary<string, object> Fields
private Dictionary<string, object> _Fields = new Dictionary<string, object>();
public Dictionary<string, object> Fields
{
get { return _Fields; }
set { _Fields = value; }
}
}EDM (CSDL) Model
<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
<Schema Namespace="ODataDynamicProperties.Dto" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EntityType Name="PersonDto" OpenType="true">
<Key>
<PropertyRef Name="id" />
</Key>
<Property Name="id" Type="Edm.Int64" Nullable="false" />
<Property Name="name" Type="Edm.String" Nullable="false" />
</EntityType>
</Schema>
<Schema Namespace="Default" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EntityContainer Name="Container">
<EntitySet Name="PersonDtos" EntityType="ODataDynamicProperties.Dto.PersonDto" />
</EntityContainer>
</Schema>
</edmx:DataServices>
</edmx:Edmx>Request/Response
(see ODataDynamicProperties.http in the repo)
https://github.com/Thomas5x01/ODataEfCoreFilterDynamicProps/blob/main/ODataDynamicProperties/ODataDynamicProperties.http
Request without filtering works
GET {{ODataDynamicProperties_HostAddress}}/personDtos
Accept: application/json
Request filtering a hardcoded property works
GET {{ODataDynamicProperties_HostAddress}}/personDtos?$filter=id eq 1
Accept: application/json
SQL-Command that is sent by EfCore:
--Executed DbCommand (1ms) [Parameters=[@__TypedProperty_0='1' (DbType = Int64)], CommandType='Text', CommandTimeout='30']
SELECT [p].[Id], [p].[Name], [p].[Birthday], [p].[LikesFootball]
FROM [Persons] AS [p]
WHERE [p].[Id] = @__TypedProperty_0Resulting JSON Response
[
{
"id": 1,
"name": "Max",
"fields": {
"Birthday": "2000-01-01T00:00:00",
"LikesFootball": false
}
}
]==Request filtering a dynamic property fails==
GET {{ODataDynamicProperties_HostAddress}}/personDtos?$filter=LikesFootball eq false
Accept: application/json
System.ArgumentException: Method 'System.Object get_Item(System.String)' declared on type System.Collections.Generic.Dictionary`2[System.String,System.Object]' cannot be called with instance of type 'System.Object'
Filter by dynamic property with EfCore works
(see PersonDtosController.Get "a small test to narrow down the problem")
List<PersonEntity> personEntityList = _DbContext.Persons
.Where(x => EF.Property<bool>(x, "LikesFootball") == true)
.ToList();Expected behavior
Our OData based web Api should be able to filter DTOs based on the value of fields / properties that are unknown at compile time / dynamic properties.
As far as I have understood, the behavior of EfCore and OData is not compatible to enable the desired behavior and a "little bridge" is needed. A sample to adress this kind of problem is described here
https://devblogs.microsoft.com/odata/customizing-filter-for-spatial-data-in-asp-net-core-odata-8/
I don't know if the OData team can solve this problem in general way to provide a solution of the box. That of course would be the best solution for me :).
Maybe at least some more advice how to write a custom FilterBinder would be good: Is it possible to write a query by hand that can be translated (see commented Where() calls)?
Maybe based on the expression, that is produced by that query, one can write a custom FilterBinder.
//from PersonDtoController
IQueryable<PersonDto> personDtosQuery = _DbContext.Persons
.Select(entity => new PersonDto()
{
Id = entity.Id,
Name = entity.Name,
Fields = new Dictionary<string, object>
{
{ "Birthday", entity["Birthday"] },
{ "LikesFootball", entity["LikesFootball"] }
}
}
);
// each of these where clauses fails
List<PersonDto> personDtoList = personDtosQuery
//.Where(x => x.Fields.ContainsKey("Birthday"))
//.Wherex => EF.Property<bool>(x, "LikesFootball") == true)
//.Where(x => EF.Property<bool>(x.Fields, "LikesFootball") == true)
.ToList();
Related issues and bugs
Filtering / Grouping on dynamic properties does not translate to valid SQL
(#689)
Exception when using filter associated with a Dictionary<string,object>
(#890)
This issue is quite similar but does not involve a projection from entity to dto in the screenshot of the solution. Propsal 1 to load data into memory is not an viaable option for our scenario.
Additional context
System.ArgumentException: Method 'System.Object get_Item(System.String)' declared on type 'System.Collections.Generic.Dictionary`2[System.String,System.Object]' cannot be called with instance of type 'System.Object'
at System.Linq.Expressions.Expression.ValidateCallInstanceType(Type instanceType, MethodInfo method)
at System.Linq.Expressions.Expression.ValidateAccessor(Expression instance, MethodInfo method, ParameterInfo[] indexes, ReadOnlyCollection`1& arguments, String paramName)
at System.Linq.Expressions.Expression.ValidateIndexedProperty(Expression instance, PropertyInfo indexer, String paramName, ReadOnlyCollection`1& argList)
at System.Linq.Expressions.Expression.MakeIndex(Expression instance, PropertyInfo indexer, IEnumerable`1 arguments)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitConditional(ConditionalExpression conditionalExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitUnary(UnaryExpression unaryExpression)
at Microsoft.EntityFrameworkCore.SqlServer.Query.Internal.SqlServerSqlTranslatingExpressionVisitor.VisitUnary(UnaryExpression unaryExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
at Microsoft.EntityFrameworkCore.SqlServer.Query.Internal.SqlServerSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.TranslateInternal(Expression expression, Boolean applyDefaultTypeMapping)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.Translate(Expression expression, Boolean applyDefaultTypeMapping)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateExpression(Expression expression, Boolean applyDefaultTypeMapping)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateLambdaExpression(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateWhere(ShapedQueryExpression source, LambdaExpression predicate)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.System.Collections.IEnumerable.GetEnumerator()
at ODataDynamicProperties.Controllers.PersonDtosController.Get(ODataQueryOptions`1 queryOptions) in C:\Repo\Git\Thomas5x01\ODataEfCoreFilterDynamicProps\ODataDynamicProperties\Controllers\PersonDtosController.cs:line 78
at lambda_method55(Closure, Object, Object[])
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
If you do the 'Where' clause first then do the 'Select', it should work as:
Here's the SQL (I switched to use SQLLite for my simplicity)
If you switch the order, EF core can't understand the 'type' in 'Where' clause? I am not sure the real root cause, but that's the behaviour.
My suggestion is to use one POCO class (Merge PersonDto and PersonEntity), make some properties for DB side, other properties for OData/Client side. Then, a request containing query option coming, customize the FilterBinder to use the DB side properties to build the Expression and generate the SQL to execute. You can refer to this sample: https://github.com/xuzhg/mydotnetconf/blob/main/OData/OData.WebApi/Models/School.cs
If the above cannot work for you, you can build two Edm models, one using PersonDto for client, the other using PersonEntity for DBSide, Then, a request containing query option coming, customize the FilterBinder to use the Edm Model using PersonEntity to generate the Expression and generate the SQL, etc.
Hope it can help.
Thank you, xuzhg. I'll try out you proposals.