Dashboard api request failed (Fielddata is disabled on [stack_id])
yang-xiaodong opened this issue · comments
Hello team,
I have deployed a new self-hosted environment with the latest versions of Elasticsearch and Exceptionless. I encountered the following error when accessing the dashboard.
Elasticsearch Version: 8.8.2
Exceptionless Version: 8.0.1
Api Request:
/api/v2/events?filter=type:error+(status:open+OR+status:regressed)&limit=15&mode=stack_frequent&offset=480m&time=last+24+hours
Server Error:
[00:50:48 ERR] Unhandled error: Error while counting
Original: [ElasticsearchClientException] Request failed to execute. Call: Status code 400 from: POST /prod-events-2023.07.17%2Cprod-events-2023.07.18/_search?typed_keys=true&ignore_unavailable=true. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" CausedBy: "Type: illegal_argument_exception Reason: "Fielddata is disabled on [stack_id] in [prod-events-2023.07.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [stack_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory." CausedBy: "Type: illegal_argument_exception Reason: "Fielddata is disabled on [stack_id] in [prod-events-2023.07.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [stack_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."""
Server Error (Index=): all shards failed
[400] POST /prod-events-2023.07.17%2Cprod-events-2023.07.18/_search?typed_keys=true&ignore_unavailable=true
{"track_total_hits":true,"aggs":{"terms_stack_id":{"aggs":{"min_date":{"meta":{"@field_type":"date"},"min":{"field":"date"}},"max_date":{"max":{"field":"date"},"meta":{"@field_type":"date"}},"sum_count":{"meta":{"@field_type":"integer"},"sum":{"field":"count","missing":1.0}},"cardinality_user":{"cardinality":{"field":"data.@user.identity.keyword"}}},"meta":{"@field_type":"keyword"},"terms":{"field":"stack_id","order":[{"sum_count":"desc"}],"size":16}}},"query":{"bool":{"filter":[{"bool":{"must":[{"range":{"date":{"gte":"2023-07-17T00:00:00Z","lte":"2023-07-18T00:50:48.5233642Z"}}},{"match":{"type":{"query":"error"}}},{"term":{"organization_id":{"value":"64b55f3690a34700013ea954"}}},{"range":{"date":{"gte":"now/d-4d","lte":"now/d+1d"}}},{"term":{"stack_id":{"value":"64b55f6b949184000f5628ba"}}}]}}]}},"size":0,"_source":{"excludes":["idx","ip","os","error"]}}
Foundatio.Repositories.Exceptions.DocumentException: Error while counting
Original: [ElasticsearchClientException] Request failed to execute. Call: Status code 400 from: POST /prod-events-2023.07.17%2Cprod-events-2023.07.18/_search?typed_keys=true&ignore_unavailable=true. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" CausedBy: "Type: illegal_argument_exception Reason: "Fielddata is disabled on [stack_id] in [prod-events-2023.07.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [stack_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory." CausedBy: "Type: illegal_argument_exception Reason: "Fielddata is disabled on [stack_id] in [prod-events-2023.07.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [stack_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."""
Server Error (Index=): all shards failed
[400] POST /prod-events-2023.07.17%2Cprod-events-2023.07.18/_search?typed_keys=true&ignore_unavailable=true
{"track_total_hits":true,"aggs":{"terms_stack_id":{"aggs":{"min_date":{"meta":{"@field_type":"date"},"min":{"field":"date"}},"max_date":{"max":{"field":"date"},"meta":{"@field_type":"date"}},"sum_count":{"meta":{"@field_type":"integer"},"sum":{"field":"count","missing":1.0}},"cardinality_user":{"cardinality":{"field":"data.@user.identity.keyword"}}},"meta":{"@field_type":"keyword"},"terms":{"field":"stack_id","order":[{"sum_count":"desc"}],"size":16}}},"query":{"bool":{"filter":[{"bool":{"must":[{"range":{"date":{"gte":"2023-07-17T00:00:00Z","lte":"2023-07-18T00:50:48.5233642Z"}}},{"match":{"type":{"query":"error"}}},{"term":{"organization_id":{"value":"64b55f3690a34700013ea954"}}},{"range":{"date":{"gte":"now/d-4d","lte":"now/d+1d"}}},{"term":{"stack_id":{"value":"64b55f6b949184000f5628ba"}}}]}}]}},"size":0,"_source":{"excludes":["idx","ip","os","error"]}}
---> Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: POST /prod-events-2023.07.17%2Cprod-events-2023.07.18/_search?typed_keys=true&ignore_unavailable=true. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" CausedBy: "Type: illegal_argument_exception Reason: "Fielddata is disabled on [stack_id] in [prod-events-2023.07.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [stack_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory." CausedBy: "Type: illegal_argument_exception Reason: "Fielddata is disabled on [stack_id] in [prod-events-2023.07.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [stack_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."""
--- End of inner exception stack trace ---
at Foundatio.Repositories.Elasticsearch.Extensions.ElasticIndexExtensions.ToCountResult[T](ISearchResponse`1 response, ICommandOptions options) in /_/src/Foundatio.Repositories.Elasticsearch/Extensions/ElasticIndexExtensions.cs:line 156
at Foundatio.Repositories.Elasticsearch.ElasticReadOnlyRepositoryBase`1.CountAsync(IRepositoryQuery query, ICommandOptions options) in /_/src/Foundatio.Repositories.Elasticsearch/Repositories/ElasticReadOnlyRepositoryBase.cs:line 492
at Exceptionless.Web.Controllers.EventController.GetInternalAsync(AppFilter sf, TimeInfo ti, String filter, String sort, String mode, Int32 page, Int32 limit, String after, Boolean usesPremiumFeatures) in /app/src/Exceptionless.Web/Controllers/EventController.cs:line 340
at Exceptionless.Web.Controllers.EventController.GetAsync(String filter, String sort, String time, String offset, String mode, Int32 page, Int32 limit, String after) in /app/src/Exceptionless.Web/Controllers/EventController.cs:line 213
at lambda_method641(Closure, Object)
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
What is the elasticsearch index mappings for the prod-events-2023.07.17 and prod-events-2023.07.18 indexes. It's almost like there was an error creating the indexes. Also are there any errors in the logs when a new days indexes are created?
Solved, When I changed EX_AppScope to any string other than "prod", it miraculously started working, I don't know why.
It appears to be the same issue as #1216.
No exception occurred when creating the index, I repeatedly cleaned /var/lib/elasticsearch multiple times for create indexes.
That doesn't make much sense to me. Did you disable the mapper size plugin if you aren't using our elasticsearch docker images? Any errors in the logs when creating those indexes? Any difference in the index mappings between prod and any other string prefix
Would you mind sharing your elasticsearch connection string that you have configured (please modify any passwords and host names before sharing).