Migrate to ecs@mappings template
zmoog opened this issue · comments
This is the Step 3: Enable ECS system tests in elastic-package of the meta issue Migrate to ecs@mappings template shipped with Elasticsearch.
Disclaimer: this is a stub description that I will update as we clarify the details in this issue's comments.
I spent some time trying to learn what we have now and what we need to do in Step 3.
What we have now
I see elastic-package, when users set dependencies.ecs.import_mappings to true, can:
- Import the ECS schema to validate the fields.
- Embed a "legacy" version of the ECS dynamic mappings.
What we need to do in the step 3
My current understanding is we should:
- Continue using ECS schema to validate fields
- Stop embedding the "legacy" ECS dynamic mappings
- Start leveraging the
ecs@mappingscomponent template when it is available for integrations.
Questions
We currently use dependencies.ecs.reference to understand which ECS version the package targets. However, once we add ecs@mappings to the integration component template, it will be available out-of-the-box from ES. Should we add some mechanism to the package-spec to avoid this from happening automatically, and allow developers to test their packages before opt-in to ecs@mappings?
Continue using ECS schema to validate fields
Yes, we need to continue doing this in elastic-package. What we can maybe change now is to somehow obtain the same mapping Elasticsearch is using, maybe copying it directly from Elasticsearch repo (from here?).
- Stop embedding the "legacy" ECS dynamic mappings
- Start leveraging the
ecs@mappingscomponent template when it is available for integrations.
I wonder if this could be handled by Fleet. The logic could be like this:
- If the manifest contains dynamic templates prefixed by
_embedded_ecs-(underelasticsearch.index_template.mappings.dynamic_templates)- These dynamic templates are not installed.
- The
ecs@mappingscomponent template is added.
Or maybe Fleet can just always add the ecs@mappings component template, and ignore dynamic mappings prefixed by _embedded_ecs-. @kpollich wdyt?
Eventually elastic-package should stop including the _embedded-ecs- dynamic mappings in any case, at least for recent versions of Kibana.
Should we add some mechanism to the package-spec to avoid this from happening automatically, and allow developers to test their packages before opt-in to
ecs@mappings?
As ecs@mappings is now it looks like a safer default for dynamic mappings in our use cases. Do we want this to continue being opt-in? Would it make sense to install it in the data streams of all packages?
Continue using ECS schema to validate fields
Yes, we need to continue doing this in elastic-package. What we can maybe change now is to somehow obtain the same mapping Elasticsearch is using, maybe copying it directly from Elasticsearch repo (from here?).
These are the ecs@mappings dynamic templates.
With "ECS schema" I was referring to https://github.com/elastic/ecs/blob/main/generated/ecs/ecs_nested.yml: Isn't elastic-package using ecs_nested.yml to validate the fields that the integration defines in the fields.yml and other .yml files?
Or maybe Fleet can just always add the ecs@mappings component template, and ignore dynamic mappings prefixed by _embedded_ecs-
Here is the related change in Kibana to always add the ecs@mappings component template: elastic/kibana#174855
Eventually elastic-package should stop including the _embedded-ecs- dynamic mappings in any case, at least for recent versions of Kibana.
Yep, we should stop including the _embedded-ecs- dynamic templates starting from the Kibana version that will begin adding the ecs@mappings to the integration index template.
On new stack versions, adding ecs@mappings will be the responsibility of Elasticsearch and Kibana, not elastic-package.
Currently, there are ~40 security-focused integrations currently leveraging import_mappings: true.
As ecs@mappings is now it looks like a safer default for dynamic mappings in our use cases. Do we want this to continue being opt-in? Would it make sense to install it in the data streams of all packages?
Yep, the consensus seems to gravitate towards installing ecs@mappings in the data streams of all packages.
We were wondering if we need to provide an opt-out mechanism for not adding ecs@mappings to the index template. This can help if an existing integration runs into an issue with ecs@mappings.
I am trying to summarize the possible following actions and some questions on aspects that probably need more work:
- Remove the Legacy ECS Mappings from elastic-package.
- Validate the fields that the integration defines in the
fields.ymland other.ymlfiles using the "ECS schema" at https://github.com/elastic/ecs/blob/main/generated/ecs/ecs_nested.yml - On new stack versions:
- Adding
ecs@mappingswill be the responsibility of Elasticsearch and Kibana, notelastic-packagethanks to elastic/kibana#174855 - installing
ecs@mappingsin the data streams of all packages.- ❓Question: should we provide a mechanism to opt-out from including
ecs@mappignsin the integration's index template (requires a change in Fleet)
- ❓Question: should we provide a mechanism to opt-out from including
- Adding
- ❓Question: should we identify the minimum ECS version supported by the integration using
dependencies.ecs.reference, or something like the min stack version atconditions.kibana.version?
cc @P1llus
❓Question: should we provide a mechanism to opt-out from including ecs@mappigns in the integration's index template (requires a change in Fleet)
I think it's a good and prudent idea to add an opt-out mechanism here. While we don't anticipate issues, and ecs@mappings only applys to fields that are not specifically mapped in the fields yml, it seems like it doesn't hurt to have an escape hatch if we do notice an unforeseen issue in a specific integration. It would give us a quicker path to resolve such issues by creating a new version of an integration that opts-out of importing ecs@mappings.
❓Question: should we identify the minimum ECS version supported by the integration using dependencies.ecs.reference, or something like the min stack version at conditions.kibana.version?
The flag to specify the minimum stack version (conditions.kibana.version) still makes sense. Integrations that rely on ecs@mappings, and don't add a specific fields.yml will need a min stack version of 8.13. However, integrations can't pin the ecs mappings to a specific version anymore. They'll just import the current version. Therefore, we'll need to ensure that the changes in ecs@mappings are strictly additive.
@jsoriano, is dependencies.ecs.reference currently used to get field definitions in the ecs.yml file AND download the ECS schema from https://github.com/elastic/ecs, right?
@P1llus, do you think we may need to migrate the ~40 security-focused integrations currently using legacy ecs@mappings embedded by elastic-package?
@jsoriano, is
dependencies.ecs.referencecurrently used to get field definitions in theecs.ymlfile AND download the ECS schema from https://github.com/elastic/ecs, right?@P1llus, do you think we may need to migrate the ~40 security-focused integrations currently using legacy ecs@mappings embedded by
elastic-package?
I would say that if we plan on removing the option in build.yml + deprecating the legacy template (which we should), then we do need to remove that line from each build.yml file and bump the package versions for any integration using it. We also need to update the min version of those packages to the version where this new template is actually available and working for integrations, so that there is no conflict.
If we dont do that then we will simply be missing all ECS mapping if users are on new integration versions but older stack versions.
@P1llus @jsoriano, we plan to:
- Remove
dependencies.ecs.import_mappings: truefrom integration using it, and set 8.13 as min stack version to switch from legacy to modernecs@mappings. - Deprecate
dependencies.ecs.import_mappings. - Document how to migrate to modern
ecs@mappings. - Share how to adopt modern
ecs@mappingsat the following integrations office hours meeting.
TIL, the index templates get an update during stack upgrades; I didn't expect this, but the index template for Azure the Blob Storage integration received the ecs@mappings from 8.12 to 8.13. See zmoog/public-notes#78 for more.
- Remove
dependencies.ecs.import_mappings: truefrom integration using it, and set 8.13 as min stack version to switch from legacy to modernecs@mappings.
If we make Fleet to ignore mappings prefixed with _embedded-ecs- when ecs@mappins is used, do we need changes in packages?
- Remove
dependencies.ecs.import_mappings: truefrom integration using it, and set 8.13 as min stack version to switch from legacy to modernecs@mappings.If we make Fleet to ignore mappings prefixed with
_embedded-ecs-whenecs@mappingsis used, do we need changes in packages?
If we ignore dynamic mappings prefixed with _embedded-ecs- 1 when ecs@mappings is used (starting from 8.13) I think we don't need to update the packages.
Footnotes
We do have tooling to modify these packages though, I dont see why we would want to leave the settings there when the amount of work to remove it is so small?
We do have tooling to modify these packages though, I dont see why we would want to leave the settings there when the amount of work to remove it is so small?
My concern is more on needing to increase the kibana version to 8.13. If we keep the current approach we keep support for more stack versions.
I see both points:
- Removing the legacy
ecs@mappingsto keep integrations lean and tidy. - Keeping the minimum required stack version low to reach as many users as possible.
They move to opposite directions.
Question: What if a package has both mappings installed?
To add a data point to facilitate a decision, I tried to answer the question the following question:
Is it a problem if a package has both mappings installed?
To find an answer, I reused the same EC deployment we used earlier to Figure out if index templates are regenerated during a stack upgrade.
Answer: Diving into Azure Blob Storage on 8.13
The EC deployment contains the Azure Blob Storage integration, which is one of the 41 integrations currently using the legacy ecs@mappings.
Here's the index template for Azure Blog Storage.
In the logs-azure_blob_storage.generic index template, we have the two legacy and modern ecs@mappings:
- The legacy ecs_mappings.yaml from elastic/elastic-package in the
logs-azure_blob_storage.generic@packagecomponent template. - The modern ecs@mappings.json from elastic/elasticsearch in the
ecs@mappingscomponent template.
We can see how they Elasticsearch combines them into one index template by simulating it in the Dev Tools:
POST /_index_template/_simulate_index/logs-azure_blob_storage.generic-default
Here is the complete output:
Click me to show the response
{
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "logs"
},
"codec": "best_compression",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_hot"
}
}
},
"mapping": {
"total_fields": {
"limit": "10000",
"ignore_dynamic_beyond_limit": "true"
},
"ignore_malformed": "true"
},
"final_pipeline": ".fleet_final_pipeline-1",
"query": {
"default_field": [
"input.type",
"tags",
"azure.storage.container.name",
"azure.storage.blob.name",
"azure.storage.blob.content_type"
]
},
"default_pipeline": "logs-azure_blob_storage.generic-1.1.0"
}
},
"mappings": {
"_meta": {
"managed_by": "fleet",
"managed": true,
"package": {
"name": "azure_blob_storage"
}
},
"dynamic_templates": [
{
"_embedded_ecs-ecs_timestamp": {
"path_match": "@timestamp",
"mapping": {
"ignore_malformed": false,
"type": "date"
}
}
},
{
"_embedded_ecs-data_stream_to_constant": {
"path_match": "data_stream.*",
"mapping": {
"type": "constant_keyword"
}
}
},
{
"_embedded_ecs-resolved_ip_to_ip": {
"match": "resolved_ip",
"mapping": {
"type": "ip"
}
}
},
{
"_embedded_ecs-forwarded_ip_to_ip": {
"match": "forwarded_ip",
"match_mapping_type": "string",
"mapping": {
"type": "ip"
}
}
},
{
"_embedded_ecs-ip_to_ip": {
"match": "ip",
"match_mapping_type": "string",
"mapping": {
"type": "ip"
}
}
},
{
"_embedded_ecs-port_to_long": {
"match": "port",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-thread_id_to_long": {
"path_match": "*.thread.id",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-status_code_to_long": {
"match": "status_code",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-line_to_long": {
"path_match": "*.file.line",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-priority_to_long": {
"path_match": "log.syslog.priority",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-code_to_long": {
"path_match": "*.facility.code",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-bytes_to_long": {
"match": "bytes",
"path_unmatch": "*.data.bytes",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-packets_to_long": {
"match": "packets",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-public_key_exponent_to_long": {
"match": "public_key_exponent",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-severity_to_long": {
"path_match": "event.severity",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-duration_to_long": {
"path_match": "event.duration",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-pid_to_long": {
"match": "pid",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-uptime_to_long": {
"match": "uptime",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-sequence_to_long": {
"match": "sequence",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-entropy_to_long": {
"match": "*entropy",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-size_to_long": {
"match": "*size",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-entrypoint_to_long": {
"match": "entrypoint",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-ttl_to_long": {
"match": "ttl",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-major_to_long": {
"match": "major",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-minor_to_long": {
"match": "minor",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-as_number_to_long": {
"path_match": "*.as.number",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-pgid_to_long": {
"match": "pgid",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-exit_code_to_long": {
"match": "exit_code",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-chi_to_long": {
"match": "chi2",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-args_count_to_long": {
"match": "args_count",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-virtual_address_to_long": {
"match": "virtual_address",
"mapping": {
"type": "long"
}
}
},
{
"_embedded_ecs-io_text_to_wildcard": {
"path_match": "*.io.text",
"mapping": {
"type": "wildcard"
}
}
},
{
"_embedded_ecs-strings_to_wildcard": {
"path_match": "registry.data.strings",
"mapping": {
"type": "wildcard"
}
}
},
{
"_embedded_ecs-path_to_wildcard": {
"path_match": "*url.path",
"mapping": {
"type": "wildcard"
}
}
},
{
"_embedded_ecs-message_id_to_wildcard": {
"match": "message_id",
"mapping": {
"type": "wildcard"
}
}
},
{
"_embedded_ecs-command_line_to_multifield": {
"match": "command_line",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"_embedded_ecs-error_stack_trace_to_multifield": {
"match": "stack_trace",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"_embedded_ecs-http_content_to_multifield": {
"path_match": "*.body.content",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"_embedded_ecs-url_full_to_multifield": {
"path_match": "*.url.full",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"_embedded_ecs-url_original_to_multifield": {
"path_match": "*.url.original",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"_embedded_ecs-user_agent_original_to_multifield": {
"path_match": "user_agent.original",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"_embedded_ecs-error_message_to_match_only": {
"path_match": "error.message",
"mapping": {
"type": "match_only_text"
}
}
},
{
"_embedded_ecs-message_match_only_text": {
"path_match": "message",
"mapping": {
"type": "match_only_text"
}
}
},
{
"_embedded_ecs-agent_name_to_keyword": {
"path_match": "agent.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-service_name_to_keyword": {
"path_match": "*.service.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-sections_name_to_keyword": {
"path_match": "*.sections.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-resource_name_to_keyword": {
"path_match": "*.resource.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-observer_name_to_keyword": {
"path_match": "observer.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-question_name_to_keyword": {
"path_match": "*.question.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-group_name_to_keyword": {
"path_match": "*.group.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-geo_name_to_keyword": {
"path_match": "*.geo.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-host_name_to_keyword": {
"path_match": "host.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-severity_name_to_keyword": {
"path_match": "*.severity.name",
"mapping": {
"type": "keyword"
}
}
},
{
"_embedded_ecs-title_to_multifield": {
"match": "title",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-executable_to_multifield": {
"match": "executable",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-file_path_to_multifield": {
"path_match": "*.file.path",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-file_target_path_to_multifield": {
"path_match": "*.file.target_path",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-name_to_multifield": {
"match": "name",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-full_name_to_multifield": {
"match": "full_name",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-os_full_to_multifield": {
"path_match": "*.os.full",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-working_directory_to_multifield": {
"match": "working_directory",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"_embedded_ecs-timestamp_to_date": {
"match": "timestamp",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-delivery_timestamp_to_date": {
"match": "delivery_timestamp",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-not_after_to_date": {
"match": "not_after",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-not_before_to_date": {
"match": "not_before",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-accessed_to_date": {
"match": "accessed",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-origination_timestamp_to_date": {
"match": "origination_timestamp",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-created_to_date": {
"match": "created",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-installed_to_date": {
"match": "installed",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-creation_date_to_date": {
"match": "creation_date",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-ctime_to_date": {
"match": "ctime",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-mtime_to_date": {
"match": "mtime",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-ingested_to_date": {
"match": "ingested",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-start_to_date": {
"match": "start",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-end_to_date": {
"match": "end",
"mapping": {
"type": "date"
}
}
},
{
"_embedded_ecs-score_base_to_float": {
"path_match": "*.score.base",
"mapping": {
"type": "float"
}
}
},
{
"_embedded_ecs-score_temporal_to_float": {
"path_match": "*.score.temporal",
"mapping": {
"type": "float"
}
}
},
{
"_embedded_ecs-score_to_float": {
"match": "*_score",
"mapping": {
"type": "float"
}
}
},
{
"_embedded_ecs-score_norm_to_float": {
"match": "*_score_norm",
"mapping": {
"type": "float"
}
}
},
{
"_embedded_ecs-usage_to_float": {
"match": "usage",
"mapping": {
"scaling_factor": 1000,
"type": "scaled_float"
}
}
},
{
"_embedded_ecs-location_to_geo_point": {
"match": "location",
"mapping": {
"type": "geo_point"
}
}
},
{
"_embedded_ecs-same_as_process_to_boolean": {
"match": "same_as_process",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-established_to_boolean": {
"match": "established",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-resumed_to_boolean": {
"match": "resumed",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-max_bytes_per_process_exceeded_to_boolean": {
"match": "max_bytes_per_process_exceeded",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-interactive_to_boolean": {
"match": "interactive",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-exists_to_boolean": {
"match": "exists",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-trusted_to_boolean": {
"match": "trusted",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-valid_to_boolean": {
"match": "valid",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-go_stripped_to_boolean": {
"match": "go_stripped",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-coldstart_to_boolean": {
"match": "coldstart",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-exports_to_flattened": {
"match": "exports",
"mapping": {
"type": "flattened"
}
}
},
{
"_embedded_ecs-structured_data_to_flattened": {
"match": "structured_data",
"mapping": {
"type": "flattened"
}
}
},
{
"_embedded_ecs-imports_to_flattened": {
"match": "*imports",
"mapping": {
"type": "flattened"
}
}
},
{
"_embedded_ecs-attachments_to_nested": {
"match": "attachments",
"mapping": {
"type": "nested"
}
}
},
{
"_embedded_ecs-segments_to_nested": {
"match": "segments",
"mapping": {
"type": "nested"
}
}
},
{
"_embedded_ecs-elf_sections_to_nested": {
"path_match": "*.elf.sections",
"mapping": {
"type": "nested"
}
}
},
{
"_embedded_ecs-pe_sections_to_nested": {
"path_match": "*.pe.sections",
"mapping": {
"type": "nested"
}
}
},
{
"_embedded_ecs-macho_sections_to_nested": {
"path_match": "*.macho.sections",
"mapping": {
"type": "nested"
}
}
},
{
"ecs_timestamp": {
"match": "@timestamp",
"mapping": {
"ignore_malformed": false,
"type": "date"
}
}
},
{
"ecs_message_match_only_text": {
"path_match": [
"message",
"*.message"
],
"unmatch_mapping_type": "object",
"mapping": {
"type": "match_only_text"
}
}
},
{
"ecs_ip": {
"path_match": [
"ip",
"*.ip",
"*_ip"
],
"match_mapping_type": "string",
"mapping": {
"type": "ip"
}
}
},
{
"ecs_wildcard": {
"path_match": [
"*.io.text",
"*.message_id",
"*registry.data.strings",
"*url.path"
],
"unmatch_mapping_type": "object",
"mapping": {
"type": "wildcard"
}
}
},
{
"ecs_path_match_wildcard_and_match_only_text": {
"path_match": [
"*.body.content",
"*url.full",
"*url.original"
],
"unmatch_mapping_type": "object",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"ecs_match_wildcard_and_match_only_text": {
"match": [
"*command_line",
"*stack_trace"
],
"unmatch_mapping_type": "object",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "wildcard"
}
}
},
{
"ecs_path_match_keyword_and_match_only_text": {
"path_match": [
"*.title",
"*.executable",
"*.name",
"*.working_directory",
"*.full_name",
"*file.path",
"*file.target_path",
"*os.full",
"email.subject",
"vulnerability.description",
"user_agent.original"
],
"unmatch_mapping_type": "object",
"mapping": {
"fields": {
"text": {
"type": "match_only_text"
}
},
"type": "keyword"
}
}
},
{
"ecs_date": {
"path_match": [
"*.timestamp",
"*_timestamp",
"*.not_after",
"*.not_before",
"*.accessed",
"created",
"*.created",
"*.installed",
"*.creation_date",
"*.ctime",
"*.mtime",
"ingested",
"*.ingested",
"*.start",
"*.end"
],
"unmatch_mapping_type": "object",
"mapping": {
"type": "date"
}
}
},
{
"ecs_path_match_float": {
"path_match": [
"*.score.*",
"*_score*"
],
"path_unmatch": "*.version",
"unmatch_mapping_type": "object",
"mapping": {
"type": "float"
}
}
},
{
"ecs_usage_double_scaled_float": {
"path_match": "*.usage",
"match_mapping_type": [
"double",
"long",
"string"
],
"mapping": {
"scaling_factor": 1000,
"type": "scaled_float"
}
}
},
{
"ecs_geo_point": {
"path_match": [
"location",
"*.location"
],
"mapping": {
"type": "geo_point"
}
}
},
{
"ecs_flattened": {
"path_match": [
"*structured_data",
"*exports",
"*imports"
],
"match_mapping_type": "object",
"mapping": {
"type": "flattened"
}
}
},
{
"all_strings_to_keywords": {
"match_mapping_type": "string",
"mapping": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
],
"date_detection": false,
"properties": {
"@timestamp": {
"type": "date",
"ignore_malformed": false
},
"azure": {
"properties": {
"storage": {
"properties": {
"blob": {
"properties": {
"content_type": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"container": {
"properties": {
"name": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
}
},
"data_stream": {
"properties": {
"dataset": {
"type": "constant_keyword"
},
"namespace": {
"type": "constant_keyword"
},
"type": {
"type": "constant_keyword"
}
}
},
"event": {
"properties": {
"agent_id_status": {
"type": "keyword",
"ignore_above": 1024
},
"ingested": {
"type": "date",
"format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
}
}
},
"input": {
"properties": {
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"log": {
"properties": {
"offset": {
"type": "long"
}
}
},
"tags": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"aliases": {}
},
"overlapping": [
{
"name": "logs",
"index_patterns": [
"logs-*-*"
]
}
]
}In the response, I see the following:
# legacy
$ pbpaste | jq -r '.template.mappings.dynamic_templates.[] | [keys[0]] | @tsv' | grep '^_embedded_ecs' | wc -l
99
# modern
$ pbpaste | jq -r '.template.mappings.dynamic_templates.[] | [keys[0]] | @tsv' | grep '^ecs_' | wc -l
12Observations
- legacy and modern ecs@mappings are similar but different
- It seems the modern ecs@mappings are not a drop-in replacement for legacy ecs@mappings
- Legacy ecs@mappings have precedence over modern ones
Similar, but different
Many dynamic templates are similar but different. For example, the dynamic templates for @timestamp are the following.
Legacy:
{
"_embedded_ecs-ecs_timestamp": {
"path_match": "@timestamp",
"mapping": {
"ignore_malformed": false,
"type": "date"
}
}
}Modern:
{
"ecs_timestamp": {
"match": "@timestamp",
"mapping": {
"ignore_malformed": false,
"type": "date"
}
}
}It seems the modern ecs@mappings are not a drop-in replacement for legacy ecs@mappings
The legacy ecs@mappings contains dynamic templates for fields that are not available in the modern. Here are a few examples:
[
{
"_embedded_ecs-go_stripped_to_boolean": {
"match": "go_stripped",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-coldstart_to_boolean": {
"match": "coldstart",
"mapping": {
"type": "boolean"
}
}
},
{
"_embedded_ecs-attachments_to_nested": {
"match": "attachments",
"mapping": {
"type": "nested"
}
}
},
{
"_embedded_ecs-segments_to_nested": {
"match": "segments",
"mapping": {
"type": "nested"
}
}
},
{
"_embedded_ecs-elf_sections_to_nested": {
"path_match": "*.elf.sections",
"mapping": {
"type": "nested"
}
}
}
]These fields come from more specific ECS schemas, like elf.
Legacy mappings have precedence over modern ones
Since the logs-azure_blob_storage.generic@packagecomponent package appears before ecs@mappings, the legacy dynamic templates appear first in the final list in the index template. Legacy mapping takes precedence over modern ones.
Conclusions
Since it seems the modern ecs@mappings are not a drop-in replacement for legacy ecs@mappings, it seems we need to do more than replace them with modern ones.
Since the legacy mappings have precedence over modern ones, it seems the 41 integration using the legacy should not have side effects if installed on 8.13.
Since the legacy mappings have precedence over modern ones, it seems the 41 integration using the legacy should not have side effects if installed on 8.13.
If I understand this correctly, this means that it is not a problem to have both mappings installed, right? If that's the case I would go in this direction. It has important benefits:
- We don't need to do any change in existing packages.
- We can support broader version ranges with the same code base.
Removing the legacy
ecs@mappingsto keep integrations lean and tidy.
This is a good point too, but:
- In the code-base it is a single line of code (
dependencies.ecs.import_mappings). - Mappings added to built packages are clearly identified by the
_embedded_ecs-prefix, and could be removed or updated by Fleet, transparently to users and developers.
If I understand this correctly, this means that it is not a problem to have both mappings installed, right? If that's the case I would go in this direction. It has important benefits:
- We don't need to do any change in existing packages.
- We can support broader version ranges with the same code base.
Since the legacy mappings have precedence over the modern ones, and modern ones are a subset of the legacy, it seems they can coexist.
- do you see problems in having both?
- do you know why the legacy mappings have a selection of ECS mapping from specific domains?
However, I want to test if this is the case.
@P1llus @jsoriano, any suggestions on effectively testing these 41 integrations on 8.13?