Include user information when Kibana sends an `x-opaque-id` header to Elasticsearch

[Robbie Ogburn]

Describe the feature:

Today, Kibana includes an x-opaque-id header for search requests originating from Lens, Discover, Maps, and Alerting. This gets propagated in Elasticsearch's search slow logs and is helpful in identifying where a slow request originated from:

"source": { "id": "c89d1ab3-b4a7-4920-a64a-22a910a413b0;kibana:application:dashboard:edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b;visualization:Vega:cb099a20-ea66-11eb-9425-113343a037e3" }

It'd be helpful to also include the user who submitted the request. Right now, we'd have to rely on Elasticsearch audit logs and match up elasticsearch.slowlog.id <--> opaque_id to learn the user behind the request.

Slow log:

{
  "@timestamp": "2024-03-12T18:56:25.036Z",
  "log.level": "TRACE",
  "elasticsearch.slowlog.id": "d439e0e2-aebc-4f3f-a7e1-b159fd855048;kibana:application:discover:new;application:discover:new;lens:lnsXY:unifiedHistogramLensComponent",
  "elasticsearch.slowlog.message": "[kibana_sample_data_ecommerce][0]",
  "elasticsearch.slowlog.search_type": "QUERY_THEN_FETCH",
  "elasticsearch.slowlog.source": "{\"size\":0,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"order_date\":{\"gte\":\"2024-03-12T18:41:24.960Z\",\"lte\":\"2024-03-12T18:56:24.960Z\",\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"boost\":1.0}},\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"fields\":[{\"field\":\"customer_birth_date\",\"format\":\"date_time\"},{\"field\":\"order_date\",\"format\":\"date_time\"},{\"field\":\"products.created_on\",\"format\":\"date_time\"}],\"script_fields\":{},\"track_total_hits\":2147483647,\"aggregations\":{\"0\":{\"date_histogram\":{\"field\":\"order_date\",\"time_zone\":\"America/New_York\",\"fixed_interval\":\"30s\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1}}}}",
  "elasticsearch.slowlog.stats": "[]",
  "elasticsearch.slowlog.took": "903.2micros",
  "elasticsearch.slowlog.took_millis": 0,
  "elasticsearch.slowlog.total_hits": "3 hits",
  "elasticsearch.slowlog.total_shards": 1,
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.index_search_slowlog",
  "process.thread.name": "elasticsearch[Roberts-MacBook-Pro.local][search][T#8]",
  "log.logger": "index.search.slowlog.query",
  "trace.id": "60536004624e660c7708f63fa0acb34c",
  "elasticsearch.cluster.uuid": "SRdr2d6zTae-OeyriWUUcA",
  "elasticsearch.node.id": "f_61tu0UTnOkZXy-hzmvKw",
  "elasticsearch.node.name": "Roberts-MacBook-Pro.local",
  "elasticsearch.cluster.name": "elasticsearch"
}

{
  "elasticsearch.slowlog.id":"d439e0e2-aebc-4f3f-a7e1-b159fd855048;kibana:application:discover:new;application:discover:new;lens:lnsXY:unifiedHistogramLensComponent"
}

Audit log:

{
  "type": "audit",
  "timestamp": "2024-03-12T14:56:25,034-0400",
  "cluster.uuid": "SRdr2d6zTae-OeyriWUUcA",
  "node.id": "f_61tu0UTnOkZXy-hzmvKw",
  "event.type": "rest",
  "event.action": "authentication_success",
  "authentication.type": "REALM",
  "user.name": "elastic",
  "user.realm": "reserved",
  "origin.type": "rest",
  "origin.address": "192.168.1.141:51609",
  "realm": "reserved",
  "url.path": "/kibana_sample_data_ecommerce/_async_search",
  "url.query": "batched_reduce_size=64&wait_for_completion_timeout=100ms&keep_on_completion=true&keep_alive=60000ms&ignore_unavailable=true&preference=1710269778143",
  "request.method": "POST",
  "request.body": "{\"aggs\":{\"0\":{\"date_histogram\":{\"field\":\"order_date\",\"fixed_interval\":\"30s\",\"time_zone\":\"America/New_York\",\"min_doc_count\":1}}},\"size\":0,\"fields\":[{\"field\":\"customer_birth_date\",\"format\":\"date_time\"},{\"field\":\"order_date\",\"format\":\"date_time\"},{\"field\":\"products.created_on\",\"format\":\"date_time\"}],\"script_fields\":{},\"stored_fields\":[\"*\"],\"runtime_mappings\":{},\"_source\":{\"excludes\":[]},\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"range\":{\"order_date\":{\"format\":\"strict_date_optional_time\",\"gte\":\"2024-03-12T18:41:24.960Z\",\"lte\":\"2024-03-12T18:56:24.960Z\"}}}],\"should\":[],\"must_not\":[]}},\"track_total_hits\":true}",
  "request.id": "n4Y285gISoGEJ6cNNzLKqA",
  "opaque_id": "d439e0e2-aebc-4f3f-a7e1-b159fd855048;kibana:application:discover:new;application:discover:new;lens:lnsXY:unifiedHistogramLensComponent",
  "trace.id": "60536004624e660c7708f63fa0acb34c"
}

{
  "opaque_id": "d439e0e2-aebc-4f3f-a7e1-b159fd855048;kibana:application:discover:new;application:discover:new;lens:lnsXY:unifiedHistogramLensComponent",
  "user.name": "elastic"
}

[benatbermejo]

Hi.

It would be nice to have the username of the user that executed the query on slowlog query log.

Especially, when the query is executed vía Kibana.

Sometimes users execute big range timestamp queries at the dashboards with large data indices, and they are not aware of what they are doing. We can neither limit the timestamp ranges of Kibana dashboards.

As the administrator, we need to know which final user is doing heavy queries to Elastic.

[Elastic Machine]

Pinging @elastic/kibana-core (Team:Core)

[Pierre Gayvallet]

I could be wrong, but my gut feeling is that this should be done on ES's side in some way. Any request performed from Kibana on behalf of a user is already passing the user's credentials to authenticate against ES. Passing the userId again in the x-opaque-id header feels somewhat redundant to me?

[Robbie Ogburn]

I could be wrong, but my gut feeling is that this should be done on ES's side in some way.

When you're right you're right! As it turns out, this functionality will be available in 8.14 elastic/elasticsearch#105621