Indices rule not working for data stream ES 7.17
mlollo opened this issue · comments
Elastic and Kibana 7.17.1
RoR Pro 1.39.0 and 1.40.0 (both tested)
Data stream logs-example
Rule :
- name: '::B16a-myrule::'
actions:
- indices:data/read/*
- cluster:monitor/main
groups:
- mygroup
indices:
- 'logs-example*'
type: allow
verbosity: info
Log :
[35mINDEX NOT FOUND req={ ID:1345477503-2013458868#286922, TYP:SearchRequest, CGR:N/A, USR:myuser (attempted), BRS:true, KDX:null, ACT:indices:data/read/search, OA:mysrcip, XFF:null, DA:mydstip, IDX:logs-example*, MET:GET, PTH:/logs-example*/_search, CNT:<N/A> (...) HIS:[::B16a-myrule::-> RULES:[groups->true, actions->true, indices->false] RESOLVED:[user=myuser;group=mygroup;av_groups=mygroup;indices=logs-example*]], }[0m
According to this thread this issue has been fixed in 1.39.0 changelog
It works with a rule without indices :
- name: '::B16a-myrule::'
actions:
- indices:data/read/*
- cluster:monitor/main
groups:
- mygroup
type: allow
verbosity: info
Log :
[36mALLOWED by { name: '::B16a-myrule::', policy: ALLOW, rules: [groups,actions] req={ ID:599000627-2013458868#284835, TYP:SearchRequest, CGR:N/A, USR:myuser, BRS:true, KDX:null, ACT:indices:data/read/search, OA:mysrcip, XFF:null, DA:mydstip, IDX:logs-example*, MET:GET, PTH:/logs-example*/_search, CNT:<N/A>, (...) HIS:[::B16a-myrule::-> RULES:[groups->true, actions->true] RESOLVED:[user=myuser;group=mygroup;av_groups=mygroup;indices=logs-example*]], }[0m
Hi, the latest release is 1.41.0! Is this still broken?
Yes, same behavior with 1.41.0. I have this empty response for GET /logs-example*/_search :
{"took":1,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0.0,"hits":[]}}
It works when I specify the exact name of the data stream : GET /logs-example-202206/_search
And with the rule :
- name: '::B16a-myrule::'
actions:
- indices:data/read/*
- cluster:monitor/main
groups:
- mygroup
indices:
- 'logs-example*'
type: allow
verbosity: info
HI @mlollo
Yes, I can confirm it was fixed. We have automated tests for this, so the regression is not very likely.
Did you check what you get without ROR installed (or with ROR disabled)?
Could you please show us a list of your indices (and data streams) and aliases, please?
indices rule has rich debug logs support. You can enable debug logs and show us the logs related to the HTTP request
Thank you for your response
I tried to enable debug logs :
[1725833192-634080377#10908] Discovered indices: logs-example*
checking request: 1725833192-634080377#10908
Various attempt to match a block
Attempting Login as: myuser rc: 1725833192-634080377#10908
Followed by logs "the request matches no rules in this block"
Then the intended block produces those logs (not in order because they all spawn at the same millisecond) :
[1725833192-634080377#10908] ... not matched. Stop!
[1725833192-634080377#10908] Checking - indices & aliases ...
[1725833192-634080377#10908] ... not matched. Continue
[1725833192-634080377#10908] Checking if all indices are matched ...
[1725833192-634080377#10908] ... not matched. Continue
[1725833192-634080377#10908] ... indices and aliases: [.ds-logs-example-202205-2022.05.27-000002,.ds-logs-example-202206-2022.06.14-000008,(...)
[33m[::B16a-myrule::] the request matches no rules in this block: { ID:1725833192-634080377#10908, TYP:SearchRequest, CGR:None, USR:myuser (attempted), BRS:true, KDX:null, ACT:indices:data/read/search, OA:mysrcip, XFF:null, DA:mydstip, IDX:logs-example*, MET:GET, PTH:/logs-example*/_search, CNT:<N/A>, (...) HIS:[::B16a-myrule::-> RULES:[groups->true, actions->true, indices->false] RESOLVED:[user=myuser;group=mygroup;av_groups=mygroup;indices=logs-example*]], } [0m
[1725833192-634080377#10908] Request processing time: 20ms
[1725833192-634080377#10908] No filter applied to query.
[35mINDEX NOT FOUND req={ ID:1725833192-634080377#10908, TYP:SearchRequest (...)
In the log "... indices and aliases: [.ds-logs-example-202205-2022.05.27-000002,.ds-logs-example-202206-2022.06.14-000008" there is a long list of all .ds-logs indices created by datastreams, the log is truncated.
There is no alias found for the data stream, I think by default the elasticsearch managed index template logs doesn't create aliases required for ROR indices rule to work (I didn't understood at the beginning that data stream are not the same as aliases). This is why it doesn't match any indices or aliases.
GET _cat/indices
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .ds-logs-example-202206-2022.06.14-000008 JSKOyLjPQXaJo3OMUyaIeA 1 0 7087 0 16.4mb 16.4mb
green open .ds-logs-example-202205-2022.05.27-000002 oRcmB6i_R4uULpITGCmp6w 1 0 67034 0 33.2mb 33.2mb
GET /_data_stream/logs-example*
{
"data_streams" : [
{
"name" : "logs-example-202205",
"timestamp_field" : {
"name" : "@timestamp"
},
"indices" : [
{
"index_name" : ".ds-logs-example-202205-2022.05.27-000002",
"index_uuid" : "oRcmB6i_R4uULpITGCmp6w"
}
],
(...)
},
{
"name" : "logs-example-202206",
"timestamp_field" : {
"name" : "@timestamp"
},
"indices" : [
{
"index_name" : ".ds-logs-example-202206-2022.06.14-000008",
"index_uuid" : "JSKOyLjPQXaJo3OMUyaIeA"
}
],
(...)
}
]
}
For now I have a workaround with this rule (with .ds-logs-example* pattern) :
- name: '::B16a-myrule::'
actions:
- indices:data/read/*
- cluster:monitor/main
groups:
- mygroup
indices:
- '.ds-logs-example*'
type: allow
verbosity: info
This request works GET /.ds-logs-example*/_search
If I need to make the rule works with aliases I would need to do for each new data stream YYYYMM :
POST _aliases
{
"actions": [
{
"add": {
"index": "logs-example*",
"alias": "logs-example"
}
}
]
}
With the block :
- name: '::B16a-myrule::'
actions:
- indices:data/read/*
- cluster:monitor/main
groups:
- mygroup
indices:
- 'logs-example*'
type: allow
verbosity: info
This request works GET /logs-example*/_search
For me you can close the issue, the answer is : indices ror rule match only indices and aliases not data stream.
Hi,
I face a very similar issue as @mlollo , with Elasticsearch 7.17 and RoR 1.41.
In summary,
- It works to give indices:data/read/* permission to a user on a datastream if the request contains the full name of the data stream:
GET mydatastreamAAA/_search
{
"query": {
"match_all": {}
}
}
RESPONSE:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 372,
"relation" : "eq"
},...- However, the same request with a data stream name that includes a * returns nothing:
GET mydatastream*/_search
{
"query": {
"match_all": {}
}
}
RESPONSE
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [ ]
}
}- If I bypass readonlyrest, the above request gives me results.
- It seems to me that readonlyrest doesn't support two levels of resolution (when there's a wildcard index name that matches datastream, or an alias that matches data stream)
Hi Guys. I can confirm there is an issue and it looks like we know the reason. Currently, we are integrating with ES 8.3.x, but when it's finished, we will fix this issue. Thanks for your reports!
ROR 1.47.0 is released