SPL BUG - Detect Rare Executables
patel-bhavin opened this issue · comments
From LinkedIN: Antony Bowesman
I'm a Splunk Trust member, working with Splunk Security @Deloitte - I've got some feedback on one of your anomaly detection searches, which is fundamentally flawed... (tstats count + rare will never work)
https://research.splunk.com/endpoint/44fddcb2-8d3b-454c-874e-7c6de5a4f7ac/
I'm looking at some of the ESCU searches for a client and noticed they have implemented the Detect Rare Executables search. As it was generating Notable events, I checked it and found that the search is flawed, both technically and conceptually.
Conceptually the time windows of the two tstats statements is the same, which seems a little odd in that the second tstats is effectively defining 'rare' as those in the current search time window, rather than determining rarity over a longer time window.
Technically, with tstats count by Processes.process_name, that will generate a single count for each process name, so when you do rare on process name, there is only ONE process name, so you will effectively get the first 30 (alphabetically ordered) items from that count NOT the ones with the lowest tstats count!
A correct search would follow tstats count with sort count | head 30.
Conceptually, the filtering of executables inside the subsearch is done AFTER the 30 'rare' items, so if you have 60 items after tstats that would be considered 'rare', then you cannot filter out the allowed ones before you determine rarity, so in practice, you may filter out all 30 left after the rare command and never find the true rare ones.
The final filter seems rather pointless in that you have already exercised two filters, but there may be other filtering requirements, so that's ok.
Given that this search is both part of ESCU and Splunk Security Essentials, it would be good to fix the technical deficiencies, as Splunk customers learn search techniques from these searches.
I hope I have made myself clear enough - if there is a bug route to file bugs in these searches, please let me know.
Cheers
Antony
The technically broken SPL is in the subsearch, it currently does
| tstats count from datamodel=Endpoint.Processes by Processes.process_name
| rare Processes.process_name limit=30
but the tstats will produce ONE row per process name, so rare process name is impossible to determine, as frequency is 1 per name. The search could do this instead of using the rare command
| tstats count from datamodel=Endpoint.Processes by Processes.process_name
| sort count
| head 30
As for conceptual flaws, an initial filter should be possible after the first tstats
| tstats `security_content_summariesonly` count values(Processes.dest) as dest values(Processes.user) as user min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes by Processes.process_name
| `detect_rare_executables_filter`
so that any rarity calculation done later only works on potential bad candidates.
As for the general concept of 'rarity', if you are searching an X minute window of executables and then use that same list as a 'rare' filter in the subsearch, you are likely to produce a higher proportion of false positives as 'rarity' will occur more frequently, especially outside high usage hours.
It seems like the only purpose of the subsearch filter is to prevent the 'loss' of the dest, user, first/last times that would occur with rare. As rare is flawed, the entire search could be
| tstats `security_content_summariesonly` count values(Processes.dest) as dest values(Processes.user) as user min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes by Processes.process_name
| rename Processes.process_name as process
| `filter_rare_process_allow_list`
| sort count
| head 30
| rex field=user "(?<user_domain>.*)\\\\(?<user_name>.*)"
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `detect_rare_executables_filter`
So, the search now does
- Find all executables in window
- Remove all allowed processes
- Take least executed 30 results
- Perform post data fixups
- Final filter
This still does not address the concept of 'rarity' within a longer term window, but at least will be functional, however, it will typically always generate 30 results unless filtering has been configured sensibly, which will be a lot of noise if these are notable event correlation searches in ESCU
Hello @bowesmana : thanks for all these details, after testing this a bit on some live data: the current search logic is definitely flawed and needs a re-haul. I tested your proposed search and it performs better! We will update this in our upcoming releases
Like you mentioned, this updated SPL doesnt address the concept of rarity which is something I want to dive into and see if we can solve it the right way. Thank you again!
updated PR : #2449