Reduce the need for the Deno runtime to collect filtered forensics

[Scott McCallum]

What new feature do you think would be cool to add to artemis?
Have the artifact collector pre-filter the data collected on key attributes based on some new TOML keys.

Describe the solution you'd like
Many of the artifacts have a dates/strings/SIDs so have some Rust based filter code look for keys in the TOML such as:

'collect_if_(modifed|created|accessed)_after=1712806012'

or

'collect_if_sid_is=S-1-5-{domain}-500'

These are just my examples and they need to be finessed to the project/language style.

Additional context
Only the collector executable and config files would be required to perform a focused collection. The orchestration server would simply generate the TOML with dynamically generated values limiting the data collected.

It should also be faster than the scripting approach for large servers, better for time-sensitive collection scenarios, and lower CPU use on production systems.

You might also allow for names on the right-hand side for common collection scenarios such as.

'collect_if_created_after=now_minus_7_days'

[puffyCid]

Hmm, im not sure this makes sense to include in Artemis, as an addition to its existing filtering options.

I will definitely, concede that the scripting runtime does add a bit of complexity and learning curve.
(I am trying to make it less complex via in-depth docs and guides)

I think it does though provide the easiest way to filter, IMO.

While creating custom TOML attributes could be useful for basic filtering. I think it may end up adding quite a lot of code complexity.
In addition, what if a user wants to try more creative filtering options/combinations?

Such as:

• Getting timestamp for a file. And then pivot to grab and filter for all event log entries 5+- mins after file creation
• Filter for specific macho binary entitlements (camera access, mic access, etc)
• Only get/list files created during a specific Windows logon session (ex: logon: 2024-04-02T01:15:27Z, logoff: 2024-04-02T14:43:22Z)

IMO I think the runtime provides greatest flexibility to do this.

For performance issues, have you encountered performance issues/concerns when using the runtime?
One of the goals for artemis is low CPU impact and memory usage, so if you are seeing consistent high CPU or memory usage that could be a bug.

On my test VMs and systems, performance has been pretty good.
But I have not tested in any real production environment. (Unless GitHub actions count)

[Scott McCallum]

I may be making this more complex than it needs to be. Image a single key filter= that is mapped to a single date field and, if present, only returns the record if it's after the supplied ISO date format.

My main concern is the additional work from the product's security audit, which involves vetting the Deno deployment into production enterprise servers. The audit would be significantly more straightforward if it was just the Rust .exe.

The desire for performance comes from seeing a server at a customer's site with over 300 million files on 20 NTFS volumes. They know they have created a monster, and most tools fail when dealing with it; it would be great if Artemis performed well at this scale.

[Scott McCallum]

I would also add that given the focus on supply chain attacks, deploying security-focused software with large base64 encoded blobs from open source sources is becoming more complex...

[puffyCid]

My main concern is the additional work from the product's security audit, which involves vetting the Deno deployment into production enterprise servers. The audit would be significantly more straightforward if it was just the Rust .exe.

You do not need to deploy Deno to any system.
The runtime is limited to whats exposed in artemis

Deno is only used to compile/convert TypeScript to JavaScript.

Only analysts or developers need to download Deno and only artemis is required to execute the JavaScript.
(Deno cannot run the scripts created for artemis, just like artemis should not be able to run code written for Deno)

Image a single key filter= that is mapped to a single date field and, if present, only returns the record if it's after the supplied ISO date format.

My main concern is increased complexity and duplicate work. The API already provides filtering for every artifact attribute. Creating a key filter for some or all artifact attributes would be very complex I think.

I will think about this a bit more.
I think plaso does filtering based on YAML files or something similar

Perhaps the approach plaso uses could be implemented.
Regardless, I am still hesitant to implement something like this