Request to support Elasticsearch response
dindoliboon opened this issue · comments
When CIF returns an Elasticsearch response, it returns a JSON string and not an object.
Get-CIF3Indicator -Confidence 8 -ResultSize 500Output:
{"hits":{"hits":[{"_source":{"reporttime":"2020-02-28T04:37:43.000000Z","provider":"0","group":["everyone"],"uuid":"3194dd4d-ff6f-4139-99b4-dce59612df96","tags":["honeypot","dionaea"],"indicator":"0.0.0.0","firsttime":"2020-02-27T00:59:38.000000Z","itype":"ipv4","lasttime":"2020-02-28T04:37:43.000000Z","tlp":"green","count":325,"confidence":8.0,"indicator_ipv4":"0.0.0.0"}},{"_source":{"reporttime":"2020-02-28T04:37:43.473257Z","provider":"0","group":["everyone"],"uuid":"32262f0d-8ff8-4851-8742-50852dcc8b9e","tags":["honeypot","dionaea"],"indicator":"0.0.0.0","firsttime":"2020-02-27T21:54:18.000000Z","itype":"ipv4","lasttime":"2020-02-28T04:37:43.000000Z","tlp":"green","count":0,"confidence":8.0,"indicator_ipv4":"0.0.0.0"}}]}}I modified an elseif statement in Format-CIF3ApiResponse.ps1 to handle those responses:
elseif ($Response.message -eq 'success' -or $null -ne $Response.data) {
Write-Verbose 'Received response from CIF API'
# Check for Elasticsearch results
# https://github.com/csirtgadgets/cifsdk-py-v3/blob/a659e84c63ff097942ed8e549340107c66886db6/cifsdk/client/http.py#L121
if ($InputObject.data -like '{"hits":{"hits":`[{"_source":*') {
$elasticsearchResult = ConvertFrom-Json -InputObject $InputObject.data
if ($null -eq $elasticsearchResult.hits.hits._source) {
Write-Error -Message "CIF API call succeeded, but responded with incorrect Elasticsearch value: $Response"
break
} else {
# set InputObject to hits.hits._sourcedata' property of Invoke-RestMethod return object for further processing
$InputObject = $elasticsearchResult.hits.hits._source
}
} else {
# set InputObject to 'data' property of Invoke-RestMethod return object for further processing
$InputObject = $InputObject.data
}
} Get-CIF3Indicator -Confidence 8 -ResultSize 2Output:
TLP : green
FirstTime : 2/28/2020 12:36:05 AM
Count : 1
Uuid : 7a5f1fe1-34d7-4815-94f2-290493402fd6
IType : ipv4
LastTime : 2/28/2020 12:38:33 AM
Provider : 0
Indicator_Ipv4 : 0.0.0.0
Group : {everyone}
Tag : {honeypot, dionaea}
ReportTime : 2/28/2020 12:38:33 AM
Confidence : 8
Indicator : 0.0.0.0
TLP : green
FirstTime : 1/31/2020 7:01:36 PM
Count : 1
Uuid : 66a9eb91-bdd1-4e39-80f9-96c25de40546
IType : ipv4
LastTime : 2/28/2020 12:38:33 AM
Provider : 0
Indicator_Ipv4 : 0.0.0.0
Group : {everyone}
Tag : {honeypot, cowrie}
ReportTime : 2/28/2020 12:38:33 AM
Confidence : 8
Indicator : 0.0.0.0
Adding the Raw parameter still returns the JSON string though. Get-CIF3Indicator.ps1 may need to be modified as well?
Get-CIF3Indicator -Confidence 8 -ResultSize 2 -Rawstatus data
------ ----
success {"hits":{"hits":[{"_source":{"reporttime":"2020-02-28T04:37:43.000000Z","provider":"0","group":["everyone"],"uuid":"3194dd4d-ff6f-4139-99b4-dce59612df96","tags":["honeypot","dionaea"],"indicator":"0.0.0.0","firsttime":"2020-02-27T00:59:38.000000Z","itype":"ipv4","lasttime":"2020-02-28T04:37:43.000000Z","tlp":"green","count":325,"confidence":8.0,"indicator_ipv4":"0.0.0.0"}},{"_source":{"reporttime":"2020-02-28T04:37:43.473257Z","provider":"0","group":["everyone"],"uuid":"32262f0d-8ff8-4851-8742-50852dcc8b9e","tags":["honeypot","dionaea"],"indicator":"0.0.0.0","firsttime":"2020-02-27T21:54:18.000000Z","itype":"ipv4","lasttime":"2020-02-28T04:37:43.000000Z","tlp":"green","count":0,"confidence":8.0,"indicator_ipv4":"0.0.0.0"}}]}}
Wow, good find @dindoliboon . Am I the only one confused why the CIF implementation between sqlite and ES stores would return diff data structures for the same data?
I'm thinking the JSON string return for -Raw may not be terrible since that param just instructs it to return the raw resp rather than formatting. What say ye?
Also, your code looks pretty good. Just a few thoughts:
-
variable confusion:
in the lines below, shouldInputObject.databeResponse.data?
if ($InputObject.data -like '{"hits":{"hits":``[{"_source":*') { $elasticsearchResult = ConvertFrom-Json -InputObject $InputObject.data -
for continuity purposes of existing PascalCase variables: could you swap elasticsearchResult to ElasticSearchResult?
-
for code readability: worth it to use
.StartsWith('{"hits":{"hits":[{"_source":')rather than-like? No escape char or wildcard needed with StartsWith method and then it's more analogous to the python cif3 client you cited. A quickmeasure-commandtest seems like it's similarly performant to the-likeoperator.
If you're good with all that, do you want to submit a PR for this?
They return different structures because they store the data totally differently.
SQLite is SQL based (obviosuly), so the data is taken and converted into JSON. Wheree ES is not. ES stores everything in JSON format so pulling the data out returns the data prepended with hits hits. It's addressed using in CIFSDK here https://github.com/csirtgadgets/cifsdk-py-v3/blob/22b3641ca2fbad646f0c58ead55db476d818af5e/cifsdk/client/zeromq.py#L90
This is expected. The python CIFSDK takes this into account:
After reading @sfinlon comments, it makes sense to leave the -Raw parameter alone. It gives the user the ability to manipulate the responses as they see fit.
Also added another if statement to check for empty ES responses. Instead of returning {}, it will break the foreach loop. This matches SQLite when formatting is used.
https://github.com/csirtgadgets/cifsdk-py-v3/blob/a659e84c63ff097942ed8e549340107c66886db6/cifsdk/client/http.py#L118
Created PR #4 for review.
Thanks everyone for the comments and suggestions!
Looks good @dindoliboon . I've merged in #4 , and it's helped draw my attention to some other items in this same area that need some TLC. Thanks!