How to query an URL through REST

Question

How to query an URL through REST

kittrCZ opened this issue 8 years ago · comments

Hi,

I also tried to post message to the mailing list, but I hope I'll be able to find answer here faster. I have problem to query otypes URL by providing URL to the search query. I have tried several approaches and nothing seems to work. Could someone please help me/advice me on how to query observables by provided URL.

I'm sure that observable http://cloud02.conquistasc.com/anexo-0029304902940-1.zip?662239604036014079 exists in the elasticache.
Examples I tried:

$ curl -v -XGET -H "Accept: application/vnd.cif.v2+json" -H "Authorization: Token token=x" "http://localhost:5000/observables?q=http%3A%2F%2Fwww.emorybox.com%2Fny%2Fall%2Fus%2Fhelp%2Fios1548%2Fuix%2F630bddcbf996914f1b1f9f9947565828%20"
* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 5000 (#0)
> GET /observables?q=http%3A%2F%2Fwww.emorybox.com%2Fny%2Fall%2Fus%2Fhelp%2Fios1548%2Fuix%2F630bddcbf996914f1b1f9f9947565828%20 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: application/vnd.cif.v2+json
> Authorization: Token token=x
> 
< HTTP/1.1 503 Service Unavailable
< Content-Length: 31
< Date: Thu, 23 Jun 2016 23:19:22 GMT
< X-CIF-Media-Type: cif.v2
< Content-Type: application/json
< Connection: close
< 
* Closing connection 0
{"message":"Malformed request"}

$ curl -v -XGET -H "Accept: application/vnd.cif.v2+json" -H "Authorization: Token token=x" --data-urlencode "q=http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828" "http://localhost:5000/observables"
=> []

$ curl -v -XGET -H "Accept: application/vnd.cif.v2+json" -H "Authorization: Token token=x" "http://localhost:5000/observables?q=http:\/\/www.emorybox.com\/ny\/all\/us\/help\/ios1548\/uix\/630bddcbf996914f1b1f9f9947565828"
* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 5000 (#0)
> GET /observables?q=http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: application/vnd.cif.v2+json
> Authorization: Token token=x
> 
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json
< X-CIF-Media-Type: cif.v2
< Date: Thu, 23 Jun 2016 23:25:59 GMT
< Connection: close
< 
* Closing connection 0
[]

$ curl -v -XGET -H "Accept: application/vnd.cif.v2+json" -H "Authorization: Token token=x" "http://localhost:5000/observables?q=http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828"
* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 5000 (#0)
> GET /observables?q=http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: application/vnd.cif.v2+json
> Authorization: Token token=x
> 
< HTTP/1.1 200 OK
< Date: Thu, 23 Jun 2016 23:17:31 GMT
< Content-Type: application/json
< X-CIF-Media-Type: cif.v2
< Content-Length: 2
< Connection: close
< 
* Closing connection 0
[]

From Elasticache:

$ curl -XGET "http://localhost:9200/cif.observables-2016.06.21/_search?q=otype:url"
{"took":4,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":20053,"max_score":2.9000893,"hits":[{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"2c9393f294908a75ac29f6c1269f0ebed5601796add777ab3332dd2b7b2198ab","_score":2.9000893,"_source":{"group":["everyone"],"firsttime":"2016-06-21T00:13:49Z","tlp":"white","confidence":85,"otype":"url","id":"2c9393f294908a75ac29f6c1269f0ebed5601796add777ab3332dd2b7b2198ab","tags":["uce","uce-url"],"provider":"csirtg.io","@timestamp":"2016-06-21T00:23:00.716Z","description":"url parsed out of the message body sourced from unsolicited commercial email (spam)","@version":2,"reporttime":"2016-06-21T00:23:00Z","altid_tlp":"white","observable":"http://printsandijital.com/3obzmj","lang":"EN","altid":"https://csirtg.io/search?q=http://printsandijital.com/3obzMJ","protocol":-1,"portlist":"","lasttime":"2016-06-21T00:13:49Z","application":""}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"1d1f125e84e9b179166c75f78518152841403bf6008879627f4c890336a16c03","_score":2.9000893,"_source":{"lang":"EN","altid":"http://malwareurls.joxeankoret.com/normal.txt","lasttime":"2016-06-21T00:23:26Z","otype":"url","id":"1d1f125e84e9b179166c75f78518152841403bf6008879627f4c890336a16c03","group":["everyone"],"firsttime":"2016-06-21T00:23:26Z","confidence":65,"tlp":"green","tags":["malware"],"provider":"malwareurls.joxeankoret.com","@version":2,"@timestamp":"2016-06-21T00:23:26.251Z","altid_tlp":"white","observable":"http://down10b.zol.com.cn/zoldownload/rdvideo8.2at81_327255.exe","reporttime":"2016-06-21T00:23:24Z"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"782b5eca6d2d586adec6a9b1e899a1117396ad4190ba59b7b8d2e6c6470f2c63","_score":2.9000893,"_source":{"lasttime":"2016-06-21T00:23:26Z","altid":"http://malwareurls.joxeankoret.com/normal.txt","lang":"EN","reporttime":"2016-06-21T00:23:24Z","observable":"http://down.downcdn.net/","altid_tlp":"white","@timestamp":"2016-06-21T00:23:26.251Z","@version":2,"tags":["malware"],"provider":"malwareurls.joxeankoret.com","firsttime":"2016-06-21T00:23:26Z","group":["everyone"],"tlp":"green","confidence":65,"id":"782b5eca6d2d586adec6a9b1e899a1117396ad4190ba59b7b8d2e6c6470f2c63","otype":"url"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"3bde03d73ed2d151a8bb03457d69c5ae2b872b1d73ca686f60f4ddc10c681bbf","_score":2.9000893,"_source":{"lang":"EN","altid":"http://malwareurls.joxeankoret.com/normal.txt","lasttime":"2016-06-21T00:23:26Z","id":"3bde03d73ed2d151a8bb03457d69c5ae2b872b1d73ca686f60f4ddc10c681bbf","otype":"url","confidence":65,"tlp":"green","firsttime":"2016-06-21T00:23:26Z","group":["everyone"],"provider":"malwareurls.joxeankoret.com","tags":["malware"],"@version":2,"@timestamp":"2016-06-21T00:23:26.251Z","observable":"http://down.cdnxiazai.pw/cx/setup.exe","altid_tlp":"white","reporttime":"2016-06-21T00:23:24Z"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"e76a2ca8bad626dbcfd25b60249811a2e0026aaacafd11049a1a290bb066ab6c","_score":2.9000893,"_source":{"@timestamp":"2016-06-21T00:23:26.251Z","@version":2,"reporttime":"2016-06-21T00:23:24Z","observable":"http://ak.imgfarm.com/images/nocache/vicinio/installers/100000428.s12245.1/519300-150715132625-s12245.1/filmfanaticauto.exe_0","altid_tlp":"white","confidence":65,"tlp":"green","firsttime":"2016-06-21T00:23:26Z","group":["everyone"],"id":"e76a2ca8bad626dbcfd25b60249811a2e0026aaacafd11049a1a290bb066ab6c","otype":"url","provider":"malwareurls.joxeankoret.com","tags":["malware"],"altid":"http://malwareurls.joxeankoret.com/normal.txt","lasttime":"2016-06-21T00:23:26Z","lang":"EN"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"f26c75686395cc8e6bf347a59f0e7dbbc46d65f21d5e2dcb1c092816a3ce8879","_score":2.9000893,"_source":{"lang":"EN","altid":"http://malwareurls.joxeankoret.com/normal.txt","lasttime":"2016-06-21T00:23:26Z","group":["everyone"],"firsttime":"2016-06-21T00:23:26Z","confidence":65,"tlp":"green","otype":"url","id":"f26c75686395cc8e6bf347a59f0e7dbbc46d65f21d5e2dcb1c092816a3ce8879","tags":["malware"],"provider":"malwareurls.joxeankoret.com","@timestamp":"2016-06-21T00:23:26.251Z","@version":2,"reporttime":"2016-06-21T00:23:24Z","altid_tlp":"white","observable":"http://dl.pocomixing.com/n/55269651-5f88-42c3-ab68-5c2f0a000013/popcorn_time.exe?secure=1437946771_5d0ab77c332db1bc964a9b6a7dc0d035"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"07ef5c23b3223b26a7b0a82b3c3d7293ad2af251949eba1416137c7d9352592c","_score":2.9000893,"_source":{"lasttime":"2016-06-21T00:23:26Z","altid":"http://malwareurls.joxeankoret.com/normal.txt","lang":"EN","observable":"http://drive-google-com.fanalav.com/300d215614243499001f04e3a899dd80","altid_tlp":"white","reporttime":"2016-06-21T00:23:24Z","@version":2,"@timestamp":"2016-06-21T00:23:26.251Z","tags":["malware"],"provider":"malwareurls.joxeankoret.com","otype":"url","id":"07ef5c23b3223b26a7b0a82b3c3d7293ad2af251949eba1416137c7d9352592c","group":["everyone"],"firsttime":"2016-06-21T00:23:26Z","confidence":65,"tlp":"green"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"e8ae165ed091ce79347fde16b5f2d627e897c1af3482b16de44f515e0f879c6e","_score":2.9000893,"_source":{"lang":"EN","lasttime":"2016-06-21T00:23:26Z","altid":"http://malwareurls.joxeankoret.com/normal.txt","tags":["malware"],"provider":"malwareurls.joxeankoret.com","firsttime":"2016-06-21T00:23:26Z","group":["everyone"],"tlp":"green","confidence":65,"otype":"url","id":"e8ae165ed091ce79347fde16b5f2d627e897c1af3482b16de44f515e0f879c6e","reporttime":"2016-06-21T00:23:24Z","observable":"http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828","altid_tlp":"white","@timestamp":"2016-06-21T00:23:26.251Z","@version":2}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"d34b50a29c449ca631bb0993e910310f89f0476a5b29e8077ab98e714c073c09","_score":2.9000893,"_source":{"lasttime":"2016-06-21T00:23:26Z","altid":"http://malwareurls.joxeankoret.com/normal.txt","lang":"EN","altid_tlp":"white","observable":"http://www48.omrtw.com/","reporttime":"2016-06-21T00:23:24Z","@version":2,"@timestamp":"2016-06-21T00:23:26.251Z","tags":["malware"],"provider":"malwareurls.joxeankoret.com","otype":"url","id":"d34b50a29c449ca631bb0993e910310f89f0476a5b29e8077ab98e714c073c09","firsttime":"2016-06-21T00:23:26Z","group":["everyone"],"confidence":65,"tlp":"green"}},{"_index":"cif.observables-2016.06.21","_type":"observables","_id":"ddbed7d5f44a574678047457eec4c4a793edd2b8f7ed7bc55511dd1c26bb7e54","_score":2.9000893,"_source":{"firsttime":"2016-06-21T00:23:26Z","group":["everyone"],"confidence":65,"tlp":"green","id":"ddbed7d5f44a574678047457eec4c4a793edd2b8f7ed7bc55511dd1c26bb7e54","otype":"url","tags":["malware"],"provider":"malwareurls.joxeankoret.com","@timestamp":"2016-06-21T00:23:26.251Z","@version":2,"reporttime":"2016-06-21T00:23:24Z","altid_tlp":"white","observable":"http://cloud02.conquistasc.com/anexo-0029304902940-1.zip?662239604036014079","lang":"EN","altid":"http://malwareurls.joxeankoret.com/normal.txt","lasttime":"2016-06-21T00:23:26Z"}}]}}

wes · Answer 1 · Mon Jun 27 2016 20:40:53 GMT+0800 (China Standard Time)

hiya,

i want to say this may have been fixed in ~RC16 ?

https://github.com/csirtgadgets/massive-octo-spice/releases/tag/2.00.00-rc.16
cd046e8

$ cif -d -q http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828
2016-06-27 08:35:08,187 - DEBUG - cifsdk.client[94] - uri: https://localhost/observables
2016-06-27 08:35:08,187 - DEBUG - cifsdk.client[95] - params: {"nolog":null,"observable":"http:\/\/www.emorybox.com\/ny\/all\/us\/help\/ios1548\/uix\/630bddcbf996914f1b1f9f9947565828","limit":500,"gzip":1}
2016-06-27 08:35:08,187 - INFO - cifsdk.client[97] - searching...
2016-06-27 08:35:08,295 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): localhost
2016-06-27 08:35:09,838 - DEBUG - requests.packages.urllib3.connectionpool[383] - "GET /observables?gzip=1&observable=http%3A%2F%2Fwww.emorybox.com%2Fny%2Fall%2Fus%2Fhelp%2Fios1548%2Fuix%2F630bddcbf996914f1b1f9f9947565828&limit=500 HTTP/1.1" 200 7153
2016-06-27 08:35:09,838 - DEBUG - cifsdk.client[101] - status code: 200
...
2016-06-27 08:35:09,839 - INFO - cifsdk.client[416] - returned: 73 records
+---------+----------+----------------------+----------------------+----------------------------------+-------+----+-----+----------+------------+-------------+---------+-------+-------+-----------------------------+----------------------------------+-----------+
|   tlp   |  group   |       lasttime       |      reporttime      |            observable            | otype | cc | asn | asn_desc | confidence | description |   tags  | rdata | rtype | provider                    |              altid               | altid_tlp |
+---------+----------+----------------------+----------------------+----------------------------------+-------+----+-----+----------+------------+-------------+---------+-------+-------+-----------------------------+----------------------------------+-----------+
| limited | everyone | 2016-04-16T00:23:58Z | 2016-04-16T00:23:46Z | http://www.emorybox.com/ny/all.. |  url  |    |     |          |     65     |             | malware |       |       | malwareurls.joxeankoret.com | http://malwareurls.joxeankoret.. |   public  |
| limited | everyone | 2016-04-17T00:23:28Z | 2016-04-17T00:23:14Z | http://www.emorybox.com/ny/all.. |  url  |    |     |          |     65     |             | malware |       |       | malwareurls.joxeankoret.com | http://malwareurls.joxeankoret.. |   public  |
| limited | everyone | 2016-04-18T00:28:34Z | 2016-04-18T00:28:21Z | http://www.emorybox.com/ny/all.. |  url  |    |     |          |     65     |             | malware |       |       | malwareurls.joxeankoret.com | http://malwareurls.joxeankoret.. |   public  |

you can see via the -d flag in the python client how the url is getting encoded "so the server recognizeds is", mine is a little diff than your's (and we try to normalize it, lower case it, and .rstrip('/') trailing '/').

let me know if RC16 doesn't solve this for you (assuming you're doing the lower(), strip, escaping properly too).

TH · Answer 2 · Fri Jul 01 2016 04:42:11 GMT+0800 (China Standard Time)

Hey @wesyoung, so good news! We updated the CIF to the latest version and following cURL is working:

curl -v -XGET -H "Accept: application/vnd.cif.v2+json" -H "Authorization: Token token=x" "http://localhost:5000/observables?q=http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828"

Thank you for your help.

Additionally, I have encountered following error SSL3_GET_SERVER_CERTIFICATE when using CIF command line tool:

root@cifserver:/etc/cif# cif --token=X -d -q http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828[2016-06-30T13:44:40,404Z][INFO][main:268]: starting up client...
[2016-06-30T13:44:40,404Z][INFO][main:303]: running search...
[2016-06-30T13:44:40,404Z][DEBUG][CIF::SDK::Client:170]: uri created: https://localhost/observables?observable=http://www.emorybox.com/ny/all/us/help/ios1548/uix/630bddcbf996914f1b1f9f9947565828&limit=50000&gzip=1
[2016-06-30T13:44:40,405Z][DEBUG][CIF::SDK::Client:171]: making request...
[2016-06-30T13:44:40,459Z][INFO][CIF::SDK::Client:175]: status: 599
[2016-06-30T13:44:40,459Z][INFO][CIF::SDK::Client:181]: response size: < 1MB
SSL connection failed for localhost: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
 at /usr/local/bin/cif line 321.

I believe that it is just a misconfiguration on my side though...

Thanks for the help!

wes · Answer 3 · Fri Jul 01 2016 19:06:18 GMT+0800 (China Standard Time)

if it's a local TLS cert, try the --no-verify-ssl flag. that should fix it...

(you're very welcome!, cheers!)