VirusTotal: "snake has encountered an Error!"
Schillers opened this issue · comments
Hi,
I have installed the VirusTotal Scale (snake-scales) and get multiple "snake has encountered an Error!" messages on the application.
Is this a known issue?
No known issues until now. Are you able to provide some of the snake.log for when that is occurring? I assume you have just installed the scale and set the API key.
Also are you on the latest build I think I addressed some poor handling for when the sample was actually not on VT.
Alex
Alex, thanks for getting back to me so quickly, i know I am not the only one with this issue.
I have installed the scale and entered my API key. The version is e8b8ae5.
I have had a look through the log an found the below which might be the source of the error?
[E 180328 15:09:10 web:1621] Uncaught exception POST /scale/virustotal/interface (xxx.xxx.xxx.xx)
HTTPServerRequest(protocol='http', host='xxx.xx.xx.xxx:5000', method='POST', uri='/scale/virustotal/interface', version='HTTP/1.1', remote_ip='xxx.xxx.xxx.xx')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/gen.py", line 1099, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/routes/scale.py", line 115, in post
output = await loop.run_in_executor(None, command, data['args'], data['sha256_digest'])
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 726, in wrapper
output = func(args=args_, file=file_storage, opts=opts, self=self)
File "/usr/local/lib/python3.5/dist-packages/snake_virustotal/interface.py", line 113, in info
'first_seen': j['first_seen'],
KeyError: 'first_seen'
I have also attached a screenshot of the issue.
Ah awesome, that is just the info I need and we can't fix things if we don't know about them so thanks for raising it as an issue. I'll go investigate this now shouldn't take too long to fix :)
Alex, thanks for looking into this. Just for a little more info, this is the full error when trying to show the virustotal interface:
[E 180328 15:35:26 web:1621] Uncaught exception POST /scale/virustotal/interface (xxx.xxx.xxx.xx)
HTTPServerRequest(protocol='http', host='xxx.xx.xx.xxx:5000', method='POST', uri='/scale/virustotal/interface', version='HTTP/1.1', remote_ip='xxx.xxx.xxx.xx')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/gen.py", line 1099, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/routes/scale.py", line 115, in post
output = await loop.run_in_executor(None, command, data['args'], data['sha256_digest'])
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 726, in wrapper
output = func(args=args_, file=file_storage, opts=opts, self=self)
File "/usr/local/lib/python3.5/dist-packages/snake_virustotal/interface.py", line 113, in info
'first_seen': j['first_seen'],
KeyError: 'first_seen'
[E 180328 15:35:26 web:2106] 500 POST /scale/virustotal/interface (xxx.xxx.xxx.xx) 12.44ms
[I 180328 15:35:27 web:2106] 304 GET /scale/virustotal/interface (xxx.xxx.xxx.xx) 1.58ms
[E 180328 15:35:27 web:1621] Uncaught exception POST /scale/virustotal/interface (xxx.xxx.xxx.xx)
HTTPServerRequest(protocol='http', host='xxx.xx.xx.xxx:5000', method='POST', uri='/scale/virustotal/interface', version='HTTP/1.1', remote_ip='xxx.xxx.xxx.xx')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/gen.py", line 1099, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/routes/scale.py", line 115, in post
output = await loop.run_in_executor(None, command, data['args'], data['sha256_digest'])
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 726, in wrapper
output = func(args=args_, file=file_storage, opts=opts, self=self)
File "/usr/local/lib/python3.5/dist-packages/snake_virustotal/interface.py", line 179, in parents
if 'compressed_parents' in j['additional_info']:
KeyError: 'additional_info'
[E 180328 15:35:27 web:2106] 500 POST /scale/virustotal/interface (xxx.xxx.xxx.xx) 14.80ms
[I 180328 15:35:27 web:2106] 304 GET /scale/virustotal/interface (xxx.xxx.xxx.xx) 2.30ms
[I 180328 15:35:27 web:2106] 200 POST /scale/virustotal/interface (xxx.xxx.xxx.xx) 12.64ms
[I 180328 15:35:27 web:2106] 304 GET /scale/virustotal/interface (xxx.xxx.xxx.xx) 2.34ms
[E 180328 15:35:27 web:1621] Uncaught exception POST /scale/virustotal/interface (xxx.xxx.xxx.xx)
HTTPServerRequest(protocol='http', host='xxx.xx.xx.xxx:5000', method='POST', uri='/scale/virustotal/interface', version='HTTP/1.1', remote_ip='xxx.xxx.xxx.xx')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/gen.py", line 1099, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/routes/scale.py", line 115, in post
output = await loop.run_in_executor(None, command, data['args'], data['sha256_digest'])
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 726, in wrapper
output = func(args=args_, file=file_storage, opts=opts, self=self)
File "/usr/local/lib/python3.5/dist-packages/snake_virustotal/interface.py", line 142, in names
return j['submission_names']
KeyError: 'submission_names'
[E 180328 15:35:27 web:2106] 500 POST /scale/virustotal/interface (xxx.xxx.xxx.xx) 13.68ms
[E 180328 15:35:28 web:1621] Uncaught exception POST /scale/virustotal/interface (xxx.xxx.xxx.xx)
HTTPServerRequest(protocol='http', host='xxx.xx.xx.xxx:5000', method='POST', uri='/scale/virustotal/interface', version='HTTP/1.1', remote_ip='xxx.xxx.xxx.xx')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/local/lib/python3.5/dist-packages/tornado-5.0.1-py3.5-linux-x86_64.egg/tornado/gen.py", line 1099, in run
value = future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/routes/scale.py", line 115, in post
output = await loop.run_in_executor(None, command, data['args'], data['sha256_digest'])
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 726, in wrapper
output = func(args=args_, file=file_storage, opts=opts, self=self)
File "/usr/local/lib/python3.5/dist-packages/snake_virustotal/interface.py", line 160, in urls
return j['ITW_urls']
KeyError: 'ITW_urls'
[E 180328 15:35:28 web:2106] 500 POST /scale/virustotal/interface (xxx.xxx.xxx.xx) 17.59ms
Hmm this is odd, looks like not all of the json has been saved into the VT entry, but I can't replicate this!
Does resetting the VT cache for the sample fix the error?
curl 'http://127.0.0.1:5000/scale/virustotal/interface' -d '{"sha256_digest":"SAMPLE_HASH_HERE","type":"pull","command":"info","args":{"cache":"false"}}' -XPOST
Otherwise what json is VT returning you for the sample?
curl --request POST --url 'https://www.virustotal.com/vtapi/v2/file/report' -d apikey=API_KEY -d 'resource=SAMPLE_HASH_HERE' -d allinfo=1
Sorry for the inconvenience.
Ah, are you using a public VirusTotal API key or a private one? If public this would make sense. I will have to tweak the scale to accommodate the limitations of public VT API.
Right, I have updated the VT scale to support the public API (WithSecureLabs/snake-scales@682168e), I also had to make a minor change to the core so you should be error free if you do the following:
sys/install.sh # Should upgrade snake to 1.0.1, let me know if not
sudo snake upgrade virustotal # Upgrade to 1.1
Reload snake and it should be good to go.
Thanks Alex, I will give it a go when I get a chance!
Alex, thanks for checking that out. Sorry for the delay in reply.
I have updated and still get the same error. I still get the same type of error as above.
Hmm, that is odd the new VT scale should remove the features unavailable when using a public API key. Are you still seeing all functions? The new scale should only show: results. Did you restart Snake after running the steps above?
All the functions are still showing in the interfaces tab.
I have restarted snake, I have noticed that when I put snake check virustotal I get this error:
Checking: virustotal
Traceback (most recent call last):
File "/usr/local/bin/snake", line 11, in <module>
load_entry_point('snake==1.0', 'console_scripts', 'snake')()
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/snake_utility.py", line 103, in main
check(args.scale[0]) # XXX: Hmm, namespace clash? nargs is 1
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/snake_utility.py", line 37, in check
scale_manager._ScaleManager__load_scales([scale]) # pylint: disable=no-member, protected-access
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/core/scale_manager.py", line 104, in __load_scales
self.__load_scale(entry_point.name, loader.path)
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/core/scale_manager.py", line 71, in __load_scale
scale.load_components()
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 526, in load_components
raise err
File "/usr/local/lib/python3.5/dist-packages/snake-1.0-py3.5.egg/snake/scale.py", line 522, in load_components
self.components['upload'] = importlib.import_module('snake.scales.' + self.name + '.upload').Upload()
AttributeError: module 'snake.scales.virustotal.upload' has no attribute 'Upload'
Looks like my sys/install.sh has not upgraded Snake, still says 1.0 there should say 1.0.1!
Does the following upgrade snake to 1.0.1?
cd snake
git pull
cd snake-core
sudo python3 setup.py install --force
So, good news and really bad news...
Good news:
The update worked and I am now running 1.0.1...
Bad news:
I am now unable to see any previous uploads and I am unable to upload a new sample. I also appear to have lost all scales in the upload menu...
Looking through the logs, I now have:
[E 180410 14:06:42 web:1621] Uncaught exception GET /store?limit=10&sort=timestamp&file_type=memory (xxx.xxx.xxx.xx)
HTTPServerRequest(protocol='http', host='xxx.xx.xx.xxx:5000', method='GET', uri='/store?limit=10&sort=timestamp&file_type=memory', version='HTTP/1.1', remote_ip='xxx.xxx.xxx.xx')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1541, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "/usr/local/lib/python3.5/dist-packages/webargs/core.py", line 448, in wrapper
force_all=force_all)
File "/usr/local/lib/python3.5/dist-packages/webargs/core.py", line 370, in parse
parsed = self._parse_request(schema=schema, req=req, locations=locations)
File "/usr/local/lib/python3.5/dist-packages/webargs/core.py", line 291, in _parse_request
argname = field_obj.data_key or argname
AttributeError: 'String' object has no attribute 'data_key'
I have a feeling that this might be due to packages being the incorrect versions (probably marshmallow being too old), I need to freeze the version numbers into setup.py, that is an error on my part, apologies. Does this work:
sudo pip3 install celery==4.1.0 celery[redis]==4.1.0 marshmallow==3.0.0b7 motor==1.2.1 python-magic==0.4.15 pyyaml==3.12 requests==2.18.4 tornado==5.0.1 webargs==2.0.0
Worked like a charm!
Tank you, snake and virustotal now working perfectly!
Thanks for riding through the teething issues :) I will push a commit to freeze the packages.