When resolving git-lfs references, json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Question

When resolving git-lfs references, json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

jbraswell opened this issue 3 years ago · comments

We are unable to access attck.enterprise.actors, attck.enterprise.techniques, etc because the code is downloading a json file that is actually a git-lfs reference and not valid json.

It fails when parsing the output of https://github.com/swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True.

We are using version 3.0.1.

Reproduction below:

Python 3.8.6 (default, Jan  8 2021, 15:34:15)
[Clang 12.0.0 (clang-1200.0.26.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyattck import Attck
>>> attck = Attck()
>>> attck.enterprise.actors
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/blah/lib/python3.8/site-packages/pyattck/attck.py", line 224, in enterprise
    self.__load_data()
  File "/blah/lib/python3.8/site-packages/pyattck/attck.py", line 285, in __load_data
    Attck.__ENTERPRISE_NIST_DATA_JSON = self.__datasets.get_data(
  File "/blah/lib/python3.8/site-packages/pyattck/datasets.py", line 66, in get_data
    return self.__get_json_data(
  File "/blah/lib/python3.8/site-packages/pyattck/datasets.py", line 48, in __get_json_data
    data = requests.get(url).json()
  File "/blah/lib/python3.8/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/blah/.pyenv/versions/3.8.6/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/blah/.pyenv/versions/3.8.6/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/blah/.pyenv/versions/3.8.6/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Josh Rickard · Answer 1 · Thu Jun 03 2021 12:57:46 GMT+0800 (China Standard Time)

I’m working on a fix for this right now. I should have 3.0.2 out tomorrow sometime.

Josh Rickard · Answer 2 · Fri Jun 04 2021 01:50:05 GMT+0800 (China Standard Time)

@jbraswell Can you try this when instantiating the Attck class:

from pyattck import Attck

attck = Attck()

attck.__ENTERPRISE_GENERATED_DATA_JSON = 'https://github.com/swimlane/pyattck/blob/master/generated_attck_data.json?raw=True'

This is a workaround for the time being.

jbraswell · Answer 3 · Fri Jun 04 2021 10:42:49 GMT+0800 (China Standard Time)

This workaround worked for me. Thank you!

Amir Jalali · Answer 4 · Fri Jun 25 2021 10:15:02 GMT+0800 (China Standard Time)

Any update on this issue? The workaround didn't work for me!

>>> from pyattck import Attck
>>> attck = Attck()
>>> attck.__ENTERPRISE_GENERATED_DATA_JSON = 'https://github.com/swimlane/pyattck/blob/master/generated_attck_data.json?raw=True'
>>> attck.enterprise.techniques
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/lib64/python3.6/site-packages/pyattck/attck.py", line 224, in enterprise
    self.__load_data()
  File ".../venv/lib64/python3.6/site-packages/pyattck/attck.py", line 282, in __load_data
    data_type='generated_data', force=force
  File ".../venv/lib64/python3.6/site-packages/pyattck/datasets.py", line 69, in get_data
    force=force
  File ".../venv/lib64/python3.6/site-packages/pyattck/datasets.py", line 48, in __get_json_data
    data = requests.get(url).json()
  File ".../venv/lib64/python3.6/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Josh Rickard · Answer 5 · Fri Jun 25 2021 22:54:45 GMT+0800 (China Standard Time)

@amirjalali65 When did you try last? We ran into a storage issue but this has been resolved for now. Please give it a try again.

Amir Jalali · Answer 6 · Sat Jun 26 2021 00:56:17 GMT+0800 (China Standard Time)

@MSAdministrator I just tried it and got the same error. Am I setting the path correctly?

attck.__ENTERPRISE_GENERATED_DATA_JSON = 'https://github.com/swimlane/pyattck/blob/master/generated_attck_data.json?raw=True'

Josh Rickard · Answer 7 · Sat Jun 26 2021 01:01:36 GMT+0800 (China Standard Time)

Yeah looks correct - which version are you using and what OS?

Amir Jalali · Answer 8 · Sat Jun 26 2021 01:37:22 GMT+0800 (China Standard Time)

I was using 3.0.0, bumping to 3.0.1 fixed the issue. Thanks

Josh Rickard · Answer 9 · Sat Jun 26 2021 01:38:19 GMT+0800 (China Standard Time)

@amirjalali65 @jbraswell Please checkout the new release which resolves this issue and adds much better handling of configuration going forward.

pip install pyattck==4.0.0