SyntaxError: invalid syntax on grr_api_shell

Question

SyntaxError: invalid syntax on grr_api_shell

siftuser opened this issue 4 years ago · comments

while running

grr_api_shell --basic_auth_username "user" --basic_auth_password "pwd" \
 --exec_code 'print "\n".join(c.client_id for c in grrapi.SearchClients(""))' \
 http://localhost:8000

getting error

  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/grr_api_shell", line 11, in <module>
    load_entry_point('grr-api-client==3.4.0.post1', 'console_scripts', 'grr_api_shell')()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/api_shell.py", line 97, in main
    exec (flags.exec_code, dict(grrapi=grrapi))
  File "<string>", line 1
    print "\n".join(c.client_id for c in grrapi.SearchClients(""))
             ^
SyntaxError: invalid syntax

FYI, rebooted the mac as suggested here

Server is running GRR Version 3.4.0.4 in a docker container locally on the osx mojave

Appreciate help

siftuser commented 4 years ago

Thanks

mbushkov · Answer 1 · Sat May 16 2020 02:33:49 GMT+0800 (China Standard Time)

GRR is Python 3 now, so unlike Python 2, where you could say print x, you have to use print(x) with the latest versions of GRR.

Corrected snippet:

grr_api_shell --basic_auth_username "user" --basic_auth_password "pwd" \
 --exec_code 'print("\n".join(c.client_id for c in grrapi.SearchClients("")))' \
 http://localhost:8000

We need to update the GRR api client readme (https://github.com/google/grr/blob/master/api_client/python/README.md) - apparently, it still has Python 2-based snippets.

siftuser · Answer 2 · Sat May 16 2020 09:19:02 GMT+0800 (China Standard Time)

@mbushkov seeing this error now :(

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 401, in _CreateMessageFromTypeUrl
    message_descriptor = pool.FindMessageTypeByName(type_name)
KeyError: "Couldn't find message ApiGetCollectedTimelineArgs"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 582, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 479, in ConvertMessage
    methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 607, in _ConvertAnyMessage
    sub_message = _CreateMessageFromTypeUrl(type_url, self.descriptor_pool)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 404, in _CreateMessageFromTypeUrl
    'Can not find message descriptor by type_url: {0}.'.format(type_url))
TypeError: Can not find message descriptor by type_url: type.googleapis.com/ApiGetCollectedTimelineArgs.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 582, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 596, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse default field: Can not find message descriptor by type_url: type.googleapis.com/ApiGetCollectedTimelineArgs..

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 567, in _ConvertFieldValuePair
    self.ConvertMessage(item, sub_message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse argsTypeDescriptor field: Failed to parse default field: Can not find message descriptor by type_url: type.googleapis.com/ApiGetCollectedTimelineArgs...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/grr_api_shell", line 11, in <module>
    load_entry_point('grr-api-client==3.4.0.post1', 'console_scripts', 'grr_api_shell')()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/api_shell.py", line 97, in main
    exec (flags.exec_code, dict(grrapi=grrapi))
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/api.py", line 32, in SearchClients
    return client.SearchClients(query, context=self._context)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/client.py", line 273, in SearchClients
    items = context.SendIteratorRequest("SearchClients", args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/context.py", line 53, in SendIteratorRequest
    first_page = next(pages)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/context.py", line 36, in _GeneratePages
    result = self.connector.SendRequest(handler_name, args_copy)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/connectors/http_connector.py", line 249, in SendRequest
    self._InitializeIfNeeded()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/connectors/http_connector.py", line 139, in _InitializeIfNeeded
    self._FetchRoutingMap()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grr_api_client/connectors/http_connector.py", line 113, in _FetchRoutingMap
    json_format.Parse(json_str, proto, ignore_unknown_fields=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 430, in Parse
    return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 450, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse items field: Failed to parse argsTypeDescriptor field: Failed to parse default field: Can not find message descriptor by type_url: type.googleapis.com/ApiGetCollectedTimelineArgs....

mbushkov · Answer 3 · Sat May 16 2020 15:02:44 GMT+0800 (China Standard Time)

The error is caused by version mismatch (the error message will be much more informative in the upcoming release).
Basically, you run the server version 3.4.0.4, but likely use grr-api-client from PIP (if you do pip install grr-api-client). Current PIP grr-api-client version is the this one: 3.4.0.1.

grr-api-client may have issues when used with a newer version of the server.

The workaround is: you either have to use the Docker image corresponding to 3.4.0.1, or build the grr-api-client from HEAD (that'd require checking out GRR and running pip install -e grr/proto; pip install -e api_client/python).

siftuser · Answer 4 · Sun May 17 2020 13:28:11 GMT+0800 (China Standard Time)

Thanks @mbushkov that was very useful.

Would you help to point reference doc to a) list all the flows that were ran against a client b) run a specific flow on a client using api cli. thank you

siftuser · Answer 5 · Mon May 18 2020 19:36:55 GMT+0800 (China Standard Time)

gentle reminder - any suggestion on how to a) list all the flows that were ran against a client and b) run a specific flow on a client using grr_api_shell. thank you

mbushkov · Answer 6 · Mon May 18 2020 23:39:58 GMT+0800 (China Standard Time)

Hey, so here are a few examples.

To list the flows:

flows = list(grrapi.Client("C.123456").ListFlows())
for f in flows:
  print(f.data)

Starting a flow:

args = grrapi.types.CreateFlowArgs("FileFinder")
print(type(args))
# Will output: grr_response_proto_flows_pb2.FileFinderArgs
# This corresponds to: https://github.com/google/grr/search?l=Protocol+Buffer&q=FileFinderArgs
# Please consult the protobuf definition and the UI to see which values should be set.
args.paths.append("/foo/*")
f = grrapi.Client("C.12345").CreateFlow("FileFinder", args)
print(f.data)

siftuser · Answer 7 · Tue May 19 2020 01:30:20 GMT+0800 (China Standard Time)

Thank you @mbushkov

FileFinder flow works like a charm! Is there a way to include Collect extended attributes flag ? Tried couple of ways but it did not like (args.action.download.collectExtAttrs = "true")

List flows code, throws error

grr_api_shell --basic_auth_username user --basic_auth_password pass \
  --exec_code 'flows = list(grrapi.Client("C.1234").ListFlows()) \
  for f in flows:
  print(f.data)' \ 
http://localhost:8000

IndentationError: unexpected indent

grr_api_shell --basic_auth_username user --basic_auth_password pass \
  --exec_code 'flows = list(grrapi.Client("C.1234").ListFlows())  for f in flows:
  print(f.data)'  \
http://localhost:8000

SyntaxError: invalid syntax

mbushkov · Answer 8 · Tue May 19 2020 01:45:52 GMT+0800 (China Standard Time)

The code snippets that I posted earlier are not meant to be ran as one-liners. You can put them in a file, though, and use the --exec_file argument instead of --exec_code (see https://github.com/google/grr/tree/master/api_client/python#using-command-line-api-shell).

If you want to use flows snippet as a one liner, you need something along the lines of:

grr_api_shell --basic_auth_username "user" --basic_auth_password "pwd" \
  --exec_code 'print("\n".join(repr(f) for f in grrapi.Client("C.1234").ListFlows()))' \
  http://localhost:1234

As for collect extended attributes parameter, you have to set it differently depending on whether you want to stat or download.

If you want to stat, you do (see this):

args.stat.collect_ext_attrs = True

If you want to download, you do (see this):

args.action.action_type = args.action.DOWNLOAD
args.action.download.collect_ext_attrs = True

siftuser · Answer 9 · Tue May 19 2020 09:10:43 GMT+0800 (China Standard Time)

Thank you @mbushkov

list flow worked as expected. thanks!

For the filefinder.Download flow, I get following error upon including collect_ext_attrs, it works without that argument ... any clue ? Thank you

  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/grr_api_shell", line 11, in <module>
    load_entry_point('grr-api-client', 'console_scripts', 'grr_api_shell')()
  File "/Users/user/Downloads/grr/api_client/python/grr_api_client/api_shell.py", line 96, in main
    exec (flags.exec_code, dict(grrapi=grrapi))
  File "<string>", line 5, in <module>
AttributeError: download

mbushkov · Answer 10 · Tue May 19 2020 15:21:14 GMT+0800 (China Standard Time)

Ah, it should args.action.download. Corrected the above snippet:

args.action.action_type = args.action.DOWNLOAD
args.action.download.collect_ext_attrs = True

siftuser · Answer 11 · Tue May 19 2020 17:39:43 GMT+0800 (China Standard Time)

Thank you @mbushkov !

download was missing from args.action.download.collect_ext_attrs earlier ... now working as expected