Strings must be encoded before hashing

Question

Strings must be encoded before hashing

namedgraph opened this issue a year ago · comments

Martynas Jusevičius commented a year ago

Just installed awscurl on EC2 (Ubuntu) and trying to access the Neptune DB endpoint, but

awscurl 'https://xxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \
        -X POST \
        --data-binary \
        -d 'query=ASK {}' \
        --region us-east-1 \
        --service neptune-db

this gives me an error:

Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 543, in main
    inner_main(sys.argv[1:])
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 516, in inner_main
    response = make_request(args.request,
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 107, in make_request
    canonical_request, payload_hash, signed_headers = task_1_create_a_canonical_request(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 219, in task_1_create_a_canonical_request
    payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/utils.py", line 20, in sha256_hash_for_binary_data
    return hashlib.sha256(val).hexdigest()
TypeError: Strings must be encoded before hashing

Am I missing something? :)

Also, is intentional that the --data-binary option does not take an argument? Because it does in plain curl.

Igor Okulist · Answer 1 · Mon Aug 07 2023 05:47:16 GMT+0800 (China Standard Time)

What’s your python version?On Aug 6, 2023, at 4:39 PM, Martynas Jusevičius ***@***.***> wrote: Just installed awscurl on EC2 (Ubuntu) and trying to access the Neptune DB endpoint, but awscurl 'https://xxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \ -X POST \ --data-binary \ -d 'query=ASK {}' \ --region us-east-1 \ --service neptune-db this gives me an error: Traceback (most recent call last): File "/home/ubuntu/.local/bin/awscurl", line 8, in <module> sys.exit(main()) File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 543, in main inner_main(sys.argv[1:]) File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 516, in inner_main response = make_request(args.request, File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 107, in make_request canonical_request, payload_hash, signed_headers = task_1_create_a_canonical_request( File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 219, in task_1_create_a_canonical_request payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data) File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/utils.py", line 20, in sha256_hash_for_binary_data return hashlib.sha256(val).hexdigest() TypeError: Strings must be encoded before hashing Am I missing something? :) Also, is intentional that the --data-binary option does not take an argument? Because it does in plain curl. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Martynas Jusevičius · Answer 2 · Mon Aug 07 2023 05:50:28 GMT+0800 (China Standard Time)

$ python3 --version
Python 3.10.12

Igor Okulist · Answer 3 · Mon Aug 07 2023 06:04:56 GMT+0800 (China Standard Time)

Interesting Btw, why are using binary, your example has a readable string (in the -d parameter)?On Aug 6, 2023, at 4:39 PM, Martynas Jusevičius ***@***.***> wrote: Just installed awscurl on EC2 (Ubuntu) and trying to access the Neptune DB endpoint, but awscurl 'https://xxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \ -X POST \ --data-binary \ -d 'query=ASK {}' \ --region us-east-1 \ --service neptune-db this gives me an error: Traceback (most recent call last): File "/home/ubuntu/.local/bin/awscurl", line 8, in <module> sys.exit(main()) File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 543, in main inner_main(sys.argv[1:]) File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 516, in inner_main response = make_request(args.request, File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 107, in make_request canonical_request, payload_hash, signed_headers = task_1_create_a_canonical_request( File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/awscurl.py", line 219, in task_1_create_a_canonical_request payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data) File "/home/ubuntu/.local/lib/python3.10/site-packages/awscurl/utils.py", line 20, in sha256_hash_for_binary_data return hashlib.sha256(val).hexdigest() TypeError: Strings must be encoded before hashing Am I missing something? :) Also, is intentional that the --data-binary option does not take an argument? Because it does in plain curl. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Martynas Jusevičius · Answer 4 · Mon Aug 07 2023 06:08:43 GMT+0800 (China Standard Time)

I was just following the examples from Using RDF and SPARQL to access the graph in Amazon Neptune and replacing curl with awscurl :)

If I remove the --data-binary arg

awscurl 'https://octopustriplestorecluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \
        -X POST \
        -d 'query=ASK {}' \
        --region us-east-1 \
        --service neptune-db

I get a Neptune error:

{"requestId":"2ec4e671-f734-5519-03fd-6971f5229051","detailedMessage":"Missing 'query' or 'update' parameter for POST request","code":"MissingParameterException"}

Igor Okulist · Answer 5 · Mon Aug 07 2023 06:22:18 GMT+0800 (China Standard Time)

Hmm, I’d have to try that. Meantime could you check if this is your issue, and see if workaround in the end works for you:How to escape AMP query parameter right? · Issue #169 · okigan/awscurlgithub.comOn Aug 6, 2023, at 5:08 PM, Martynas Jusevičius ***@***.***> wrote: I was just following the examples from Using RDF and SPARQL to access the graph in Amazon Neptune and replacing curl with awscurl :) If I remove the --data-binary arg awscurl 'https://octopustriplestorecluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \ -X POST \ -d 'query=ASK {}' \ --region us-east-1 \ --service neptune-db I get a Neptune error: {"requestId":"2ec4e671-f734-5519-03fd-6971f5229051","detailedMessage":"Missing 'query' or 'update' parameter for POST request","code":"MissingParameterException"} —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

Martynas Jusevičius · Answer 6 · Mon Aug 07 2023 06:39:38 GMT+0800 (China Standard Time)

This works!

awscurl 'https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql' \
        -X POST \
        -d 'query=ASK {}' \
        --region us-east-1 \
        --service neptune-db \
        --header 'Content-Type: application/x-www-form-urlencoded'

Martynas Jusevičius · Answer 7 · Mon Aug 07 2023 14:15:57 GMT+0800 (China Standard Time)

@okigan what about --data-binary? Why does it differ from plain curl?

John Walker · Answer 8 · Mon Aug 07 2023 18:02:15 GMT+0800 (China Standard Time)

The following works for me for to send a CONSTRUCT query and requesting response as N-Triples (using URL from your example for consistency):

awscurl "https://xxxxxxxxxxxxcluster.cluster-ro-cnol6sn9sq5j.us-east-1.neptune.amazonaws.com:8182/sparql" \
  --service neptune-db \
  --data @query.rq \
  -H "Content-Type: application/sparql-query" \
  -H "Accept: application/n-triples"

Note that with query via POST with URL-encoded parameters the content type header of the HTTP request must be set to application/x-www-form-urlencoded. Perhaps awscurl does not automatically add that header using -d and --data, you could run in verbose mode to check that.