ls command fails while parsing response

Question

ls command fails while parsing response

ramonorru opened this issue a year ago · comments

I'm using s3cmd (version 2.3.0, homebrew installed) to connect to a ceph radosgw ( 17.2.3 quincy release). Bucket creation and PUT are working while la and ls are failing.

Here's the output with the debug flag on:

s3cmd ls -d
DEBUG: s3cmd version 2.3.0
DEBUG: ConfigParser: Reading file '/Users/***/.s3cfg'
DEBUG: ConfigParser: access_key->PQ...17_chars...Q
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->default
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->False
DEBUG: ConfigParser: check_ssl_hostname->False
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: connection_max_age->5
DEBUG: ConfigParser: connection_pooling->True
DEBUG: ConfigParser: content_disposition->
DEBUG: ConfigParser: content_type->
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/local/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->kk...0_chars...k
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.lnf.infn.it:8080
DEBUG: ConfigParser: host_bucket->%(bucket).s3.lnf.infn.it:8080
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: kms_key->
DEBUG: ConfigParser: limit->-1
DEBUG: ConfigParser: limitrate->0
DEBUG: ConfigParser: list_allow_unordered->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: long_listing->False
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: multipart_copy_chunk_size_mb->1024
DEBUG: ConfigParser: multipart_max_chunks->10000
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: public_url_use_https->False
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->65536
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: requester_pays->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: restore_priority->Standard
DEBUG: ConfigParser: secret_key->eT...37_chars...0
DEBUG: ConfigParser: send_chunk->65536
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: signurl_use_https->False
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: ssl_client_cert_file->
DEBUG: ConfigParser: ssl_client_key_file->
DEBUG: ConfigParser: stats->False
DEBUG: ConfigParser: stop_on_error->False
DEBUG: ConfigParser: storage_class->
DEBUG: ConfigParser: throttle_max->100
DEBUG: ConfigParser: upload_id->
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_http_expect->False
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Command: ls
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(None): s3.lnf.infn.it:8080
DEBUG: canonical_headers = host:s3.lnf.infn.it:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20231106T222915Z

DEBUG: Canonical Request:
GET
/

host:s3.lnf.infn.it:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20231106T222915Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20231106T222915Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=*********************/20231106/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=4b039cb0a0482d7f092eb7bd125493c01e034f69244814856d56a222a6e8209b', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): s3.lnf.infn.it:8080
DEBUG: ConnMan.get(): creating new connection: https://s3.lnf.infn.it:8080
DEBUG: Using ca_certs_file None
DEBUG: Using ssl_client_cert_file None
DEBUG: Using ssl_client_key_file None
DEBUG: Disabling SSL certificate checking
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: non-proxied HTTPSConnection(s3.lnf.infn.it, 8080)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='GET', uri='/', headers={'x-amz-date': '20231106T222915Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=*************************/20231106/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=4b039cb0a0482d7f092eb7bd125493c01e034f69244814856d56a222a6e8209b', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://s3.lnf.infn.it:8080#1)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns='
         b'"http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>rorru</ID><Disp'
         b'layName>Ronfffffff</DisplayName></Owner><Buckets><Bucket><Name>test<'
         b'/Name><CreationDate>2023-11-06T19:01:50.469Z</CreationDate></Bucket>'
         b'<Bucket><Name>testbuck</Name><CreationDate>2023-11-06T22:17:05.836Z<'
         b'/CreationDate></Bucket></Buckets></ListAllMyBucketsResult>',
 'headers': {'content-type': 'application/xml',
             'date': 'Mon, 06 Nov 2023 22:29:15 GMT',
             'transfer-encoding': 'chunked',
             'x-amz-request-id': 'tx00000a435c88f7d301d9b-00654968bb-6538-default'},
 'reason': 'OK',
 'status': 200}

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd ls -d
Problem: <class 'TypeError: sequence item 1: expected str instance, bytes found
S3cmd:   2.3.0
python:   3.12.0 (main, Oct  5 2023, 15:48:48) [Clang 14.0.0 (clang-1400.0.29.202)]
environment LANG=NOTSET

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3286, in <module>
    rc = main()
         ^^^^^^
  File "/usr/local/bin/s3cmd", line 3183, in main
    rc = cmd_func(args)
         ^^^^^^^^^^^^^^
  File "/usr/local/bin/s3cmd", line 171, in cmd_ls
    subcmd_all_buckets_list(s3)
  File "/usr/local/bin/s3cmd", line 176, in subcmd_all_buckets_list
    response = s3.list_all_buckets()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/S3.py", line 327, in list_all_buckets
    response["list"] = getListFromXml(response["data"], "Bucket")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py", line 277, in getListFromXml
    tree = getTreeFromXml(xml)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py", line 263, in getTreeFromXml
    xml, xmlns = stripNameSpace(encode_to_s3(xml))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/s3cmd/2.3.0/libexec/lib/python3.12/site-packages/S3/BaseUtils.py", line 255, in stripNameSpace
    xml = RE_XML_NAMESPACE.sub("\\1\\2", xml, 1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sequence item 1: expected str instance, bytes found

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Kelson Vibber · Answer 1 · Fri Nov 10 2023 10:46:23 GMT+0800 (China Standard Time)

Duplicate of #1343 ?

ramonorru · Answer 2 · Fri Nov 10 2023 17:12:41 GMT+0800 (China Standard Time)

Duplicate of #1343 ?

Yes, I can confirm I'm using 2 machines both running python 3.12.0 (Sonoma and Monterey), and the error is exactly the same.
The workaround supplied by https://github.com/sveltema at #1343 (comment) worked like a charm.

Thanks @kvibber , I didn't manage to find the duplicate before.

I'm closing the issue #

ramonorru · Answer 3 · Fri Nov 10 2023 17:13:05 GMT+0800 (China Standard Time)

Duplicate of #1343