piskvorky / smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support for wasb/wasbs protocols

utkarsharma2 opened this issue · comments

Problem description

When trying to run the below code:

smart_open.open("wasb://someContainer/test.csv", transport_params={...})
or
smart_open.open("wasbs://someContainer/test.csv", transport_params={...})

Output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tati/Code/astro-sdk-new/astro-sdk/sql-cli/.nox/test-3-8-airflow-2-5/lib/python3.8/site-packages/smart_open/smart_open_lib.py", line 224, in open
    binary = _open_binary_stream(uri, binary_mode, transport_params)
  File "/home/tati/Code/astro-sdk-new/astro-sdk/sql-cli/.nox/test-3-8-airflow-2-5/lib/python3.8/site-packages/smart_open/smart_open_lib.py", line 399, in _open_binary_stream
    submodule = transport.get_transport(scheme)
  File "/home/tati/Code/astro-sdk-new/astro-sdk/sql-cli/.nox/test-3-8-airflow-2-5/lib/python3.8/site-packages/smart_open/transport.py", line 93, in get_transport
    raise NotImplementedError(message)
NotImplementedError: Unable to handle scheme 'wasb', expected one of ('', 'azure', 'file', 'gs', 'hdfs', 'http', 'https', 's3', 's3a', 's3n', 's3u', 'scp', 'sftp', 'ssh', 'viewfs', 'webhdfs'). Extra dependencies required by 'wasb' may be missing. See <https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst> for details.

ref: https://datacadamia.com/azure/wasb

Version: smart-open==6.3.0

@piskvorky is there any timeline to fix this?

@sunank200 I don't think there's anything to fix - it's just not supported yet.