piskvorky / smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

zstd write does not work with `wb` mode

djudd opened this issue · comments

Problem description

FileLikeProxy, and therefore smart_open, produces an AttributeError when trying to open a zstd S3 file in wb mode

Cousin to #810 maybe?

Steps/code to reproduce the problem

>>> smart_open.open('s3://my-real-bucket-was-here/a-path.zst', mode='wb')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/smart_open/smart_open_lib.py", line 251, in open
    return so_utils.FileLikeProxy(decoded, binary)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/smart_open/utils.py", line 213, in __init__
    self.__inner = inner
    ^^^^^^^^^^^^
AttributeError: 'zstd.ZstdCompressionWriter' object has no attribute '_FileLikeProxy__inner'

Only occurs with wb mode, not w mode

Versions

>>> import platform, sys, smart_open
>>> print(platform.platform())
Linux-5.4.209-116.367.amzn2.x86_64-x86_64-with-glibc2.35
>>> print("Python", sys.version)
Python 3.11.6 (main, Mar 11 2024, 06:41:25) [GCC 11.4.0]
>>> print("smart_open", smart_open.__version__)
smart_open 7.0.3
>>> print("wrapt", wrapt.__version__)
wrapt 1.15.0

Checklist

Before you create the issue, please make sure you have:

  • Described the problem clearly
  • Provided a minimal reproducible example, including any required data
  • Provided the version numbers of the relevant software