Error on save file

Question

Error on save file

heckad opened this issue 4 years ago · comments

Hi all!

  File "C:\Users\Asus\Documents\Projects\python\gelio_vostok_backend\files\views.py", line 12, in upload_file
    file = FileModel.objects.create(
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 782, in save_base
    updated = self._save_table(
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 924, in _do_insert
    return manager._insert(
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1391, in execute_sql
    for sql, params in self.as_sql():
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1334, in as_sql
    value_rows = [
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1335, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1335, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1286, in pre_save_val
    return field.pre_save(obj, add=True)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\fields\files.py", line 288, in pre_save
    file.save(file.name, file.file, save=False)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\fields\files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\core\files\storage.py", line 51, in save
    name = self.get_available_name(name, max_length=max_length)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 65, in do_wrap_path_impl
    return _to_posix_path(func(self, _to_sys_path(name), *args, **kwargs))
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 339, in get_available_name
    return super(S3Storage, self).get_available_name(name, max_length=max_length)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\core\files\storage.py", line 82, in get_available_name
    while self.exists(name) or (max_length and len(name) > max_length):
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 370, in exists
    return self.exists(name + "/")
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 359, in exists
    results = self.s3_connection.list_objects_v2(
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\botocore\client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\botocore\client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: Unknown

Seems it's incorrect handle file not exists error!

Dave Hall · Answer 1 · Thu Jul 23 2020 17:27:48 GMT+0800 (China Standard Time)

Can I see your S3 settings please? Feel free to blank out the access keys, but everything else would be useful.

…

On Wed, 22 Jul 2020 at 18:29, Kazantcev Andrey ***@***.***> wrote: Hi all! File "C:\Users\Asus\Documents\Projects\python\gelio_vostok_backend\files\views.py", line 12, in upload_file file = FileModel.objects.create( File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\query.py", line 433, in create obj.save(force_insert=True, using=self.db) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 745, in save self.save_base(using=using, force_insert=force_insert, File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 782, in save_base updated = self._save_table( File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 887, in _save_table results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\base.py", line 924, in _do_insert return manager._insert( File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\query.py", line 1204, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1391, in execute_sql for sql, params in self.as_sql(): File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1334, in as_sql value_rows = [ File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1335, in <listcomp> [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields] File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1335, in <listcomp> [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields] File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\sql\compiler.py", line 1286, in pre_save_val return field.pre_save(obj, add=True) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\fields\files.py", line 288, in pre_save file.save(file.name, file.file, save=False) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\db\models\fields\files.py", line 87, in save self.name = self.storage.save(name, content, max_length=self.field.max_length) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\core\files\storage.py", line 51, in save name = self.get_available_name(name, max_length=max_length) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 65, in do_wrap_path_impl return _to_posix_path(func(self, _to_sys_path(name), *args, **kwargs)) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 339, in get_available_name return super(S3Storage, self).get_available_name(name, max_length=max_length) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django\core\files\storage.py", line 82, in get_available_name while self.exists(name) or (max_length and len(name) > max_length): File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 370, in exists return self.exists(name + "/") File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\django_s3_storage\storage.py", line 359, in exists results = self.s3_connection.list_objects_v2( File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\botocore\client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "C:\Users\Asus\.virtualenvs\gelio_vostok_backend-XP0MyTwt\lib\site-packages\botocore\client.py", line 635, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: Unknown Seems it's incorrect handle file not exists error! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#112>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCG73ENACUDNLA4DHEDR44OXTANCNFSM4PE5ZBUA> .

Kazantcev Andrey · Answer 2 · Thu Jul 23 2020 18:43:30 GMT+0800 (China Standard Time)

# The AWS region to connect to.
AWS_REGION = "cdnspa.ams3"

# The full URL to the S3 endpoint.
AWS_S3_ENDPOINT_URL = "https://cdnspa.ams3.digitaloceanspaces.com"

Dave Hall · Answer 3 · Thu Jul 23 2020 18:50:04 GMT+0800 (China Standard Time)

Oh. That doesn't look like an S3 endpoint. That looks like a 3rd party S3-like thing. Are you using the AWS_S3_KEY_PREFIX setting?

…

On Thu, 23 Jul 2020 at 11:43, Kazantcev Andrey ***@***.***> wrote: # The AWS region to connect to. AWS_REGION = "cdnspa.ams3" # The full URL to the S3 endpoint. AWS_S3_ENDPOINT_URL = "https://cdnspa.ams3.digitaloceanspaces.com" — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCEHR2VDYIBL2TKJI3LR5AH6HANCNFSM4PE5ZBUA> .

Kazantcev Andrey · Answer 4 · Thu Jul 23 2020 18:52:47 GMT+0800 (China Standard Time)

No, what is this?

Strange, it crashes on name availability check. First time with 404 error, wich converted to OsError and trying to do self.exists(name + "/") although it should have returned False from exists method.

Dave Hall · Answer 5 · Thu Jul 23 2020 18:58:22 GMT+0800 (China Standard Time)

Just a hunch I had about the cause of the error. Since you're not using S3, but some 3rd part copy, I don't really have any ability to test. I'm happy to review and merge a MR that fixes your issue. This looks like some compatibility problem between S3 and digital ocean's S3 clone.

…

On Thu, 23 Jul 2020 at 11:53, Kazantcev Andrey ***@***.***> wrote: No, what is this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCFCDXFNJSXI6KBNK53R5AJA3ANCNFSM4PE5ZBUA> .

Kazantcev Andrey · Answer 6 · Thu Jul 23 2020 19:03:42 GMT+0800 (China Standard Time)

Please describe what should to do exists method. Should it return False if file not exists (now seems to returns None) and can I send False if I got 404 error?

Dave Hall · Answer 7 · Thu Jul 23 2020 19:09:10 GMT+0800 (China Standard Time)

The "exists" method should check if the file "exists", and return True if it does, and False if it does not. I don't think the current implementation returns None anywhere. https://github.com/etianen/django-s3-storage/blob/master/django_s3_storage/storage.py#L354 https://docs.djangoproject.com/en/3.0/ref/files/storage/#django.core.files.storage.Storage.exists

…

On Thu, 23 Jul 2020 at 12:04, Kazantcev Andrey ***@***.***> wrote: Please describe what should to do exists method. Should it return False if file not exists (now seems to returns None) and can I send False if I got 404 error? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCAHJJ7WHKOGL7WPFA3R5AKJ5ANCNFSM4PE5ZBUA> .

Kazantcev Andrey · Answer 8 · Thu Jul 23 2020 19:29:15 GMT+0800 (China Standard Time)

Is it correct that do _wrap_errors have applied only for meta method but when we do check on directory we don't do any check on exception?

Kazantcev Andrey · Answer 9 · Thu Jul 23 2020 19:47:25 GMT+0800 (China Standard Time)

After fix works as expected!