OverflowError: Python int too large to convert to SQLite INTEGER
jayvdb opened this issue · comments
Similar to #226, using schemathesis, I run into a lot of
OverflowError: Python int too large to convert to SQLite INTEGER
There is one issue in DRF about this
encode/django-rest-framework#7134
And I intent to raise another one, and submit a PR for one, but it occurs all over the place and need a few local patches until Django core takes up the issue themselves.
Here are a few of them
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 239, in get
return self.list(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/mixins.py", line 38, in list
queryset = self.filter_queryset(self.get_queryset())
File "/usr/lib/python3.8/site-packages/oscarapi/views/basket.py", line 293, in get_queryset
basket = self.check_basket_permission(self.request, basket_pk=basket_pk)
File "/usr/lib/python3.8/site-packages/oscarapi/views/utils.py", line 54, in check_basket_permission
basket = generics.get_object_or_404(Basket.objects, pk=basket_pk)
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 19, in get_object_or_404
return _get_object_or_404(queryset, *filter_args, **filter_kwargs)
File "/usr/lib/python3.8/site-packages/django/shortcuts.py", line 76, in get_object_or_404
return queryset.get(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 411, in get
num = len(clone)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 258, in __len__
self._fetch_all()
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 115, in debug_sql
sql = self.db.ops.last_executed_query(self.cursor, sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/operations.py", line 152, in last_executed_query
params = self._quote_params_for_last_executed_query(params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/operations.py", line 141, in _quote_params_for_last_executed_query
return cursor.execute(sql, params).fetchone()
OverflowError: Python int too large to convert to SQLite INTEGER
[03:12:36][ERROR] django.request log.py:log_response:222 | Internal Server Error: /oscarapi/baskets/9223372036854775808/lines/
[03:12:36][ERROR] django.server basehttp.py:log_message:157 | "GET /oscarapi/baskets/9223372036854775808/lines/ HTTP/1.1" 500 164Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 285, in put
return self.update(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/mixins.py", line 65, in update
instance = self.get_object()
File "/usr/lib/python3.8/site-packages/oscarapi/views/basic.py", line 91, in get_object
basket = super(BasketDetail, self).get_object()
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 96, in get_object
obj = get_object_or_404(queryset, **filter_kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 19, in get_object_or_404
return _get_object_or_404(queryset, *filter_args, **filter_kwargs)
File "/usr/lib/python3.8/site-packages/django/shortcuts.py", line 76, in get_object_or_404
return queryset.get(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 411, in get
num = len(clone)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 258, in __len__
self._fetch_all()
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 115, in debug_sql
sql = self.db.ops.last_executed_query(self.cursor, sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/operations.py", line 152, in last_executed_query
params = self._quote_params_for_last_executed_query(params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/operations.py", line 141, in _quote_params_for_last_executed_query
return cursor.execute(sql, params).fetchone()
OverflowError: Python int too large to convert to SQLite INTEGER
[03:28:26][ERROR] django.request log.py:log_response:222 | Internal Server Error: /oscarapi/baskets/9223372071222935553/
[03:28:26][ERROR] django.server basehttp.py:log_message:157 | "PUT /oscarapi/baskets/9223372071222935553/ HTTP/1.1" 500 164Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 285, in put
return self.update(request, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/rest_framework/mixins.py", line 65, in update
instance = self.get_object()
File "/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 83, in get_object
queryset = self.filter_queryset(self.get_queryset())
File "/usr/lib/python3.8/site-packages/oscarapi/views/basket.py", line 325, in get_queryset
basket = get_object_or_404(operations.editable_baskets(), pk=basket_pk)
File "/usr/lib/python3.8/site-packages/django/shortcuts.py", line 76, in get_object_or_404
return queryset.get(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 411, in get
num = len(clone)
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 258, in __len__
self._fetch_all()
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.8/site-packages/django/db/backends/utils.py", line 115, in debug_sql
sql = self.db.ops.last_executed_query(self.cursor, sql, params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/operations.py", line 152, in last_executed_query
params = self._quote_params_for_last_executed_query(params)
File "/usr/lib/python3.8/site-packages/django/db/backends/sqlite3/operations.py", line 141, in _quote_params_for_last_executed_query
return cursor.execute(sql, params).fetchone()
OverflowError: Python int too large to convert to SQLite INTEGER
[05:40:41][ERROR] django.request log.py:log_response:222 | Internal Server Error: /oscarapi/baskets/9223372036854775808/lines/0/
[05:40:41][ERROR] django.server basehttp.py:log_message:157 | "PUT /oscarapi/baskets/9223372036854775808/lines/0/ HTTP/1.1" 500 164
Thanks for the effort and the report but this has unfortunately nothing to do with OscarAPI.
I also think that this is not something DRF should handle, as it's something specific to sqlite3 and it's integer value limitations. It's a low level exception that says to you that your having a serious problem. So the "Internal Server Error" is actually something you want to see (and not be replaced in any other message) as this is really a bad thing and should be fixed in the correct layer.
I guess that if this needs some specific attention, the sqlite3 backend of django is the place for it.
Thanks for merging #228 - your reasoning there is why I put up that patch - it at least channels these errors through one pathway.
Your reasoning here is exactly why I didnt put up the hackier fixes. I agree the problem isn't OscarAPI, as I see it in other apps also.
However IMO it "should" be fixed in DRF core, as the very large ID that it is processing can not possibly be in the underlying data on the connected database, due to the same underlying limitation causing the OverflowError, so 404 is a reasonable quick answer, and would have been the answer if connected to a different backend. Where to detect the overflow may still need fine tuning, and fingers crossed it can be fine tuned enough to eliminate any cases where 404 might not be an appropriate answer.
(I also suspect that the sqlite driver may be part of the problem, so I am not saying you are wrong there - I dont know enough about it to say for sure.)
A 404 in a GET query would something I'd say is worth discussing in the implementation of the DRF specific part where it occurs. You already opened up encode/django-rest-framework#7448 which is the right place for it I guess.