cockroachdb / django-cockroachdb

CockroachDB Backend for Django

Home Page:http://cockroachlabs.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

add SmallAutoField support

timgraham opened this issue · comments

Django 3.0 adds support for SmallAutoField, however, cockroachdb's unique_rowid() generates 8-byte values that are too large for int2 (2-byte). I'm not sure how to proceed.

Test failure:

======================================================================
ERROR: test_fk_to_smallautofield (many_to_one.tests.ManyToOneTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.NumericValueOutOfRange: integer out of range for type int2 (column "country_id")


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tim/code/django/tests/many_to_one/tests.py", line 583, in test_fk_to_smallautofield
    City.objects.create(country=us, name='Chicago')
  File "/home/tim/code/django/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/tim/code/django/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/tim/code/django/django/db/models/base.py", line 746, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/tim/code/django/django/db/models/base.py", line 784, in save_base
    force_update, using, update_fields,
  File "/home/tim/code/django/django/db/models/base.py", line 886, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/tim/code/django/django/db/models/base.py", line 925, in _do_insert
    using=using, raw=raw,
  File "/home/tim/code/django/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/tim/code/django/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/tim/code/django/django/db/models/sql/compiler.py", line 1344, in execute_sql
    cursor.execute(sql, params)
  File "/home/tim/code/django/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/tim/code/django/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/tim/code/django/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/tim/code/django/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/tim/code/django/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: integer out of range for type int2 (column "country_id")

smallserial is used for SmallAutoField:

CREATE TABLE "smallauto_country" ("id" smallserial NOT NULL PRIMARY KEY, "name" varchar(50) NOT NULL);

However, it looks like cockroachdb uses unique_rowid() for that column. The 8-byte id is stored without error despite the fact that smallserial is 2-byte on PostgreSQL . The error comes when using the large id in a foreign key (which uses smallint):

CREATE TABLE "smallauto_city" ("id" bigserial NOT NULL PRIMARY KEY, "name" varchar(50) NOT NULL, "country_id" smallint NULL);

I wonder if we should change the smallserial alias in CockroachDB so it uses a different function that generates a smaller int? Like implement a unique_smallrowid() function, perhaps. I don't know if there are other implications with doing that.

FWIW, there also is a cluster setting called serial_normalization that allows the SERIAL type to use a sequence instead of unique_rowid(). I don't think we want to use it, but just listing it as an option. Details on the setting:
https://www.cockroachlabs.com/docs/releases/v2.1.0-beta.20180827.html
https://www.cockroachlabs.com/docs/stable/cluster-settings.html

cc @jordanlewis @apantel

I'm guessing that unique_smallrowid() might be the way to go since CockroachDB's documentation says, "SERIAL is provided only for compatibility with PostgreSQL. New applications should use real data types and a suitable DEFAULT expression."

Django's AutoField and BigAutoField are both using integer data type with DEFAULT unique_rowid() rather than serial and bigserial.