apply_initial_migration fails
Hafnernuss opened this issue · comments
Hi,
I've just discovered this package and wanted to test a new migration that I am writing. So far, the project consists of a few apps, and the app I want to write migration tests for has currently > 100 applied migrations.
For the sake of simplicity, I set migrate_from and migrate_to to an already applied migration (~101 -> 102, only a change in a field type) just to fiddle arround. Curiously, it failed.
Traceback:
`MySQLdb._exceptions.OperationalError: (3730, "Cannot drop table 'wagtailcore_collectionviewrestriction' referenced by a foreign key constraint 'wagtailcore_collecti_collectionviewrestri_47320efd_fk_wagtailco' on table 'wagtailcore_collectionviewrestriction_groups'.")
env/lib/python3.6/site-packages/MySQLdb/connections.py:239: OperationalError
The above exception was the direct cause of the following exception:
env/lib/python3.6/site-packages/django_test_migrations/contrib/unittest_case.py:36: in setUp
self.migrate_from,
env/lib/python3.6/site-packages/django_test_migrations/migrator.py:46: in apply_initial_migration
sql.drop_models_tables(self._database, style)
env/lib/python3.6/site-packages/django_test_migrations/sql.py:32: in drop_models_tables
get_execute_sql_flush_for(connection)(database_name, sql_drop_tables)
env/lib/python3.6/site-packages/django/db/backends/base/operations.py:405: in execute_sql_flush
cursor.execute(sql)
env/lib/python3.6/site-packages/django/db/backends/utils.py:68: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
env/lib/python3.6/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
return executor(sql, params, many, context)
env/lib/python3.6/site-packages/django/db/backends/utils.py:86: in _execute
return self.cursor.execute(sql, params)
env/lib/python3.6/site-packages/django/db/utils.py:90: in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
env/lib/python3.6/site-packages/django/db/backends/utils.py:84: in _execute
return self.cursor.execute(sql)
env/lib/python3.6/site-packages/django/db/backends/mysql/base.py:74: in execute
return self.cursor.execute(query, args)
env/lib/python3.6/site-packages/MySQLdb/cursors.py:209: in execute
res = self._query(query)
env/lib/python3.6/site-packages/MySQLdb/cursors.py:315: in _query
db.query(q)
and a lot more other errors related to drop queries. As I understand it, during setup, django-test-migrations takes the current test_db and deletes all models in the database, and then reapplies all migrations up and including to migrate_from. In my case, this seems to fail. I cannot imagine what I could have done wrong in my testcase, as the error occurs during the setUp function of the MigratorTestCase.
I don't think that it has anything to do with wagtail, as the testcase seems to fail with different errors on each run:
self = <_mysql.connection closed at 0x35419e8>
query = b'DROP TABLE `auth_group` CASCADE'
def query(self, query):
# Since _mysql releases GIL while querying, we need immutable buffer.
if isinstance(query, bytearray):
query = bytes(query)
> _mysql.connection.query(self, query)
E django.db.utils.OperationalError: (3730, "Cannot drop table 'auth_group' referenced by a foreign key constraint 'auth_group_permissions_group_id_b120cbf9_fk_auth_group_id' on table 'auth_group_permissions'.")
Testcase:
class TestMigrations(MigratorTestCase):
migrate_from = ('myapp', None)
migrate_to = ('myapp', '001_initial')
def prepare(self):
pass
def test_migration001(self):
self.assertTrue(True)Maybe I am missing something crucial and obvious, so here is what I did:
install django-test-migrations with pip
wrote a unittest testcase, as provided in the example (the actual tc is just an assertTrue(True)
ran manage.py test
I did not add anything to INSTALLED_APPS whatsoever.
Any ideas?
DB: MYSQL 8.0.19 mysqlclient (1.4.6)
Django: 3.0.2
django-test-migrations: 1.0.0
@skarzi my wild guess is that it can be mysql related failure. What do you think?
I did a quick test with the same project on another system that uses sqlite as backend. The error is different there and only occurs in the tearDown of the testcase:
self = <django.db.backends.utils.CursorWrapper object at 0x7f71205fd198>
sql = 'CREATE TABLE "wagtailsearch_editorspick" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "sort_order" integer NULL,...ription" text NOT NULL, "page_id" integer NOT NULL REFERENCES "wagtailcore_page" ("id") DEFERRABLE INITIALLY DEFERRED)'
params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.sqlite3.base.DatabaseWrapper object at 0x7f7122080780>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f71205fd198>})
def _execute(self, sql, params, *ignored_wrapper_args):
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
# params default might be backend specific.
> return self.cursor.execute(sql)
env/lib/python3.6/site-packages/django/db/backends/utils.py:84:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x7f711fe83d38>
query = 'CREATE TABLE "wagtailsearch_editorspick" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "sort_order" integer NULL,...ription" text NOT NULL, "page_id" integer NOT NULL REFERENCES "wagtailcore_page" ("id") DEFERRABLE INITIALLY DEFERRED)'
params = None
def execute(self, query, params=None):
if params is None:
> return Database.Cursor.execute(self, query)
E sqlite3.OperationalError: table "wagtailsearch_editorspick" already exists
env/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py:394: OperationalError
The above exception was the direct cause of the following exception:
env/lib/python3.6/site-packages/django_test_migrations/contrib/unittest_case.py:50: in tearDown
self._migrator.reset()
env/lib/python3.6/site-packages/django_test_migrations/migrator.py:69: in reset
call_command('migrate', verbosity=0, database=self._database)
env/lib/python3.6/site-packages/django/core/management/__init__.py:168: in call_command
return command.execute(*args, **defaults)
env/lib/python3.6/site-packages/django/core/management/base.py:369: in execute
output = self.handle(*args, **options)
env/lib/python3.6/site-packages/django/core/management/base.py:83: in wrapped
res = handle_func(*args, **kwargs)
env/lib/python3.6/site-packages/django/core/management/commands/migrate.py:233: in handle
fake_initial=fake_initial,
env/lib/python3.6/site-packages/django/db/migrations/executor.py:117: in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
env/lib/python3.6/site-packages/django/db/migrations/executor.py:147: in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
env/lib/python3.6/site-packages/django/db/migrations/executor.py:245: in apply_migration
state = migration.apply(state, schema_editor)
env/lib/python3.6/site-packages/django/db/migrations/migration.py:124: in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
env/lib/python3.6/site-packages/django/db/migrations/operations/models.py:92: in database_forwards
schema_editor.create_model(model)
env/lib/python3.6/site-packages/django/db/backends/base/schema.py:324: in create_model
self.execute(sql, params or None)
env/lib/python3.6/site-packages/django/db/backends/base/schema.py:142: in execute
cursor.execute(sql, params)
env/lib/python3.6/site-packages/django/db/backends/utils.py:68: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
env/lib/python3.6/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
return executor(sql, params, many, context)
env/lib/python3.6/site-packages/django/db/backends/utils.py:86: in _execute
return self.cursor.execute(sql, params)
env/lib/python3.6/site-packages/django/db/utils.py:90: in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
env/lib/python3.6/site-packages/django/db/backends/utils.py:84: in _execute
return self.cursor.execute(sql)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x7f711fe83d38>
query = 'CREATE TABLE "wagtailsearch_editorspick" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "sort_order" integer NULL,...ription" text NOT NULL, "page_id" integer NOT NULL REFERENCES "wagtailcore_page" ("id") DEFERRABLE INITIALLY DEFERRED)'
params = None
def execute(self, query, params=None):
if params is None:
> return Database.Cursor.execute(self, query)
E django.db.utils.OperationalError: table "wagtailsearch_editorspick" already exists
env/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py:394: OperationalError
but unlike with mysql, the error does not change on successive runs. implementing a dummy tearDown causes the test to pass.
@skarzi my wild guess is that it can be mysql related failure. What do you think?
The issue described by @Hafnernuss in the first comment is related to MySQL.
In django.db.backends.sql.operations.DatabaseOperations.sql_flush django disables FOREIGN_KEY_CHECKS before executing SQL FLUSH and the same behaviour should be used to trigger SQL DELETE queries.
Currently, we have django_test_migrations.db module, where we can move django_test_migrations.sql module and refactor it a bit to make adding such features per DB backend easier.
That's also a great time to focus on #107, so performing above mentioned refactoring will be easier.
@Hafnernuss could you please try to reproduce error described in your second comment without depending on wagtail?
It's quite big lib with a lot of migrations and it will make debugging much more complicated, so simpler and smaller example will be very welcome!
Sorry for the late reply. I had a little bit of time and played around.
What I did:
- fresh mysql db
- removed all wagtail references from the app
- ran all migrations
- created following testcase (same as above):
class TestMigrations(MigratorTestCase):
migrate_from = ('myapp', None)
migrate_to = ('myapp', '001_initial')
def prepare(self):
pass
def test_migration001(self):
self.assertTrue(True)Here is the error I get:
self.migrate_from,
env/lib/python3.6/site-packages/django_test_migrations/migrator.py:46: in apply_initial_migration
sql.drop_models_tables(self._database, style)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
database_name = 'default'
style = <django.core.management.color.Style object at 0x7fcc0ed13518>
def drop_models_tables(
database_name: str,
style: Optional[Style] = None,
) -> None:
"""Drop all installed Django's models tables."""
style = style or no_style()
connection = connections[database_name]
tables = connection.introspection.django_table_names(
only_existing=True,
include_views=False,
)
sql_drop_tables = [
connection.SchemaEditorClass.sql_delete_table % {
'table': style.SQL_FIELD(connection.ops.quote_name(table)),
}
for table in tables
]
if sql_drop_tables:
> get_execute_sql_flush_for(connection)(database_name, sql_drop_tables)
E TypeError: execute_sql_flush() takes 2 positional arguments but 3 were given
env/lib/python3.6/site-packages/django_test_migrations/sql.py:32: TypeError
I am completly clueless.
Edit: Unfortunately, I had to migrate to django 3.1.1 in the meantime.
Apparently, the error above is indeed caused by Django 3.1. I have created a MRE that runs on Django 3.0. The problem does not occur when usig SQLITE, however, it does when using MYSQL. The migration does not even have to be applied.
All you have to do is create a env (requirements provided) and change the database credentials in the settings.py file.
The error I recieve:
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_migration001 (sample.tests.TestPopulatePlayerPositions)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 74, in execute
return self.cursor.execute(query, args)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (3730, "Cannot drop table 'django_content_type' referenced by a foreign key constraint 'auth_permission_content_type_id_2f476e4b_fk_django_co' on table 'auth_permission'.")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django_test_migrations/contrib/unittest_case.py", line 35, in setUp
self.old_state = self._migrator.apply_initial_migration(
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django_test_migrations/migrator.py", line 46, in apply_initial_migration
sql.drop_models_tables(self._database, style)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django_test_migrations/sql.py", line 32, in drop_models_tables
get_execute_sql_flush_for(connection)(database_name, sql_drop_tables)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/base/operations.py", line 405, in execute_sql_flush
cursor.execute(sql)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 74, in execute
return self.cursor.execute(query, args)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (3730, "Cannot drop table 'django_content_type' referenced by a foreign key constraint 'auth_permission_content_type_id_2f476e4b_fk_django_co' on table 'auth_permission'.")
----------------------------------------------------------------------
Ran 1 test in 0.510s
FAILED (errors=1)
Tested on python 3.6 and 3.8.
@Hafnernuss thank you for the great and detailed investigation!
I will try to fix issues related to MySQL in the following days.
Fix for django>=3.1 is already on master, but we need to fix a few other issues before releasing the new version.
Great to hear that! I suspected that the error was related to 3.1.x ;)
I think it is somehow related to foreign keys and the order in which models are deleted. I wonder especially how this can be solved if someone uses on_delete=PROTECT
Maybe this could also help?
SET FOREIGN_KEY_CHECKS=0; since the whole db is cleared, doesn't matter if the checks fail or not. Has to be reenabled though ;)
@skarzi anything I can help with to get the next release out? I'm getting hit by this also.
hi @tmm,
sure, thank you so much for your help! 👍
You can prepare PR with changes mentioned by me in #122 (comment).
Let's create some base class like BaseDatabaseConfiguration, but for database operations, so we can move there functions from django_test_migrations.sql and simply subclass and extend it for all supported vendors (currently only MySQL needs a different implementation of drop_models_table - we need to disable and then enable FOREIGN_KEY_CHECKS).
If something is not clear or you have some other ideas, please share it here