Error when trying to insert data into migrated database

[Florian Hanzer]

I tried to migrate a database from 0.7.5 to 0.8.0 using terracotta migrate, however I am running into a sqlalchemy.exc.CompileError: Unconsumed column names: path when trying to insert data into the migrated database. The issue seems to be that the column filepath in the datasets table has been renamed to path in 0.8.0, which is not handled by the migration script.

Minimal example:

pip install 'terracotta==0.7.5'

python -c '
import terracotta as tc
driver = tc.get_driver("db.sqlite3")
driver.create(("a", "b", "c"))
'

pip install 'terracotta==0.8.0'
terracotta migrate db.sqlite3

python -c '
import terracotta as tc
driver = tc.get_driver("db.sqlite3")
driver.insert((1, 2, 3), "dummy.tif", skip_metadata=True)
'

results in the following error:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/terracotta/drivers/terracotta_driver.py", line 237, in insert
    self.meta_store.insert(keys=keys, path=override_path or path, metadata=metadata)
  File "/home/flo/miniconda3/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/terracotta/drivers/base_classes.py", line 35, in inner
    return fun(self, *args, **kwargs)
  File "/home/flo/miniconda3/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/terracotta/drivers/relational_meta_store.py", line 402, in insert
    conn.execute(datasets_table.insert().values(**keys, path=path))
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute
    return meth(
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1627, in _execute_clauseelement
    compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 703, in _compile_w_cache
    compiled_sql = self._compiler(
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 315, in _compiler
    return dialect.statement_compiler(dialect, self, **kw)
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1426, in __init__
    Compiled.__init__(self, dialect, statement, **kwargs)
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 867, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 912, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 143, in _compiler_dispatch
    return meth(self, **kw)  # type: ignore  # noqa: E501
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 5636, in visit_insert
    crud_params_struct = crud._get_crud_params(
  File "/home/flo/.virtualenvs/tmp-5ffc7c8d179e425/lib/python3.9/site-packages/sqlalchemy/sql/crud.py", line 335, in _get_crud_params
    raise exc.CompileError(
sqlalchemy.exc.CompileError: Unconsumed column names: path

After renaming the column manually (sqlite3 db.sqlite3 'alter table datasets rename column filepath to path') everything works as expected.

[Dion Häfner]

Thanks for the great bug report! Should be fixed as soon as #316 is merged.