Support converting (or at least ignoring) generated fields

Question

Support converting (or at least ignoring) generated fields

rasa opened this issue a year ago · comments

Ross Smith II commented a year ago

Describe the bug

MySQL transfer failed inserting data into table tablename: 1054 (42S22): Unknown column 'name' in 'field list'

Expected behaviour

Accept (or ignore) generated fields.

Actual result

See above

System Information

| software               | version                              |
|------------------------|--------------------------------------|
| sqlite3-to-mysql       | 1.4.16                               |
|                        |                                      |
| Operating System       | Windows 10                           |
| Python                 | CPython 3.10.9                       |
| MySQL                  | MySQL client not found on the system |
| SQLite                 | 3.39.4                               |
|                        |                                      |
| click                  | 8.1.3                                |
| mysql-connector-python | 8.0.29                               |
| pytimeparse            | 1.1.8                                |
| simplejson             | 3.18.4                               |
| six                    | 1.16.0                               |
| tabulate               | 0.9.0                                |
| tqdm                   | 4.64.1                               |

Additional context

I'm comfortable with Python, so if you could give me some pointers on how to implement this fix (enhancement?), I'd be happy to try to cobble together a PR.

Klemen Tusar · Answer 1 · Tue Mar 28 2023 13:47:08 GMT+0800 (China Standard Time)

Hi. Can you provide some more detail on your table structure?

Feel free to play around with the source code and submit a PR, but please don't forget to add tests.

Ross Smith II · Answer 2 · Tue Mar 28 2023 14:23:33 GMT+0800 (China Standard Time)

The field

name TEXT GENERATED ALWAYS AS (TRIM(first_name) || ' ' || TRIM(last_name)) STORED

Caused the error.

Klemen Tusar · Answer 3 · Tue Mar 28 2023 14:59:46 GMT+0800 (China Standard Time)

Yeah, that's not supported.

Feel free to submit a PR.

Klemen Tusar · Answer 4 · Mon May 29 2023 02:21:51 GMT+0800 (China Standard Time)

So I finally found some time and did some digging into generated columns.

Essentially, there are 3 gotchas:

There is no way of extracting the expression of the generated column, and even if there were the expressions themselves aren't compatible between the databases.
The only way to successfully transfer generated data from SQLite to MySQL is to create these generated columns as normal visible columns and then transfer all the data into them. This ensures that the columns aren't skipped and no data is lost.
The method described above only works with a SQLite version that supports PRAGMA table_xinfo("table_name") which means that it has to be SQLite v3.26.0 or higher.