Ignore generated columns when export data

Question

Ignore generated columns when export data

cdxf opened this issue a year ago · comments

Is your feature request related to a problem? Please describe.

MYSQL can't import the exported file if the table contains generated columns:
ERROR 3105 (HY000) at line 9402: The value specified for generated column 'column' in table 'table' is not allowed.

https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html

Describe the solution you'd like

Ignore generated columns when export data or set the value to DEFAULT

For INSERT, REPLACE, and UPDATE, if a generated column is inserted into, replaced, or updated explicitly, the only permitted value is DEFAULT.

Wisser · Answer 1 · Wed Jan 25 2023 16:33:25 GMT+0800 (China Standard Time)

Generated columns are ignored when exporting if they are marked as "virtual" in the data model(editor):

Virtual columns should be detected automatically. But this works only if the database driver is JDBC 4 compatible. Which driver do you use?

cdxf · Answer 2 · Wed Jan 25 2023 16:41:01 GMT+0800 (China Standard Time)

It is not detected automatically I think

I use mysql-connector-java-8.0.21

Wisser · Answer 3 · Wed Jan 25 2023 16:44:34 GMT+0800 (China Standard Time)

That's strange. I just tested this successfully with exactly this driver version.
Could you show me the DDL for this table (or at least that of the generated column)?

cdxf · Answer 4 · Wed Jan 25 2023 16:53:47 GMT+0800 (China Standard Time)

DDL:

is_hidden tinyint(1) GENERATED ALWAYS AS expression_removed_for_brevity STORED`

Step to reproduce:

New Connection (use remote db connection)
Analyze Database
Check the table details

Wisser · Answer 5 · Wed Jan 25 2023 17:25:00 GMT+0800 (China Standard Time)

Thanks for the info.
I tested this again (also with the current MySQL driver) and found out the following:

a column:
GENERATED ALWAYS AS ...
is specified as "generated" by the driver.

A column:
GENERATED ALWAYS AS ... STORED
however not. (Which is a driver bug in my opinion).

I will try to find another way than via driver to find out in MySQL if a column is generated.
Currently the only solution to your problem is to set the "virtual" flag manually.

Wisser · Answer 6 · Wed Jan 25 2023 18:02:30 GMT+0800 (China Standard Time)

An alternative way to get info about generated columns is to query the "INFORMATION_SCHEMA" tables.

I have adapted the configuration (based on version 14.1.1) for this purpose.

If you replace the file "jailer.xml" in your installation with the one in the attachment, your column should also be "virtual" after the analysis.

(Unpack the file for this purpose. GitHub does not allow XML files in the attachment. The destination folder is either ".jailer" in your user directory, or where you unzipped "jailer-n.zip". Depending on what you have installed)

Please do a test on this.
Thanks in advance,
Ralf

jailer.xml.zip

Wisser · Answer 7 · Sat Jan 28 2023 19:13:46 GMT+0800 (China Standard Time)

Fixed in Release 14.2

cdxf · Answer 8 · Sat Feb 18 2023 03:25:55 GMT+0800 (China Standard Time)

An alternative way to get info about generated columns is to query the "INFORMATION_SCHEMA" tables.

I have adapted the configuration (based on version 14.1.1) for this purpose.

If you replace the file "jailer.xml" in your installation with the one in the attachment, your column should also be "virtual" after the analysis.

(Unpack the file for this purpose. GitHub does not allow XML files in the attachment. The destination folder is either ".jailer" in your user directory, or where you unzipped "jailer-n.zip". Depending on what you have installed)

Please do a test on this. Thanks in advance, Ralf

jailer.xml.zip

Sorry, I forget about this thread. Tried again and it works now in the latest version.
Thank for your work

Graham · Answer 9 · Thu Jan 04 2024 02:01:14 GMT+0800 (China Standard Time)

In Jailer 15.4, using SQL2022 I find that by default it tries to specify values for the generated temporal columns. You can create a table that should repro the issue like this:

CREATE TABLE [dbo].[Person] (
    [Id] INT NOT NULL PRIMARY KEY IDENTITY,
    [RecordStartTime]         DATETIME2 (7) GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
    [RecordEndTime]           DATETIME2 (7) GENERATED ALWAYS AS ROW END HIDDEN   NOT NULL,
    PERIOD FOR SYSTEM_TIME ([RecordStartTime], [RecordEndTime])
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE=[dbo].[PersonHistory], DATA_CONSISTENCY_CHECK=ON));

This query shows the column in case that helps:

SELECT *
FROM sys.columns
WHERE generated_always_type <> 0
AND object_id = OBJECT_ID('Person')

The workaround of manually marking virtual also worked fine.

Thanks for the great project!

Wisser · Answer 10 · Thu Jan 04 2024 19:23:31 GMT+0800 (China Standard Time)

Thanks @GrahamTheCoder
I have updated the request for the virtual columns accordingly.
You can find it here: https://sourceforge.net/projects/jailer/files/stage15.4.2/

Please try this out.

Graham · Answer 11 · Tue Jan 09 2024 05:07:12 GMT+0800 (China Standard Time)

Yes that's working fine now, thanks again!

Wisser · Answer 12 · Tue Jan 09 2024 19:15:45 GMT+0800 (China Standard Time)

Fixed in Release 15.5