Don't break TYPO3 database compare
DanielSiepmann opened this issue · comments
TYPO3 allows to compare the current database structure with a target structure. This is available via Install Tool as Web UI as well via https://packagist.org/packages/helhum/typo3-console through command line. The command allows to clean things up via '*' which is more or less the same as checking all checkboxes within web ui.
It is also possible to use multiple databases with TYPO3. One can configure the database to use per table.
We can't execute the '*' scenario, or blindly check all checkboxes right now. We don't get a plain nothing to do, but instead:
Remove tables (rename with prefix)
select/deselect all
ALTER TABLE `fe_users_address` RENAME TO `zzz_deleted_fe_users_address`
ALTER TABLE `fe_users_list` RENAME TO `zzz_deleted_fe_users_list`
ALTER TABLE `fe_users_list_type` RENAME TO `zzz_deleted_fe_users_list_type`
Rows in table: 5
ALTER TABLE `fe_users_property` RENAME TO `zzz_deleted_fe_users_property`
ALTER TABLE `fe_users_property_type` RENAME TO `zzz_deleted_fe_users_property_type`
ALTER TABLE `mshop_customer` RENAME TO `zzz_deleted_mshop_customer`
ALTER TABLE `mshop_customer_address` RENAME TO `zzz_deleted_mshop_customer_address`
ALTER TABLE `mshop_customer_group` RENAME TO `zzz_deleted_mshop_customer_group`
ALTER TABLE `mshop_customer_list` RENAME TO `zzz_deleted_mshop_customer_list`
ALTER TABLE `mshop_customer_list_type` RENAME TO `zzz_deleted_mshop_customer_list_type`
ALTER TABLE `mshop_customer_property` RENAME TO `zzz_deleted_mshop_customer_property`
ALTER TABLE `mshop_customer_property_type` RENAME TO `zzz_deleted_mshop_customer_property_type`
Drop fields (really!)
select/deselect all
ALTER TABLE `madmin_cache_tag` DROP FOREIGN KEY `fk_macacta_tid`
ALTER TABLE `mshop_attribute_list` DROP FOREIGN KEY `fk_msattli_pid`
ALTER TABLE `mshop_attribute_property` DROP FOREIGN KEY `fk_msattpr_pid`
ALTER TABLE `mshop_catalog_list` DROP FOREIGN KEY `fk_mscatli_pid`
ALTER TABLE `mshop_coupon_code` DROP FOREIGN KEY `fk_mscouco_pid`
ALTER TABLE `mshop_locale` DROP FOREIGN KEY `fk_msloc_currid`
ALTER TABLE `mshop_locale` DROP FOREIGN KEY `fk_msloc_langid`
ALTER TABLE `mshop_locale` DROP FOREIGN KEY `fk_msloc_siteid`
ALTER TABLE `mshop_media_list` DROP FOREIGN KEY `fk_msmedli_pid`
ALTER TABLE `mshop_media_property` DROP FOREIGN KEY `fk_msmedpr_pid`
ALTER TABLE `mshop_order` DROP FOREIGN KEY `fk_msord_baseid`
It turns out that Classes/Setup.php only checks one database connection. Adding the 2nd solves some problems and ends up in:
Drop fields (really!)
select/deselect all
ALTER TABLE `fe_users_address` DROP FOREIGN KEY `fk_t3feuad_pid`
ALTER TABLE `fe_users_list` DROP FOREIGN KEY `fk_t3feuli_pid`
ALTER TABLE `fe_users_property` DROP FOREIGN KEY `fk_t3feupr_pid`
ALTER TABLE `mshop_customer_address` DROP FOREIGN KEY `fk_mscusad_pid`
ALTER TABLE `mshop_customer_list` DROP FOREIGN KEY `fk_mscusli_pid`
ALTER TABLE `mshop_customer_property` DROP FOREIGN KEY `fk_mcuspr_pid`
ALTER TABLE `madmin_cache_tag` DROP FOREIGN KEY `fk_macacta_tid`
ALTER TABLE `mshop_attribute_list` DROP FOREIGN KEY `fk_msattli_pid`
ALTER TABLE `mshop_attribute_property` DROP FOREIGN KEY `fk_msattpr_pid`
ALTER TABLE `mshop_catalog_list` DROP FOREIGN KEY `fk_mscatli_pid`
ALTER TABLE `mshop_coupon_code` DROP FOREIGN KEY `fk_mscouco_pid`
ALTER TABLE `mshop_locale` DROP FOREIGN KEY `fk_msloc_currid`
ALTER TABLE `mshop_locale` DROP FOREIGN KEY `fk_msloc_langid`
ALTER TABLE `mshop_locale` DROP FOREIGN KEY `fk_msloc_siteid`
ALTER TABLE `mshop_media_list` DROP FOREIGN KEY `fk_msmedli_pid`
ALTER TABLE `mshop_media_property` DROP FOREIGN KEY `fk_msmedpr_pid`
ALTER TABLE `mshop_order` DROP FOREIGN KEY `fk_msord_baseid`
It would be cool if one could auto compare the database with '*' option during deployment when using aimeos with TYPO3 and two databases. Let me know if I can help somehow. I couldn't find a solution for the foreign keys yet.
Used Versions:
TYPO3 v11.5.23
aimeos/ai-admin-jqadm 2021.10.x-dev 62c3fd5 Aimeos Vue.js+Bootstrap admin interface
aimeos/ai-admin-jsonadm 2021.10.5 Aimeos ai-admin-jsonadm extension
aimeos/ai-client-html 2021.10.18 Aimeos ai-client-html extension
aimeos/ai-client-jsonapi 2021.10.5 Aimeos JSON API extension
aimeos/ai-controller-frontend 2021.10.6 Aimeos ai-controller-frontend extension
aimeos/ai-controller-jobs 2021.10.9 Aimeos ai-controller-jobs extension
aimeos/ai-gettext 2021.10.3 Aimeos Gettext extension
aimeos/ai-payments 2021.10.x-dev 8a0b51b Payment extension for Aimeos web shops and e-commerce solutions
aimeos/ai-typo3 2021.10.7 TYPO3 adapter for Aimeos web shops and e-commerce solutions
aimeos/aimeos-core 2021.10.x-dev acaca78 Full-featured e-commerce components for high performance online shops
aimeos/aimeos-typo3 2021.10.x-dev b0e92f2 Professional, full-featured and high performance TYPO3 e-commerce extension for online shops and complex B2B projects
aimeos/map 2.6.1 Easy and elegant handling of PHP arrays as array-like map objects similar to jQuery and Laravel Collections
aimeoscom/ai-customergroups 2021.10.4 Aimeos ai-customergroups extension
aimeoscom/ai-vatcheck 2021.10.4 Aimeos ai-vatcheck extension
doctrine/dbal 2.13.9
Configured table mapping:
$GLOBALS['TYPO3_CONF_VARS']['DB']['TableMapping'] = [
'madmin_cache' => 'Aimeos',
'madmin_cache_tag' => 'Aimeos',
'madmin_job' => 'Aimeos',
'madmin_log' => 'Aimeos',
'madmin_queue' => 'Aimeos',
'mshop_attribute' => 'Aimeos',
'mshop_attribute_list' => 'Aimeos',
'mshop_attribute_list_type' => 'Aimeos',
'mshop_attribute_property' => 'Aimeos',
'mshop_attribute_property_type' => 'Aimeos',
'mshop_attribute_type' => 'Aimeos',
'mshop_catalog' => 'Aimeos',
'mshop_catalog_list' => 'Aimeos',
'mshop_catalog_list_type' => 'Aimeos',
'mshop_coupon' => 'Aimeos',
'mshop_coupon_code' => 'Aimeos',
'mshop_index_attribute' => 'Aimeos',
'mshop_index_catalog' => 'Aimeos',
'mshop_index_cgroup' => 'Aimeos',
'mshop_index_customer' => 'Aimeos',
'mshop_index_price' => 'Aimeos',
'mshop_index_supplier' => 'Aimeos',
'mshop_index_text' => 'Aimeos',
'mshop_locale' => 'Aimeos',
'mshop_locale_currency' => 'Aimeos',
'mshop_locale_language' => 'Aimeos',
'mshop_locale_site' => 'Aimeos',
'mshop_media' => 'Aimeos',
'mshop_media_list' => 'Aimeos',
'mshop_media_list_type' => 'Aimeos',
'mshop_media_property' => 'Aimeos',
'mshop_media_property_type' => 'Aimeos',
'mshop_media_type' => 'Aimeos',
'mshop_order' => 'Aimeos',
'mshop_order_base' => 'Aimeos',
'mshop_order_base_address' => 'Aimeos',
'mshop_order_base_coupon' => 'Aimeos',
'mshop_order_base_product' => 'Aimeos',
'mshop_order_base_product_attr' => 'Aimeos',
'mshop_order_base_service' => 'Aimeos',
'mshop_order_base_service_attr' => 'Aimeos',
'mshop_order_status' => 'Aimeos',
'mshop_plugin' => 'Aimeos',
'mshop_plugin_type' => 'Aimeos',
'mshop_price' => 'Aimeos',
'mshop_price_list' => 'Aimeos',
'mshop_price_list_type' => 'Aimeos',
'mshop_price_property' => 'Aimeos',
'mshop_price_property_type' => 'Aimeos',
'mshop_price_type' => 'Aimeos',
'mshop_product' => 'Aimeos',
'mshop_product_list' => 'Aimeos',
'mshop_product_list_type' => 'Aimeos',
'mshop_product_property' => 'Aimeos',
'mshop_product_property_type' => 'Aimeos',
'mshop_product_type' => 'Aimeos',
'mshop_review' => 'Aimeos',
'mshop_service' => 'Aimeos',
'mshop_service_list' => 'Aimeos',
'mshop_service_list_type' => 'Aimeos',
'mshop_service_type' => 'Aimeos',
'mshop_stock' => 'Aimeos',
'mshop_stock_type' => 'Aimeos',
'mshop_subscription' => 'Aimeos',
'mshop_supplier' => 'Aimeos',
'mshop_supplier_address' => 'Aimeos',
'mshop_supplier_list' => 'Aimeos',
'mshop_supplier_list_type' => 'Aimeos',
'mshop_tag' => 'Aimeos',
'mshop_tag_type' => 'Aimeos',
'mshop_text' => 'Aimeos',
'mshop_text_list' => 'Aimeos',
'mshop_text_list_type' => 'Aimeos',
'mshop_text_type' => 'Aimeos',
];We use the following patch:
diff --git a/Classes/Setup.php b/Classes/Setup.php
index 1ef0094..be1b744 100644
--- a/Classes/Setup.php
+++ b/Classes/Setup.php
@@ -187,39 +187,46 @@ class Setup implements UpgradeWizardInterface, RepeatableInterface, ChattyInterf
{
$ctx = self::getContext();
$dbm = $ctx->getDatabaseManager();
- $conn = $dbm->acquire();
+ $connectionNames = array_keys($ctx->getConfig()->get( 'resource'));
+ $connectionNames = array_filter($connectionNames, fn (string $key): bool => str_starts_with($key, 'db'));
- try
- {
- $tables = [];
+ foreach ($connectionNames as $connectionName) {
+ $conn = $dbm->acquire($ctx->getConfig()->get( 'resource/default', $connectionName ));
- foreach( ['fe_users_', 'madmin_', 'mshop_'] as $prefix )
+ try
{
- $result = $conn->create( 'SHOW TABLES like \'' . $prefix . '%\'' )->execute();
+ $tables = [];
- while( ( $row = $result->fetch( \Aimeos\MW\DB\Result\Base::FETCH_NUM ) ) !== null ) {
- $tables[] = $row[0];
- }
- }
+ foreach( ['fe_users_', 'madmin_', 'mshop_'] as $prefix )
+ {
+ $result = $conn->create( 'SHOW TABLES like \'' . $prefix . '%\'' )->execute();
- foreach( $tables as $table )
- {
- $result = $conn->create( 'SHOW CREATE TABLE ' . $table )->execute();
+ while( ( $row = $result->fetch( \Aimeos\MW\DB\Result\Base::FETCH_NUM ) ) !== null ) {
+ $tables[] = $row[0];
+ }
+ }
- while( ( $row = $result->fetch( \Aimeos\MW\DB\Result\Base::FETCH_NUM ) ) !== null )
+ foreach( $tables as $table )
{
- $str = preg_replace( '/,[\n ]*CONSTRAINT.+CASCADE/', '', $row[1] );
- $str = str_replace( '"', '`', $str );
+ $result = $conn->create( 'SHOW CREATE TABLE ' . $table )->execute();
+
+ while( ( $row = $result->fetch( \Aimeos\MW\DB\Result\Base::FETCH_NUM ) ) !== null )
+ {
+ $str = $row[1];
- $sql[] = $str . ";\n";
+ $str = str_replace( '"', '`', $str );
+ $str = preg_replace( '#CONSTRAINT `\w+` #', '', $str );
+
+ $sql[] = $str . ";\n";
+ }
}
- }
- $dbm->release( $conn );
- }
- catch( \Exception $e )
- {
- $dbm->release( $conn );
+ $dbm->release( $conn );
+ }
+ catch( \Exception $e )
+ {
+ $dbm->release( $conn );
+ }
}This works for V11 LTS. But I doubt that the approach to retrieve all connections will work in all projects. But this can be a reference / example for others.
You can use several database connections without problem when you configure the TableMapping correctly. What's the real problem are the foreign key constraint that are still not understood by the database analyzer and will always result in:
Drop fields (really!)
select/deselect all
ALTER TABLE `fe_users_address` DROP FOREIGN KEY `fk_t3feuad_pid`
...
If you let the database analyzer remove the foreign keys, you will get stale records in the list and property tables which then results in being unable to upgrade to newer versions because the migration fails when the Aimeos setup tasks try to add the constraints again.
@aimeos how should the mapping look like? I've pasted our above. Also a fix without touching the mapping any further. TYPO3 moves all none mapped tables to default connection which is what we expect. The issue seemed that aimeos didn't check the default TYPO3 connection and didn't add their schema. This was solved by not only fetching structure for one but all connections with above patch.
The keys were also fixed with above patch.
Did you configure Aimeos to use the second database connection like described here?
https://aimeos.org/docs/latest/typo3/optimize/#databases
Yes we did. The system is already working for some years. We just wanted to improve our deployment and add '*' to clean up our database.
We use a dedicated aimeos database for most tables, but some are stored within TYPO3 database.
I don't see where EXT:aimeos would check the structure for the TYPO3 database, it doesn't provide any database name, falling back to "db". But we also have "db-customer" which is configured to use TYPO3 database. I guess because of joins to TYPO3 fe_users table.
Our above patch solves the issue with the key, and it also solves all other issues. But it expects all database connection to be prefixed with db- which might not be the case in all setups?
It furthermore removed the tables on initial compare, but works later on, for some reason I didn't find yet.
You must have at least two databases configured in your LocalConfiguration.php (probably "Default" and "Aimeos"). In the project specific Aimeos extension (however it's named) the TYPO3 connections are most likely mapped to the Aimeos db* connections in ./Resources/Private/Config/resource.php.
All Aimeos database connections are prefixed by db (without the dash, with dash will skip the main/fallback DB)
If you can create a PR for the current dev-master branch, we will merge it :-)
Two potential issues are in your patch.
1.) If resource/default is set, it's value is always used regardless of what $connectionName contains:
$conn = $dbm->acquire($ctx->getConfig()->get( 'resource/default', $connectionName ));
2.) The FK-Constraints are most likely not removed completely from the output:
$str = preg_replace( '#CONSTRAINT `\w+` #', '', $str );
This only matches CONSTRAINT "fk_mproli_parentid" but not CONSTRAINT "fk_mproli_parentid" ON DELETE CASCADE ON UPDATE CASCADE
We have two DBs within TYPO3, 'Default' and 'Aimeos' as referenced in the mentioned docs.
So I can use db (without dash) as detection, that's cool and I'll improve the code change.
Foreign keys are actually supported (at least in TYPO3 V11) only constraints are not supported. I plan to create a PR once I've finished all tests and integration in our setup. Our setup currently is a bit messy, that's why we want to clean things up. So might take time until next week.
Thanks for your notes.