opencart / opencart

What version of OpenCart are you reporting this for?
v3.0.3.9

Describe the bug
When a database field contains an escaped line break (like "\r\n"), it becomes an actual line break when using the Restore function in the Admin Console. Generally, this affects any database field used for HTML rendering, where the escaped vs. interpreted character changes the way the page renders. More specifically, if you edit a Theme for a layout that contains Javascript with escaped line breaks, any Backups of that template will break when you Restore later.

For example, there are currently over 30 default TWIG files that have this AJAX error handling duplicated over 100 times:

error: function(xhr, ajaxOptions, thrownError) {
  alert(thrownError + "\r\n" + xhr.statusText + "\r\n" + xhr.responseText);
}

If someone uses the Theme Editor to override those TWIG files, does a Backup, then a Restore, every instance of that error handler will look like this:

error: function(xhr, ajaxOptions, thrownError) {
  alert(thrownError + "
" + xhr.statusText + "
" + xhr.responseText);
}

Visiting the page and trying to do anything that requires JavaScript will fail with the exception:
Uncaught SyntaxError: "" string literal contains an unescaped line break

To Reproduce
Base Case:

In the Admin Console, go to Theme Editor and pick any template, like account/account.
Erase the existing template and type in <script>alert("\r\n");</script>. Click 'Save'.
Find the raw field in the database. It is correct: <script>alert("\r\n");</script>
In the Admin Console, run a Backup.
Find the exported field in the Backup SQL script. It is correct: <script>alert("\r\n");</script>
In the Admin Console, run a Restore using the SQL script.
Find the raw field in the database. It is incorrect because the escaped line breaks have become real ones:

&lt;script&gt;alert(&quot;
&quot;);&lt;/script&gt;

Workaround
When I edit Themes that contain Javascript line break strings, I now replace those linebreaks with another character like /.

Expected behavior
I would expect the database field to be identical before Backup and after Restore. I presume that Backup needs to escape the escape slash so the Restore doesn't interpret it.

Server / Test environment:

Test environment deployed in AWS EC2
Amazon Linux 2023
PHP 8.2.9
Apache 2.4.58

Additional context
None.

I am unable to reproduce your issue. Just tested it with the 3.0.x.x admin's design editor for the product.twig, and the oc_theme DB table stores it correctly escaped:

alert(thrownError + "\r\n" + xhr.statusText + "\r\n" + xhr.responseText);

The frontend reads it from the database, and the frontend source view correctly shows this:

alert(thrownError + "\r\n" + xhr.statusText + "\r\n" + xhr.responseText);

Hello, can I confirm that you also did the Backup / Restore steps in the middle?

Thank you.

Yes, I did. All working fine.

I have found another location where this problem occurs for me: If you create, backup, and restore an HTML module containing line breaks.

I know that this Issue is already closed and you couldn't reproduce it -- that's totally fine, I'm not trying to change that decision! :) I just want to add additional context in case it helps someone else who stumbles upon this Issue from Google in the future.

I'm appending to this Issue since the root cause circumstances seem to be the same, and I don't want to flood the Issue Tracker with similar Issues.

To Reproduce

Create a new HTML Module and use the "Code view" to insert line breaks. Save it.
Examine the field in the database and you'll see the escaped line breaks stored correctly.
Run a Backup of the ps_module table. The export SQL is correct.

TRUNCATE TABLE `ps_module`;
INSERT INTO `ps_module` (`module_id`, `name`, `code`, `setting`) VALUES ('5', 'TestText', 'html', '{"name":"TestText","module_description":{"1":{"title":"TestText","description":"&lt;p&gt;Test&lt;\\/p&gt;\r\n&lt;p&gt;Test&lt;\\/p&gt;"}},"status":"1"}');

Now run a Restore of the export SQL. Examine the field in the database and you'll see that the line breaks are no longer escape characters.
Visit the Extensions > Extensions > Modules page. The Restored HTML module is incorrectly shown as "Disabled" even though it's Enabled in the database. Click into it to edit and all of the fields are blank even though there is data for it in the database. Also, if you try to add the Module to a layout and then save, the module is not shown during page rendering and does not show up when you edit the layout again. I presume that the Admin Console is unable to load this corrupted Module for editing.

Workaround
Write your HTML module on 1 line.

Expected behavior
I would expect the contents of a field in the database to be identical before and after Backup/Restore.

Server / Test environment
Same as above.

The HTML module itself is fine.

However, I managed to reproduce the issue with the Backup/Restore. The code in the admin/model/tool/backup.php appears to be quite convoluted. We'll have to look into it. Probably the same issue in OpenCart 4.0.2.4 (master branch), too.

OK, I tested a quick patch for the admin/model/tool/backup.php, in function backup:

					.....
					foreach (array_values($result) as $value) {
						if ($value !== null) {
							$value = str_replace(array("\x00", "\x0a", "\x0d", "\x1a"), array('\0', '\n', '\r', '\Z'), $value);
							$value = str_replace(array("\n", "\r", "\t"), array('\n', '\r', '\t'), $value);
							$value = str_replace('\\', '\\\\',	$value);
							$value = str_replace('\'', '\\\'',	$value);
							$value = str_replace('"', '\\"',	$value);
//							$value = str_replace('\\\n', '\n',	$value);
//							$value = str_replace('\\\r', '\r',	$value);
//							$value = str_replace('\\\t', '\t',	$value);
							$values .= '\'' . $value . '\', ';
						} else {
							$values .= 'NULL, ';
						}
					}
					.....

Needs more testing of course, but seems to work for the oc_module table.

Can you or anyone else do some more testing please?

Update: Just tested the backup and restore for a whole OC database, and then another backup, and compared the first backup with the 2nd one, they are identical, except for my own session expiry value. Tested on a Linux client and server. Need someone to test the patch on a Windows system, too, before doing a final commit and merge.

Confirmed that this fix absolutely works on oc_module.

Does not work on oc_theme. Example case shown below.

Test theme:

After Backup/Restore:

Escape character 4 is preserved as desired.
Escape character 1 incorrectly gets rendered as text.
Escape characters 2, 3, 5, and 6 incorrectly get rendered in the script tags and break the Javascript.

I'm guessing that a blanket comment-out in the backup code will be an insufficient fix, since where the escape character appears in a theme (e.g. inside of a Javascript string) matters (and surely no one wants to do an assessment of the whole schema to find other tables affected!).

It might be safer to to keep the Backup exactly as-is and then make the blanket adjustment on the Restore side (e.g., Restore sees "\r" so it literally stores the string "\r" in the field, because that's exactly what was in the DB before the Backup). Maybe some string manipulation (or preventing any implicit manipulation) in admin/controller/tool/backup.php import()?

Thank you for looking into this again. Since it's a side effect of Restore rather than core functionality, I can work around it while there are higher priority bug fixes out there!

Yeah, may need some further patches for the import function in file admin/controller/tool/backup.php

Actually, the import appears to be fine. It's likely to be an issue with the theme editor, not being able to read correctly from the DB oc_theme table.

Update: Try these changes:

file admin/controller/design/theme.php, in function template:

		.....
		$theme_info = $this->model_design_theme->getTheme($store_id, $theme, $path);

		if ($theme_info) {
			$json['code'] = html_entity_decode($theme_info['code']);
			$json['code'] = str_replace(array('\\r','\\n','\\t'),array("\r","\n","\t"),$json['code']);
		.....

And in function save:

		......
		if (!$json) {
			$this->load->model('design/theme');

			$pos = strpos($path, '.');
			$code = str_replace(array("\r","\n","\t"),array('\\r','\\n','\\t'),$this->request->post['code']);

			$this->model_design_theme->editTheme($store_id, $theme, ($pos !== false) ? substr($path, 0, $pos) : $path, $code);

			$json['success'] = $this->language->get('text_success');
		}
		......

And finally, for file catalog/controller/event/theme.php, in function index:

		.....
		$theme_info = $this->model_design_theme->getTheme($route, $directory);

		if ($theme_info) {
			$code = html_entity_decode($theme_info['code'], ENT_QUOTES, 'UTF-8');
			$code = str_replace(array('\\r','\\n','\\t'),array("\r","\n","\t"),$code);
		}
		.....

The theme editor look fine to me. The issue looks to be the following lines.

opencart/upload/admin/model/tool/backup.php

Lines 48 to 49 in 3f696d0

    
           $value = str_replace(array("\x00", "\x0a", "\x0d", "\x1a"), array('\0', '\n', '\r', '\Z'), $value); 
        
           $value = str_replace(array("\n", "\r", "\t"), array('\n', '\r', '\t'), $value);

If the data has both actual CR and LF and escaped versions, such as CRLR\r\n. It will converted to \r\n\r\n. Where it should be \r\n\\r\\n.

\ should probably be escaped first.

Well, I tested all my suggested modifications for 3.0.x.x, and it all works fine now. I haven't yet committed and merged these suggested modifications.

@mhcwebdesign Yes it will work as you are effectively escaping the \ in front of some characters before the other backup escaping. It's just the wrong place to do it. It needs to be done in the backup as there is other data in the database that needs escaping correctly. Such as the HTML module.

I agree that fixing this at the controller/model level feels like a band-aid -- it will definitely solve my reported problem, but it doesn't address the underlying issue: The Backup/Restore process does not result in a 100% identical database afterwards.

The application works correctly on data in the tables before a Backup.
Restore changes the data unexpectedly.
The application doesn't always handle this changed data well.

Instead of trying to handle the changed data on a case by case basis (themes, modules, etc), it may be more robust to fix Backup/Restore itself -- either changing Backup to serve better SQL, or changing Restore to account for quirks in the existing SQL. Or a combination of both that also allows backwards compatibility with SQL backups that already exist out in the wild? Maybe I'm just inventing work now :D

The base test case should be: "Every field in the database before the Backup looks identical after the Restore". A corollary: "If a field contains \r\n before the Backup, it should still contain \r\n after the Backup, and not the control character versions of those escaped line breaks.

I recognize that I'm not a contributor here, just an interested user, so I totally get it if the "pure" approach is too big to tackle. Just offering my opinion! Thank you.

What is probably needed is to replace.

opencart/upload/admin/model/tool/backup.php

Lines 48 to 54 in 3f696d0

    
           $value = str_replace(array("\x00", "\x0a", "\x0d", "\x1a"), array('\0', '\n', '\r', '\Z'), $value); 
        
           $value = str_replace(array("\n", "\r", "\t"), array('\n', '\r', '\t'), $value); 
        
           $value = str_replace('\\', '\\\\',	$value); 
        
           $value = str_replace('\'', '\\\'',	$value); 
        
           $value = str_replace('\\\n', '\n',	$value); 
        
           $value = str_replace('\\\r', '\r',	$value); 
        
           $value = str_replace('\\\t', '\t',	$value);

With.

$value = str_replace(array('\\', "\x00", "\n", "\r", "\x1a", '\'', '"'), array('\\\\', '\0', '\n', '\r', '\Z', '\\\'', '\"'), $value);

@briuri Could you give that a try?

Confirmed that @ADDCreative 's fix (alone, without any previous modifications) seems to work correctly.

HTML Modules are identical before and after backup/restore.

Themes are identical before and after backup/restore.

Both types of business object can be edited in the Admin Console and rendered in the UI after Restore!

yep, @ADDCreative is a genius, his bugfix works perfectly. I have now created a pull request and merged it to the 3.0.x.x, see #13709

	$value = str_replace(array("\x00", "\x0a", "\x0d", "\x1a"), array('\0', '\n', '\r', '\Z'), $value);
	$value = str_replace(array("\n", "\r", "\t"), array('\n', '\r', '\t'), $value);

	$value = str_replace(array("\x00", "\x0a", "\x0d", "\x1a"), array('\0', '\n', '\r', '\Z'), $value);
	$value = str_replace(array("\n", "\r", "\t"), array('\n', '\r', '\t'), $value);
	$value = str_replace('\\', '\\\\', $value);
	$value = str_replace('\'', '\\\'', $value);
	$value = str_replace('\\\n', '\n', $value);
	$value = str_replace('\\\r', '\r', $value);
	$value = str_replace('\\\t', '\t', $value);

[3.0.x.x] Escaped line breaks in database fields are interpreted during Restore process