magento / magento2

Prior to making any Submission(s), you must sign an Adobe Contributor License Agreement, available here at: https://opensource.adobe.com/cla.html. All Submissions you make to Adobe Inc. and its affiliates, assigns and subsidiaries (collectively “Adobe”) are subject to the terms of the Adobe Contributor License Agreement.

Home Page:http://www.magento.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Major 2.1.3 regression bug breaks js translations, caused by overly aggressive performance optimizations of the static content deployment

hostep opened this issue · comments

Preconditions

  1. Magento CE 2.1.3
  2. PHP 7.0.14

Steps to reproduce

  1. Install Magento CE 2.1.3 using composer
  2. Install a couple of language packs (for example one for nl_NL and fr_FR)
  3. Setup multiple storeviews, each one in a different language
  4. Put your store in production mode (I used the --skip-compilation flag)
  5. Run setup:static-content:deploy --theme=Magento/luma en_US nl_NL fr_FR
  6. Go to each storeview in the frontend and go to the 'create an account' page
  7. Don't fill in the fields, just press the 'Create an Account' button

Expected result

  1. The warnings about the required fields are translated on each different storeview in its own language

Actual result

  1. The warnings about the required fields on all different storeviews are all in English (the first language of the setup:static-content:deploy command)

Discussion

Magento 2.1.3 has optimized the static content deployment so it only generates the content for the first language and then blindly copies that directory to the other requested locales.
This is really great, because 99% of the files in those directories are always the same.
Except for the javascript translations...
And now we have this problem caused by this optimization.

More info

In Magento 2.1.2, if I take a diff of the nl_NL & fr_FR directories, I see 4 files which differ:

....pub/static/frontend/Vendor/theme $ diff -rwq nl_NL fr_FR
Files nl_NL/css/email-inline.css and fr_FR/css/email-inline.css differ
Files nl_NL/css/email.css and fr_FR/css/email.css differ
Files nl_NL/css/styles-m.css and fr_FR/css/styles-m.css differ
Files nl_NL/js-translation.json and fr_FR/js-translation.json differ

I'm not sure if the changes in the email css files are important, the changes in the styles-m.css files only differ by the order in which they are defined (which is weird, but whatever), but the most important file is that js-translation.json file.

Workaround

You can work around this bug by executing the setup:static-content:deploy command for every language separately:

php bin/magento setup:static-content:deploy --theme=Magento/luma en_US
php bin/magento setup:static-content:deploy --theme=Magento/luma nl_NL
php bin/magento setup:static-content:deploy --theme=Magento/luma fr_FR

But this means you can't depend on the performance optimization which runs these processes in parallel, which was introduced in 2.1.1.

@ Magento team: seriously, can you actually test stuff before releasing them into a new version? We aren't waiting on new "performance optimization" features, we just want bug fixes for now, not new features with new bugs.

@woosley-mg @benmarks: I expect this major bug the be resolved in maximum 2 weeks (or let's make it 3 due to the upcoming holidays). Thank you!

@hostep I think their testing procedures are broken. Just going by TravisCI, even pull requests always fail with the same error. I'm of the opinion that this error is hiding a lot of other major errors. @woosley-mg @benmarks can version 2.1.4 be completely clean please?

2.1.3. Also happens in backend. Confirmed. I noticed this also. I had to disable static versioning in Admin Configuration Development, else "translations-nb.js" was not loading. I got the error when I tried to list products in admin - empty result and stops at loading spinner because the translation JS file was not found.

@petterkj: sorry, but your issue has nothing to do with what I describe above. In my case the js-translation.json file is getting created but contains the wrong translations.
Your issue is more related to this one I think: #7869

Another issue with this optimization is that when you did all steps as it described in install guide, the js and css don't work. Magento 2.1.3 looking for it in some strange location for example http://magento2ce.loc/pub/static/version1482322390/adminhtml/Magento/backend/en_US/requirejs/require.js. Why it 'version1482322390' path is appeared in my url? I do not have it directory, so, js and css could not work.

@molneek: this again has nothing to do with what I describe above.

What you now experience is that Magento has switched the static signing of assets to on by default now.
See the release notes: http://devdocs.magento.com/guides/v2.1/release-notes/ReleaseNotes2.1.3CE.html:

Versioning of static files (including CSS, JS, font, and image files) is now enabled by default.

You can disable this in the backend under Advanced => Developer => Sign Static Files

@hostep Thanks, but who can explain why it versioning was necessary enabled by default?
The

Versioning of static files (including CSS, JS, font, and image files) is now enabled by default.

is located in middle of release notes. After install new version I expect that all previous features work as well as before but every time I get new issues and should looking for solutions for fix this instead of do my work.

@Ctucker9233 See tweet stream here: https://twitter.com/tonypiper/status/809328942968664064 – also, the semantic version was increased to say BC, but the notes said no BC. They have confirmed the bug (affecting Travis) is in test and not in M2 code, but you're right, it could be hiding all sorts of other nasties.

Hi @hostep

Thanks for reporting.
We've created internal ticket MAGETWO-62660 to investigate and fix this issue.

Hi @hostep

We've added the bug and your workaround to Known Issues for both CE and EE Release Notes (Magento 2.1.3).

Thanks for your support!

I have same problem with 2.1.3 js-translations.

I am experiencing the same issues also on 2.0.11 when using: setup:static-content:deploy en_US fr_FR
(Single store mode)

@mvmenno

It's work if you do

setup:static-content:deploy en_US
and then
setup:static-content:deploy fr_FR

@phillipvandebriel ah i noticed this works. Well i am deploying using capistrano for multiple installations. So i have to write a new task for that, thank you!

The suggested workaround works indeed. However, I noticed the js-translation file also remain empty when requesting one language in combination with a specific theme or area. In other words, js-translation files also contain an empty string when using one the following commands (and probably commands with other extra parameters):

bin/magento setup:static-content:deploy nl_NL --area frontend
bin/magento setup:static-content:deploy nl_NL --area frontend --exclude-theme Magento/blank
bin/magento setup:static-content:deploy nl_NL --theme Magento/luma

Same problem.
Server: Ubuntu 16.04, Nginx 1.10, PHP 7, MySQL 5.7, Magento 2.1.3 CE clear.
Command php /var/www/html/bin/magento setup:static-content:deploy --theme Magento/luma ru_RU don't help. I think Magento 2 will never work well. Throughout the year, I'm trying to run project on Magento 2. Every step - errors in Magento 2.

@ALL btw. if you only have 1 language for a site it also helps setting all admin users locale to the locale of the frontend store. That way you don't have to compile the en_US language ,it saves the hassle. Afterwards you deploy with static-content:deploy (language code)

@redelschaap are you sure? Because we run something like this when we deploy and it works perfectly fine:

bin/magento setup:static-content:deploy --theme Magento/backend en_US
bin/magento setup:static-content:deploy --theme Vendor/custom nl_NL
bin/magento setup:static-content:deploy --theme Vendor/custom fr_FR

(the area is not strictly necessary I believe, Magento is able to figure it out on its own)

@hostep My steps:
Copy file ru_RU to theme, or create pack to app/i18n.

cd /var/www/agestor.ru/html
rm -rf pub/static/*
rm -rf var/view_preprocessed/*
bin/magento cache:clean
bin/magento setup:static-content:deploy --theme Magento/backend en_US
bin/magento setup:static-content:deploy --theme Agestor/default ru_RU
bin/magento cache:clean

Result: all js phrases on English (minicheckout, checkout).

@Flamestyle: sorry, you must be doing something wrong, because it works over here.

  1. Install Magento CE 2.1.3 using composer
  2. Install a Russian language pack using composer (I took this one: https://github.com/Magento2Translations/language_ru_ru)
  3. Set your locale in the Magento backend to ru_RU
  4. Run php bin/magento setup:static-content:deploy --theme Magento/luma ru_RU
  5. Check the js-translation.json file:
$ head -c 100 pub/static/frontend/Magento/luma/ru_RU/js-translation.json
{"Close":"\u0417\u0430\u043a\u0440\u044b\u0442\u044c","Loading...":"\u0417\u0430\u0433\u0440\u0443\u

after cleaning up the escaped characters (just informational, the above should already work):

{"Close": "Закрыть", "Loading...": "Загрузка...",

@hostep I don't using composer with Russian pack. Only add file ru_RU to theme. Or make my pack same official documentation.
You try Manually method?
js-translation.json - empty...
I can't use this pack - 26% ready.
Magento 2 have docs - this not works.
Please write steps, who can correct use translation with files (composer.json, language.hml, registration.php, xx_XX.csv) or only one file xx_XX in theme.

@hostep Yes I'm sure. Just tested this again. It only works for me when I delete the js-translation.json file manually and request one language without any other parameters.

@hostep Please detail your steps.
I try 2 methods:
Method 1: Create a language package same official docs, create 4 files: composer.json, language.hml, registration.php, ru_RU.csv.
After make this commands:

cd /var/www/html
rm -rf pub/static/*
rm -rf var/view_preprocessed/*
php /var/www/html/bin/magento cache:clean
bin/magento setup:static-content:deploy --theme Magento/backend en_US
bin/magento setup:static-content:deploy --theme Agestor/default ru_RU
php /var/www/html/bin/magento cache:clean

Method 2: Add ru_RU to my theme. Path: app/design/frontend/Agestor/default/i18n/.
After make this commands:

cd /var/www/html
rm -rf pub/static/*
rm -rf var/view_preprocessed/*
php /var/www/html/bin/magento cache:clean
bin/magento setup:static-content:deploy --theme Magento/backend en_US
bin/magento setup:static-content:deploy --theme Agestor/default ru_RU
php /var/www/html/bin/magento cache:clean

Same problem: js phrases are not translated.

Check in your server configuration that the jzon files are processed by Nginx as a static file.

Under the nginx/apache settings in plesk, I tried turning on "Serve static files directly by nginx" with the following default files active:

ac3 avi bmp bz2 css cue dat doc docx dts eot exe flv gif gz htm html ico img iso jpeg jpg js mkv mp3 mp4 mpeg mpg ogg pdf png ppt pptx qt rar rm svg swf tar tgz ttf txt wav woff woff2 xls xlsx zip xml json

This has nothing to do with nginx configs. Magento checks those files via the file system.

Confirmed this bug in Magento 2.1.4 when having more than 1 language
e.g. running bin/magento setup:static-content:deploy en_US nl_NL creates empty js-translation.json
Running these commands seperate:
bin/magento setup:static-content:deploy en_US
bin/magento setup:static-content:deploy nl_NL
makes the translation work again.

I re-cofirm @PascalBrouwers's issue. The Translation class should not be a singleton I guess. Took me a while to understand why this happens.

I can confirm this bug is still present in version 2.1.6. Will it be fixed in version 2.2?

This bug present in 2.15 and 2.16

I'm not 100% sure if this is the same bug, but after debugging the static content deployment process for a couple of hours, I found out that that in Magento\Framework\Translate#setConfig() on line 212, getId() always returns null. This is because the loaded themes don't have a theme_id in their _data since they're loaded from the filesystem (in 2.1.x, this code was in Magento\Theme\Model\Theme\Collection) instead of the database. That leads to this if statement to always return true and never loading the theme translations.

(I used a random commit for the links so the line numbers don't change when the branch is updated.)

Hmm, even after running the commands seperate I sometimes get that the js-translation.json contains an empty array.

Removing the file js-translation.json and running bin/magento setup:static-content:deploy nl_NL did the trick to get the translations back.

Note that I do not have theme translations and thus not experiencing issues other people run into that they have to change vendor/magento/module-translation/Model/Json/PreProcessor.php

This issue has already been fixed in develop branch.
Internal ticket for ver. 2.1. - MAGETWO-63309 - it is planned for one of the next releases.

There is a next release for 2.1? Do you know when?

@veloraven, in which commit? We are testing the latest version of the develop branch for 2.2 and we can still recreate the bug.

@veloraven
First language deployment is still overrides js-translation.json in all languages.
php bin/magento setup:static-content:deploy en_CA en_US
Got the canada js-translation.json in both static path.

on my magento 2.2.0-rc i have no translation validation messages on registation page even use only one language when enabled: production mode+Merge JavaScript Files+Enable JavaScript Bundling +Minify JavaScript Files (on every other pages validation messages are translated properly)

ps: same problem with /catalogsearch/advanced/ page

i found that when form validation is done by javascript fe:

var dataForm = $('#form-validate');
dataForm.mage('validation', {}); 

validation messages are not translated for all this page, but where form tag has code: data-mage-init='{"validation":{}}', they are working properly, can you check this please?

@davidalger have you confirmed that the issue is in fact fixed in 2.1.8? we updated to your 0.7.0 release on 2.1.8 and for us the issue is still regressed

@chaohe: it is still broken in Magento 2.1.8, the release notes seem to suggest that it is fixed, but it definitly is not:

Static content deployment (SCD) now works when multiple languages are specified. Previously, Magento displayed an error if you tried to deploy static content in more than one language (for example, bin/magento setup:static-content:deploy en_CA fr_CA de_DE).

I don't really understand what they mean with this, they fixed something about an error being displayed when using multiple locale's, but I've never seen an error. But the statement that says 'SCD now works when multiple languages are specified' is definitely incorrect since this bug isn't fixed yet.
/cc: @jfrontain?

Also: since MAGETWO-62660 got merged in the Magento 2.2.0 release candidates I was hoping it would be fixed over there, but that is also still not the case: #10673 :(

@chaohe / @hostep This is what I get for relying on the release notes! Ouch.

I'm reverting that change in the gem and including it in v0.7.1, which I'll push up shortly.

@hostep, thank you for your report.
The issue is already fixed in 2.2.0

@magento-engcom-team: well no, it's not really fixed in 2.2.0, see: #10673

@okorshenko: Well yes, maybe, probably, I don't know...

It is mentioned in this thread that MAGETWO-62660 fixes this bug. I have personally never tested those commits, so I don't know for sure if it is true or not.
But MAGETWO-62660 is included in 2.2.0-RC2.2, while 2.2.0-RC2.2 still seems to have the same problem, but maybe caused by a new bug? That's why I created a new issue, because I assumed it would be a new bug. Otherwise this issue would have been closed - like it was today - and nobody would have noticed it was still an issue in 2.2.0-RC2.2.

So it might be fixed if you apply MAGETWO-62660 on top of 2.1-develop (again: untested), but it's definitely not fixed in 2.2.0-RC2.2.

Sorry, I have no more info about this, I was hoping you guys knew more about the status of this :)

It would be great if this could get fixed before 2.2.0 GA is released, since its part of the "new" pipeline deployment, so I would have expected that this static deployment would have been tested a lot better with the release of 2.2.0, but it doesn't really seem to be the case unfortunately.

They are not duplicates

@hostep, thank you for your report.
We've created internal ticket(s) MAGETWO-62660 to track progress on the issue.

On my side, the problem comes from the LocaleQuickDeploy which copy the js translation file from first locale to the new local without regeneration.

@magento-engcom-team - Any update on this please?

@BenSpace48 I've checked the internal ticket and have seen some progress on this issue. I've chased again to get a better understanding on where this sits.

Thanks as always! <3

Hi @TomashKhamlai. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if your want to validate it one more time, please, go though the following instruction:

  • 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 2. Verify that the issue is reproducible on 2.3-develop branch

    Details- Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 3. Verify that the issue is reproducible on 2.2-develop branch.

    Details- Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

  • 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

Hi @engcom-backlog-tomash. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if your want to validate it one more time, please, go though the following instruction:

  • 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 2. Verify that the issue is reproducible on 2.3-develop branch

    Details- Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 3. Verify that the issue is reproducible on 2.2-develop branch.

    Details- Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

  • 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

Hi @engcom-backlog-nazar. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if your want to validate it one more time, please, go though the following instruction:

  • 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

  • 2. Verify that the issue is reproducible on 2.3-develop branch

    Details- Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
    - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

  • 3. Verify that the issue is reproducible on 2.2-develop branch.

    Details- Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
    - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

  • 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

Still a problem in 2.3 installed from composer create project can someone please give an example of what the contents of the js-translation.json file looks like as mine is just json []

@LiamKarlMitchell: I'm pretty sure you are running against another bug. This bug here is around the js-translation.json file containing data, but incorrect data (wrong translations).

Even though I'm like 99% sure this was fixed in Magento 2.2.2 and higher the labels added by @engcom-backlog-nazar seem to contradict this. Haven't tested it on 2.3 though...

I think I know this error and I solved it by the following patch :

--- a/vendor/magento/module-translation/Model/Json/PreProcessor.php
+++ b/vendor/magento/module-translation/Model/Json/PreProcessor.php
@@ -77,11 +77,12 @@ public function process(Chain $chain)
             if ($context instanceof FallbackContext) {
                 $themePath = $context->getThemePath();
                 $areaCode = $context->getAreaCode();
-                $this->translate->setLocale($context->getLocale());
             }
 
             $area = $this->areaList->getArea($areaCode);
             $area->load(\Magento\Framework\App\Area::PART_TRANSLATE);
+
+            $this->translate->setLocale($context->getLocale())->loadData($areaCode, true);
 
             $chain->setContent(json_encode($this->dataProvider->getData($themePath)));
             $chain->setContentType('json');

Thx man! You save my day :)

Closing as this was fixed somewhere in Magento 2.2.x

The only reason this ticket was still open is because it wasn't fixed yet in Magento 2.1
But Magento 2.1 will be EOL after its final release 2.1.18 next week, so let's just close this ticket.