Upgrade to latest CKEditor v4 (eg. ~4.11.3)

[Greg Hurrell]

I'm creating this issue to lay out a plan for moving us from the version of CKEditor that we are currently using in Liferay DXP (v4.6.0 + our patches) to the latest upstream v4 release (at the time of writing, that's v4.11.3). The corresponding internal ticket is: https://issues.liferay.com/browse/LPS-91153

How things work right now

This repo is a fork of the upstream ckeditor-dev repo, based on the v4.6.0 release, with a number of patches applied on top in the ee-4.6.x branch. As an indication of the magnitude of the changes we've applied, this is the git diff --stat summary:

 33 files changed, 1210 insertions(+), 40 deletions(-)

Because this is a fork of the upstream repository, we can use its build.sh script in conjunction with our config to produce a final build that bundles about 50 standard plug-ins. We do this with a gulpfile that specifies an artifactName of "ckeditor" and offers these tasks:

[13:08:44] Tasks for ~/code/liferay-ckeditor/gulpfile.js
[13:08:44] ├── clean-maven-dist
[13:08:44] ├── prepare-maven-install
[13:08:44] ├── prepare-maven-snapshot
[13:08:44] ├── init-maven-install
[13:08:44] ├── install-maven-release
[13:08:44] ├── prepare-maven-artifact
[13:08:44] ├── publish-maven-artifact
[13:08:44] ├── publish-maven-snapshot
[13:08:44] ├── maven-install
[13:08:44] ├── maven-publish
[13:08:44] └── maven-publish-snapshot

Using these tasks, the built module gets published to maven and is consumed in frontend-editor-ckeditor-web module of the main Liferay DXP repo. It downloads the webjar, massages the paths to remove various levels of wrapping directories, strips some but not all superfluous files, applies a number of diffs — including adding some extra plugins — and produces a build that can be used in DXP and which is active by default. You can see here where we inject a script tag that loads the built ckeditor/ckeditor.js when the module is loaded.

We also offer AlloyEditor as a module in DXP. The AlloyEditor repo embeds the CKEditor repo as a submodule and has a script for creating a new CKEditor build, which just delegates to the upstream submodule build script to produce a release and a debug build. At the time of writing, AlloyEditor is using CKEditor v4.11.2 with no patches applied on top, and a less extensive config which specifies fewer additional plug-ins.

The AlloyEditor module in DXP has an implicit dependency on the CKEditor one because, as you can see here, it expects ckeditor/ckeditor.js to be available. You can confirm this by unloading the CKEditor bundle in the Gogo Shell, which will cause attempts to use AlloyEditor to fail because the resource is no longer available. In DXP, this is the line that is responsible for selecting the alloy-editor-no-ckeditor-min.js build — ie. the one which is not bundled with CKEditor.

Fun fact: you can configure DXP to use AlloyEditor, CKEditor, or both (in different places). When both modules are active, you will see ckeditor.js being requested twice, but my understanding is that in production the ETag headers will match and the browser will only actually download the file once despite the fact that the URLs differ due to a timestamp in teh query string.

Problems with the current arrangement

• Our CKEditor build is very out of date (4.6.0 came out over two years ago, in November 2016). It is possible that some of the bugs that we are currently patching may already have been fixed upstream.
• Updating to a new version of CKEditor is very hard (look at the repeated history of merge commits), such that it has become so hard that we don't do it any more; the current delta between 4.6.0 and 4.11.3 as reported by git diff --stat is 6981 files changed, 440364 insertions(+), 37450 deletions(-).
• When working in the AlloyEditor project we may have a false sense of confidence that changes will work in DXP because it is easy to forget that the version of CKEditor that is used in the test suite, in CI, in the demo, is not the same one that is eventually used when running in DXP.
• Relatedly, in order to update "Liferay's version of CKEditor", we have to do it in two places.
• Our build process takes us from a JS codebase to Maven via a set of custom Gulp tasks, we then download a JAR artifact (containing a bunch of JS) and transform it with a Java-based toolchain back into JS, in order for it to be delivered via a Java platform to clients.
• It is not straightforward to test a debug using a build of CKEditor in DXP.

Attributes we'd like to see in a possible future arrangement

• We'd like to be using the latest release of CKEditor v4 in DXP.
• We'd like to have a more streamlined update process such that we don't find ourselves stuck on the same version for years at a time.
• We'd like AlloyEditor and DXP to be using the same version of CKEditor.
• We'd like to have a single place to update CKEditor when moving to a new version, and have that take effect in both AlloyEditor and DXP.
• We'd like a simpler build process.
• We'd like to be able to run a debug build of CKEditor just like we can in AlloyEditor (at least, like we can in the open source project).

Proposed future arrangement

0. (Nice to have:) In the name of symmetry, we could update the CKEditor build config used by AlloyEditor to match the one used in this repo.
1. Create a brand new, empty branch in this repo in which to carry out this work. It could be called develop, or next or experimental/ckeditor-update or whatever; the name doesn't actually matter.
2. Let this be a new NPM package called liferay-ckeditor (which would be good because it matches the repo name) or @liferay/ckeditor (which would be good because namespacing allows us to completely forget about collisions with third parties).
3. Copy the build approach used in AlloyEditor — that is, add the upstream CKEditor repo as a submodule and use a wrapper script and a config to produce debug and release build variants — and start by building v4.11.3 of CKEditor. (Later, after testing in DXP, we may find that no patches are needed anymore, but in the event that we still need to apply patches, this repo/branch will be the place to keep them.)
4. Instead of publishing to Maven with Gulp, just publish to NPM. This package will be very lightweight, basically only containing the main ckeditor.js build and accompanying resources (skins, lang files etc).
5. Update the DXP build process to download the NPM package instead of the Webjar. Continue to augment the downloaded artifact with the additonal plug-ins from the _diffs/ folder. One thing to note about this: the webjar contains a lot of redundant content (eg. plug-in files) that wind up in modules/apps/frontend-editor/frontend-editor-ckeditor-web/tmp/META-INF/resources/ckeditor/plugins and may cause confusion; these plug-in files are already built in to modules/apps/frontend-editor/frontend-editor-ckeditor-web/tmp/META-INF/resources/ckeditor/ckeditor.js by the build in this repo, and the only plug-ins we should be seeing in there are the extras that we've added. Additionally, some of the steps that the current build is doing (such as excluding some files) will no longer be required because we won't be shipping the unwanted files in the published NPM package.
6. Remove the custom CKEditor build from AlloyEditor and instead add a dependency on "liferay-ckeditor" package, thus solving the problem of running different versions in different places.
7. (Nice to have:) Figure out how to switch to a CKEditor debug build for testing purposes inside DXP. (This one may require us to concede and ship additional files along side the production builds, negating what I said above about the package being "very lightweight".
8. If all of this works, pick a definitive name for whatever this branch is and set it to be the default so that people will see the branch and the README explaining what all this is when they come to the repo.

Options considered by not chosen

• Given that we have an up-to-date build of CKEditor in AlloyEditor, we thought about inverting the dependency relationship here to make "frontend-editor-ckeditor-web" depend on AlloyEditor instead of the other way around. This would meet some of our goals (like being up-to-date, and updating in one place instead of two), but there are a couple of problems with it. One is that given that AlloyEditor wraps CKEditor, it would be somewhat confusing to have the DXP version of CKEditor depend on a-thing-that-depends-on-CKEditor. Another is that we'd like to preserve the ability to determine the fate of AlloyEditor independently of what we do with CKEditor; that is, just say we wanted to move to a different editor in DXP 7.5, it would be nice if we could move away from AlloyEditor without that meaning having to rework our CKEditor support. Yet another reason is that we'd still need a place to apply patches to CKEditor, in all likelihood; at the moment that place is this repo, but if we made DXP depend on the AlloyEditor package instead, where would those patches go? They could go in the AlloyEditor repo but we'd be back in the same boat again of tying the fates of CKEditor and AlloyEditor together more than is strictly necessary (AlloyEditor will always depend on CKEditor, unless we undertake a grand rewrite, but there is no reason for CKEditor to depend on AlloyEditor).
• Just do a big merge with 4.11.3 and leave everything else as-is. This one may not be realistic due to the size of the delta that has opened up over the last two+ years — ie. it may literally not be possible for us to pull off the merge as mere mortals; it is very hard to determine what changes were made even in this repo (because many commits were cherry picked from upstream with rewritten messages), and the scope of the changes on the upstream side is staggering — and also because we'll be leaving a lot of the value on the table in terms of streamlining the build process, getting to debuggable builds, having one place to update etc.

[Greg Hurrell]

cc'ing @jbalsas and @julien to sanity check and see if I've forgotten anything (I probably have; I'll have another read over this when I come back from lunch and see if there is anything to add/change).

[Chema Balsas]

(Nice to have:) In the name of symmetry, we could update the CKEditor build config used by AlloyEditor to match the one used in this repo.

The asymmetry in this case comes from AlloyEditor needing less plugins because most of the code is unnecessary/unused or replaced by custom plugins/features. The AlloyEditor config was streamlined so when used as a standalone project would create as smallest footprint as possible.

When using CKEditor, however, the expectation (from past versions) is different and much more functionality is expected by default, so we probably need to keep it fatter.

We could probably take the opposite direction, where we match AlloyEditor config to this one... or if we're going to consume this project, keep a couple configs (config.js and config-lite.js) producing different builds for different purposes.

[Greg Hurrell]

Added milestone here for tracking individual tasks: https://github.com/liferay/liferay-ckeditor/milestone/1

[Greg Hurrell]

Done in d6d2bd3#diff-b9cfc7f2cdf78a7f4b91a753d10865a2 and related commits visible on the "master" branch. Will address any follow-up that's needed in separate issues.