Code Splitting

Question

Code Splitting

sokra opened this issue 12 years ago · comments

GWT has a cool feature named Code Splitting which loads parts of the code on demand. This is also appliable for webmake.

Here my idea:

// a.js:
var b = require("b");
// ...
bindEventHandlerForSomeRarEvent(function() {
  // the magic keyword "process.nextTick", which behaves normally in node.js
  // but so some special thing in webmake
  process.nextTick(function(err) {
    // Some code loading err handler
    // err is undefined in node.js
    if(err) { /* ... */ }
    var c = require("c"); // really big libary, which is only needed in this rar case
    // ...
  });
});

webmake core would provide the magic process.nextTick function to the compiled code.
process.nextTick do this in the compiled code:

store the callback function
adds a script tag to the document
the script contains JSONP
callback function (JSONP) adds some new modules to the context
and calls the stored callback function.
in case of an error (ex. timeout) the stored callback function is called with some kind of error parameter

Here is a quick hint on the compiled files:

// output.js
( /* core code */
({
  "a": function(/* ... */, process) {
     // a.js:
     var b = require("b");
     // ...
     bindEventHandlerForSomeRarEvent(function() {
       // the magic keyword, which behaves normally in node.js
       // but so some special thing in webmake
       process.nextTick(function(err) {
         // Some code loading err handler
         // err is undefined in node.js
         if(err) { /* ... */ }
         var c = require("c"); // really big libary, 
               // which is only needed in this rar case
         // ...
       });
     });
  },
  "b": function /* content of b.js */
  // here is no "c": !!
  // which saves bandwidth

// may be named "a.output.js":
webmake_magic_jsonp_function( // the jsonp callback function
"a", // origin of code loading (used for identifing the stored function)
{
  "c": function(/* ... */, process) {
    // the content of the big libary
  },
  "d": function(/* ... */, process) {
    // d.js is here a dependency of c.js (optional)
  }
}

This feature may makes the compile process a big complexer.

It may be useful for:

loading parts of your webapp when accessing them
loading polyfills only when necessary
decrease startup time

Mr.Wiley commented 3 years ago

wow!

Mariusz Nowak · Answer 1 · Sun Feb 26 2012 20:23:40 GMT+0800 (China Standard Time)

Code splitting is indeed most missing major feature and I plan to add it with next big update.
However in first place I imagine it just as a possibility of creating few different bundles that reuse same modules which are loaded once with one of the bundles, that's it.
I don't think it's good idea to change semantics of process.nextTick for client-side for that. If we would really need some function to do async modules bundle load on client side it should be specific to client-side and be used in code written specifically for client-side.

Phoscur · Answer 2 · Sun Feb 26 2012 22:32:32 GMT+0800 (China Standard Time)

Isn't setTimeout(fn, 0) the browser replacement for process.nextTick(fn) ?

Phoscur · Answer 3 · Sun Feb 26 2012 22:43:27 GMT+0800 (China Standard Time)

var c = require("c"); // really big library, which is only needed in this rar case

Maybe this is the problem, you should not include a big library if you don't really need it.

I'm fine with having a little more startup time and then a more fluent interface. I would like to have an option in the opposite direction, so that module code can be preexecuted to be ready when it is required, but that's probably not worth the effort.

This does not mean that I don't like the idea of sending code later to the client, I see usecases for that. But it's not to reduce startup time. It would be nice to be able to rollout new features without a whole app refresh.

Tobias Koppers · Answer 4 · Sun Feb 26 2012 23:56:43 GMT+0800 (China Standard Time)

@medikoo
i don't think this changes the semantics of process.nextTick. process.nextTick executes some code in the near future in node.js. It you implement Code Splitting as i propossed it has the same semantics. It executes some code in the near future. There is a code loading in between on client side, but this is transparent to the user.

@Phoscur yes setTimeout(fn, 0) is the replacement, but i don't want to use a existing function (on client side) because this could impact exisiting code. I think process.nextTick is the best choice because it has nearly no impact on node.js. The only impact is that the code is executed later. This semantic stays the same on client side (Ok, the delay is bigger on client side)

Consider a big webapp with several features. The features are in subpages, but because you do not want a whole page reload you have all features in one codebase.
Without Code Splitting this has a negative impact on startup time because you compiled your codebase with webmake into a big js file.
To optimize, you want to split your code and you only want to load if you navigate into the subpage. So insert process.nextTick somewhere in your code (i should be easy in ansynchronous functions). Now your compiled codebase will be splitted into pieces and only the first piece is loaded on webapp startup. This decreases startup time in compare to webmake without this feature.

This feature is optional because you don't need to use process.nextTick, but if you decide to use it it is transparent to the programmer.

Phoscur · Answer 5 · Mon Feb 27 2012 02:05:44 GMT+0800 (China Standard Time)

Why wouldn't you want all features in one codebase?
You can pack about 300k codelines into one megabyte minified scriptfile, how big is your webapp going to get? I think the overhead to request a few bytes when you need them is bigger than a little slower loading time in the beginning.
Keep in mind that the code is only loaded and parsed in the beginning, it is lazy-executed when it's needed.

Creating some shims for node specific functions like process.nextTick is not a bad idea, we can probably steal those from somewhere else anyways.

Mariusz Nowak · Answer 6 · Wed Feb 29 2012 22:51:08 GMT+0800 (China Standard Time)

@sokra I would definitely not mess with process.nextTick it's Node specific and was not intended for asynchrounous background loading.
Anyway as I mentioned, soon we'll have the way to create many separate bundles that share modules and load them on demand. It can be helpful when you have really big application with clearly separated functionalities.

Tobias Koppers · Answer 7 · Thu Mar 08 2012 23:05:54 GMT+0800 (China Standard Time)

I reimplemented webmake from scratch to reduces some limitations of it and add code splitting.
It is just a start and was made in a 2 day hacking session.

It do need some additional love but the most things are working really good.

See https://github.com/sokra/modules-webpack for some docs
and https://github.com/sokra/modules-webpack/tree/master/example for a example

Here is a list of differences to the original webmake:

Module and file names are replaced with a unique id per file
=> different versions of the same module do not overlap anymore
=> no need for resolving on client side => smaller

It is allowed to require outside to current package scope

Browser replacements
web_modules is preferred to node_modules
xxx.web.js is preferred to xxx.js if require("./xxx") without extension

Code splitting with require.ensure
as in http://wiki.commonjs.org/wiki/Modules/Async/A
see links above

Mariusz Nowak · Answer 8 · Sat Mar 10 2012 01:05:37 GMT+0800 (China Standard Time)

@sokra thanks for input.

Your approach is very interesting, but you introduced big limitation with that. You didn't take into account specific cases when modules are not required directly.
In code you may pass generated string to require or run require functions indirectly. Such require calls of course cannot be picked statically but in webmake there's include option through which you can force to include such modules, so there's back door to deal with that, in your fork I guess it's not possible.

In projects I currently work with this is the way templates are loaded, indirectly by template engine, I wouldn't be able to use your solution there.

Other goal, when working on webmake, was not to change source code of modules, it's not that major concern as one above, but still it's certainly better for development and debugging to keep module names and paths intact.

If you decide to continue development of your fork please change it's name, as you pointed it's very different solution.

Thanks!

Tobias Koppers · Answer 9 · Sat Mar 10 2012 03:19:58 GMT+0800 (China Standard Time)

@medikoo thanks for your feedback

The limitation, you pointed out, is a problem, but i think a "back door" can be added with some config options. Do you have a example of a use case? Can you post it?

The changes in the source code are really small so i don't think that this is a big problem.

The point of posting it here is a aspect of code splitting which may can used in your webmake (also without the other changes). It's the same approach as in the inital post, but with require.ensure instead of process.nextTick. (As you argued correctly process.nextTick is more a node specific thing)

I'll rename it if you want so. It was orginally intended as proof of concept for this issue.

Mariusz Nowak · Answer 10 · Sat Mar 10 2012 05:23:44 GMT+0800 (China Standard Time)

@sokra I'm not sure if any back door is available in your solution. Code can only be scanned statically, but all requires should work programmatically, so e.g. you should provide some sort of back door, so following could work:

var indirect = require;
indirect(pathGeneratedByFn());

When you do all paths resolution on server side, and for client you don't provide any mechanism to do real path resolution then it's no place for back door.

For example here https://github.com/medikoo/domjs/blob/master/lib/domjs.js#L92 I call require indirectly.

Tobias Koppers · Answer 11 · Sat Mar 10 2012 06:34:53 GMT+0800 (China Standard Time)

Hmm... this really does not seem to work. This stays a limitation.

What do think about the code splitting?

Mariusz Nowak · Answer 12 · Sat Mar 10 2012 17:25:17 GMT+0800 (China Standard Time)

To have code splitting, in first place we need to add possibility of creating different bundles that share same modules, form that point you can upload such bundles on demand in many different ways.

By example, code of additional bundle that we may need to load, will probably look like:

require.push({
  // extra modules
})
("invoke/one/of/the/added/modules"); // optional

Where require is global variable exposed by bundle that was initially loaded.

I'm still not sure do any load on demand functionality should be included in webmake, or should it be left to external tools how to handle that. We'll figure it out in near future.

Tobias Koppers · Answer 13 · Sat Mar 10 2012 20:15:27 GMT+0800 (China Standard Time)

I've moved it to https://github.com/webpack/webpack

Hope that is ok...

Mariusz Nowak · Answer 14 · Sat Mar 10 2012 20:32:58 GMT+0800 (China Standard Time)

That's ok, thanks

Tobias Koppers · Answer 15 · Tue May 08 2012 06:14:32 GMT+0800 (China Standard Time)

A few moths ago, I worked much on webpack and it's working really good.

Here is a example web app: http://webpack.github.com/example-app/

Tobias Koppers · Answer 16 · Tue Oct 23 2012 02:44:26 GMT+0800 (China Standard Time)

I don't need this issue anymore. Should I close it or do you want it to stay open?

Mariusz Nowak · Answer 17 · Sun Jun 23 2013 17:14:06 GMT+0800 (China Standard Time)

I plan to add this feature, with Webmake v0.4, so I'll keep it open until it's done.

Deleted user · Answer 18 · Mon Dec 12 2016 03:08:35 GMT+0800 (China Standard Time)

@medikoo Over 3 years later... is there anything done here?

Mariusz Nowak · Answer 19 · Mon Dec 12 2016 04:03:38 GMT+0800 (China Standard Time)

@JHGitty nothing yet, but you may expect something 1st quarter next year.

Abhinav Singi · Answer 20 · Fri Dec 15 2017 18:36:43 GMT+0800 (China Standard Time)

So that's how it all started!
https://twitter.com/TheNicoKoenig/status/941410877290491905

Aman Bhargava · Answer 21 · Sun May 02 2021 03:19:27 GMT+0800 (China Standard Time)

Like seriously! this is how Webpack was born 😄 @sokra

Ashish Duklan · Answer 22 · Tue Dec 14 2021 15:05:01 GMT+0800 (China Standard Time)

Genesis 😃

JinPeng · Answer 23 · Thu Sep 29 2022 16:51:33 GMT+0800 (China Standard Time)

火钳刘明

Ajay Pathak · Answer 24 · Tue May 02 2023 00:57:18 GMT+0800 (China Standard Time)

Happy to see the history in a thread.

xuejiangping · Answer 25 · Wed Aug 23 2023 16:17:21 GMT+0800 (China Standard Time)

hello webpack