asfktz / autodll-webpack-plugin

Perhaps I'm being optimistic but ... it would be pretty fantastic if we could leverage the webpack loaders we were already using to allow bringing CSS / images into the DLL bundle. Not even sure if its possible, but with packages that are mixed mediums and unchanging (think font-awesome) the inclusion would be pretty sweet.

Not an issue, more a polite request / fervent hope.

EDIT: I noticed that you could add plugins as you do in the webpack config, so I tried adding rules in the same way and it appears to not impact anything. :(

Hi, @tquetano-r7!

That's a great idea! never thought about inheriting the loaders from the config before.
It can really solve the problem with packages like font-awesome.
Let me sleep on it.

I noticed that you could add plugins as you do in the webpack config, so I tried adding rules in the same way and it appears to not impact anything. :(

Yeah, only plugins were added. I believe in adding a feature only when I know for sure there's a use case for it. Plugins, for example, were added because people wanted to use the DLL in production and needed a way to pass it thru the uglify plugin.

Hi, @tquetano-r7,

I'm playing around with this idea in the loaders branch.

I would like to share my thoughts with you if you don't mind.

First, I can allow the user to pass the module option to the DLL compiler.
Which is the same as webpack's module

new AutoDllPlugin({
  filename: '[name].dll.js',
  entry: { vendor: [ ... ] },
  module: {
    strictExportPresence: true,
    rules: [
      {
        oneOf: [
          {
            test: [/\.bmp$/, /\.gif$/, /\.jpe?g$/, /\.png$/],
            loader: require.resolve('url-loader'),
            options: {
              limit: 10000,
              name: 'static/media/[name].[hash:8].[ext]'
            }
          },
          {
            exclude: [/\.js$/, /\.html$/, /\.json$/],
            loader: require.resolve('file-loader'),
            options: {
              name: 'static/media/[name].[hash:8].[ext]'
            }
          }
        ]
      }
    ]
  }
});

But because it makes a lot of sense to reuse the loaders from your own config, I thought about adding an inherit options, which basically copy the module option from your config to the DLL config:

module.exports = {
  entry: {
    app: './src/index.js'
  },

  output: {
    filename: '[name].bundle.js',
    path: path.resolve(__dirname, 'dist'),
    publicPath: '/'
  },

  module: {
    strictExportPresence: true,
    rules: [
      {
        oneOf: [
          {
            test: [/\.bmp$/, /\.gif$/, /\.jpe?g$/, /\.png$/],
            loader: require.resolve('url-loader'),
            options: {
              limit: 10000,
              name: 'static/media/[name].[hash:8].[ext]'
            }
          },
          {
            exclude: [/\.js$/, /\.html$/, /\.json$/],
            loader: require.resolve('file-loader'),
            options: {
              name: 'static/media/[name].[hash:8].[ext]'
            }
          }
        ]
      }
    ]
  }

  plugins: [
    new AutoDllPlugin({
      filename: '[name].dll.js',
      entry: { vendor: [ ... ] },
      module: 'inherit'
    });
  ]
};

I thought about doing the same with plugins, but it can get a bit awkward (Adding an AutoDLL plugin to the DLL itself.. hmm.. 🙃)

I still need to test it, a lot.
Because if it will turn out to be a bad idea, it will be hard to deprecate it later.

What do you think? I would love to hear your feedback.

Also, If you can help me get a few good test cases I would appreciate it a lot (like what loaders font-awesome needs, and what other libs you know that require a special loader)

Asaf

It would also help to supply config options which get merged with the default config.
I currently have the case that I'm bundling some modules which also import NodeJS internals like fs.

In the main webpack config I just do this to avoid webpack throwing "Cannot find module":

{
    node: {
        fs: 'empty',
        module: 'empty',
        net: 'empty',
    },
}

But when I now try to use this plugin, I cannot set this config, which results in the exception mentioned above.

So maybe it would be possible instead of just allowing modules to be specified that every config can be specified. Even better would be if the complete webpack config is taken as is and then modify entry, output and plugins for the dll bundle.
Although if I think about it that some plugins should probably not be part of the dll config, like CommonsChunkPlugin. So a separate config is probably the best option?

Maybe something like this could work:

    new AutoDllPlugin({
      filename: '[name].dll.js',
      entry: { vendor: [ ... ] },
      webpack: {
         // my custom webpack config
      }
    });

Hi @danez,
Thanks for bringing this up!
It's an important use case to consider.

maybe it would be possible instead of just allowing modules to be specified that every config can be specified.

You are right, It will be much better to find a generic solution instead.
Allowing the user an option to pass his own config is the right way to go, but I still search for the most intuitive way do so.

To better understand the challenges, here's what happens behind the scenes.

If this is your config:

new AutoDllPlugin({
  entry: {
    vendor:['react', 'react-dom']
  },
  filename: '[name].dll.js',
  plugins: [
    new SomePlugin({
      foo: 'bar'
    })
  ]
});

The generated DLL config will look like that:

{
  entry: {
    vendor: [
      'react',
      'react-dom',
    ],
  },
  output: {
    filename: '[name].dll.js',
    library: '[name]_[hash]',
    path: '/abs-path-to-your-project/.cache/node_modules/autodll-webpack-plugin/environment-name__unique-hash',
  },
  plugins: [
    DllPlugin {
      options: {
        name: '[name]_[hash]',
        path: '/abs-path-to-your-project/node_modules/.cache/autodll-webpack-plugin/environment-name__unique-hash/[name].manifest.json',
      },
    },
    SomePlugin {
      options: {
        foo: 'bar'
      },
    }
  ],
  resolve: {
    extensions: [
      '.js',
      '.jsx',
      '.json'
    ],
  },
}

This is how the user's option will be merged into:

entry passed exactly as it is
filename is output.filename
plugins merged with the DllPlugin

So if we'll take your suggestion for example:

new AutoDllPlugin({
  entry: {
    vendor:['react', 'react-dom']
  },
  filename: '[name].dll.js',
  webpack: {
    entry: {
      vendor: [ ... ],
      other:  [ ... ]
    },
    output: {
       path: path.resolve(__dirname, "dist"),
       filename: "bundle-dll.js",
       publicPath: "/assets/",
       library: "MyLibrary",
       libraryTarget: "umd",
     }
     module: { ... },
     plugins: [ ... ],
     node: { ... }
  }
});

Now we got a few dilemmas to think about:

entry is now duplicated
filename is duplicated too, but that's not so transparent to the user
output, I really don't want the users to mess with it (it's the cache output, not the user's output), but they may want to change output.library and output.libraryTarget in some advanced configurations.

Even better would be if the complete webpack config is taken as is and then modify entry, output and plugins for the dll bundle

Yes, this is something I really want the plugin to have, the ability to inherit from the user's config.

I don't know how much intuitive it will be as the default behavior (although it seems like it does, judging from the latest issues)

module.exports = {
  entry: {
    app: './src/index.js'
  },

  output: {
    filename: '[name].bundle.js',
    path: path.resolve(__dirname, 'dist'),
    publicPath: '/'
  },

  module: {
    rules: [ ... ]
  }

  node: node: {
      fs: 'empty',
      module: 'empty',
      net: 'empty',
  },

  performance: {
    hints: false,
  },

  devtool: 'cheap-module-source-map',

  plugins: [
    new AutoDllPlugin({
      filename: '[name].dll.js',
      entry: { vendor: [ ... ] },

      inherit: true // <---
      config: {
        plugins: [
          new UglifyJsPlugin()
        ]
      }
    }),
    new HtmlPlugin({
      template: "index.ejs"
    }),
    new webpack.DefinePlugin({
      PRODUCTION: JSON.stringify(true),
      VERSION: JSON.stringify("5fa3b9")
    })
  ]
};

By setting inherit: true the user's own config will be copied into the DLL config, except:

entry
devServer
plugins

Then, he can override it using the config options.
You don't want to inherit module? set config: { module: {} }

Plugins are the most problematic.
How do we know which plugins should be excluded?
AutoDll of course, but what about HtmlPlugin or CommonChunk?

Of course, the user can override that by settings config: { plugins : [] },
But what if he doesn't? bad things will happen.

The best option I can think of is to not inherit plugins in the first place.

I would love to hear your opinion,
it is very important for me to get it right.

Asaf

I assumed that this (autodll-webpack-plugin using my webpack config's loaders when bundling the DLL) would be done automatically by default. I was surprised when I saw that wasn't the case. I think the proposed "inherit" option would be a nice solution (from my admittedly very limited experience with the plugin).

Technically, loaders can also apply to vendor dependencies. For example, it was previously necessary (in webpack 1.x) to add JsonLoader manually for certain node modules to be loaded properly.

In my case, I have a home-grown library (that we modify very infrequently) that I need to bundle using specific loaders. autodll-webpack-plugin chokes on that one.

Is there a solution for this apart from reverting to a manual DllPlugin configuration?

@fortinmike I have the same needs, and was quite disapointed too. I guess everyone thinks their use case is a common one ;)

@asfktz IMHO inherit: true would make so much sense to me. I'm really looking forward to this.
I'm using two "home-made vendors" deps, one with much SCSS in it, and one with a big load of JS. The DLL plugin is then choking on the SCSS import on my main bundle.

Please let me know if I can help.

Ultimately, thanks for this hard work, simplyfing is really hard 👍

@fortinmike @thomasbertet,
Hi! thanks for letting me know how important that feature is. I think so too.

It's important to me to be aware of more advanced use cases so I can keep them in mind while I add new features.

I apologize that it takes so long, My life got a bit too busy lately and I hope to get more free time soon.

I almost finished working on that one. hope to publish it on the weekend.
I'll really appreciate your help testing this feature if you feel like (:

Asaf

I would like to confirm that inherit makes a lot of sense. Making DLLs would be useful for us to not only split vendors out of the chunks, but also parts of our app itself (we usually don't work on the whole app at the same time, but only on an individual functionality).

Hi @niieani,
Good to have you here (:

Yeah, I agree. inherit makes a lot of sense for me too.

I implemented both of the features (inherit and config) on the next branch,
They can be used like so:

module.exports = {
  context: __dirname,
  entry: {
    main: './src/index.js',
  },

  module: {
    rules: [ ... ]
  },

  plugins: [
    new UglifyJsPlugin()
    new AutoDllPlugin({
      filename: '[name].[hash].js',
      entry: {
        dllBundle: [ ... ],
      },

      inherit: true, // inherit from the parent config
      config: {} // extend the config
    })
  ]
};

It works! but there are issues that prevent me from shipping it:

Inhering plugins
Currently, plugins are excluded by default.
I'm not sure it's a good idea to merge plugin instances from one config to another.

Loaders are not working properly
I believe that I'm not handling the assets paths correctly.
I made a simple example to demonstrate the problem.

Steps to reproduce the bug:

git clone git@github.com:asfktz/autodll-webpack-plugin.git
git checkout next
npm i
npm run build
cd examples/inherit
npm i
npm start
Open http://localhost:8080/
You should see a square instead of an icon and 404 errors in the network.
Open examples/inherit/webpack.config.js and comment out line 51
(remove ./src/awesome-module.js from the dll)
npm start again
You should see the icon now.

I believe this has something to do with the why I assign the assets created by the DLL to the compilation.assets.

autodll-webpack-plugin/src/plugin.js

Lines 78 to 96 in 2044537

    
           compiler.plugin('emit', (compilation, callback) => { 
        
             const dllAssets = memory 
        
               .getAssets() 
        
               .reduce((assets, { filename, buffer }) => { 
        
                 const assetPath = path.join(settings.path, filename); 
        
                 return { 
        
                   ...assets, 
        
                   [assetPath]: new RawSource(buffer) 
        
                 }; 
        
               }, {}); 
        
             compilation.assets = merge( 
        
               compilation.assets, 
        
               dllAssets 
        
             ); 
        
             callback(); 
        
           });

Have any idea what can cause this?
I could really appreciate some help with it (:

Has there been any movement on this?
I'd like to help out with this as soon as I get the chance @asfktz.

I have a huge react codebase of >25 apps that I want to split into DLLs to improve build times. If I understand correctly I wont be able to do that currently with autodll-webpack-plugin since I can't run loaders agains the DLLs (babel loader with react and env presets & syntax-object-rest-spread plugin).

Stepping through a debugger during compilation shows that file-loader (called by url-loader which loads the font-awesome fonts and svg) always executes before autodll-webpack-plugin. Which means that the publicPaths from file-loader will not get the DLL path prepended to it.
Everything works if

autodll-webpack-plugin/examples/inherit/webpack.config.js

Line 47 in a8b44f7

path: './dll',

is set to path:'' because in that case the file-loader publicPath doesn't need anything prepended.

file-loader is ultimately calling emitFile in webpack NormalModule which sets the module assets? I'm not sure why we can't override those assets in

autodll-webpack-plugin/src/plugin.js

Lines 90 to 92 in a8b44f7

    
           compilation.assets = merge( 
        
             compilation.assets, 
        
             dllAssets

like you would expect. It might be a bug in using a combination of DLLs + css-loader + file-loader.

I will look into this further when I have some time

Hi, @tryggvigy
Your help on that issue will be much appreciated!

It seems like you have much more direction than I had.

That will be so amazing if you manage to solve it.

Here are some tips for better debugging:

open 2 terminal windows,
in the first one, run BABEL_ENV=debug npm run build:watch
in the second one, run:

cd examples/inherit/
node --inspect --inspect-brk node_modules/.bin/webpack or webpack-dev-server
open chrome and go to chrome://inspect
click on Open dedicated DevTools for Node

That's the debugging workflow I use (btw, @niieani, It might be useful for you too).

Also npm run test:inspect is useful for debugging tests.

Let me know if you need further clarifications.

Thanks!
Asaf

Curious if there was any more movement on this. We would also like to leverage autodll for separate yarn workspaces we have in our app.

@asfktz would you mind updating the next branch to sync up with master? I can send a PR (rebase or merge) if you'd like.

Hi @switz,

Currently, there's no progress on this.

You're more than welcome to send a PR.
The next branch is no longer in use, you can use the master branch instead.

Thanks!

@asfktz Thanks for this great plugin, very much in need of loaders support in the option atleast, instead of inheriting them from parent config, just like plugins

I think this is the missing piece that will bring auto dll plugin complete, because external css and assets can rather be big and the build can speed up much by bundling them to dlls.

It works

new AutoDllPlugin({
      filename: '[name].[hash].js',
      entry: {
      },
      inherit: true, // inherit from the parent config
      config: {} // extend the config
    })

inherit: true // <---

Thanks!

	compiler.plugin('emit', (compilation, callback) => {
	const dllAssets = memory
	.getAssets()
	.reduce((assets, { filename, buffer }) => {
	const assetPath = path.join(settings.path, filename);

	return {
	...assets,
	[assetPath]: new RawSource(buffer)
	};
	}, {});

	compilation.assets = merge(
	compilation.assets,
	dllAssets
	);

	callback();
	});

Leverage loaders for use with CSS / images / etc

It works! but there are issues that prevent me from shipping it: