Babel caching issues

[Ole Treichel]

Hey,

First of all thank you for this awesome plugin, it solves a lot of things I need for MDX-Documentation at one go.

From what I can understand the extractFrontMatter function is only executed on startup / restart of the webpack dev server. This means that when I add a new mdx file in pages the frontmatter is not generated before I manually restart the dev server.

Are there any workarounds for this? How do you use the plugin in development?

I am pretty new to babel/webpack stuff, sorry if this is a noobie question. Any help would be much appreciated, thank you :)

[Jeff Escalante]

Right now, restart the watcher.

This plugin is most useful in concert with babel-import-glob-array, which is cached by babel, but when you add a new file the babel cache doesn't update since it's watching the file contents, not the external filesystem. There are a few issues open for this that I am tracking, but at the moment there's no concrete solution.

Additionally, there is a ton of overhead that comes with scanning the full file tree on every change to every file. We were unable to find a way to get webpack to tell us specifically that a new file has been added, and not an existing file changed etc, so there are only two options: scan the file tree every time there's a change to anything, which is slow, or scan only on initial run. We default to only the initial run, but that can be disabled via this parameter to the underlying dependency. The performance hit is so significant though, that we made it so that you have to explicitly enable it. This could be solved if we/anyone could figure out how to get from webpack that specifically a new file has been added - the only way I have seen this done elsewhere is caching every file in the project and whenever a change comes in comparing it against the previous state to determine if the file exists, which has overhead as well. But very much open to contributions there.

If you're not using the import glob array plugin, this problem can be solved but with a large perf hit by adding an option to change the file tree scan frequency to the prebuild plugin, and could be solved with less of a perf hit if anyone can figure out how to get webpack to tell you whether a changed file is newly added or not within a plugin. If you are using the import glob array plugin, we would also need to solve the babel caching issue on top of that in order for everything to work smoothly.

In the interest of continuing to move forward, and given the speed of a watcher restart and the infrequency of new file adds in the middle of a dev server watch, we basically decided to table this.
As a note, if you are using the import glob array plugin, you'll want to clear out the babel cache before each dev mode run with something like rm -rf node_modules/.cache/babel-loader/ - which you can build right into your npm command in package.json.

Hopefully this helps, and sorry this isn't 100% as smooth as could be!

[Ole Treichel]

Wow, thank you for this super detailed explanation! I do understand a lot more now. Will try the mentioned parameter and see how performance goes with. The plugin still is very awesome 👍

[Ole Treichel]

So I found this webpack-plugin which lets you configure additional folders / files to be watched.

If I configure it within my next.config.js like this:

webpack: (config) => {
    config.plugins.push(
      new ExtraWatchPlugin({
        dirs: [
          path.join(config.context, 'pages')
        ]
      })
    )

    return config
  }

Newly created pages are parsed and frontmatter is generated without a restart of the dev server. If you think this would be useful to add to the plugin I am happy to make a PR :)

[Jeff Escalante]

Oh nice, this is a really great discovery - I'd like to dig a bit deeper on why this works, since pages is already watched by default in nextjs. Will get back to this as soon as I can.

[Pedro Duarte]

@ole-treichel thanks for this!

I tried it out but couldn't get it to work.

const path = require('path');
const ExtraWatchWebpackPlugin = require('extra-watch-webpack-plugin');

module.exports = withPlugins(
  [
    withMdxEnhanced({
      layoutPath: 'layouts',
      defaultLayout: true,
    })({
      pageExtensions: ['ts', 'tsx', 'js', 'jsx', 'md', 'mdx'],
      webpack: (config) => {
        config.plugins.push(
          new ExtraWatchWebpackPlugin({
            dirs: [path.join(config.context, 'pages')],
          })
        );

        return config;
      },
    }),
    withOptimizedImages,
  ],
  {
    // Next.js config
  }
);

I also tried:

module.exports = withPlugins(
  [
    withMdxEnhanced({
      layoutPath: 'layouts',
      defaultLayout: true,
    })({
      pageExtensions: ['ts', 'tsx', 'js', 'jsx', 'md', 'mdx'],
    }),
    withOptimizedImages,
  ],
  {
    // Next.js config
    webpack: (config) => {
      config.plugins.push(
        new ExtraWatchWebpackPlugin({
          dirs: [path.join(config.context, 'pages')],
        })
      );

      return config;
    },
  }
);

My folder structure is:

- pages
  - blog
    - file.mdx

I also tried replacing path.join(config.context, 'pages') with path.join(config.context, 'pages/blog')

[Steve Ruiz]

As a note, if you are using the import glob array plugin, you'll want to clear out the babel cache before each dev mode run with something like rm -rf node_modules/.cache/babel-loader/ - which you can build right into your npm command in package.json.

I wasn't able to get this working with node_modules/.cache/babel-loader/, but it did work with .next/cache/next-babel-loader.

For future readers using this library together with babel-plugin-import-glob-array, my package.json now includes:

  "scripts": {
    "dev": " next dev",
    "build": "next build",
    "start": "next start",
    "refresh": "rm -rf .next/cache/next-babel-loader"
  },