COMUNICA: troubles using quadstore-comunica with rollup

Question

COMUNICA: troubles using quadstore-comunica with rollup

retog opened this issue 4 years ago · comments

Trying to use Quadstore (v7.0.1-alpha.9) with a rollup project I get errors like the following:

Uncaught ReferenceError: _webpack_ignored__comunica_actor_http_memento is not defined

or

Uncaught ReferenceError: require$$0$3 is not defined

Is there any known project succesfully using Quadstore with rollup?

Jacopo Scazzosi · Answer 1 · Mon Oct 19 2020 20:52:02 GMT+0800 (China Standard Time)

Hello @retog! I'm not aware of any project using quadstore with rollup, at the moment. However, this has likely something to do with our custom configuration of Comunica. I have explicitly excluded the @comunica/actor_http_memento from our configuration and I'm surprised to see that something is trying to use it. Two questions:

Do these errors come up as soon as you require() quadstore into your project or do they come up at query time?
~~Can you check whether rollup is hitting the index.bundle.js file from quadstore-comunica as opposed to index.js?~~ It's clear from the fact that you're getting that specific ReferenceError that rollup is correctly hitting index.bundle.js, my bad.

Jacopo Scazzosi · Answer 2 · Tue Oct 20 2020 00:47:10 GMT+0800 (China Standard Time)

Could be related to comunica/comunica#656 ?

Reto Gmür · Answer 3 · Tue Oct 20 2020 06:15:21 GMT+0800 (China Standard Time)

Hello @jacoscaz! I get the error as soon as I add the import to my code

import {Quadstore} from 'quadstore'

I don't need to create an instance or a call a function for the error to appear.

Reto Gmür · Answer 4 · Tue Oct 20 2020 06:34:29 GMT+0800 (China Standard Time)

comunica/comunica#656 manifests in an error at build time, here I get the error at runtime in the browser or in the electron app, so I don't see a relation between the two problems.

A near minimal project showing this problem: https://github.com/retog/rollup-comunica/tree/quadstore (the quadstore branch shows this problem, master shows comunica/comunica#656).

Jacopo Scazzosi · Answer 5 · Wed Oct 21 2020 04:50:06 GMT+0800 (China Standard Time)

While working on our configuration of comunica I noticed that Webpack, unless explicitly instructed not to do so in its webpack.config.json, always ends up bundling all of Comunica's modules and dependencies, regardless of which ones I include in the configuration.

This seems to happen with both Webpack v4.x and v5.x and I only managed to work around it (rather than fix it) using externals. I am still not sure as to whether there's a bug somewhere or whether it's me doing something wrong in Webpack's configuration and/or in Comunica's.

There might be something common to these:

comunica/comunica#656 is about component.js ending up in the bundle when using Rollup, which should not happen;
my own issues with Webpack revolve around unused modules ending up in the bundle, even though I have explicitly dropped them from the Comunica configuration;
this issue revolves around Rollup producing a bundle that contains unexpected require()s to modules that are not in the bundle because they're are not in quadstore-comunica's bundle, having been been flagged to Webpack as modules to ignore due to no. 2.

@retog: given that we publish our Comunica configuration to NPM already bundled, what you might want to try is to tell Rollup to use quadstore-comunica/index.js rather than the main file quadstore-comunica/index.bundle.js and install @comunica/actor-init-sparql as a dependency of your project. This should lead Rollup to ignore our pre-built Webpack bundle, which in the best case might give you a working bundle and in the worst case should lead you straight to reproducing comunica/comunica#656. You'll need to have a CommonJS loader plugin of some sorts, though, as our TS build currently targets CommonJS modules.

See also:

our webpack configuration;
our Comunica configuration

@rubensworks any ideas?

Just FYI, I'm feeling a little under the weather and I won't be as responsive over the next day or two. Please tag me again if the conversation continues and I don't come back to it within a few days.

Reto Gmür · Answer 6 · Wed Oct 21 2020 17:06:44 GMT+0800 (China Standard Time)

given that we publish our Comunica configuration to NPM already bundled, what you might want to try is to tell Rollup to use quadstore-comunica/index.js rather than the main file quadstore-comunica/index.bundle.js.

Thanks for your suggestion. Unfortunately, I don't know how to do this. I tried manually editing node_modules/quadstore-comunica/package.json to change the main file, but now I'm getting:

Uncaught ReferenceError: index_browser_1 is not defined

Ruben Taelman · Answer 7 · Wed Oct 21 2020 17:19:35 GMT+0800 (China Standard Time)

use quadstore-comunica/index.js rather than the main file quadstore-comunica/index.bundle.js.

As a non-user of Rollup, this looks like something that should fix the error, as Rollup seems to be using the Webpacked bundle, which is bound to cause problems.
Also not sure here how to configure that in Rollup.

In any case, I also suspect comunica/comunica#656 will pop up here again.

Jacopo Scazzosi · Answer 8 · Wed Oct 21 2020 21:10:00 GMT+0800 (China Standard Time)

As a non-user of Rollup, this looks like something that should fix the error, as Rollup seems to be using the Webpacked bundle, which is bound to cause problems.

Why, though? The bundle included in quadstore-comunica does work and I currently install it via NPM and require() it into quadstore like any other CommonJS module.

@retog, I've made a couple of little changes to partially test the above. Could you please try to use quadstore@7.0.1-alpha.11, which I've just published to NPM, and slightly change your Rollup configuration so as to add the following ignore parameter to the CommonJS plugin?

commonjs({
  ignore: id => id.startsWith('_webpack_ignored_')
}),

No need to change the package.json.

Reto Gmür · Answer 9 · Thu Oct 22 2020 01:11:24 GMT+0800 (China Standard Time)

Hi @jacoscaz, I've tried with the changes you suggested but I'm still getting:

Uncaught ReferenceError: require$$0$1 is not defined
   <anonymous> http://localhost:5000/bundle.js:27130

Jacopo Scazzosi · Answer 10 · Fri Oct 23 2020 14:53:13 GMT+0800 (China Standard Time)

Hi @retog! Just FYI, I'm testing with Rollup myself to see whether I can conjure a configuration that correctly bundles quadstore-comunica. It's taking longer than I had hoped but I hope I'll have something to share about this soon enough.

Reto Gmür · Answer 11 · Sat Oct 24 2020 03:58:51 GMT+0800 (China Standard Time)

I've tried using webpack, and getting similar errors:

$ npx webpack
[webpack-cli] Compilation finished
Hash: 7af3cd92dcc63946470a
Version: webpack 4.44.2
Time: 13741ms
Built at: 10/23/2020 9:47:04 PM
  Asset      Size  Chunks  Chunk Names
main.js  1.32 MiB       0  main
Entrypoint main = main.js
  [5] (webpack)/buildin/global.js 472 bytes {0} [built]
  [8] (webpack)/buildin/module.js 497 bytes {0} [built]
 [11] ./node_modules/events/events.js 13.8 KiB {0} [built]
 [19] ./node_modules/quadstore/dist/lib/types/index.js 963 bytes {0} [built]
 [36] ./node_modules/asynciterator/dist/asynciterator.js + 1 modules 64 KiB {0} [built]
      | ./node_modules/asynciterator/dist/asynciterator.js 63.3 KiB [built]
      | ./node_modules/asynciterator/dist/taskscheduler.js 682 bytes [built]
 [38] ./node_modules/quadstore/dist/lib/utils/index.js 5.39 KiB {0} [built]
 [63] ./node_modules/timers-browserify/main.js 1.97 KiB {0} [built]
 [64] ./node_modules/assert/assert.js 15.6 KiB {0} [built]
 [66] ./node_modules/quadstore/dist/lib/serialization/index.js 710 bytes {0} [built]
[133] ./node_modules/lodash/lodash.js 530 KiB {0} [built]
[135] ./src/index.js 250 bytes {0} [built]
[136] ./node_modules/quadstore/dist/index.js 303 bytes {0} [built]
[137] ./node_modules/quadstore/dist/lib/quadstore.js 11.3 KiB {0} [built]
[148] ./node_modules/levelup/lib/levelup.js 7.86 KiB {0} [built]
[174] ./node_modules/quadstore/dist/lib/get/index.js 6.95 KiB {0} [built]
    + 254 hidden modules

ERROR in ./node_modules/quadstore-comunica/index.bundle.js
Module not found: Error: Can't resolve '_webpack_ignored__comunica_actor_http_memento' in '/mnt/e/Documents/3rdparties/rollup-starter-app/node_modules/quadstore-comunica'
 @ ./node_modules/quadstore-comunica/index.bundle.js 2:418603-418659
 @ ./node_modules/quadstore/dist/lib/quadstore.js
 @ ./node_modules/quadstore/dist/index.js
 @ ./src/index.js

ERROR in ./node_modules/quadstore-comunica/index.bundle.js
Module not found: Error: Can't resolve '_webpack_ignored__comunica_actor_http_native' in '/mnt/e/Documents/3rdparties/rollup-starter-app/node_modules/quadstore-comunica'
 @ ./node_modules/quadstore-comunica/index.bundle.js 2:418693-418748
 @ ./node_modules/quadstore/dist/lib/quadstore.js
 @ ./node_modules/quadstore/dist/index.js
 @ ./src/index.js

ERROR in ./node_modules/quadstore-comunica/index.bundle.js
Module not found: Error: Can't resolve '_webpack_ignored__comunica_actor_query_operation_sparql_endpoint' in '/mnt/e/Documents/3rdparties/rollup-starter-app/node_modules/quadstore-comunica'
 @ ./node_modules/quadstore-comunica/index.bundle.js 2:418782-418857
 @ ./node_modules/quadstore/dist/lib/quadstore.js
 @ ./node_modules/quadstore/dist/index.js
 @ ./src/index.js

[...]

Jacopo Scazzosi · Answer 12 · Sat Oct 24 2020 04:00:46 GMT+0800 (China Standard Time)

Hello @retog. What a coincidence! I'm working on a reference repository for bundling quadstore with Webpack right now and will have that running within the next hour :)

Jacopo Scazzosi · Answer 13 · Sat Oct 24 2020 04:33:50 GMT+0800 (China Standard Time)

Hello again.

Let's start with the bad news first. I haven't been able to make quadstore (and quadstore-comunica in particular) work with Rollup. I did get a little further than the errors you reported but nowhere close to a working bundle, mostly because the commonjs plugin seems unable to pack the dependencies in the correct order.

That said, the good news: I've created a whole new repository that will act as a reference point for how to use Webpack with quadstore: https://github.com/beautifulinteractions/node-quadstore-webpack-bundle . It contains a prebuilt UMD bundle and the Webpack configuration that is used to built it.

Hopefully this will make your life easier - let me know how it goes!

Reto Gmür · Answer 14 · Sun Oct 25 2020 02:16:58 GMT+0800 (China Standard Time)

Hi @jacoscaz, that's a good news indeed! With your webpack configuration, it was easy to get it working. Thanks a lot!

Jacopo Scazzosi · Answer 15 · Sun Oct 25 2020 04:49:07 GMT+0800 (China Standard Time)

Happy to hear that @retog . I'll close this issue as we have no plans to support Rollup in the short term and we do provide an alternative for browser-side usage. By the way, I'd be happy to hear your use-case for quadstore if you have the time and the inclination to do so. We've recently got accepted in the GitHub Discussions beta: https://github.com/beautifulinteractions/node-quadstore/discussions