Giters

locutusjs / locutus

Bringing stdlibs of other programming languages to JavaScript for educational purposes

Home Page:https://locutus.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ES Modules?

niksy opened this issue · comments

commented

Is there any plan on converting from CommonJS format to ES Modules format?

There are benefits to using ES Modules for distribution code (e.g. tree-shaking of unused exports, scope hoisting, industry standard), and there are tools which can help in that.

commented

Hi there, the downside for the time being is that stable Node.js versions (haven't kept track so well but I don't think 8 or 10 have support for it?) don't support it. So anybody wanting to grab a function would then first have to use e.g. Rollup in order for it to function, which is not so common on servers yet as it is for webbrowsers. I do think that Rollup supported tree-shaking for commonjs requires or no?

commented

The process would definitely use Rollup to prepare CommonJS exports as default and ESM exports as optional.

For example, PHP number_format would look like this for CommonJS:

const numberFormat = require('locutus/php/strings/number_format');

And for ESM:

import numberFormat from 'locutus/esm/php/strings/number_format'

This way Locutus could bump minor version with ESM support as opposed to switching to ESM as default which would require major version bump.

You are correct in that Node 8 doesn’t support ESM, but plug-and-play support can be achieved with ES Module loader.

Does this make sense to you? I have some experience with this and I could probably prepare initial PR for this.

commented

Thanks! I realize that when folks install from npm we could ship a built/bundled version. In fact we already do so we can use es6 and not worry about older Node platforms. So I guess we could rewrite imports to requires at that stage also.

It's just that, currently, you'd be able to even copy paste a function in node, and that goes out the window until module supports hits a good majority of node.js installs (unless we ask everybody to install ES module loader with their apps, and hack the global require). I think on that trade-off, it's not worth it just yet to move to es6 modules. If we revisit in a year or so, and node supports import out of the box, I'd be very happy to change all requires in a heartbeat. When I'm balancing the pros and cons i don't think it's the right time for a project that also caters to the serverside (i'm very much in favor of imports for browser/bundle-only projects today already). Or what do you think?

commented

I'd love to have an ESM version. Of course our code fits the example that it's bundled with Webpack and runs in the browsers. So I understand the want/need to keep it compatible for native Node applications that can't take advantage of the ESM syntax. We're using all ESM syntax where possible, which is for the majority of our code.

I think having an ESM path in Locutus sounds good. With Webpack for example it can easily alias locutus/php/strings to locutus/esm/php/strings for imports.

Another option would be to create an ESM branch in Git and then publish the branch with Node's beta tag https://docs.npmjs.com/getting-started/using-tags Then the ESM branch wouldn't be installed unless the user specified @beta during the install.

The only downside I can think of this is that two files have to be managed instead of one, because you can't even merge the master branch into the ESM branch. If that's not a concern, though, then I don't see why this isn't a viable option. Then when Node natively supports it, you can merge the ESM branch into the master branch.

Interestingly, I don't know if it has to do with the types declaration from DefinitelyTyped or Webpack, but we're able to import with the ESM syntax.

commented
commented

Yes, I understand the part with the complexity that would come up with the project, but I think it could be flexible with the proper distribution method.

  • The source code is in the src/ directory and is never published to npm
  • The Rollup prepares two versions: CommonJS that goes into the root project (as it is right now) and the ES Modules version that goes into esm/ directory
  • Documentation and module consumation remains the same for all those who have used CommonJS approach, but we document that ES Modules are available under the esm/ "namespace".

Another option would be to create an ESM branch in Git and then publish the branch with Node's beta tag

I think this could create larger maintenance issues.

won’t roll up also treeshake requires?

Rollup has tree shaking and scope hoisting for ES Modules only, not for CommonJS.

And won’t you be able to include single functions, meaning treeshaking is less of a requirement?

True, but still ES Modules have their benefits over CommonJS, especially in browser landscape.

In the end, it’s your decisions and I’m glad you’re open to the option for using ES Modules by default!

commented

Thanks, I didn't know that about RollUp! I thought only Webpack had the constraint on ESmodules. And RollUp would treeshake whatever you threw at it. That doesn't change though that folks are requiring single functions and that treeshaking isn't so beneficial then.

True, but still ES Modules have their benefits over CommonJS, especially in browser landscape.

Can you eloborate on that, glancing over the article it mostly seems to address bundlesizes and performance when bundled via different bundlers? I don't see how requiring vs importing a single function would impact this.

Documentation and module consumation remains the same for all those who have used CommonJS approach, but we document that ES Modules are available under the esm/ "namespace".

Once we go all in with Locutus, we'll have to deprecate and move everybody's code back from using esm/, or keep pushing out clones under that namespace. Since I don't see overly compelling upsides for this particular project, this bit is rather unattractive to me. I'm also considering to have a module per function, like @locutus/php-strings-sprintf via Lerna.js. While perfectly possible, pushing out the extra namespace per module seems like it would increase chances-of-headaches and build-times.

commented

I don't see how requiring vs importing a single function would impact this.

Importing 10 CommonJS single functions results in 10 function calls for that modules exports. Importing 10 ES Modules single functions, with proper scope hoisting support (Rollup, Webpack and Parcel support that) results in flattened 1 function call. It’s probably negligible in the case of 10 modules, but it can show performance degradations in large codebases (the article goes in depth for this situation).

Once we go all in with Locutus, we'll have to deprecate and move everybody's code back from using esm/, or keep pushing out clones under that namespace.

True, this could be major problem for usage, but on the other hand it also means major version bump which at least expects of developers to revisit their codebase and make proper changes.

I'm also considering to have a module per function, like @locutus/php-strings-sprintf via Lerna.js.

Oh, this could be great! I suppose majority of Locutus users are actually using some, and not all functions in their codebase.

While perfectly possible, pushing out the extra namespace per module seems like it would increase chances-of-headaches and build-times.

Do you mean @locutus or esm/ namespace? For former, this makes sense since it’s ecosystem standard and I don’t see problems with that. For latter, esm/ in case of "one function per package" can (and should) be avoided with main and module property of package.json so you can have both distribution files in the package root directory.

commented

For latter, esm/ in case of "one function per package" can (and should) be avoided with main and module property of package.json so you can have both distribution files in the package root directory.

That sounds very reasonable and would solve a big part of my concerns. Would you like to take a swing at that? So every function is a module with a main and module entry point?

commented

Do you have some baseline we can work with?

Is the plan to have project structure similar to the one Babel uses, so PHP’s number_format is actually inside packages/php-strings-number_format (or packages/locutus-php-strings-number_format) and scoped to @locutus/php-strings-number_format?

commented
commented

I think @locutus/<language>-<module> sounds good. I don't see any benefit from having it be more verbose with the path in the name.

commented
commented

When you put it that way, then I guess it makes sense.

commented

As for internal directory structure, do you think we can keep the nesting category?

I suppose we’re going the way of Lerna, so first step would be to see if Lerna has some option to automatically create distribution directories for packages. Keeping files in the current directory state clashes with the functionality of package.json, so if we don’t have proper relocation process in place, directory structure should be packages/php/strings/number_format/index.js and inside that directory package.json for that module at least, test infrastructure at most (see Babel structure).

I wonder if in the case of php we should make an exception and rip out the middle part so that it becomes php-sprintf for example

I wouldn’t go that way. This could introduce a lot of edge cases and confusing moments. @locutus/php-array-array_reverse should be okay :)

commented

Ok, I'm cool with packages/php/strings/number_format/index.js and consistently using the class in module names @locutus/php-array-array_reverse. Tests are generated and extracted from header comments, I think we should probably not output the generated mocha tests inside the packages/php/strings/number_format/ to avoid people from hacking on those generated files. Likely this also makes/keeps the tests faster.

Another approach could maybe also be to regard Lerna packages as build residue. e.g. we'd keep everything as is, but as part of the build step, which already does AST parsing and crazy things, it would populate a .lerna/packages directory with index.js files and package.jsons were needed , just so we can publish individual modules to npm, but we don't let that decision change how we currently work on/automate our functions.

commented

Wondering if @kukawski has any strong opinions or insights on this matter?

commented

@kvz I find it very nice to have each function as separate module published under a namespace and maintained from single repository.
Regarding our test setup that relies on comments in the source files - sometimes I found it quite limiting, because I would like to test a function thoroughly, which would end up with infinitely long comment. Maybe we should consider writing test files manually?
Of course there are things to consider or double-check for feasibility - like the folder structure in the repository.
Currently, IIRC, user can require whole category of functions, because we generate index.js. Do we want to keep it?
How do we deal with shared code? Where do we keep it?
How do we deal with module dependecies? Some functions rely on other functions in the locutus package.
Then, how do we deal with existing users, if suddenly the way how to use the module changes?
Can we change the locutus module to be a namespace in npm? I've never checked that.

commented

Regarding our test setup that relies on comments in the source files - sometimes I found it quite limiting, because I would like to test a function thoroughly, which would end up with infinitely long comment. Maybe we should consider writing test files manually?

For starters, maybe use what currently exists and then switch to manual writing, push MVP and then iterate?

Currently, IIRC, user can require whole category of functions, because we generate index.js. Do we want to keep it?

I think this could be possible, maybe event with Lerna as main tool.

How do we deal with shared code? Where do we keep it?
How do we deal with module dependecies? Some functions rely on other functions in the locutus package.

If we go the way of Lerna, we should probably check what other projects do in regards to that (e.g. Babel).

Then, how do we deal with existing users, if suddenly the way how to use the module changes?
Can we change the locutus module to be a namespace in npm?

npm has deprecation process for existing packages. If we publish current implementation (everyting in index.js) under @locutus/locutus, I suppose majority of users will have easier transition period (I think npm offers solution where locutus could be aliased to @locutus/locutus). For everything else (single function) users should use namespaced modules. This should probably be major version bump. What do you think?

commented

With regards to Babel 7+ they use a namespace structure

{
"devDependencies": {
	"@babel/core": "7.1.0",
	"@babel/plugin-proposal-class-properties": "7.1.0",
	"@babel/plugin-proposal-export-namespace-from": "7.0.0",
	"@babel/plugin-proposal-object-rest-spread": "7.0.0",
	"@babel/plugin-syntax-dynamic-import": "^7.0.0",
	"@babel/plugin-transform-modules-commonjs": "7.1.0",
	"@babel/preset-env": "^7.1.0",
}

Instead of every function being it's own package, would it be better to have them grouped by the language that they emulate?

commented

One more case of popular utility library switching to ESM and sharing their process: https://blog.date-fns.org/post/ecmascript-modules-in-date-fns-v2-i1v007ou81rm

commented

Have there been any thoughts about this since October?

One thing that could be done is to write it in ES6 first, and then use Babel to transpile it to CommonJS for distribution on npm. That way up front there's only one code base to work in, and then production distribution would have ESM syntax and CommonJS.

commented

We can already write in ES6 as we do indeed also already transpile to ES5 so that could be a viable option yes. I don't have the time for it, but if anybody feels strongly enough to implement this PRs are welcome

commented

So does this mean I can't use import?

commented

If you set up Babel in your project you can.