Investigate supporting ES6/JSX
nzakas opened this issue · comments
After many requests and some soul-searching, I've come around to believing that ESLint should optionally support JSX. My rationale is based on:
- The growing popularity of React/JSX
- The existence of Esprima-FB and estraverse-FB
There is still the issue of escope not supporting es6 scoping rules and relying on estraverse directly, but that may be solvable.
The way I'm envisioning this working is by allowing you to specify one of four JavaScript modes:
- ECMAScript 3
- ECMAScript 5
- ECMAScript 6
- JSX
That means ECMAScript 6 support is separate from JSX support. You opt-in to JSX, you get whatever is in Esprima-FB vs. opting in to ES6, which should not support JSX syntax.
This issue is to gather investigation data to determine how this can be achieved and what blockers might exist.
ES4? I hope you meant ES5.
Yikes, yes.
ECMAScript 6 is a superset of ECMAScript 5, which is a superset of ECMAScript 3. I see only two options: ES6 and JSX.
Please note that we're committed to JSX and in order to make sure that the integration with tooling such as eslint is as easy as possible we created a specification for JSX.
- JSX Specification: http://facebook.github.io/jsx/
- JSX AST Specification: https://github.com/facebook/jsx/blob/master/AST.md
@vjeux good to know! The spec is useful, but not as useful as libraries that we can plug in and have "just work". :) Have you (all) done any work around creating an escope that's compatible with JSX?
@nzakas: It's worth noting that esprima-fb is the esprima harmony branch + JSX (it tracks the upstream harmony branch, we periodically merge in updates from upstream, and we always send harmony-specific PRs to upstream first) -- so if you just always use esprima-fb for ES6, you're supporting both ES6 and JSX. If you were to provide a toggle between esprima-fb and esprima/harmony, the only distinction you'd see (barring bugs) is that esprima/harmony would give a parse error if you threw JSX at it.
On the other hand, you still have the option of rejecting JSX in code if you so desire -- even if the parser parses it (if you see the presence of a JSX node while traversing the syntax tree, you error).
Would it make sense to simply provide a no-JSX lint than a toggle on which parser to use?
The only issue using esprima-fb (or the esprima harmony branch) applied on ES5 JS code is the following:
lib/rules/no-extra-parens.js
43:12 error Expected a "break" statement before "case" no-fallthrough
53:12 error Expected a "break" statement before "case" no-fallthrough
The harmony branch has the fallthrough comment at a different position in the AST. Shouldn't be hard to get the lint rule accept both positions.
In order to lint ES6 or JSX code, there are a few rules that need to be updated. For example the unused variable rule doesn't consider to be a 'use' and marks it as unused.
We're currently playing around switching from jslint after pre-processing to eslint. So far we've got something "working" by disabling a few rules. It doesn't seem that it would require drastic changes and only a handful of pull requests would be needed to support it properly.
In order to lint ES6 or JSX code, there are a few rules that need to be updated. For example the unused variable rule doesn't consider to be a 'use' and marks it as unused.
This is tricky because in React's JSX, writing <Button> counts as a use of the variable Button but since JSX itself doesn't specify semantics, that might not be true with a different transformer.
The harmony branch has the fallthrough comment at a different position in the AST
This sounds like a bug we should just get fixed -- not quite sure why they differ
For example the unused variable rule doesn't consider to be a 'use' and marks it as unused.
This would be a React-specific lint rule. JSX is just syntax -- React applies meaning to that syntax; So given that the lint-rule would be react-specific anyway, specifying the react-specific contextually-local variables in the lint rule makes sense.
@jeffmo ah, that's very helpful information to know! I didn't realize the relationship between esprima and esprima-fb was so well-defined. Your point about erroring on JSX syntax is valid, too. It's kind of nice not to have to do anything and have the parser blow up, but in lieu of that, a hand-crafted approach that just looks for JSX and reports an error is definitely feasible.
A followup question: is ES5 + JSX a valid use case for anyone? I was under the (perhaps mistaken) impression that ES6 was a requirement. If that's not the case, then it would be better to have a JSX opt-in in addition to specifying which ES version you want.
@vjeux that's great. Feel free to file issues for anything that would have to be tidied up to work with esprima-fb. (We require issues for all code-based PRs, so best to get started early. :) )
@jeffmo @vjeux also agree that the comment being in a different place seems like a bug, can one of you follow up on that?
@nzakas: I don't think there's anything that says React+JSX strictly requires ES6 today, but it's likely to start building on more and more newer ES concepts (classes, spread operators, arrows, etc).
On the other hand (as @michaelficarra mentioned), in the same way that esprima-fb is a superset of esprima/harmony, ES6 is a superset of ES5 -- so I'd imagine you could apply the same regressive thoughts around ES5+JSX just working off of the esprima-fb parser (unless you're just relying on the parser to blow up if it hits ES6). Though even if you did want such a parser (ES5+JSX, but no ES6), we don't have any plans to build or support such a parser on our end.
On the comment-node issue: I'll try to take a look tomorrow
Regarding the problem with the comment location, looks like eslint uses esprima 1.2.2. On esprima master (2.0.0-dev) the output differs from esprima 1.2.2. I used following source to test
switch(foo) { case 0: a(); /* falls through */ case 1: b(); }
esprima-fb has the same output as esprima 2.0.0-dev
@frantic yup, that's exactly the problem. There's no reason for the comment location to differ between the two branches.
I took a look at dropping-in esprima-fb and estraverse-fb and basically found two issues:
- The location of comments isn't just different for the
casestatement mentioned previously, it's also different in the treating of top-level comments. This basically looks howestraverse.attachComments()deals with comments rather than the way esprima 1.2.x deals with comments, which makes many rules dealing with comments break in those cases. - esprima-fb is a bit slower than esprima 1.2.x. I'm not sure if that's because the enhancements from 1.2.x didn't land in the Harmony branch, but it's a non-trivial performance reduction. You can try yourself running
npm run perf.
Here's with esprima 1.2.2:
CPU Speed is 3093 with multiplier 7500000
Performance Run #1: 1719.9273899999998ms
Performance Run #2: 1716.118489ms
Performance Run #3: 1708.291122ms
Performance Run #4: 1688.523239ms
Performance Run #5: 1706.549722ms
Performance budget ok: 1708.291122ms (limit: 2424.8302618816683ms)
Here's with esprima-fb:
CPU Speed is 3093 with multiplier 7500000
Performance Run #1: 1893.5033819999999ms
Performance Run #2: 1884.1219959999999ms
Performance Run #3: 1904.216641ms
Performance Run #4: 1897.484768ms
Performance Run #5: 1899.810498ms
Performance budget ok: 1897.484768ms (limit: 2424.8302618816683ms)
I started some investigative work around this on the esprimafb branch. Not anywhere near complete, but gives a decent idea of what I'm thinking.
Filed a ticket with Esprima about the comment attachment problem: https://code.google.com/p/esprima/issues/detail?id=607
I was playing around with this today - the only tests that are failing after rebasing onto eslint master is valid-jsdoc, as the comment location is different. It seems like every other rule now works.
Is that related to the change I made regarding the new lines?
On 26 Oct 2014 03:06, "Nicholas C. Zakas" notifications@github.com wrote:
I was playing around with this today - the only tests that are failing
after rebasing onto eslint master is valid-jsdoc, as the comment location
is different. It seems like every other rule now works.—
Reply to this email directly or view it on GitHub
#1291 (comment).
I'm not sure but my guess is no. Most of the errors related to nodes and where they appeared.
Just sounded the same symptoms I saw for things. Nodes were on different
lines.
I just fixed some linting errors about no-fallthrough in
"no-extra-parens.js" by splitting out the cases into functions do you want
me to submit a pull request for the esprimafb branch?
I couldn't see the errors you were mentioning did you use a particular
config?
On 26 October 2014 17:10, Nicholas C. Zakas notifications@github.com
wrote:
I'm not sure but my guess is no. Most of the errors related to nodes and
where they appeared.—
Reply to this email directly or view it on GitHub
#1291 (comment).
I don't want to change anything unless Esprima says they won't fix the comment attachment. We should just be able to drop in EsprimaFB and have everything work. If we start changing things to hide those differences, we are just applying a bandaid without fixing the root cause.
I just ran npm test to see the valid-jsdoc errors with esprima fb.
@nzakas totally agree with that I was just inquiring to see if that were the cause and perhaps we could justify it with the Esprima team that way.
I had not realised you didn't push the rebase so after doing that I see none of the errors you mentioned or the previous ones I saw either... very odd:
> eslint@0.9.1 test /home/jonathan/eslint
> node Makefile.js test
Validating Makefile.js
Validating JSON Files
Validating JavaScript files
Validating JavaScript test files
Validating rules
Missing link to documentation for rule valid-syntax in index
Missing default setting for valid-syntax in eslint.json
npm ERR! Test failed. See above for more details.
npm ERR! not ok code 0
That because the build is failing while linting and isn't going on to run the unit tests.
(drooling) this would be so awesome
The difference is in the function processComment(node) { function in Esprima, replacing it with the 1.2 version makes the UTs pass, I will investigate some more. My guess is it's either this checkin or this one
I've made a PR to Esprima which changes some of the comment code back to the way it was in the 1.2 branch. This change allows the UTs to pass when using esprima-fb.
Sweet, thanks @briandipalma.
Hello,
First, thanks for all your efforts to add JSX support into ESLint, we're really looking forward for this feature to integrate ESLint into our workflow and be able to validate seamlessly our JS and JSX codebase.
But while testing the esprimafb branch I noticed a couple of issues (aside the comment attachment problem):
- ESLint only validate the .js files (lib/cli-engine.js#L220 and lib/cli-engine.js#L230). I think it should now also validate .jsx files. There is 2 ways to do this:
- Adding .jsx to the hardcoded list of allowed extensions with .js
- Adding a new option to allow the user to specify the extensions to validate
- the no-unused-vars rule does not find the variables used in the JSX code. So some used variables are marked as unused;
Example:
var HelloMessage = React.createClass({
render: function() {
var world = 'World';
return <div>Hello {world}</div>;
}
});
React.render(<HelloMessage />, document.body);$ eslint example.jsx
example.jsx
1:4 error HelloMessage is defined but never used no-unused-vars
3:8 error world is defined but never used no-unused-vars
✖ 2 problemsThe @andreypopp eslint-jsx fork include a fixed no-unused-vars.js rule but based on the rule before this refactoring and despite my efforts I was not able to add the fix to the refactored one.
Yannick is correct.
@yannickcr While you wait, you can throw --tolerant to get it to scan every file type.
@yannickcr thanks for the feedback. We are aware that what's on the branch right now doesn't work correctly and we are waiting for fixes in a few dependencies before moving forward. Es6/JSX support will come incrementally, so we likely won't fix all issues in one release.
@nzakas Are there ways that outsiders can help?
The best way to help is by helping with our dependencies. The Esprima comment location bug is one example. We also will need a ES6/JSX compatible version of escope in order to get all the rules working correctly, so helping in such an effort would be huge.
@nzakas Is Esprima still necessary for JSX support? It looks like React has stopped using it facebook/react@bf0a857
jstransform depends on esprima: https://github.com/facebook/jstransform/blob/master/package.json#L24
@maxhoffmann Still, eslint would not need to integrate esprima-fb directly. It does look like the bits of esprima JSTransform replaces have the exact same API, but to complicate things further, the JSTransform README recommends to consider https://github.com/benjamn/recast instead.
Okay, let's bring this conversation back to the topic at hand.
We can drop-in esprima-fb and estraverse-fb and have things mostly work. The things that don't work right now are comment location (due to Esprima changing where those are located in the Harmony version) and anything related to scoping of variables (because that's determined by escope).
So the question for those who are looking for ES6/JSX support: is it okay to support the syntax but not necessarily all the rules?
So the question for those who are looking for ES6/JSX support: is it okay to support the syntax but not necessarily all the rules?
Definitely! It's better to have partial support than none at all.
So the question for those who are looking for ES6/JSX support: is it okay to support the syntax but not necessarily all the rules?
For what it's worth, the only thing that's holding my team from jumping into ES6 is not having ESLint available.
Yes partial support is very wanted!
On Nov 11, 2014, at 5:54 PM, J. Roberto Vidal notifications@github.com wrote:
So the question for those who are looking for ES6/JSX support: is it okay to support the syntax but not necessarily all the rules?
For what it's worth, the only thing that's holding my team from jumping into ES6 is not having ESLint available.
—
Reply to this email directly or view it on GitHub.
If we can have a flag to enable support so that other users aren't affected while we wait on other projects to be updated that seems reasonable.
Yeah, that's the plan: have a command line flag that allows opting in to ES6 and one for opting in to JSX.
Partial or beta ES6 and JSX support would be wonderful
@nzakas: they can't be independent though, right? If you want JSX, you will get ES6 whether you like it or not.
Sure they can, we just report an error if a JSX node is found and you didn't expliticitly say you wanted it.
@nzakas: @michaelficarra described the opposite scenario.
@mzgol can you explain more? I'm not following.
@nzakas @michaelficarra described a scenario where you'd want JSX support without wanting ES6 support. How would you prevent one using ES6 constructs when they opt into JSX?
Same approach - report an error if an ES6 node is detected outside of ES6 mode.
What is an "ES6 node", though? The list of not only new constructs but also extensions of current ones in ES6 is rather long; I expect a lot of plumbing needed to be able to detect such things.
It's much easier to detect JSX considering it's just one additional construct.
What is the benefit of having a mode for the linter core that doesn't understand ES6? If the linter core always understood es6, but you wanted to restrict (either certain or all) ES6 features, it seems like you could build a lint to error upon encountering them, right?
If eslint had a "non-es6" mode that actually switched off to some other parser, at best you'd get a cryptic parser error that says "syntax error" (rather than something more useful like "sorry, we don't support es6 syntax like class expressions in this project")
The benefit is that those who don't want to use ES6 don't have to. ES6 introduces a bunch of new syntax, which means it will break in any ES5 or earlier environment. We can't assume everyone wants ES6 enabled by default. So, we need to broke a mode that is compatible with what ESLint does right now. The benefit is we can give better error messages than an ES5 parser would.
The benefit is that those who don't want to use ES6 don't have to.
I definitely agree that those who don't want to use it shouldn't have to, but that remains to be the case even with a core library that understands it. Could the means of not wanting ES6 and/or JSX (whether by default or by opt-in) just be to have lints that clearly communicate that desire?
It's also worth mentioning that ES7 is coming along as well, and that's going to be even more rapid and iterative. TC39 is moving to a train model for shipping future versions of the spec post-ES6, so having ES5/ES6/ES7/...ESN modes might get a bit tough.
Anyway -- do what you think is right :) Just wanted to point out that two modes won't be the extent of all options for too long; Having some lints (maybe shipped with eslint and on by default?) for these kinds of modes might be a good way to keep them extensible and fluid.
It's also worth mentioning that esprima-fb already has support for some ES7-targeted syntax features like async/await syntax and object-spread. As new syntax rolls in for ES7, it's likely to get added to esprima-fb as people want them as well.
My plan is to switch to esprima-fb and build a graph that maps node types to ECMAScript versions so that we'll be able to easily flag unwanted nodes. Whether that's done in a rule or in the core is still TBD.
Good to know re: ES7 pieces. Are there SpiderMonkey node types for those already?
Nothing official for anything ES7 AFAIK, but that goes for quite a bit of ES6 syntax as well.
I think the spidermonkey spec has tried to reflect commonly implemented parser node structures (to the extent that they make sense) in the past, but I'm probably not the best authority to say how that'll work for sure.
I think I'm going to investigate using acorn-jsx instead of esprima-fb. I'm not getting anywhere with the issue I filled for esprima and I'm getting a bit tired of having such an unresponsive dependency. Acorn also has ECMAScript version support so we don't need to worry about implementing it. I'm going to spend a little time evaluating how feasible this is.
So happy! Thank you for your hard work @nzakas!
Instead of supporting JSX, couldn't we recommend that people use something like react-hyperscript instead?
https://github.com/mlmorg/react-hyperscript
I'm honestly amazed that others working with react prefer JSX and don't see the value in doing all this stuff in pure JS. It seems a bit tedious at first until you realize that it's so much easier to encapsulate stuff and keep things dry when everything is pure javascript. The default react stuff without JSX is verbose, but that doesn't mean it has to be verbose, which is why react-hyperscript was written.
JSX breaks an abstraction boundary for a little perceived convenience specific to one and only one framework and JSX has become a cost center that a lot of other library/tool writers now need to deal with (not just eslint). I can't help but get the impression that JSX support is taking development effort away from more important things like fixing bug #1482. There are plenty of bugs to be fixed and JSX increases complexity and creates more room for bugs.
Now that 0.12 has createFactory and you can no longer really mix javascript with JSX, I can't help but think that this won't be immensely simpler if JSX code is simply transformed to replace the JSX parts with it's javascript equivalent before feeding the code to eslint and then correcting the line numbers of the eslint output to account for the difference between the JSX parts and the JSX equivalent parts.
This is your project @nzakas and you can do as you see fit (you may be a react user yourself), but I'm casting one anti-vote against JSX support since it's starting to feel like its hijacking eslint to the detriment of all other eslint users.
Be wary with acorn-jsx vs esprima-fb . You dont want to break the eslint interface causing third party eslint lint rules to stop working.
If acorn & esprima are identical at the AST level and everything else that is important for third party linters then it should be no problem, however making a breaking change to the AST or the linter interface for JSX support is incredibly expensive.
@malandrew thanks for chiming in. This decision has already been made based on many requests from ESLint users. It will be opt-in, so it won't affect your code unless you want it to.
@Raynos yup, I'm keenly aware of the compatibility issues and will not do anything that will cause existing rules to break. Acorn produces the same AST as esprima, so this shouldn't be an issue. If it is, we have over 1,000 unit tests for ESLint that should warn us.
Update on Acorn: The tokens have a different format, which means we'd have to munge them into the same format as Esprima in order to get everything working. That's a non-starter for this effort. So we'll go with Esprima-FB for now and look at possibly switching to Acorn post-1.0.0.
Any update on this? I'm seeing commit under the yannickcr/eslint#deezer fork, but can't make it works (I've tried the "valid-syntax" option with jsx to true :/)
What kind of an update are you looking for?
There's no valid-syntax rule. See http://eslint.org/blog/2014/11/es6-jsx-support/ for how to configure correctly and what's going on.
I just found a way (not sure why) to force jsx
{
"settings": { "jsx": true }, // had to add this
"valid-syntax": [2, { "jsx": true }],
}Well not sure where I found that haha. Sorry for this
The other part was to do that
$ eslint . --ext=.js --ext=.jsxBut thanks now it's working correctly! Awesome work.
Update: after much deliberation, I've decided the best way to move forward is to fork Esprima and make the changes we need. We have been held back by Esprima bugs for too long. So here is our brand new adding Esprima fork called Espree: https://github.com/eslint/espree
More details on the readme, but basically I'm starting with esprima 1.2.2 and adding in ES6 and JSX features. 1.2.2 is the version that ESLint runs on today, so it provides the best compatibility for us going forward.
Unlike Esprima, Espree will support JSX as an option directly. @vjeux if Facebook folks are willing to help add JSX support to Espree, I'll happily accept the contribution.
For ES6, it's just a matter of pulling in features one by one from the Esprima harmony branch. I've already added a way to specify the ECMAScript version to use, restricted let and const to ES6 more, and added support for the regular expression u and y flags. Overall progress is tracked here: eslint/espree#10
It's disappointing to need to create a new parser, but JSHint and JSLint already have their own parsers, so this brings us inline and allows us to move much faster.
Could you use Facebook's fork of esprima, which already has full support for JSX, instead of forking? I don't have a lot of context here, so forgive my ignorance.
That's what we've been trying to do, but Esprima-FB is based on the Esprima harmony branch, which is where all the problems lie. It's just not worth continuing down that path.
@nzakas Are you sure you're not going with Acorn? Acorn would give you full es6 support, better parsing speed (especially with location info enabled), and error recovery. I'm currently considering changing from jshint to eslint on c9.io, but particularly that lack of error recovery is pretty obvious when all warnings disappear whenever I'm typing and there's a parse error :( (Also, if you would switch to Acorn I would get the benefit of using just one parser for all javascript analysis rather than having two of them ^_^;; But then I might run into #1025, I guess.)
I'd love to use acorn, but it has several incompatibilities with esprima that would require changes to almost every rule we have (not to mention it would break custom rules that people have written).
An alternative is to punt on jsx entirely and use a library like https://github.com/alexmingoia/jsx-transform/ to convert the jsx to hyperscript and then run that file through the linter. The one complexity that remains is making sure that the jsx conversion produces hyperscript code that follows the same rules as your eslint configuration.
@lennartcl The esprima API does allow you to pass in an optional tolerant flag on parse that when true makes parse return "an extra array containing all errors found, attempts to continue parsing when an error is encountered". See http://esprima.org/doc/index.html
Is the esprima tolerant mode sufficient for your needs in c9.io?
@nzakas if you've forked esprima and made bugfixes, why not send a PR back upstream as the esprima-fb developers do, instead of renaming your esprima fork to espree?
@michaelficarra I'm pretty sure the answer (as is stated in the thread) is that the Harmony branch is pretty incompatible with main branch used by ESLint and it's not fair to ESLint rule makers out there to change the AST at this point.
See also: eslint/espree#12
I'd like to see a list of those incompatibilities. They're probably things that esprima would care to fix.
@michaelficarra I've forked off of 1.2.2, which is pretty different than 2.0.0, so it's not going to be easy to reconcile the changes. I've also been filing bugs on the Esprima bug tracker and getting no updates and no indication if a fix is forthcoming. It's not fair to keep holding ESLint back waiting for changes that may or may not happen.
In any event, and to be clear to everyone: this decision has been made. I've been trying to work with Esprima for a while now and have reached the point where I don't see using Esprima as a benefit any longer. ESLint needs to move forward and the best way to do that is to control the parser we're using. Hence Espree.
@nzakas are these outstanding issues eslint has with esprima all related to getting JSX support in eslint or are there unresolved issues with esprima with vanilla javascript linting as well?
@malandrew the Esprima issues have nothing to do with JSX support. The fact that esprima-fb is based on the Esprima harmony branch means any issues from Esprima leak through. That's the problem (more details throughout this thread).
@malandrew I experimented with the "tolerant" option. But for most things I throw at it, it just fails with an exception anyway. It might be nice for eslint to at least try enable that option by default though, if only to show multiple parse errors at the same time.
If you're interested in discussing the tolerant option, please open a separate issue. That topic is orthogonal to this one.
Closing as 0.12.0 introduced support for JSX and partial ES6 support.
🏆