JSX Syntax highlighting broken?

Question

JSX Syntax highlighting broken?

cozos opened this issue 8 years ago · comments

Hey,

I was redirected here by support@github.com - it seems that JSX Highlighting is broken on Github.

For example: https://gist.github.com/cozos/d6636a93cba29c77e3221f2ca30212b0

import React from 'react';

var ExampleApplication = React.createClass({
  render: function() {
    var elapsed = Math.round(this.props.elapsed  / 100);
    var seconds = elapsed / 10 + (elapsed % 10 ? '' : '.0' );
    var message =
      'React has been successfully running for ' + seconds + ' seconds.';

    return <p>{message}</p>;
  }
});

var start = new Date().getTime();

setInterval(function() {
  ReactDOM.render(
    <ExampleApplication elapsed={new Date().getTime() - start} />,
    document.getElementById('container')
  );
}, 50);

If it helps, I'm on the latest build of Google Chrome.

Arwin Tio commented 8 years ago

👌

Arwin Tio commented 8 years ago

#3178

Paul Chaignon · Answer 1 · Wed Jun 08 2016 05:16:30 GMT+0800 (China Standard Time)

Linguist only selects the grammar to use for highlighting. For JSX, GitHub uses gandm/language-babel. You should open an issue there if there isn't already one. Sorry for the two redirects :S

Arfon Smith · Answer 2 · Wed Jun 08 2016 11:07:56 GMT+0800 (China Standard Time)

Closing as this is an upstream issue.

Graham · Answer 3 · Wed Jun 08 2016 12:30:48 GMT+0800 (China Standard Time)

@arfon @pchaigno Author of language-babel here. I've just checked my grammar using the version on linguist - 2.25.1 - inside Atom and it appears to work so I don't know what is going on.

e.g.

but on github

/***
  XMLHttpRequest example
 */


import 'js/web.jsx';
import 'console.jsx';

class _Main {
    static function main(args : string[]) : void {
        var xhr = new XMLHttpRequest();
        var path = dom.document.location.pathname;
        xhr.open("GET", path.replace(/\/[^\/]*$/, "") + "/hello.txt");
        xhr.onreadystatechange = function (e) {
            if (xhr.readyState == xhr.DONE) {
                _Main.update(xhr.responseText);
            }
        };
        xhr.onerror = function (e) {
            console.error("XHR error");
        };
        xhr.send();
    }

    static function update(text : string) : void {
        var output = dom.id("output");
        var textNode = dom.document.createTextNode(text);
        output.appendChild(textNode);
    }
}


// vim: set expandtab:

John Gardner · Answer 4 · Wed Jun 08 2016 16:28:01 GMT+0800 (China Standard Time)

@gandm Not 100% sure if this is the issue, but your end pattern for #literal-module-import has an unbalanced closing bracket at the end. That's the pattern that's being matched at the first line.

Also, I couldn't help noticing you've escaped semicolons in your expressions (\\;). Any particular reason why?

Graham · Answer 5 · Wed Jun 08 2016 16:36:01 GMT+0800 (China Standard Time)

@Alhadis The end pattern looks here fine to me. I don't see any unbalanced parens.

No reason for escaping the semi colons.

John Gardner · Answer 6 · Wed Jun 08 2016 16:38:39 GMT+0800 (China Standard Time)

Look a bit closer:

Graham · Answer 7 · Wed Jun 08 2016 16:45:01 GMT+0800 (China Standard Time)

I just ran it through RegexBuddy and it's OK.

Graham · Answer 8 · Wed Jun 08 2016 16:46:49 GMT+0800 (China Standard Time)

\s_(?:(?:(\bfrom\b)?+\s++(('|")([^\"']_)(\k<-3>)))|(?=;|^\s*\b(if|switch|try|var|let|const|static|function|return|class|do|for

\s*(?:(?:(\bfrom\b)?+\s++(('|\")([^\"']*)(\k<-3>)))|(?=\;|^\s*\b(if|switch|try|var|let|const|static|function|return|class|do|for|while|debugger|export|import|yield|type|declare|interface)\b|\)|}))

Options: Case sensitive; Exact spacing; Dot doesn’t match line breaks

Match a single character that is a “whitespace character” (ASCII space, tab, line feed, carriage return, vertical tab, form feed) \s*
- Between zero and unlimited times, as many times as possible, giving back as needed (greedy) *
Match the regular expression below (?:(?:(\bfrom\b)?+\s++(('|\")([^\"']*)(\k<-3>)))|(?=\;|^\s*\b(if|switch|try|var|let|const|static|function|return|class|do|for|while|debugger|export|import|yield|type|declare|interface)\b|\)|}))
- Match this alternative (attempting the next alternative only if this one fails) (?:(\bfrom\b)?+\s++(('|\")([^\"']*)(\k<-3>)))
  - Match the regular expression below (?:(\bfrom\b)?+\s++(('|\")([^\"']*)(\k<-3>)))
    - Match the regex below and capture its match into backreference number 1 (\bfrom\b)?+
      - Between zero and one times, as many times as possible, without giving back (possessive) ?+
      - Assert position at a word boundary (position preceded or followed—but not both—by a Unicode letter, digit, or underscore) \b
      - Match the character string “from” literally (case sensitive) from
      - Assert position at a word boundary (position preceded or followed—but not both—by a Unicode letter, digit, or underscore) \b
    - Match a single character that is a “whitespace character” (ASCII space, tab, line feed, carriage return, vertical tab, form feed) \s++
      - Between one and unlimited times, as many times as possible, without giving back (possessive) ++
    - Match the regex below and capture its match into backreference number 2 (('|\")([^\"']*)(\k<-3>))
      - Match the regex below and capture its match into backreference number 3 ('|\")
        
        Match this alternative (attempting the next alternative only if this one fails) '
        
        Match the character “'” literally '
        
        Or match this alternative (the entire group fails if this one fails to match) \"
        
        Match the character “"” literally \"
      - Match the regex below and capture its match into backreference number 4 ([^\"']*)
        
        Match any single character NOT present in the list below [^\"']*
        
        Between zero and unlimited times, as many times as possible, giving back as needed (greedy) *
        
        The literal character “"” \"
        
        The literal character “'” '
      - Match the regex below and capture its match into backreference number 5 (\k<-3>)
        
        Match the same text that was most recently matched by capturing group number 3 (case sensitive; fail if the group did not participate in the match so far) \k<-3>
- Or match this alternative (the entire group fails if this one fails to match) (?=\;|^\s*\b(if|switch|try|var|let|const|static|function|return|class|do|for|while|debugger|export|import|yield|type|declare|interface)\b|\)|})
  - Assert that the regex below can be matched, starting at this position (positive lookahead) (?=\;|^\s*\b(if|switch|try|var|let|const|static|function|return|class|do|for|while|debugger|export|import|yield|type|declare|interface)\b|\)|})
    - Match this alternative (attempting the next alternative only if this one fails) \;
      - Match the character “;” literally \;
    - Or match this alternative (attempting the next alternative only if this one fails) ^\s*\b(if|switch|try|var|let|const|static|function|return|class|do|for|while|debugger|export|import|yield|type|declare|interface)\b
      - Assert position at the beginning of a line (at beginning of the string or after a line break character) (line feed) ^
      - Match a single character that is a “whitespace character” (ASCII space, tab, line feed, carriage return, vertical tab, form feed) \s*
        
        Between zero and unlimited times, as many times as possible, giving back as needed (greedy) *
      - Assert position at a word boundary (position preceded or followed—but not both—by a Unicode letter, digit, or underscore) \b
      - Match the regex below and capture its match into backreference number 6 (if|switch|try|var|let|const|static|function|return|class|do|for|while|debugger|export|import|yield|type|declare|interface)
        
        Match this alternative (attempting the next alternative only if this one fails) if
        
        Match the character string “if” literally (case sensitive) if
        
        Or match this alternative (attempting the next alternative only if this one fails) switch
        
        Match the character string “switch” literally (case sensitive) switch
        
        Or match this alternative (attempting the next alternative only if this one fails) try
        
        Match the character string “try” literally (case sensitive) try
        
        Or match this alternative (attempting the next alternative only if this one fails) var
        
        Match the character string “var” literally (case sensitive) var
        
        Or match this alternative (attempting the next alternative only if this one fails) let
        
        Match the character string “let” literally (case sensitive) let
        
        Or match this alternative (attempting the next alternative only if this one fails) const
        
        Match the character string “const” literally (case sensitive) const
        
        Or match this alternative (attempting the next alternative only if this one fails) static
        
        Match the character string “static” literally (case sensitive) static
        
        Or match this alternative (attempting the next alternative only if this one fails) function
        
        Match the character string “function” literally (case sensitive) function
        
        Or match this alternative (attempting the next alternative only if this one fails) return
        
        Match the character string “return” literally (case sensitive) return
        
        Or match this alternative (attempting the next alternative only if this one fails) class
        
        Match the character string “class” literally (case sensitive) class
        
        Or match this alternative (attempting the next alternative only if this one fails) do
        
        Match the character string “do” literally (case sensitive) do
        
        Or match this alternative (attempting the next alternative only if this one fails) for
        
        Match the character string “for” literally (case sensitive) for
        
        Or match this alternative (attempting the next alternative only if this one fails) while
        
        Match the character string “while” literally (case sensitive) while
        
        Or match this alternative (attempting the next alternative only if this one fails) debugger
        
        Match the character string “debugger” literally (case sensitive) debugger
        
        Or match this alternative (attempting the next alternative only if this one fails) export
        
        Match the character string “export” literally (case sensitive) export
        
        Or match this alternative (attempting the next alternative only if this one fails) import
        
        Match the character string “import” literally (case sensitive) import
        
        Or match this alternative (attempting the next alternative only if this one fails) yield
        
        Match the character string “yield” literally (case sensitive) yield
        
        Or match this alternative (attempting the next alternative only if this one fails) type
        
        Match the character string “type” literally (case sensitive) type
        
        Or match this alternative (attempting the next alternative only if this one fails) declare
        
        Match the character string “declare” literally (case sensitive) declare
        
        Or match this alternative (the entire group fails if this one fails to match) interface
        
        Match the character string “interface” literally (case sensitive) interface
      - Assert position at a word boundary (position preceded or followed—but not both—by a Unicode letter, digit, or underscore) \b
    - Or match this alternative (attempting the next alternative only if this one fails) \)
      - Match the character “)” literally \)
    - Or match this alternative (the entire group fails if this one fails to match) }
      - Match the character “}” literally }
Your regular expression may find zero-length matches
- Ruby 1.9 allows a zero-length match at the position where the previous match ends.
- Ruby 1.9 advances one character through the string before attempting the next match if the previous match was zero-length.

Created with RegexBuddy

Paul Chaignon · Answer 9 · Wed Jun 08 2016 16:48:41 GMT+0800 (China Standard Time)

@Alhadis Maybe you should try submitting a pull request (with a test on Lightshow) to @gandm's repository...?

Graham · Answer 10 · Wed Jun 08 2016 16:52:49 GMT+0800 (China Standard Time)

What exactly are we trying to prove? Of course I would expect that end statement to match import "blah"

John Gardner · Answer 11 · Wed Jun 08 2016 17:16:45 GMT+0800 (China Standard Time)

I said I wasn't sure if it was the cause of the problem. But I believe I've found an issue @gandm can investigate: there seems to be an issue somewhere in the end pattern:

Since the first import statement failed to find a matching end, it's essentially swallowed up the rest of the document.

Graham · Answer 12 · Wed Jun 08 2016 18:13:44 GMT+0800 (China Standard Time)

It works in atom, so I suggest you remove it from linguist and find another package.

From: John Gardnermailto:notifications@github.com
Sent: ‎08/‎06/‎2016 10:16
To: github/linguistmailto:linguist@noreply.github.com
Cc: Grahammailto:graham.f.clark54@gmail.com; Mentionmailto:mention@noreply.github.com
Subject: Re: [github/linguist] JSX Syntax highlighting broken? (#3044)

I said I wasn't sure if it was the cause of the problem. But I believe I've found an issue @gandm can investigate: there seems to be an issue somewhere in the end pattern:

[Doesn't work](https://github-lightshow.herokuapp.com/?utf8=%E2%9C%93&scope=from-text&grammar_url=&grammar_text=%7B%0D%0A%09%22name%22%3A+%22Test%22%2C%0D%0A%09%22scopeName%22%3A+%22source.test%22%2C%0D%0A%09%22fileTypes%22%3A+%5B%5D%2C%0D%0A%09%22patterns%22%3A+%5B%0D%0A%09%09%7B%0D%0A%09%09%09%22begin%22%3A+%22%5C%5Cs*%2B%28%3F%3C%21%5C%5C.%29%5C%5Cb%28import%29%28%3F%21%5C%5Cs*%3A%29%5C%5Cb%22%2C%0D%0A%09%09%09%22end%22%3A+%22%5C%5Cs*%28%3F%3A%28%5C%5Cbfrom%5C%5Cb%29%3F%5C%5Cs%2B%28%28%27%7C%5C%22%29%28%5B%5E%5C%22%27%5D*%29%28%5C%5C3%29%29%29%7C%28%3F%3D%3B%7C%5E%5C%5Cs*%5C%5Cb%28if%7Cswitch%7Ctry%7Cvar%7Clet%7Cconst%7Cstatic%7Cfunction%7Creturn%7Cclass%7Cdo%7Cfor%7Cwhile%7Cdebugger%7Cexport%7Cimport%7Cyield%7Ctype%7Cdeclare%7Cinterface%29%5C%5Cb%7C%5C%5C%29%7C%5C%5C%7D%5C%5C%29%22%2C%0D%0A%09%09%09%22beginCaptures%22%3A+%7B%0D%0A%09%09%09%09%221%22%3A+%7B+%22name%22%3A+%22keyword.control.module.js%22+%7D%0D%0A%09%09%09%7D%2C%0D%0A%09%09%09%22endCaptures%22%3A+%7B%0D%0A
%09%09%0
9%09%221%22%3A+%7B+%22name%22%3A+%22keyword.control.module.js%22+%7D%2C%0D%0A%09%09%09%09%223%22%3A+%7B+%22name%22%3A+%22punctuation.definition.string.begin.js%22+%7D%2C%0D%0A%09%09%09%09%224%22%3A+%7B+%22name%22%3A+%22string.quoted.module.js%22+%7D%2C%0D%0A%09%09%09%09%225%22%3A+%7B+%22name%22%3A+%22punctuation.definition.string.begin.js%22+%7D%0D%0A%09%09%09%7D%0D%0A%09%09%7D%0D%0A%09%5D%0D%0A%7D%0D%0A&code_source=from-text&code_url=&code=import+%27js%2Fweb.jsx%27%3B%0D%0Aimport+%27console.jsx%27%3B%0D%0A)
[Works](https://github-lightshow.herokuapp.com/?utf8=%E2%9C%93&scope=from-text&grammar_url=&grammar_text=%7B%0D%0A%09%22name%22%3A+%22Test%22%2C%0D%0A%09%22scopeName%22%3A+%22source.test%22%2C%0D%0A%09%22fileTypes%22%3A+%5B%5D%2C%0D%0A%09%22patterns%22%3A+%5B%0D%0A%09%09%7B%0D%0A%09%09%09%22begin%22%3A+%22%5C%5Cs*%2B%28%3F%3C%21%5C%5C.%29%5C%5Cb%28import%29%28%3F%21%5C%5Cs*%3A%29%5C%5Cb%22%2C%0D%0A%09%09%09%22end%22%3A+%22%3B%22%2C%0D%0A%09%09%09%22beginCaptures%22%3A+%7B%0D%0A%09%09%09%09%221%22%3A+%7B+%22name%22%3A+%22keyword.control.module.js%22+%7D%0D%0A%09%09%09%7D%2C%0D%0A%09%09%09%22endCaptures%22%3A+%7B%0D%0A%09%09%09%09%221%22%3A+%7B+%22name%22%3A+%22keyword.control.module.js%22+%7D%2C%0D%0A%09%09%09%09%223%22%3A+%7B+%22name%22%3A+%22punctuation.definition.string.begin.js%22+%7D%2C%0D%0A%09%09%09%09%224%22%3A+%7B+%22name%22%3A+%22string.quoted.module.js%22+%7D%2C%0D%0A%09%09%09%09%225%22%3A+%7B+%22name%22%3A+%22punctuation.definition.string.begin.js%22+%7D%0D%0A%09
%09%09%7
D%0D%0A%09%09%7D%0D%0A%09%5D%0D%0A%7D%0D%0A&code_source=from-text&code_url=&code=import+%27js%2Fweb.jsx%27%3B%0D%0Aimport+%27console.jsx%27%3B%0D%0A)

Since the first import statement failed to find a matching end, it's essentially swallowed up the rest of the document.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#3044 (comment)

Graham · Answer 13 · Wed Jun 08 2016 22:14:44 GMT+0800 (China Standard Time)

I've just run code that works in oniguruma but does nothing in lightshow. I use this type of back referencing a lot in the grammar. Maybe this is the issue?

Test Grammar

{
  "name": "Test capture group",
  "scopeName": "source.js.jsx",
  "patterns": [
    { 
       "name": "invalid.illegal.string.js",
       "match": "(('|\")([^\"']*)(\\k<-3>))" 
    }
  ]
}

against code

"hello world"

should highlight in reverse red.

Seems like backreferencing isn't working in whatever Regex engine is used by Lightshow but does in Oniguruma engine as used by Atom.

John Gardner · Answer 14 · Wed Jun 08 2016 22:42:56 GMT+0800 (China Standard Time)

That was my theory, but it still wasn't matching when I replaced it with an ordinary backreference.

Graham · Answer 15 · Wed Jun 08 2016 22:49:35 GMT+0800 (China Standard Time)

Yeah I think back referencing is broken in whatever engine is used in lightshow and I guess by extension in linguist. What engine is it?

John Gardner · Answer 16 · Wed Jun 08 2016 22:55:02 GMT+0800 (China Standard Time)

I don't know, nobody outside GitHub does. The damn thing is closed source.

Graham · Answer 17 · Wed Jun 08 2016 23:07:40 GMT+0800 (China Standard Time)

Well I could fix this single issue by changing the end for imports regex but the grammar uses this in other places where it isn't possible to change it. Therefore it may be prudent to look at other grammars that parse jsx but don't use back refererencing and use one of these inside linguist.

Arfon Smith · Answer 18 · Wed Jun 08 2016 23:25:16 GMT+0800 (China Standard Time)

@Alhadis is this the same issue as described in #2703 ?

John Gardner · Answer 19 · Wed Jun 08 2016 23:53:54 GMT+0800 (China Standard Time)

Nope, it's a separate issue. Basically, Lightshow seems to be powered with a different regex engine that only supports a subset of Oniguruma's extensions.

I say "subset" because most TextMate grammars already use Oniguruma extensions in the form of (?x:...) or \\G (well, which were taken from Perl, at least).

Arfon Smith · Answer 20 · Thu Jun 09 2016 00:14:41 GMT+0800 (China Standard Time)

Nope, it's a separate issue. Basically, Lightshow seems to be powered with a different regex engine that only supports a subset of Oniguruma's extensions.

OK thanks.

Graham · Answer 21 · Fri Jun 10 2016 06:50:17 GMT+0800 (China Standard Time)

I just noticed that @cozos, who raised the issue, appears to be using jsx to mean the statically typed OO language JSX and not the JSX extension to JavaScript as used by facebook's React. Maybe linguist has better support for this language using another suffix that he could use?

It makes no difference to the underlying issue, as React JSX would also get this problem due to the incompatibility between the linguist regex engine and Atom's Oniguruma.

Arwin Tio · Answer 22 · Fri Jun 10 2016 08:56:50 GMT+0800 (China Standard Time)

@gandm I apologize, I actually meant the JSX extension to JavaScript used by Facebook's React - I was just looking for some examples of JSX on Github and seeing that the syntax highlighting issue was also happening on the statically typed JSX I carelessly assumed it was the React JSX.

I will edit the example to this:

https://gist.github.com/cozos/d6636a93cba29c77e3221f2ca30212b0

import React from 'react';

var ExampleApplication = React.createClass({
  render: function() {
    var elapsed = Math.round(this.props.elapsed  / 100);
    var seconds = elapsed / 10 + (elapsed % 10 ? '' : '.0' );
    var message =
      'React has been successfully running for ' + seconds + ' seconds.';

    return <p>{message}</p>;
  }
});

var start = new Date().getTime();

setInterval(function() {
  ReactDOM.render(
    <ExampleApplication elapsed={new Date().getTime() - start} />,
    document.getElementById('container')
  );
}, 50);

Spencer · Answer 23 · Wed Jun 29 2016 04:08:40 GMT+0800 (China Standard Time)

Happy to see this issue getting some discussion, but I'm not sure where it landed... Was this reopened because it's a bug linguist is going to fix?

Arwin Tio · Answer 24 · Wed Jun 29 2016 04:41:46 GMT+0800 (China Standard Time)

@spalger
Seems to be going into the backlog :P basically Linguist's regex engine can't handle language-babel's back referencing, which works in Oniguruma.

I would also like to mention to @arfon and @Alhadis that JSX syntax highlighting was working on Github probably around a month and a half ago - so perhaps there was some changes on linguist's end?

Graham · Answer 25 · Wed Jun 29 2016 04:56:51 GMT+0800 (China Standard Time)

@spalger @cozos it would have worked about a month ago but then I refactored the import/export part of the grammar to use a certain back reference notation that oniguruma supports but linguist doesn't appear to. Actually, I've always used this form of back reference in other parts of the grammar that handled flow syntax but I guess no one has picked up on it not working as flow isn't widely used.

Maybe I could change the grammar to support something that linguist can use but I'm reluctant on two counts. One, it makes the code less readable and more unique for each regex, and secondly, I believe linguist should be text mate compatible and it isn't.

Arwin Tio · Answer 26 · Wed Jun 29 2016 06:39:19 GMT+0800 (China Standard Time)

Why doesn't linguist just use https://github.com/gandm/language-babel/releases/tag/v2.24.4 then? @gandm, 2.24.x is the pre back reference notation version right? Seems to be a good compromise.

Lucas Parry · Answer 27 · Wed Jun 29 2016 08:55:11 GMT+0800 (China Standard Time)

Is there a workaround for this? Some sort of comment/pragma mark we can put in jsx files to get them to highlight correctly and ignore the faulty autodetected language for the files?

Arwin Tio · Answer 28 · Wed Jun 29 2016 08:59:45 GMT+0800 (China Standard Time)

I made a PR to version lock language-babel to an earlier release: #3091

Lucas Parry · Answer 29 · Tue Aug 30 2016 08:30:04 GMT+0800 (China Standard Time)

Still broken on github PRs :/

Is there another issue I should be tracking for that

Arwin Tio · Answer 30 · Tue Aug 30 2016 11:04:14 GMT+0800 (China Standard Time)

I guess we should close this one and continue discussion there? Up to the maintainers I guess.

Trent · Answer 31 · Mon Jun 19 2017 13:03:41 GMT+0800 (China Standard Time)

Still seeing broken highlighting in diffs for some sections of JSX. I'm a bit confused by the trail of issues and this seemed like a reasonable place to make a comment. Seems like the issue has gone a bit stale?

Apologies if this is the wrong place.

Lewis Blackwood · Answer 32 · Thu Jun 22 2017 21:47:58 GMT+0800 (China Standard Time)

Is this resolved? It seems that highlighting only shows correctly for files with the .jsx extension - see example gist here.

I'm using Create-React-App, which supports but does not recommend the .jsx extension.

Edit: most recent discussion for the above is at #3677.

Colin Seymour · Answer 33 · Sat Jun 24 2017 01:51:43 GMT+0800 (China Standard Time)

@lewisblackwood yup, once #3677 is merged, JSX files with a .js extension should start to be highlighted.

Lewis Blackwood · Answer 34 · Sat Jun 24 2017 01:55:50 GMT+0800 (China Standard Time)

@lildude wonderful, thanks team!

Colin Seymour · Answer 35 · Sat Jun 24 2017 01:58:17 GMT+0800 (China Standard Time)

If the syntax highlighting is then still broken, blame @Alhadis

Seriously, we probably need to switch to using an externally maintained JSX syntax package as I don't think it's right nor feasible in the long run to rely on a package that isn't maintained by someone with intimate knowledge of the language. I realise this is easier said than done.

John Gardner · Answer 36 · Sat Jun 24 2017 02:07:31 GMT+0800 (China Standard Time)

What are you talking about...? Who doesn't have an intimate knowledge of what?

Colin Seymour · Answer 37 · Sat Jun 24 2017 02:12:59 GMT+0800 (China Standard Time)

Ooops, my mistake. 😊 For some reason I thought you were responsible for the fork at https://github.com/github-linguist/language-babel and were actively attempting to maintain it. I now see that's not the case as we're just using it to peg a version. Sorry about that. 🙇

stale · Answer 38 · Tue Nov 06 2018 15:14:31 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had activity in a long time. If this issue is still relevant and should remain open, please reply with a short explanation (e.g. "I have checked the code and this issue is still relevant because ___."). Thank you for your contributions.

Jason Gornall · Answer 39 · Fri Nov 09 2018 01:35:25 GMT+0800 (China Standard Time)

bump this is still broken

Paul Chaignon · Answer 40 · Fri Nov 09 2018 02:27:10 GMT+0800 (China Standard Time)

@jasongornall As far as I can see, the bug reported in the original post is fixed. Do you have another case of incorrect highlighting? Maybe we should open a new issue to keep things clear?

Rodrigo Graças · Answer 41 · Mon Nov 26 2018 08:27:35 GMT+0800 (China Standard Time)

@jasongornall Recently, I've also had the problem of having jsx in a .js file committed to a Github repository as a .js file. The code was highlighted in red and I thought it was a syntax error (missing semi-colon or some lint error..), but no. I was using single-quotes (') and I guess this broke the syntax.. I've removed and now is fine, with no red highlight.

stale · Answer 42 · Thu Dec 27 2018 04:27:53 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had activity in a long time. If this issue is still relevant and should remain open, please reply with a short explanation (e.g. "I have checked the code and this issue is still relevant because ___."). Thank you for your contributions.

Andrew Zamora · Answer 43 · Sat Jan 05 2019 00:04:49 GMT+0800 (China Standard Time)

Unfortunately, I believe this is still an issue. A single apostrophe was the cause for me. This stackoverflow post helped me figure it out. As you can see in the commit below the jsx in my js file is being highlighted in red even though there is no error.
AndrewZamora/Whos-On-What@8f45eda

When I changed the file extension to .jsx the red highlighting went away as you can see here and in the image below.

I changed the file extension back to .js and removed and apostrophe from the <h2> tag and the red highlighting went away as you can see here and below.

I hope this was helpful.

Colin Seymour · Answer 44 · Wed Jan 09 2019 01:23:24 GMT+0800 (China Standard Time)

Unfortunately, I believe this is still an issue. A single apostrophe was the cause for me. This stackoverflow post helped me figure it out. As you can see in the commit below the jsx in my js file is being highlighted in red even though there is no error.

This is going to be a problem with the grammar itself and not Linguist. You're using the .js extension so the https://github.com/atom/language-javascript grammar is being used. When you use .jsx, the https://github.com/github-linguist/babel-sublime grammar is used.

I know @Alhadis has been working on a grammar to combine support for both of these into the same grammar though I'm not sure of the status.

John Gardner · Answer 45 · Wed Jan 09 2019 01:34:30 GMT+0800 (China Standard Time)

Ditch Atom's grammar, and switch to Babel's until I can get something battle-ready. There'll be no visible discrepancies between JS and JSX files (caused by different scope choices in either grammar), and all valid JavaScript can be guaranteed to be valid JSX.

Colin Seymour · Answer 46 · Fri Feb 01 2019 00:56:05 GMT+0800 (China Standard Time)

Ditch Atom's grammar, and switch to Babel's until I can get something battle-ready. There'll be no visible discrepancies between JS and JSX files (caused by different scope choices in either grammar), and all valid JavaScript can be guaranteed to be valid JSX.

Hmm, this seems easier said than done. A metric 🚣‍♂️ load of other grammars expect to find source.js and they can't as soon as I switch the grammars resulting in the grammar checker reporting a ton of errors like:

repository vendor/grammars/language-ruby (from https://github.com/atom/language-ruby) (2 errors)
- Missing include in grammar: source.ruby (in grammars/ruby.cson) attempts to include source.js but the scope cannot be found
- Missing include in grammar: source.ruby (in grammars/ruby.cson) attempts to include source.js.jquery but the scope cannot be found

Seems the grammar compiler isn't pulling in all of the files in the grammar so isn't finding the source.js scope defined in one of the files as we can see from the diff of grammars.yml:

diff --git a/grammars.yml b/grammars.yml
index 0ff1dcd9..73b18475 100755
--- a/grammars.yml
+++ b/grammars.yml
@@ -448,11 +448,6 @@ vendor/grammars/language-html:
 - text.html.basic
 vendor/grammars/language-inform7:
 - source.inform7
-vendor/grammars/language-javascript:
-- source.js
-- source.js.regexp
-- source.js.regexp.replacement
-- source.jsdoc
 vendor/grammars/language-jison:
 - source.jison
 - source.jisonlex

The removal of source.js.regexp also means the babel-sublime grammar itself can't be added without forcing:

$ script/add-grammar --replace language-javascript https://github.com/github-linguist/babel-sublime
Checking docker is installed and running
$ docker ps
Deregistering: vendor/grammars/language-javascript
$ git submodule deinit vendor/grammars/language-javascript
$ git rm -rf vendor/grammars/language-javascript
$ script/grammar-compiler update -f
Registering new submodule: vendor/grammars/babel-sublime
$ git submodule add -f https://github.com/github-linguist/babel-sublime vendor/grammars/babel-sublime
$ script/grammar-compiler add vendor/grammars/babel-sublime
  > latest: Pulling from linguist/grammar-compiler
  > Digest: sha256:159ff655c832de0e3ab3ea1366992606b5f604bcb64f8757b6c575d4a7628bc6
  > Status: Image is up to date for linguist/grammar-compiler:latest
  > The new grammar repository `vendor/grammars/babel-sublime` (from https://github.com/github-linguist/babel-sublime) contains 1 errors:
  >     - Missing include in grammar: `source.js.jsx` (in `JavaScript (Babel).tmLanguage`) attempts to include `source.regexp.js` but the scope cannot be found
  >
  > failed to compile the given grammar
$

John Gardner · Answer 47 · Fri Feb 01 2019 16:56:09 GMT+0800 (China Standard Time)

Ah heck, that's bad. 😓 Didn't expect there'd be a circular dependency involved... that complicates everything.

@50Wliu, any idea if the Atom team would be averse to removing the rule which highlights unterminated strings as errors? Since Atom's grammars currently ship with both TextMate and Tree-sitter grammars (the latter of which Linguist ignores), I imagine the impact to users is minimal…

Winston Liu · Answer 48 · Thu Feb 07 2019 23:38:11 GMT+0800 (China Standard Time)

@Alhadis that's probably fine. We don't actively maintain the TextMate variants anymore, but I think this is something we'll merge a PR for.

John Gardner · Answer 49 · Sat Feb 09 2019 13:57:13 GMT+0800 (China Standard Time)

Great. Would you mind patching this? I can't submit a PR because I'm blocked from the @atom org.

diff --git a/grammars/javascript.cson b/grammars/javascript.cson
index 469fb4b..fca11f9 100644
--- a/grammars/javascript.cson
+++ b/grammars/javascript.cson
@@ -1344,10 +1344,6 @@
           {
             'include': '#string_escapes'
           }
-          {
-            'match': "[^']*[^\\n\\r'\\\\]$"
-            'name': 'invalid.illegal.string.js'
-          }
         ]
       }
       {
@@ -1364,10 +1360,6 @@
           {
             'include': '#string_escapes'
           }
-          {
-            'match': '[^"]*[^\\n\\r"\\\\]$'
-            'name': 'invalid.illegal.string.js'
-          }
         ]
       }
       {

stale · Answer 50 · Mon Mar 11 2019 14:18:12 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had activity in a long time. If this issue is still relevant and should remain open, please reply with a short explanation (e.g. "I have checked the code and this issue is still relevant because ___."). Thank you for your contributions.

Colin Seymour · Answer 51 · Fri Mar 15 2019 17:47:03 GMT+0800 (China Standard Time)

I've opened atom/language-javascript#642 with @Alhadis's suggested change.

Colin Seymour · Answer 52 · Sat Mar 30 2019 01:32:40 GMT+0800 (China Standard Time)

@Alhadis I don't know if you can see atom/language-javascript#642 but if not, the latest comment from @50Wliu for your changes is:

I thought about this for a bit: wouldn't this still make it so everything will be matched as a string until the next quote? I suppose that's better than angry red highlighting, but still not that ideal.

The answer would be "yes" unless you can think of a better solution.

John Gardner · Answer 53 · Sat Mar 30 2019 01:44:51 GMT+0800 (China Standard Time)

I wasn't notified that I was mentioned at the atom/language-javascript repo... I guess being blocked org-wide affects that too. 👎

Yes, it would affect unclosed strings, but only affecting users who have Tree Sitter Parsers disabled in their settings:

While I've not tried this for myself, I believe users who leave this setting enabled by default will continue to see error highlighting for unclosed strings. I might be wrong though.

John Gardner · Answer 54 · Sat Mar 30 2019 01:54:10 GMT+0800 (China Standard Time)

There's only one other possibility I can think of, one that would involve extra work on Atom (or GitHub's) end: add unclosed as a scope so the full scope-name becomes invalid.illegal.unclosed.string.js. Then ask the Primer team to modify their scope-to-CSS-class handling to avoid applying background colours to the .pl-ii class in JavaScript files only.

The only advantage this has over the previous solution is that Atom users who have Tree Sitter grammars turned off can continue to see error highlighting for unterminated strings. Which, given the complexity involved, probably isn't worth the effort.

Colin Seymour · Answer 55 · Sat Mar 30 2019 01:54:12 GMT+0800 (China Standard Time)

I wasn't notified that I was mentioned at the atom/language-javascript repo... I guess being blocked org-wide affects that too. 👎

Thought as much 😞

Yes, it would affect unclosed strings, but only affecting users who have Text Mate Parsers disabled in their settings:

You mean tree sitter parsers 😉

~~Do you think there's an alternate solution for our case, ie TextMate, or is it very much an all or nothing approach that we're already proposing?~~

Answered as our updates crossed.

John Gardner · Answer 56 · Sat Mar 30 2019 01:57:04 GMT+0800 (China Standard Time)

You mean tree sitter parsers 😉

Oops, yes, of course. 😀 What the hell.

Answered as our updates crossed.

I was following-up with a third comment saying "See above, our answers were posted at the same time", but I saw you notice and cross that sentence out. WebSockets are a wonderful thing, aren't they?

Colin Seymour · Answer 57 · Tue Apr 02 2019 22:11:31 GMT+0800 (China Standard Time)

🎉 🤞 this finally puts this issue to 🛏 once and for all.

John Gardner · Answer 58 · Tue Apr 02 2019 22:31:42 GMT+0800 (China Standard Time)

Awesome! 🎉 Though I had to check Graphemica.com to figure out what this meant:

Michael Cooper · Answer 59 · Tue Apr 16 2019 00:42:43 GMT+0800 (China Standard Time)

Unfortunately, it seems like this is still not fixed. I'm not sure if linguist has been updated with the new version of language-javascript, but I expected it would be after about two weeks. This PR still shows the red-highlighted false-positive unclosed strings.

John Gardner · Answer 60 · Tue Apr 16 2019 12:19:04 GMT+0800 (China Standard Time)

@mythmon It's yet to take effect on GitHub. Changes to Linguist (including the grammars it uses for highlighting) only become live once a new release of Linguist has been cut, which happens roughly once a month.

Colin Seymour · Answer 61 · Sun Apr 21 2019 20:50:07 GMT+0800 (China Standard Time)

This PR still shows the red-highlighted false-positive unclosed strings.

Not any more 😀

From a quick look over previously reported issues, all appear to be rendering correctly now so I'm considering this issue now resolved and closing. If a new instance of a problem comes to light, open a new issue.