Regex delimiters from `js` not included in `jsx`?

Question

Regex delimiters from `js` not included in `jsx`?

karlhorky opened this issue 4 years ago · comments

Hi @wooorm , sorry for all the issues lately!

Working on link and highlighting tooling a bit right now...

Here's something that I found interesting: the js language returns some regex delimiter tokens, but the jsx language does not... should it be like that? I thought those extra tokens came from js-extras, maybe this is not included with jsx or typescript or tsx?

Ran the following on Runkit:

var refractor = require("refractor")
console.log(JSON.stringify(refractor.highlight('/\bno\-highlight\b/', 'js'), null, 2))
console.log(JSON.stringify(refractor.highlight('/\bno\-highlight\b/', 'jsx'), null, 2))

...and received the following:

JS

[
  {
    "type": "element",
    "tagName": "span",
    "properties": {
      "className": [
        "token",
        "regex"
      ]
    },
    "children": [
      {
        "type": "element",
        "tagName": "span",
        "properties": {
          "className": [
            "token",
            "regex-delimiter"
          ]
        },
        "children": [
          {
            "type": "text",
            "value": "/"
          }
        ]
      },
      {
        "type": "element",
        "tagName": "span",
        "properties": {
          "className": [
            "token",
            "language-regex"
          ]
        },
        "children": [
          {
            "type": "text",
            "value": "\bno-highlight\b"
          }
        ]
      },
      {
        "type": "element",
        "tagName": "span",
        "properties": {
          "className": [
            "token",
            "regex-delimiter"
          ]
        },
        "children": [
          {
            "type": "text",
            "value": "/"
          }
        ]
      }
    ]
  }
]

JSX

[
  {
    "type": "element",
    "tagName": "span",
    "properties": {
      "className": [
        "token",
        "regex"
      ]
    },
    "children": [
      {
        "type": "text",
        "value": "/\bno-highlight\b/"
      }
    ]
  }
]

Titus · Answer 1 · Thu Aug 27 2020 00:35:50 GMT+0800 (China Standard Time)

Refractor is supposed to do what prism does (albeit virtual). Cab you confirm whether prism (does not) result in the same output?

Karl Horky · Answer 2 · Thu Aug 27 2020 00:43:58 GMT+0800 (China Standard Time)

On the test page and on a sandbox, the javascript language results in no extra tokens:

Test Page: https://prismjs.com/test.html#language=javascript
Demo: https://codesandbox.io/s/prismjs-forked-0ygs8?file=/src/index.js

Karl Horky · Answer 3 · Thu Aug 27 2020 00:45:12 GMT+0800 (China Standard Time)

So I suppose if there's a bug here, it's that js adds the extra tokens in refractor but not in Prism.js.

Titus · Answer 4 · Thu Aug 27 2020 01:03:19 GMT+0800 (China Standard Time)

I can confirm, javascript, jsx.

regex-delimiter is added in JavaScript, due to these Prism lines. They don’t include JSX.

The normal build of refractor includes everything, so regex runs, which modifies JavaScript. But that apparently doesn’t happen in these Prism cases? 🤔

Titus · Answer 5 · Thu Aug 27 2020 01:06:54 GMT+0800 (China Standard Time)

This seems to be due to the global state of Prism:

require("prismjs")
require("prismjs/components/prism-jsx")
require("prismjs/components/prism-regex")

console.log('js', Prism.highlight('/\bno\-highlight\b/', Prism.languages.javascript, 'javascript'))
console.log('jsx:', Prism.highlight('/\bno\-highlight\b/', Prism.languages.jsx, 'jsx'))

Yields:

js: <span class=\"token regex\"><span class=\"token regex-deli…pan><span class=\"token regex-delimiter\">/</span></span>
jsx: <span class=\"token regex\">/\bno-highlight\b/</span>

But moving prism-regex to before prism-jsx makes them the same.

Karl Horky · Answer 6 · Thu Aug 27 2020 03:44:27 GMT+0800 (China Standard Time)

The normal build of refractor includes everything

But moving prism-regex to before prism-jsx makes them the same

Ah, so the way to address this will be to enable all regex and other add-on features like js-extras for jsx too?

Wonder if this also affects typescript and tsx...

Titus · Answer 7 · Thu Aug 27 2020 04:13:12 GMT+0800 (China Standard Time)

Unfortunately, this is really hard to solve, because it’s a problem that comes from Prism. Prism works with one whole global object, and has intricate interdependencies. We’re already doing a lot to figure out which languages depend on each other. We used to do regexes, but since this version we’re using Prism-provided information to figure out what depends on what.

Sometimes there are direct dependencies. But in this case, it’s if X is defined, do something to it, otherwise ignore it.
If you load regex, it’ll mutate javascript if it exists (one of the few that always does). If you then load jsx, it uses the mutated JavaScript object. If you do it the other way around, jsx clones javascript before regex extends javascript. 🤔

I’m guessing it also affects ts(x) indeed. And I’m sure there are others.

Karl Horky · Answer 8 · Thu Aug 27 2020 23:56:17 GMT+0800 (China Standard Time)

since this version we’re using Prism-provided information to figure out what depends on what

Ah interesting... couldn't find the commit / PR for this.

Edit: Or do you mean this PR #32 ? With the commit c130059 (#32) ?

Karl Horky · Answer 9 · Thu Aug 27 2020 23:57:16 GMT+0800 (China Standard Time)

@RunDevelopment you wouldn't have any suggestions here would you?

If I'm understanding correctly, refractor wants to be able to load all Prism languages in the correct order for their dependencies, and is currently using some Prism-provided info for this.

Michael Schmidt · Answer 10 · Fri Aug 28 2020 00:37:05 GMT+0800 (China Standard Time)

Oops. Seems like I messed up with the dependencies here. I'll fix it.

Titus · Answer 11 · Fri Aug 28 2020 00:44:40 GMT+0800 (China Standard Time)

Ohh, that’s great to hear! <3

Titus · Answer 12 · Sun Oct 11 2020 15:50:17 GMT+0800 (China Standard Time)

Can confirm that this is now solved in the latest refractor.

Thanks @RunDevelopment!

Karl Horky · Answer 13 · Mon Oct 12 2020 21:39:45 GMT+0800 (China Standard Time)

Nice, thanks to both of you!

Titus · Answer 14 · Fri Apr 23 2021 17:01:49 GMT+0800 (China Standard Time)

@RunDevelopment Hmm, I’m running into this again. JS is included here by default. Regex isn‘t.

Because optional dependencies are passed as reference (example of regex in JS), the import order matters.

For Prism bundles, y’all are solving it very smartly (dependencies.js).
That makes it impossible (I think) to load a few languages first and more languages later (which is what I do here).

One solution I can think of is for the grammars to use:

'regex-source': {
  pattern: /^(\/)[\s\S]+(?=\/[a-z]*$)/,
  lookbehind: true,
  alias: 'language-regex',
  inside: 'regex'
},

...which is then resolved to Prism.languages.regex at runtime.
Would something like that be acceptable for Prism?

Alternatively, I could make more bundles for refractor: 0, common languages, and all languages. Then I can leave it up to users.

Michael Schmidt · Answer 15 · Fri Apr 23 2021 19:36:31 GMT+0800 (China Standard Time)

That makes it impossible (I think) to load a few languages first and more languages later (which is what I do here).

No, it's possible. The trick is that loading a language again will overwrite the old version. I.e. if your bundle includes JS but not Regex, you can just load Regex and then load JS again.

This isn't a particularly nice solution but it works. Our loadLanguages function uses exactly this strategy.

You did point out one of the painful consequences of Prism's current load-order-based dependency system. For Prism v2, we plan to adopt ESM into our codebase, at which point we will most likely have to phase out the current solution for optional dependencies and implement something (hopefully) nicer.

Titus · Answer 16 · Fri Apr 23 2021 20:23:54 GMT+0800 (China Standard Time)

Ah, so I could add:

regex.optionalDependants = ['javascript', 'vala']

And then when regex loads it’ll re-initialize javascript and vala when available? 🤔
A bit more complex would be to reload coffeescript and such, which depend on javascript.

Interesting, thanks, I’ll try something along those lines!

For Prism v2, we plan to adopt ESM into our codebase, at which point we will most likely have to phase out the current solution for optional dependencies and implement something (hopefully) nicer.

Nice! Is there an ETA on when that would be ready?

Michael Schmidt · Answer 17 · Fri Apr 23 2021 21:11:35 GMT+0800 (China Standard Time)

Ah, so I could add:

regex.optionalDependants = ['javascript', 'vala']

Sorry, but I don't know what that is about.

And then when regex loads it’ll re-initialize javascript and vala when available? 🤔
A bit more complex would be to reload coffeescript and such, which depend on javascript.

I suggest using our getLoader API. It will handle all of that for you.

getLoader(components, languagesToLoad, loadedLanguages)
    .load(id => { /* load language with the given id */ });

That's all you have to do. getLoader will ensure that everything works.

It's a little harder for asynchronous operations. Some languages can be loaded in parallel but others cannot. load will also take care of that for you, see the Promise example.

However, it might be a problem that getLoader needs to know all possible dependencies upfront.

Is there an ETA on when that would be ready?

Unfortunately, no.

Titus · Answer 18 · Fri May 07 2021 22:56:01 GMT+0800 (China Standard Time)

Thanks!

Sorry, but I don't know what that is about.

I generate custom bundles of each language, so that they can be registered on their own. For example, javascript.js.

However, I was hoping with my previous comment that it would be possible for this case to work:

Assume all languages except for YAML are loaded
Load YAML, which knows which other languages have be reloaded (such as markdown, and many others)

Unfortunately I found that a change in certain languages requires a lot of re-registrations, and resulted in refractor taking 1.3 seconds to load.
The code I used to find the things that would have to be reregistered was this pseudo code:

var list = getLoader(components, [theLanguage], allLanguages).getIds()
var reregister = list.slice(list.indexOf(theLanguage) + 1)