Loading All Language Lexers Even When Only One is Needed
Innei opened this issue · comments
Validations
- Follow our Code of Conduct
- Read the Contributing Guide.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
Describe the bug
Description
I am using Shiki's code highlighting library to implement code splitting and lazy loading for language lexers and themes. However, despite specifying only one language in the configuration, Shiki is loading all the configured language lexers, leading to unnecessary loading of resources.
Steps to Reproduce
Set up the getHighlighterCore
with multiple languages configured as lazy-loadable modules.
Use codeHighlighter function specifying only one language (e.g., 'typescript') to highlight.
Observe that all configured language modules are loaded, not just the one required.
async function main() {
const codeHighlighter = await (async () => {
const [{ getHighlighterCore }, getWasm] = await Promise.all([
import('shiki/core'),
import('shiki/wasm').then((m) => m.default),
])
const shiki = await getHighlighterCore({
themes: [
import('shiki/themes/github-light.mjs'),
import('shiki/themes/github-dark.mjs'),
],
langs: [
() => import('shiki/langs/javascript.mjs'),
() => import('shiki/langs/typescript.mjs'),
() => import('shiki/langs/css.mjs'),
() => import('shiki/langs/tsx.mjs'),
() => import('shiki/langs/jsx.mjs'),
() => import('shiki/langs/json.mjs'),
() => import('shiki/langs/sql.mjs'),
() => import('shiki/langs/rust.mjs'),
() => import('shiki/langs/go.mjs'),
() => import('shiki/langs/cpp.mjs'),
() => import('shiki/langs/c.mjs'),
() => import('shiki/langs/markdown.mjs'),
() => import('shiki/langs/vue.mjs'),
() => import('shiki/langs/html.mjs'),
() => import('shiki/langs/asm.mjs'),
() => import('shiki/langs/shell.mjs'),
() => import('shiki/langs/ps.mjs'),
],
loadWasm: getWasm,
})
return (o: { lang: string; attrs: string; code: string }) =>
shiki.codeToHtml(o.code, {
lang: o.lang,
theme: 'github-light',
})
})()
document.querySelector<HTMLDivElement>('#app')!.innerHTML = `
<div>
${codeHighlighter({
lang: 'typescript',
attrs: '',
code: `import typescriptLogo from './typescript.svg'`,
})}
</div>
`
}
main()
For the code above, I registered many languages, but I only used one.
But it will download all the js files
Expected Behavior
Only the lexer for the specified language should be loaded. For example, if 'typescript' is specified, only the Typescript lexer should be dynamically imported and used.
Or I hope shiki can add automatic language recognition and download lexical js files like prism.
Actual Behavior
All lexers specified in the getHighlighterCore setup are being loaded regardless of the language specified in codeHighlighter.
Reproduction
https://github.com/Innei/shiki-bundle-and-import-repro
Contributes
- I am willing to submit a PR to fix this issue
- I am willing to submit a PR with failing tests
Yes because highlighting is sync (required in many integrations), while the loading is async, so it has to be done ahead of time.
If your integration are acceptable for async context, you could do loading yourself
if (!shiki.getLanguages().includes('typescript'))
await shiki.loadLanguage(import('shiki/langs/typescript.mjs'))
const html = shiki.codeToHtml(code, { lang: 'ts' })
Thank you for your response.
I am using this method now and he is working fine.
use(
useMemo(async () => {
async function loadShikiLanguage(language: string, languageModule: any) {
const shiki = codeHighlighter?.codeHighlighter
if (!shiki) return
if (!shiki.getLoadedLanguages().includes(language)) {
await shiki.loadLanguage(await languageModule())
}
}
const { bundledLanguages } = await import('shiki/langs')
if (!language) return
const importFn = (bundledLanguages as any)[language]
if (!importFn) return
return loadShikiLanguage(language || '', importFn)
}, [codeHighlighter?.codeHighlighter, language]),
)