[FR] Expose option to load custom languages for shiki

[Moritz Jung]

I would like to use Expressive Code in my starlight project, especially since starlight now ships with it enabled by default, but I found no way to register custom tmlanguages for shiki for Expressive Code. And since starlight fails to render a page with expressive code enabled if there exists a code block of a language that is unknown to shiki, I sadly can't use Expressive Code in my project.

This FR is a 2-in-1 package

1. I would like to be able to register custom tmlanguages to shiki used by expressive code
2. Expressive code should fall back to no highlighting (and not error) for a code block where shiki does not have a tmlanguage for the code block language (idk if this is a starlight issue, the starlight people directed me to this repo)

For context: The error trace in starlight for a codeblock of a language that is unknown to shiki looks like this

[Hippo]

Thank you for this feature request!

I agree that a console warning and rendering as plain text would be the better way to handle unknown languages. Throwing an error is too harsh for this type of issue.

Regarding the custom language registration, I'm pasting some additional context here that was provided on Discord. You were using roughly the following configuration in Astro:

import { defineConfig } from 'astro/config'
import { bundledLanguages } from 'shikiji'

const MetaBind = {
  // defined here: https://github.com/mProjectsCode/obsidian-meta-bind-plugin-docs/blob/master/metaBindLanguage.js
}

export default defineConfig({
  // ...
  markdown: {
    shikiConfig: {
      langs: [
        ...Object.keys(bundledLanguages),
        {
          id: 'meta-bind',
          scopeName: 'source.meta-bind',
          grammar: MetaBind,
          aliases: [],
        },
      ],
    },
  },
})

To fix this part of the issue, the markdown.shikiConfig config option must be recognized by astro-expressive-code and passed down to @expressive-code/plugin-shiki, which must then use the contained settings when creating the highlighter.

[Moritz Jung]

Regarding the config, I noticed that astro allows passing the grammar in a wrapper object like in the example in your message and passing the grammar directly, so like the following example.

import { defineConfig } from 'astro/config'
import { bundledLanguages } from 'shikiji'

const MetaBind = {
  // defined here: https://github.com/mProjectsCode/obsidian-meta-bind-plugin-docs/blob/master/metaBindLanguage.js
}

export default defineConfig({
  // ...
  markdown: {
    shikiConfig: {
      langs: [
        ...Object.keys(bundledLanguages),
        MetaBind
      ],
    },
  },
})

[Hippo]

Part 2 of your feature request (don't throw an error but instead log a warning and fall back to plaintext when encountering unknown languages) has been completed in version 0.29.4 that I just released to NPM.

Part 1 (custom language support) will be added soon. I'd like astro-expressive-code to be able to transparently pick up and use custom languages defined in the Astro config. However, Astro migrated from Shiki to Shikiji, and their lang config property types differ, as you have also already pointed out. Instead of developing a custom compatibility function that attempts to convert between the types, I'd rather migrate to Shikiji as well, and this will be done in the next minor release.

[Hippo]

Custom language support has landed in version 0.30.0! :)
(Update: 0.30.1 including a parallelism fix)

As described in my previous message, astro-expressive-code now automatically picks up Astro's markdown.shikiConfig.langs config option (if set) and forwards it to plugin-shiki. Alternatively, you can also set this option directly in Expressive Code's options (shiki: { langs: [...] }).

Please note that only the flattened Shikiji version of the langs array is supported, not the old Shiki version, as I didn't want to introduce the overhead of a conversion function for an option that's probably only used by a few selected people. :) So, passing the grammars directly without a wrapper object is the way to go.

I'll create a PR in the Starlight repo to bump the EC dependency to ^0.30.1. If you want to use this feature within Starlight before the PR has been created and merged, you can use PNPM overrides to upgrade right away:

// package.json
{
  "name": "your-docs-repo",
  "type": "module",
  "version": "0.0.1",
  "scripts": {
    "dev": "astro dev",
    "start": "astro dev",
    "build": "astro check && astro build",
    "preview": "astro preview",
    "astro": "astro"
  },
  "dependencies": {
    "@astrojs/starlight": "^0.13.1",
    "astro": "^3.6.4",
+   "astro-expressive-code": "^0.30.1",
    "sharp": "^0.32.6"
  },
+ "pnpm": {
+   "overrides": {
+     "astro-expressive-code": "^0.30.1"
+   }
+ }
}

After making these changes and running pnpm i, your Starlight project will use the latest astro-expressive-code version immediately.

Please let me know if you encounter any issues. Closing this as completed now.

[Moritz Jung]

I did the version override with npm since i don't use pnpm and it works great. Thanks for implementing this :)

[Moritz Jung]

The only thing I noticed is that the theme seems to be different.

Expressive code on

Expressive code off

[Hippo]

Great to hear that it works for you!

That Expressive Code's Starlight integration comes with a set of new default themes is intended. The previous Starlight default theme was based on the css-variables Shiki theme which only supports very limited grammar rules. With Expressive Code enabled in Starlight, we were no longer restricted to use this limited theme, so we picked a new set of default themes and decided upon Night Owl & Light Owl by Sarah Drasner, a fairly popular set of themes.

You are of course free to pick your own set of dark & light themes and put them into Starlight's expressiveCode: { themes: [...] } config option. If choose your own themes, you can optionally also set useStarlightUiThemeColors: true to automatically tweak your selected themes' background colors to match the rest of your Starlight site, which should allow them to fit as nicely into your site as the default themes.

[Moritz Jung]

Did some more tmlanguage trickery to get my language to run as part of the markdown language and now my docs look mostly the same as my plugin (which the docs are for) 🎉.

Docs:

My Plugin:

[Hippo]

Very nice! Can't wait to see it live!

[Moritz Jung]

And it's live 🎉

[Hippo]

Looks great! Very nice usage of the line & text highlighting as well, this really makes the examples easy to understand.