Expected a valid instance of ExpressiveCodeAnnotation, but got ... (`InlineStyleAnnotation`)

[Johan Fagerberg]

Problem description

After upgrading to astro-expressive-code@0.33.4, I'm getting the error

Plugin "Shiki" caused an error in its "performSyntaxAnalysis" hook. Error message: Expected a valid instance of ExpressiveCodeAnnotation, but got {"inlineRange":{"columnStart":0,"columnEnd":88},"renderPhase":"earliest","color":"#99A0A6","italic":false,"bold":false,"underline":false,"styleVariantIndex":0}

when rendering the codeblock

```javascript
// The ASCII characters we can use in our password are represented by char codes 32-126,
// and the sextets we want are in the range 26-51
const asciiCodes = [32..127];
const permittedBits = new Set([26..52].map(i => i.toBinary(wordLength=6)));
// Using this we can recursively generate all the passwords that encode to the wanted format
const generateStrings = (currentString, targetLength) => {
  if (targetLength <= 0) { return [currentString]; }
  return getAllowedCodes(currentString)
    .flatMap(code => {
      const candidateStr = currentString + char(code);
      // abort if it doesn't encode correctly
      const binary = candidateStr.toBinary(wordLength=8);
      for (let i = 0; i + 6 < binary.length; i += 6) {
        if (!permittedBits.has(binary[i..i+6])) { return []; }
      }
      // otherwise dive into this branch
      return generateStrings(candidateStr, targetLength - 1)
    });
};
```

Analysis

After digging around a bit, I've found that this happens when @expressive-code/plugin-shiki attempts to add an InlineStyleAnnotation, whose definition it has imported from its dependency on @expressive-code/core:

expressive-code/packages/@expressive-code/plugin-shiki/src/index.ts

Lines 113 to 126 in 2469749

 codeLines[lineIndex]?.addAnnotation( 
 	new InlineStyleAnnotation({ 
 		styleVariantIndex, 
 		color: token.color || theme.fg, 
 		italic: ((fontStyle & FontStyle.Italic) as FontStyle) === FontStyle.Italic, 
 		bold: ((fontStyle & FontStyle.Bold) as FontStyle) === FontStyle.Bold, 
 		underline: ((fontStyle & FontStyle.Underline) as FontStyle) === FontStyle.Underline, 
 		inlineRange: { 
 			columnStart: charIndex, 
 			columnEnd: tokenEndIndex, 
 		}, 
 		renderPhase: 'earliest', 
 	}) 
 ) 

This instance is then passed to the calling code, and eventually makes its way into @expressive-code/core's validateExpressiveCodeAnnotation, which validates it using instanceof ExpressiveCodeAnnotation:

expressive-code/packages/@expressive-code/core/src/common/annotation.ts

Line 219 in 2469749

 if (!(annotation instanceof ExpressiveCodeAnnotation)) throw 'Not an ExpressiveCodeAnnotation instance' 

Unfortunately there is a subtle trick of package management here, which causes this code to fail its intended function: when plugin-shiki instantiates an InlineStyleAnnotation, it does so based on the class definition imported from its dependency (node_modules/@expressive-code/plugin-shiki/node_modules/@expressive-code/core). When @expressive-code/core (the root dependency, not the dependency from plugin-shiki) runs the instanceof, it does so using the class definition in its own package (node_modules/@expressive-code/core).

Although node_modules/@expressive-code/core and node_modules/@expressive-code/plugin-shiki/node_modules/@expressive-code/core contains the same code, they are not referentially equivalent. For this reason, annotation instanceof ExpressiveCodeAnnotation returns false.

flowchart TD
    subgraph shiki [plugin-shiki]
        subgraph shikicore [core]
            shikiInline[[InlineStyleAnnotation]]
            shikiAnno[[ExpressiveCodeAnnotation]]
            shikiInline -.->|Extends| shikiAnno
        end
        annotation -.->|Instance of| shikiInline
    end
    subgraph core [core]
        anno[[ExpressiveCodeAnnotation]]
        validate(["validateExpressiveCodeAnnotation(annotation)"])

        validate -.->|Checks inheritance of| anno
    end

    annotation -->|Passed to|validate

Loading

[Johan Fagerberg]

Reproduction

When trying to create a reproduction repo for this issue, I found that installing a fresh install of Astro and Expressive Code didn't replicate the issue. When looking at the installed node_modules, there was no node_modules/@expressive-code/plugin-shiki/node_modules.

It looks like this issue depends a lot on whether NPM decides to install @expressive-code/core as a dependency of @expressive-code/plugin-shiki, or keeps it deduplicated in the root node_modules. Usually NPM only does this if the version specified in dependencies of plugin-shiki doesn't match the version specified in the root dependencies, but in my case it looks like it got confused.

One way I've found of replicating this behavior:

1. Check out or create a workspace of the reproduction repo (a simple Astro starter from version ^3.4.3, with astro-expressive-code and @expressive-code/plugin-collapsible-sections on version ^0.26.2)
2. Run npx @astrojs/upgrade (or change the version of Astro in package.json to ^4.4.15 and run npm install). This causes a peer resolution warning, which must happen for NPM to get confused.

   "dependencies": {
   +   "astro": "^4.4.15",
       "astro-expressive-code": "^0.26.2",
       "@expressive-code/plugin-collapsible-sections": "^0.26.2",
       "typescript": "^5.4.2"
   }

3. Change the version of astro-expressive-code to ^0.33.4 while removing @expressive-code/plugin-collapsible-sections. This is the crucial step - upgrading one package while removing an old plugin seems to get NPM confused in the install order.

   "dependencies": {
       "astro": "^4.4.15",
   +   "astro-expressive-code": "^0.33.4",
   -   "@expressive-code/plugin-collapsible-sections": "^0.26.2",
       "typescript": "^5.4.2"
   }

4. Run npm install
5. Verify that node_modules/@expressive-code/plugin-shiki/node_modules exists, and that /test throws the error from this issue when rendered.

Workaround

The workaround looks to be to delete package-lock.json and running npm install again:

rm package-lock.json && rm -rf node_modules && npm install

This seems to force NPM back to its intended functionality, where it installs @expressive-code/core only in the root node_modules, not also in plugin-shiki/node_modules.

I'd still consider this a bug in Expressive Code, since plugin-shiki doesn't work in some valid dependency configurations, but I think it's so low-priority that I'll close the issue for now. Feel free to re-open if you think otherwise.