[Bug]: Vitest mocks not working

[Viktor Luft]

What does the bug relate to?

• Plugin's core
• Debugger

Describe the bug

When mocking an optimized tree-shaked entrypoint in Vitest the mock doesn't work.

E.g. imagine

// util/foo.ts
export const Util = { determineState: () => null };

// util/index.ts
export { Util } from "./foo";

And in the test file:

vi.mock("./util");

Will no longer mock deeply nested functions like "determineState", which works without using the plugin.

---

Moreover, the following used to work in plugin versions prior to 0.4.0.

// util/foo.ts
export const determineState = () => null;

// util/index.ts
export { determineState } from "./foo";

And in the test file:

vi.mock("./util", () => ({ determineState: () => "changed" }));

But now it only works when doing:

vi.mock("./util/foo", () => ({ determineState: () => "changed" }));

To Reproduce

• Use Vite-Config that uses the plugin even when process.env.NODE_ENV === "test"
• Use "vitest" as test runner
• Create file examples as above

Vite version

5.2.7

Additional context

I started to use this plugin also for test execution as it tremendously improves test execution times in our repo. Roughly by factor 20 for many tests.

I can imagine that the plugin is rewriting the code in a way that it replaces mocked instances.

[n028]

Now I have to admit I feel quite dumb because I've never had the idea to use this plugin for tests myself, that sounds like a great idea and I'm really glad about the benefits you've observed around execution times. That said, I really want to thank you for all these tests and detailed issues, they really are helpful ❤️ and I'm really happy to have users like you around, although I feel a bit guilty for making you waste that much time and efforts with the late v0.4.x release and the issues you've ran into, my apologies… :')

Back to the issue

TL;DR

While I do reproduce the problems you mentioned, I really can't figure out how your second scenario could work in older versions because there hasn't been any drastic changes on the way transform logic is being applied on candidates. Still, there definitely are problems around mocking/test environments that I'll need to think of, but right now I'm not 100% sure about the best approach to handle these, and I ended up obsessively trying to understand what could have changed between latest and older versions, I'd be really grateful if you could provide further guidance on how to get the second scenario working with older versions 🙏

Details

Note

You stated in the reproduction steps that it needed a Vite config that runs the plugin even though the NODE_ENV environment variable is set to "test". However, the only determining parameter here should be Vite's execution context (build or serve) which is - I gather - independent from NODE_ENV. AFAIK, "dev-only" plugins should still be picked by Vitest when running test suites, and from all what I've tested, it is! Am I missing something here?

Okay I feel like this one might be a bit trickier. I originally wanted to tackle this step-by-step, starting with what you said was working prior to v0.4.x since it is quite easy to compare two versions' inner working and catch potential regressions that may have occurred in-between. Unfortunately, whatever the scenario is, it seems like I'm always getting the same output with either v0.4.x or older versions zum beispiel - yeah I do like german - e.g. v0.3.3 or v0.3.0. I might be missing something about your own usage, or I even might be misusing Vitest mocks, that wouldn't be the first time huh, please let me know!

Anyway, to illustrate all what I've came up with, I set up an issue-45 branch which includes two examples:

• vitest which directly consumes the v0.4.2 version (through branch's workspace code).
• vitest-old which consumes the v0.3.3 version (though even older versions seem to behave the same).

Each of these examples consist in the following:

Code

• a util/index.ts file which is being handled as an entry by the plugin and exposes two entities from two different modules:

export { Foo } from './foo'; // your original issue
export { determineBarState } from './bar'; // your "moreover" part

• a Foo namespace/object that is re-exported from util/foo.ts which itself exposes a determineFooState function through a VariableDeclaration > ObjectLiteralExpression (this corresponds to your original issue). Basically, you'd invoke such a function either way:

// Recall: utils/foo.ts
export const Foo = { determineFooState: () => null };

// Through intermediate entry file
import { Foo } from './util';
Foo.determineFooState(); // or by destructuring first

// Through direct file
import { Foo } from './util/foo';
Foo.determineFooState(); // or by destructuring first

• a determineBarState function that is re-exported from util/bar.ts which itself exposes that function through a VariableDeclaration > ArrowFunction (this corresponds to your "moreover" scenario). Basically, you'd invoke such a function either way:

// Recall: utils/bar.ts
export const determineBarState = () => null;

// Through intermediate entry file
import { determineBarState } from './util';
determineBarState(); // or using `import * as X` then X.determineBarState();

// Through direct file
import { determineBarState } from './util/bar';
determineBarState(); // or using `import * as X` then X.determineBarState();

Tests

For the record, I've tested the above scenarii with two different variants:

1. directly testing the targeted functions (determineFooState and determineBarState)
2. testing targeted functions with an intermediate module/function (run.ts: runFoo and runBar)
   (Keep in mind that original implementations all return null as described in your repro!)

Tests for both examples include:

• a foo.test.ts: (original issue) this mocks the util/index.ts entry module entirely without specifying any factory, which means all of its exported functions are mocked and therefore transformed as MockFns. As you stated, it doesn't work properly when using the plugin (whatever the version is).
• a bar.test.ts ("moreover" part of this issue, not working): this mocks the util/index.ts, but specifies the determineBarState implementation within a factory: which means the determineBarState function is mocked but not transformed as a MockFn (not a vi.fn() nor a spy) but rather as a plain implementation of the mocked module's function, this doesn't seem to work whatever the version is (but once again, you said it was working before, so we might have a different scenario here).
• a bar-alt.test.ts ("moreover" part of this issue, working), this directly mocks the target module without using the intermediate entry file, this should work both in v0.4.x and older versions, and from what I've tested it does, I'm just keeping it for further references.

Test results (just for record)

v0.4.2

• With plugin 5 failed | 3 passed
• Without plugin 8 passed

v0.3.3

• With plugin 5 failed | 3 passed
• Without plugin 8 passed

Warning

Note to myself: whatever solution I come up with, I should definitely test the scenario where a user reuses original implementation through a factory's importOriginal arg (or maybe I should just cry, pray and bribe our beloved Anthony Fu)

[Viktor Luft]

First of all, I don't feel like putting too much work into it. The plugin provided a lot of benefit to our setup and hopefully becomes part of Vite itself. Also, recommended that to the Vite engineers.

Secondly, I can just give it back ❤️ It feels very refreshing to see package maintainers who really react to issues and are fast to tackle them.

Now into the details:

• I put a comment here where I could find a deviation from my "old" setup
• the null return value has no particular meaning, just wanted to return the simplest thing not being undefined to identify if the function was mocked or not
• I mentioned process.env.NODE_ENV === "test" only as hint, because in our more elaborated vite config, we don't use the plugin for building the application bundle and only activate it for Local-Dev and Cypress Component-Testing and Unit-Testing, but for the sake of reproduction you can configure the plugin unconditionally of course.

And thank you again for taking so much care 😃

[n028]

Thanks a lot for the kind words, it is really motivating :)

That one is hard as I first imagined, I'm already bald but still tearing out my hair out

I've tried to downgrade Vite as you suggested, and couldn't get those bar tests working either :/ I've taken a more drastic approach by progressively downgrading both Vite and Vitest (while syncing Vitest's version with Vite's own codebase per-version to prevent any incompatibility), but still didn't get any successful result! If you have any other ideas on that, I'd be happy to hear them beacause there's definitely something odd here!

I mentioned process.env.NODE_ENV === "test" only as hint, because in our more elaborated vite config, we don't use the plugin for building the application bundle and only activate it for Local-Dev and Cypress Component-Testing and Unit-Testing, but for the sake of reproduction you can configure the plugin unconditionally of course.

It's not required to conditionally setup the plugin based on the environment as its logic only applies to the serve/dev/test context! When building, it's completely ignored thanks to Vite's apply property which is effectively set to serve. Well obviously I don't know what your whole setup looks like, you may already know about this, but I thought it was worth noting just in case it could help :)

Let's roll

To be clear before I keep going with my reasoning, even though I really can't figure out how that could work properly before (see below), I'm really not disclosing by any way the fact that it may have worked before on your side, please don't misread the following as a "you lied to me" haha, definitely not the intention here :D

Still, this is a bit frustrating not being able to figure out what could have changed and I end up wondering if I shouldn't just focus on how to make it work now instead of obsessively look back at past circumstances! I'll detail my reasoning below for my own sake, later reference and in case it just clicks anything on your side. A few notes:

• Right now I'm just focusing on the bar.test.ts scenario

• This is quite detailed, don't feel the need to read it all it probably only helps me BUT
• May I still ask your insight on the "important" annotation I wrote down below?

For context: recall on how the plugin's logic applies

Fundamentally, the logic behind this plugin is quite straight-forward and dumb: okay buddy, please analayse specified entry files, then transform any requested file that imports entities from those entry files to rewrite import statements to be as precise as possible. Basically, it only cares about import/export statements in the codebase, that's it! What happens within vi.mock (and specifically in our case its first argument - path to the module we want mocked) is and was never taken into consideration (I'm 100% confident about this).

Early considerations

The first thing that came to my mind is: well, we're probably both targeting ts extensions (I mean through the plugin's extensions option). It therefore includes *.test.ts files (I've double-checked it to be sure, nothing changed about the way the extensions option is being handled). Basically, in our bar scenario, it means that bar.test.ts is being transformed by the plugin and that this import from ./util therefore gets rewritten:

// Before
import { runBar } from './run';
import { determineBarState } from './util';

// After
import { runBar } from './run';
import { determineBarState } from 'path/to/util/bar.ts';

Obviously, the run.ts is also being transformed:

// Before
import { determineBarState } from './util';

// After
import { determineBarState as determineBarState } from  'path/to/src/util/bar.ts';

Out of curiosity, I've explicitly ignored all /\.test.ts$/ files through the ignorePattern option to see what would happen and well: (1) the second test now passes, (2) but the third one still fails. IMO this very simple test provides a major lead that is worth investigating.

1. Why is the second test now passing?

Right now, I can't tell for sure, but since bar.test.ts is not transformed anymore, the import remains as ./util, which matches our vi.mock's ./util path. Is it a coincidence? I don't think so but we'll need to clarify on this and dig a bit further into it later. Because if it's not a coincidence, there's an additional question:

If the requirement to get it working is that both the imported entity's and the mocked module's paths should match, then are we talking about the "raw" path or the resolved one? This one is easy to answer using an alias for either path, and the result is: it's the resolved path that matters.

To sum it up, in a mock context:

// a == b
import { determineBarState as a } from './util' // Resolves to `./src/util/index.ts`
import { determineBarState as b } from '@util' // Resolves to `./src/util/index.ts`

// a != b even though they actually end up matching the same function
import { determineBarState as a } from './util/bar' // Resolves to `./src/util/bar.ts`
import { determineBarState as b } from './util' // Resolves to `./src/util/index.ts`

Maybe we'll have to further investigate Vitest's mock implementation.

Important

I may be confused because it's being late, but the more I think about this test scenario, the more I feel like it's the closest I can get to your past "it used to work" experience, that would explain a lot and help me sleep better! I first got kinda blind-folded by the third test but I realize it could be completely irrelevant to your own usage (rn I haven't a single clue on how I could handle this), whereas this one seems more plausible!

My only reservation however is that in such a scenario, I'm not sure how you would observe a ~x20 performance improvement on execution times since not transforming test files would require adding all imports' subsequent modules to Vite's dependency graph (though I just probably don't know enough about how Vitest works under the hood). If that scenario is still relevant, one hypothesis is that you test a lot of functions that rely on code exposed by entry files. In other words, tested functions eventually consumes (directly or not) code from an entry-shaken entry file. This plugin's performance implications are hard to estimate because it's all based on a ratio between entry-analysis time and requested module's transform time, this all depends on the codebase

2. Why is the third test still failing?

Based on our previous observation, we'd indeed expect that third test to fail because resolved paths do not match:

// (not transforming test files) a != b even though they actually end up matching the same function
import { determineBarState as a } from './util' // Resolves to `./src/util/index.ts`
import { runBar } from './run' // Resolves to `./src/run.ts`

// (transforming test files) a != b even though they actually end up matching the same function
import { determineBarState as a } from './util/bar' // Resolves to `./src/util/bar.ts`
import { runBar } from './run' // Resolves to `./src/run.ts`

However, this third test passes when not using the plugin, so there's ineluctably something else going on here, it's not all about the resolved paths mismatching within a test file. With those two different considerations, we can safely state that run.ts's own transformation breaks the mocking behaviour. As I would expect, no more excuses, we'll definitely have to investigate Vitest's mock implementation!

Into the wild

…

(I'll complete this tomorrow and in the following days, at least I now have one serious lead to follow, hopefully it isn't a dead end 🙏)

[Viktor Luft]

[...] wondering if I shouldn't just focus on how to make it work now instead of obsessively look back [...]

Yes 😅

[...] one hypothesis is that you test a lot of functions that rely on code exposed by entry files [...]

To be more precise: We have entry-files for the following things:

• UI component library
• UI tools
• Generated UI models / endpoints ~ 3000 different files from 10 entry points
• Backend-Clients exposing ~ 1000 endpoints from 10 entry points

Used across all different places in the 30 different UI applications.

Running a single test using some entry files

• without plugin: 953 ms
• with plugin: 30 ms

Since Vitest builds the dependency graph of all imports from a single test file and this for each test file, the plugin has great impact on the test runs.

Early considerations

I think it was a typo that path to bar.ts was different in both examples? path/to/src/util/bar.ts vs. path/to/util/bar.ts

Afaik the Vitest mock implementation transforms modules maintained by a lookup map where the key matches the passed string of the first argument (the import path that will be used by the rest of the code). Therefore, how you specify the first param and how you import the code has to be identical. Also, the preparation of these modules runs before the rest of the code runs even though you do it as part of the file. Kind of magic they are doing for this. Didn't look into their implementation 🙃

But the key is: If you are rewriting the import paths with the plugin, the lookup using the passed path just doesn't work anymore. Maybe this could be the root cause.

---

And FYI: Even with this bug in place I would still prefer continuing to use your plugin and having to write mocks with strange imports instead of going back.

[n028]

I think it was a typo that path to bar.ts was different in both examples? path/to/src/util/bar.ts vs. path/to/util/bar.ts

That was indeed a typo, good catch!

---

Well I first lost myself in Vitest's source code and it was pretty unnecessary because in the end, it seems to all make sense and I've got my mind a bit clearer about the whole situation. Tbh I'm not sure why I got that much confused in the first place, I probably needed a bit of rest, sorry about that :')

So basically the key thing here is which module is actually being mocked as we both stated in our previous posts. For the example's sake, I'll stick with that util case (and hopefully avoid any typo).

To sum it up

Let's say you're mocking the entry file (util/index.ts) directly in a test file. You would do it using vi.mock and setting its first argument as either an absolute path, a relative path or using aliases, it doesn't really matter as long as it resolves to the module which is solely mocked.

vi.mock('util', { determineBarState: vi.fn(() => 'mocked') });
vi.mock('util/index.ts', { determineBarState: vi.fn(() => 'mocked') });
vi.mock('@util', { determineBarState: vi.fn(() => 'mocked') });
vi.mock('@util/index.ts', { determineBarState: vi.fn(() => 'mocked') });
vi.mock('/path/to/util', { determineBarState: vi.fn(() => 'mocked') });
vi.mock('/path/to/util/index.ts', { determineBarState: vi.fn(() => 'mocked') });

If you're willing to test the determineBarState function (e.g. to assert that it is being called), it needs to be imported from the very same mocked module, otherwise the imported function implementation (i.e. the actual one) doesn't match the one from the mocked module. That's the case whether you're using the plugin or not. Hopefully it makes sense here?

vi.mock('util/index.ts', { /** mocked */ determineBarState: vi.fn(() => 'mocked') })
import { determineBarState as actualModuleFn } from 'util/bar.ts'; // actualModuleFn is not mocked
import { determineBarState as mockedModuleFn } from 'util/index.ts'; // mockedModuleFn is mocked
// actualModuleFn !== mockedModuleFn since `util/index.ts` is mocked but not `util/bar.ts`

In our bar.test.ts, all three tests work without the plugin because they all resolve to the util/index.ts file:

1. determineBarState is directly imported from util/index.ts
2. runBar uses the determineBarState which is imported from the same util/index.ts

With the plugin and util/index.ts set as a target, you'd have:

1. determineBarState import rewritten from util/index.ts to util/bar.ts which is not part of the mocked entry module.
2. runBar imported from run.ts that uses determineBarState originally being imported from util/index.ts but eventually rewritten by the plugin to an import from util/bar.ts which is, again, not part of the mocked entry module.

My understanding is that when mocking+testing, you either need:

One

Not to transform anything in order to prevent unexpected and hard-to-debug behaviours. But somehow it clashes the purpose of this plugin (which is fundamentally designed for dev mode) and would probably hit the performance benefits you've mentioned.

Two

Make sure you explicitly mock the actual final path of the function you're testing (here util/bar.ts). But while it could improve performance using this plugin's logic, this may end up quite hard to maintain because:

• It supposes you have a precise knowledge of your codebase (to make sure you're importing the right path and not the entry when testing any function as explained above). It's the end-user responsibility after all, but still, this may be rather confusing!
• There are so many possible cases where transforming any path through this plugin and relying on mocks could lead to unexpected behaviours: the simplest one being our example's third test that relies on an intermediate run module. I'm not even considering side-effects.

Either way

To properly benefit from this plugin within test files, one have to understand how the plugin behaves, how vite's resolver behaves and have a clear overview of their codebase.

My conclusion

The more I think about this, the more I feel like I should add a note in the README regarding the use of this plugin in a test environment since using it alongside mocks could end up with tests failing when they shouldn't.

I really don't like to give up users' reported needs and do try my best to address them. For example #34 was quite challenging but I eventually came up with a trade-off solution that addresses the original issue. But with this one, unfortunately, I don't see any viable solution. I mean, I considered adding an opt-in behaviour that could transform vi.mock's first argument to reflect and align with plugin's logic but as you'd probably imagine, it's a really bad idea for many reasons.

Es tut mir leid, but once again, I'd be happy to hear your thoughts on this!

[Viktor Luft]

I tend to agree. I don't see how it could be done with automatic transformations of the code without causing other issues downstream and cover all corner cases. If I knew the details on how this plugin worked earlier, I might have already anticipated that this issue is not solvable without changing the vitest-mocking itself.

To keep this plugin simple the following might be good alternatives:

1. Adding a hint in the README about this gotcha
2. (optional) Failing the test run when a registered entry file is tried to be used in a vi.mock() function as first argument. Maybe less challenging, but would fail also if the entry file wasn't just re-exporting stuff. It would just improve the DevX when writing tests, because otherwise the error is hard to understand.

To sum it up: Mach dir keinen Kopf. Your help was highly appreciated and you may close the issue 🙂

[n028]

Hey Viktor, sorry I couldn't reply before!

I don't see how it could be done with automatic transformations of the code without causing other issues downstream and cover all corner cases.

You summed it up perfectly and I'm glad we've came up with the same insight, I'll definitely update the README pretty soon to mention that caveat! I'm a bit disappointed I couldn't find any convenient solution but I'm still really glad you brought such a case to my attention because it's definitely something I've never thought of in the first place!

I'll consider your second option as well but my guess is that it would still require an additional layer that isn't really test runner-agnostic (if that makes any sense). While most Vite users are now probably using Vitest as a test runner, I don't feel like restricting to such an assumption because well… that may not be the case!

Anyway, you're very welcome and once again thank you very much for your help, tests and understanding, really appreciated! I'll probably close the issue once I've updated the README but I'll make sure to add a label and a reference this issue because you've definitely got a great point and the whole discussion was interesting! Danke sehr 🙏