Support for Project References

[Brad Zacher]

Currently, we do not respect project references - the TS compiler APIs we use do not respect them, and we try to avoid doing tsconfig parsing directly ourselves.

This means that when two projects depended on one-another via project references, we create a program for project A, causing TS to load the .ts files for project A, and .d.ts files for project B. Then we create a program for project B, causing TS to load the .ts files for project B.

This obviously wastes time and memory on duplicated work (the .d.ts files) (#1192), and makes it harder to work on and lint two separate but referenced projects simultaneously, as you have to rebuild one to lint the other (#1973).

We currently do not have a good solution for this right now because the way projects work in TS isn't exposed to us in the APIs that we use. We'll have to build something - though what that is exactly we don't have a clear picture of.

Note that the way we work is different to TS. TS will build and check one project at a time when you run the CLI, whereas we will "build" every single project required to lint your codebase - meaning we are subject to different constrains compared to TS.
Likely there would need to be work done within TS itself so that it can deduplicate and manage things for us.

There are really two ways we can go about this:

1. treat project references as implicit and automatic parserOptions.project entries.
2. work with TS to deduplicate effort by sharing work when project references exist.

[Miroslav Bajtoš]

I came here via #1192 from loopbackio/loopback-next#5590, my primary motivation is to fix out-of-memory errors when linting a sizeable monorepo.

As the first step, I created a small monorepo where I can run and debug typescript-eslint, see https://github.com/bajtos/learn-a. To use a dev version of typescript-eslint, I manually created a symlink from my local clone of typescript-eslint to my playground:

cd learn-a/node_modules/@typescript-eslint
ln -s ~/src/typescript-eslint/packages/typescript-estree .

Then I modified DEFAULT_COMPILER_OPTIONS and added useSourceOfProjectReferenceRedirect: true as recommended by @bradzacher in the issue description.

Finally, I added more logging to getProgramsForProjects to understand what kind of data is returned by program.getResolvedProjectReferences().

What I found: getResolvedProjectReferences returns an array of project references. For each reference, we have sourceFile property of type ts.SourceFile which provides a path to the target tsconfig file (e.g. ./A/tsconfig.json), a list of project references and few other properties.

Example sourceFile (click to expand)

    sourceFile: SourceFileObject {
      pos: 0,
      end: 341,
      flags: 33685504,
      modifierFlagsCache: 0,
      transformFlags: 0,
      parent: undefined,
      kind: 294,
      statements: [Array],
      endOfFileToken: [TokenObject],
      fileName: '/users/bajtos/src/learn-a/packages/pkg3/tsconfig.json',
      text: '{\n' +
        '  "$schema": "http://json.schemastore.org/tsconfig",\n' +
        '  "extends": "../../tsconfig.settings.json",\n' +
        '  "compilerOptions": {\n' +
        '    "composite": true,\n' +
        '    "rootDir": "src",\n' +
        '    "outDir": "dist"\n' +
        '  },\n' +
        '  "include": ["src"],\n' +
        '  "references": [\n' +
        '    {\n' +
        '      "path": "../pkg1/tsconfig.json"\n' +
        '    },\n' +
        '    {\n' +
        '      "path": "../pkg2/tsconfig.json"\n' +
        '    }\n' +
        '  ]\n' +
        '}\n',
      languageVersion: 2,
      languageVariant: 1,
      scriptKind: 6,
      isDeclarationFile: false,
      hasNoDefaultLib: false,
      externalModuleIndicator: undefined,
      bindDiagnostics: [],
      bindSuggestionDiagnostics: undefined,
      nodeCount: 37,
      identifierCount: 0,
      identifiers: [Map],
      parseDiagnostics: [],
      referencedFiles: [],
      typeReferenceDirectives: [],
      libReferenceDirectives: [],
      amdDependencies: [],
      pragmas: Map(0) {},
      version: '055a6ddc0d7ff3d20781fe161599a0cdc0af42192771b2ec70393e240bdbd06e',
      extendedSourceFiles: [Array],
      path: '/users/bajtos/src/learn-a/packages/pkg3/tsconfig.json',
      resolvedPath: '/users/bajtos/src/learn-a/packages/pkg3/tsconfig.json',
      originalFileName: '/users/bajtos/src/learn-a/packages/pkg3/tsconfig.json'
    },
    references: [ [Object], [Object] ]

As I understand this data structure, it allows us (typescript-estree) to automatically add referenced projects to the list of projects we are aware of. For example, if we have a monorepo with root-level tsconfig.json that's referencing packages/core/tsconfig.json and packages/app/tsconfig.json, then we can implement a feature where it's enough to provide project: 'tsconfig.json in eslint config file and eslint can automatically learn about core and app sub-projects.

However, it's not clear to me how can this improve eslint's performance and avoid out-of-memory errors. As I understand the current state, there is no API that would allow us to obtain the already-created ts.Program instance for a referenced project. All we have is a path to the referenced tsconfig file, therefore we have to call createWatchProgram to create a new program instance from scratch. This is pretty much the same behavior as we already have, it does not solve the performance issue.

It seems to me that we need few more improvements in TypeScript API, specifically we need to include getProgram()-like member in ts.ResolvedProjectReference type. Unless there is an existing API that I am not aware of, one that would allow us to obtain a ts.Program instance for a referenced project from the parent ts.Program/ts.BuilderProgram/ts.WatchOfConfigFile<ts.BuilderProgram> instance.

@bradzacher what's your take on this? Do you have any more knowledge than would help me to find a way how to implement fix for out-of-memory errors? Would it make sense to send a small pull request to enable useSourceOfProjectReferenceRedirect, as the first incremental step? Does it make sense to refactor getProgramsForProjects to use getResolvedProjectReferences when such change is not going to solve the out-of-memory issue?

[Brad Zacher]

This is my understanding of how it works:

Assume project A references project B, and a file X.ts in project A that references a file Y.ts in project B.

When we ask typescript to gather types for X.ts, it somehow has to get the types for Y.ts as well.
Without project references, that means TS will reference the Y.d.ts file for Y.ts.
If we then ask typescript to gather types for Y.ts, there won't be any record of Y.ts, so it will then gather the types for it separately.

So in memory we now have the following: X.ts, Y.d.ts and Y.ts. Notice the duplication of Y. Obviously Y.d.ts will be "lighter" than Y.ts because it only contains exported types, but it's still a decent chunk of memory, and it takes time to parse the file.

Now if we extrapolate this basic example into a fully fledged project, where Y.ts has some dependencies, that has some dependencies, etc - you'll find that essentially you've wasted time and memory parsing and storing two entire copies of project B - the .d.ts version, and the .ts version.

---

From my understanding based on what @uniqueiniquity has told me, useSourceOfProjectReferenceRedirect does two things:

1. it makes it so that when we ask typescript for project A, it will do a complete parse of project B - the .ts version, not the .d.ts version.
2. it makes it so that if we ask the program for project A for Y.ts, it will search the dependent projects and provide us with the type information.

This means that we don't ever have to manually parse project B, as it's been implicitly parsed for us.

---

I've been doing some thinking about this and was going to start on an implementation, but then ESLint merged their optional chaining representation, so updating to support that has taken priority.

Working this functionality into the existing code path will be problematic.

To illustrate - another example. Extending the above example to include a 3rd project - project C, which has no dependencies on A or B. Imagine the user has passed us parserOptions.project = ['A/tsconfig.json', 'B/tsconfig.json', 'C/tsconfig.json'].

When we attempt to parse a file from A, we will load the program for A (which also loads the program for B), and it will find the file and return the result.
When we attempt to parse a file from B, we will load the program for A (which also loads the program for B), and it will find the file and return the result.
When we attempt to parse a file from C, we will load the program for A (which also loads the program for B), and we won't find anything. We will then continue through the array, and separately load the program for B, and we won't find anything. We will then continue through the array, and load the program for C, and it will find the file and return the result.

Note here that because of the way the user specified the config, we have now made the memory and performance substantially worse because we've parsed the .ts version of B twice (before we had a .ts and .d.ts version, which is less memory).

So we have two options:

1. implement checks to understand that the user's manually specified B/tsconfig.json entry has been duplicated, so we can ignore it completely.
2. implement a better way of resolving projects that requires config that isn't as error-prone.

It might seem like (1) is the best option because it requires the smallest change, however I know that some users have odd setups where they might do something instead like specify B/tsconfig.eslint.json, whilst A references B/tsconfig.json. If they do something like this, there's no way for us to determine that B/tsconfig.json references the same files as B/tsconfig.eslint.json (because they may not be exactly the same, and they're different files).

So I think that the best course of action here is to implement this functionality behind a brand new configuration style.

I was thinking that we can add a new variant of parserOptions.project - the value true.

When parserOptions.project === true, we shall attempt to automatically find the tsconfig.json for the given file.

This can be done via traversing up the directory tree looking for either a tsconfig.eslint.json or a tsconfig.json. Each time we find one of those, we shall load it into memory, and check if it matches the given file. If it doesn't, we continue searching until we reach tsconfigRootDir.

Side note - typescript has this functionality built into the TS language server, and it's that logic that VSCode (and other IDEs) use to find the tsconfigs. Unfortunately that code is not exposed in a way that we can consume, which means that we can either: (a) PR typescript to expose the logic and wait for it to be released (earliest would be 4.1.0 in ~4 months), (b) implement this logic ourselves.

Now that we are automatically resolving the tsconfig, we now have complete control over what we load.

The algorithm would be as follows:

1. eslint asks to parse file X.ts
2. if parserOptions.project !== true - perform the old codepath (i.e. exit this algorithm)
3. if parserOptions.project === true - perform the new codepath
4. check the user's typescript version
   1. if they are on < TS3.9, throw an error and exit
5. iterate through our known program map. For each program:
   1. check to see if the program contains X.ts
      1. if it does - return the matching program and exit
6. traverse up the directory tree. For each folder:
   1. look for a tsconfig.eslint.json in the folder, if it exists, then
      1. create a program for it, and check if it contains X.ts
         1. if the program contains X.ts, return the program and exit
         2. if the program does not contain X.ts, continue traversing
   2. look for a tsconfig.json in the folder, if it exists, then
      1. create a program for it, and check if it contains X.ts
         1. if the program contains X.ts, return the program and exit
         2. if the program does not contain X.ts, continue traversing
   3. if folder === parserOptions.tsconfigRootDir, then throw an error and exit

[Miroslav Bajtoš]

@bradzacher Thank you for a detailed explanation! ❤️

useSourceOfProjectReferenceRedirect (...) makes it so that if we ask the program for project A for Y.ts, it will search the dependent projects and provide us with the type information.
This means that we don't ever have to manually parse project B, as it's been implicitly parsed for us.

Ah, this is the piece I was missing! I'll try to find some time next week 🤞🏻 to verify this behavior in my playground.

Based on this information, it makes me wonder if we need to deal with the complexity of detecting duplicate projects. Let me explain an alternative I am thinking of.

In loopback-next, we have a top-level tsconfig.json file which has project references for all monorepo sub-projects. This is necessary to allow a single-step build (tsc -b .) and watch mode for entire monorepo (tsc -b -w .).

• tsconfig.json references A/tsconfig.json and B/tsconfig.json
• A/tsconfig.json references B/tsconfig.json
• (We also have a script to automatically update project references based on package.json dependencies/devDependencies, it's executed from post-install hook.)

Now if useSourceOfProjectReferenceRedirect behaves as described, then it should be enough to configure eslint with a single TypeScript project - the top-level tsconfig referencing all monorepo sub-projects (project: './tsconfig.json) and let the compiler load all referenced projects automatically.

So I am thinking that perhaps we can tell typescript-eslint users to use the same approach when linting monorepos based on project references?

However I know that some users have odd setups where they might do something instead like specify B/tsconfig.eslint.json, whilst A references B/tsconfig.json. If they do something like this, there's no way for us to determine that B/tsconfig.json references the same files as B/tsconfig.eslint.json (because they may not be exactly the same, and they're different files).

In my proposal, I would ask such users to create top-level tsconfig.eslint.json file that's referencing {A,B}/tsconfig.eslint.json files and also create {A}/tsconfig.eslint.json file to reference {B}/tsconfig.eslint.json file. While maintaining two sets of tsconfig files may be a bit more work, this should be easy to automate and has the benefits of keeping the user in full control of how TypeScript projects are resolved by eslint.

[Brad Zacher]

So I am thinking that perhaps we can tell typescript-eslint users to use the same approach when linting monorepos based on project references?

A few reasons that I don't think this is the best approach:

1
If there's one thing I've learned in my time as maintainer - it's that a large portion of users do not read the documentation, so they will set it up either (a) however they want via trial and error or (b) based on some 2 year old medium article.
By this I mean that if you make it so users can misconfigure the tooling, they will misconfigure it and raise issues.

2
Lumping everything into a single "solution" project means we cannot split or defer the work at all. This means that if you only want to check files from project A, we'll be forced to first parse and typecheck all files from all projects in the workspace before eslint can even run the linter over a single file from A.

We have an open issue to implement some purging mechanism so that we can purge projects from memory once we've parsed/linted every file (#1718). But this purging would be based on the tsconfigs - if the user only has one tsconfig, then we can't ever purge it.

This also leads into...

3
Memory usage.
For a project of your size, this might be a great solution! But for larger projects, it might not be feasible to load every single project into memory at once (#1192).

I know some larger projects have had to resort to splitting their lint runs per project instead of doing a single monorepo run because they just have so many projects or so much code.

Yes, this particular issue will likely alleviate this problem for a chunk of those users, but no doubt some of them will still be too large for that approach to work.

[Miroslav Bajtoš]

if you make it so users can misconfigure the tooling, they will misconfigure it and raise issues.

Yeah, I am not surprised that people often misconfigure their projects and then report "silly" issues that are wasting maintainers' time. It makes me wonder if this concern could be alleviated by detecting a problematic setup and reporting a helpful warning or error message?

Lumping everything into a single "solution" project means we cannot split or defer the work at all. This means that if you only want to check files from project A, we'll be forced to first parse and typecheck all files from all projects in the workspace before eslint can even run the linter over a single file from A.

I see, this is a valid argument 👍🏻

3
Memory usage.
For a project of your size, this might be a great solution! But for larger projects, it might not be feasible to load every single project into memory at once (#1192).

At the moment, we have an eslint-specific TypeScript project covering the entire monorepo and eslint works fine this way (even if it's a bit slow).

When I tried to rework eslint configs to use existing TypeScript projects we have for each package, I run into OOM as described in #1192. I guess we are on the edge between projects that are small enough to be handled at once and projects that are way too large to fit into memory.

My primary motivation was to get rid of createDefaultProgram option that was slowing our linting time, and secondary I want to have a cleaner tsconfig/eslint config. After hitting OOM errors in loopbackio/loopback-next#5983, I realized I can get rid of createDefaultProgram while keeping a single monorepo-level TypeScript project (loopbackio/loopback-next#5590). So the lack of support for project references is no longer a pressing issue for us.

I am happy to leave this feature up to you @bradzacher to implement, you have clearly much better understanding of the problem and the landscape.

[Avi Vahl]

This issue came up as a possible blocker to move to project references several times on my team.
We really want to use project references, but the current behavior of project references with typescript-eslint leaves a lot to be desired.

perhaps @DanielRosenwasser @andrewbranch @sheetalkamat can assist? I know the typescript team is also using typescript-eslint in their repository, which contains project references.

[Oliver Joseph Ash]

Is there any way to workaround the lack of support for project references? I did see this in the docs:

If you use project references, TypeScript will not automatically use project references to resolve files. This means that you will have to add each referenced tsconfig to the project field either separately, or via a glob.

https://github.com/typescript-eslint/typescript-eslint/blob/9c3c686b59b4b8fd02c479a534b5ca9b33c5ff40/packages/parser/README.md#user-content-parseroptionsproject:~:text=If%20you%20use%20project%20references%2C%20TypeScript,either%20separately%2C%20or%20via%20a%20glob.

I am following this advice however I still get unexpected errors when I try to run ESLint. Here is a reduced test case: https://github.com/OliverJAsh/typescript-eslint-project-references.

$ eslint app/main.ts

/Users/oliverash/Development/typescript-eslint-project-references/app/main.ts
  3:5  error  Unexpected any value in conditional. An explicit comparison or type cast is required  @typescript-eslint/strict-boolean-expressions

✖ 1 problem (1 error, 0 warnings)

Note that if I run tsc --build app/tsconfig.json before linting then it succeeds. However I didn't expect I would need to do this, because I'm including all referenced projects in the parserOptions.project setting, as recommended in the docs. Furthermore, this means I have to run tsc --build every time I make changes to referenced files, which means we won't get accurate errors inside the IDE as we're making unsaved changes.

I'm happy to open a separate issue if you think it's necessary. I just decided to post here because I saw a lot of the issues surrounding project references were merged into this one.

[Avi Vahl]

As a first step, we could add:
watchCompilerHost.useSourceOfProjectReferenceRedirect = () => true;
to createWatchProgram.ts (src of @typescript-eslint/typescript-estree/dist/create-program/createWatchProgram.js)

Right before the call to:
const watch = ts.createWatchProgram(watchCompilerHost);

This will give memory-wasteful project reference support. One could argue that's much better than having errors for that same scenario.
That initial step will also allow vscode-eslint to not go bananas if you didn't build a referenced project, and will work in-memory, just like type checking.

It'll probably go OOM rather fast in large references scenarios. But that could be later optimized with algorithms like the one @bradzacher suggested.

I doubt it'll support root tsconfigs with files: [] and only "references", but still, a crucial first step which can be further built upon.

EDIT: just to clarify, I'm not a maintainer here or anything... The above is only a suggestion.

[Sheetal Nandi]

I had discussed with @uniqueiniquity that useSourceOfProjectReferenceRedirect might be better way especially for editing scenarios where we use this now. microsoft/TypeScript#37370 was specifically done for that reason. Howeve, for command line you would want to ensure that you dont use that flag so you use .d.ts instead. @uniqueiniquity was going to look into this further so he would be best to comment on what he has researched.

[Avi Vahl]

When I tested useSourceOfProjectReferenceRedirect locally, on two different mono-repos with project references, it worked without building even when I executed eslint from the command line.

That was awesome. That flag is not necessarily only suited for the editing experience.

I did have to point typescript-eslint to all the tsconfigs, as it doesn't have the logic to follow and track the references yet, but the experience itself, both in vscode and in the command line, was great (with the flag in-place).

[Brad Zacher]

Quick-and-dirty exposure of a flag to test this feature #2669

[Brad Zacher]

I just merged #2669, it'll be live on the canary tag shortly.
Give it a go and LMK what sort of improvements you see.

Turning it on should just require setting parserOptions.EXPERIMENTAL_useSourceOfProjectReferenceRedirect = true.

Note that this implementation contains no safe-guards for the user, so you will need to be aware of your project tree and configure accordingly to remove duplicates.

[Avi Vahl]

Tested 4.4.2-alpha.1 in https://github.com/AviVahl/ts-tools
Added the EXPERIMENTAL_useSourceOfProjectReferenceRedirect flag, and everything started working without me having to build.
Running eslint just worked, and editing in the IDE stopped showing errors. 👍

One small annoyance that I noticed, is that I couldn't specify "project": "./tsconfig.json" (which references all other tsconfigs), and had to leave it as "project": "packages/*/{src,test}/tsconfig.json".

EDIT: oh, and the "peerDependency" range of the packages doesn't include the alpha versions, so installing canary using npm@7 required passing --legacy-peer-deps to npm.

[Avi Vahl]

@bradzacher this flag is going to make my coding experience so much nicer. thank you so much.

Regarding the root tsconfig.json issue, I've checked the typescript api and behavior, and it seems possible to extract all the referenced tsconfigs paths from the provided ones in "project". Having to give it custom globs makes it a bit coupled to the inner project structure and harder to manage. Would be great if it could pick up and parse those referenced tsconfigs as well, and get the rootNames of each config for a possible future Program initialization (when a file in one of them is being linted).

Should probably be noted that those paths can be ./packages/my-package/src or ./packages/my-package/src/tsconfig.json (typescript understands both).

[Brad Zacher]

Yeah, there's the program.getResolvedProjectReferences() call which we can use to get all of the information. I believe this even has the resolved references to the tsconfig paths in it.

[Avi Vahl]

Here's a generator to make life easier:

/**
 * @param {readonly (ts.ResolvedProjectReference | undefined)[] | undefined} references
 * @param {Set<ts.ResolvedProjectReference>} visited
 * @returns {Generator<ts.ResolvedProjectReference, void, unknown>}
 */
function* filterReferences(references, visited = new Set()) {
  if (!references) {
    return;
  }
  for (const ref of references) {
    if (!ref || visited.has(ref)) {
      continue;
    }
    visited.add(ref);
    yield ref;
    yield* filterReferences(ref.references, visited);
  }
}

Now you can just:

  for (const ref of filterReferences(
    watchProgram.getProgram().getProgram().getResolvedProjectReferences()
  )) {
    // extract needed data from ref (rootName, config path, etc) 
  }

Handles duplicates and cyclic. ignores undefined.

[Oliver Joseph Ash]

I can report that enabling the experimental flag fixed all of the issues I ran into here: #2094 (comment).

[Brad Zacher]

I spent some time playing around with this feature and it's certainly non-trivial and has foot-guns.

In this repo we have two tsconfigs per folder:

• a tsconfig.build.json which specifies the config/files specifically for a build and nothing more
• a tsconfig.json which specifies the config/files, including test/tooling files.

Our ESLint config references the tsconfig.json in package, because we want to lint the source, tests and tools.
With #2669 I added references from each of our tsconfig.json to the relevant tsconfig.build.json.
This worked fine - typescript was able to resolve the .d.ts files to the .ts files correctly.

I was just playing around with de-duplication. I first tried updating each of the tsconfig.jsons to reference the other tsconfig.jsons (instead of the build variant).
When I ran the lint again, typescript was no longer able to resolve the .d.ts files to the .ts files.

(below is all just spitballing - I have no real idea how TS works under the hood)

I believe that this is because of the rootDir compiler option. In the build variant, the rootDir is set to src. In the non-build variant, the rootDir is set to ., so that it encompases our tests folder as well.
This compiler option affects the compiled output paths. I believe that TS no longer knows how to resolve dist/index.d.ts to src/index.ts, because the new rootDir would imply that the mapping would instead be to dist/src/index.d.ts.

I spent some time playing around with various combinations of references.
I tried converting the tsconfig.json to a "solution" file (a tsconfig with no include and only references), and adding separate tsconfig.jsons for the tests and tools directories. This didn't work.
I believe that it didn't work because typescript acts lazily - it does not create a program for a project if none of the files within the project are referenced (i.e. a project reference to B is only used when project A is found to import a file from B). Because there are no files in a solution project, there are no imports, thus there is no need to create a program for the referenced project.

If I'm right, then this represents a large issue for setting up this feature without any footguns! Why? Because we cannot trust that every file from every project in getResolvedProjectReferences has actually been included in the program.

There is still a lot of investigation and work required before we make this feature public.
I'd appreciate any insights from someone like @uniqueiniquity here, as I'm really just exploring in the dark 😅

For reference, these are the changes I was testing against:
https://github.com/typescript-eslint/typescript-eslint/compare/experimental-remove-footguns-for-project-refs

[Sheetal Nandi]

If I'm right, then this represents a large issue for setting up this feature without any footguns! Why? Because we cannot trust that every file from every project in getResolvedProjectReferences has actually been included in the program.

This is correct. Just because you add project reference does not mean that all the files from referenced project will be in your program.

One hint in determining which program file belongs to is checking resolvedReferenced.commandLine.fileNames. if project is composite (in all cases except for solution, referenced projects have to be composite) the list of filenames is the actual program files. rest are all .d.ts files that dont need emit.

I am not very familiar with what this repo does for project management as i have worked on this code base only once in past, but i will chat with @uniqueiniquity and we will post what comes out of that chat.

[Brad Zacher]

I am not very familiar with what this repo does for project management

It does nothing at all! This is our first foray into understanding project references.

[Joe Pea]

For stumblers, follow the steps in #890 if you have project references (for now).

The gist: make sure parserOptions.projects includes all tsconfig.json files that reference all your TS files.

[Simen Bekkhus]

FWIW, I tried enabling type info in the Jest monorepo, quite quickly running into an OOM (without enabling any rules we don't already run)

We get an OOM both with and without the EXPERIMENTAL_useSourceOfProjectReferenceRedirect flag. Probably not super helpful, but I can post the stacks I got (afaict no JS stack in either)

Without experimental

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1012d8685 node::Abort() (.cold.1) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 2: 0x1000a6309 node::Abort() [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 3: 0x1000a646f node::OnFatalError(char const*, char const*) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 4: 0x1001e8cc7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 5: 0x1001e8c63 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 6: 0x100395b65 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 7: 0x10039760a v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 8: 0x100392d35 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 9: 0x100390660 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
10: 0x10039ed4a v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
11: 0x10039edd1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
12: 0x10036cea7 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
13: 0x1006ebd38 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
14: 0x100a71679 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
15: 0x1743627f8562
16: 0x174362bbbf6f
17: 0x174362d144ec
18: 0x1743627d48c5

With experimental

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1012d8685 node::Abort() (.cold.1) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 2: 0x1000a6309 node::Abort() [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 3: 0x1000a646f node::OnFatalError(char const*, char const*) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 4: 0x1001e8cc7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 5: 0x1001e8c63 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 6: 0x100395b65 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 7: 0x10039760a v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 8: 0x100392d35 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
 9: 0x100390660 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
10: 0x10039ed4a v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
11: 0x10039edd1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
12: 0x10036cea7 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
13: 0x1006ebd38 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
14: 0x100a71679 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/simen/.nvm/versions/node/v14.15.3/bin/node]
15: 0x2379e31c37c5

Config I added:

parserOptions: {
  project: ['./tsconfig.json', './packages/*/tsconfig.json'],
  sourceType: 'module',
  tsconfigRootDir: __dirname,
},

[Rémi Alvergnat]

Is there any plan to rename EXPERIMENTAL_useSourceOfProjectReferenceRedirect flag to useSourceOfProjectReferenceRedirect ? To let you know, I have enabled the flag in my typescript monorepo and it's working as expected, I can run eslint without building anything before.

[Brad Zacher]

No, because it's not ready for production use for the most part. There are a lot of cases where this will break the lint types or cause OOMs.

[swandir]

Project references handling affects not only performance, but correctness too.

Currently, type-aware rules may change behavior depending on whether the project is built or not (whether .d.ts files are available).

Enabling EXPERIMENTAL_useSourceOfProjectReferenceRedirect makes the behavior consistent.

Here's a repro:
https://github.com/swandir/typescript-eslint-project-references
@typescript-eslint/no-unnecessary-condition is affected by difference in strictNullChecks between two TS projects and does not report errors if projects are not built.

[swandir]

Enabling EXPERIMENTAL_useSourceOfProjectReferenceRedirect makes the behavior consistent.

Though I'm not sure what exactly is going on here.
Could it be an unrelated issue?

[Jack Works]

Also tried this, gets OOM in our project. Here is our project https://github.com/DimensionDev/Maskbook/tree/eslint-proj-ref if anyone is interested.

Note: our project contains 70 tsconfigs

👀

[Cefn Hoile]

The experimental flag doesn't resolve eslint's tsconfig complaints for me, in a repo where tsc --build works (project references are functional) but eslint doesn't follow the references.

Project references look like this...

tsconfig.json -> packages/sum/tsconfig.json -> [ ./tsconfig.esm.json, ./tsconfig.cjs.json ]

...and the rootDir and include definitions are at the 'leaf' tsconfigs currently which are references-of-references see 5b310e4

I get an error as follows that points me to a non-existent help page...

Parsing error: ESLint was configured to run on `<tsconfigRootDir>/packages/sum/src/index.ts` using `parserOptions.project`: <tsconfigRootDir>/tsconfig.json
However, that TSConfig does not include this file. Either:
- Change ESLint's list of included files to not include this file
- Change that TSConfig to include this file
- Create a new TSConfig that includes this file and include it in your parserOptions.project
See the TypeScript ESLint docs for more info: https://typescript-eslint.io/docs/linting/troubleshooting##i-get-errors-telling-me-eslint-was-configured-to-run--however-that-tsconfig-does-not--none-of-those-tsconfigs-include-this-file

The eslint config is https://github.com/cefn/starter/blob/5b310e4636d0fb0b0f7b25dea903d2c721ba1503/.eslintrc.json and includes parserOptions "EXPERIMENTAL_useSourceOfProjectReferenceRedirect":true

[swandir]

Yeah, the flag does not ensure real project references support.

Although, it seems that the parser picks the first tsconfig listed in parserOptions.project that matched the file being processed. Given that, placing leaf tsconfigs before the ones which depend on them might serve as a workaround.

[Patrick Roza]

I use the flag but with an eslint config per project and it seems to work fine. I use js formatted configs, place a .eslintrc.base.js in root and share it between my projects

…

________________________________ From: swandir ***@***.***> Sent: Friday, December 9, 2022 3:20:49 AM To: typescript-eslint/typescript-eslint ***@***.***> Cc: Patrick Roza ***@***.***>; Manual ***@***.***> Subject: Re: [typescript-eslint/typescript-eslint] Support for Project References (#2094) Yeah, the flag does not ensure real project references support. Although, it seems that the parser picks the first tsconfig listed in parserOptions.project that matched the file being processed. Given that, placing leaf tsconfigs before the ones which depend on them might serve as a workaround. — Reply to this email directly, view it on GitHub<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftypescript-eslint%2Ftypescript-eslint%2Fissues%2F2094%23issuecomment-1343753552&data=05%7C01%7C%7Cf534499809d94aba855908dad98bff2b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638061492556551678%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YLsXQKMuTN8QhjykHy9%2FyI25NJ%2FHwmnDtOXkzZGh4ic%3D&reserved=0>, or unsubscribe<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAAKNJPP6S4AHG4X756B6WDWMKJQDANCNFSM4NJSSTNA&data=05%7C01%7C%7Cf534499809d94aba855908dad98bff2b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638061492556551678%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YlK5ORR7g5xrPFPwe%2FIb0eCs3PdLwWYaGmHtzRfVrVM%3D&reserved=0>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Cefn Hoile]

Thanks @swandir not sure what you mean by the first tsconfig...that matched as the only ones that match are the leaf configs (inside the project, which explicitly define the ESM and CommonJS output steps). Everything above them has an empty file list, so I wouldn't expect the traversal to stop there. https://github.com/cefn/starter/blob/5b310e4636d0fb0b0f7b25dea903d2c721ba1503/packages/multiply/tsconfig.esm.json#L3

Thanks @patroza for now I seem to be getting by with just a top-level eslintrc but maybe I can expect to hit some problems in my pipeline from trying to do this, or maybe it's less efficient somehow? I guess I can override per-project when there's framework-specific behaviours needed, but so far I've found all these work together (e.g. I can afford to have all the Vue rules in a top-level definition even where there are React projects in scope).

In the end I removed the flag and added a wildcarded pattern for project paths, which works well...

https://github.com/cefn/starter/blob/562a709922de4d86c0849c07c46c12941012e17b/.eslintrc.json#L21

[swandir]

Well, I'm not completely sure how matching works exactly too 😅. But I noticed that reordering tsconfigs in parsetOptions.project helps at least in some cases.

[Shogun Sea]

Could we please call out this in https://typescript-eslint.io/linting/typed-linting or the faq page? I spent a whole day debugging why tslint doesn't seem to honor all the references field in my tsconfig and eventually saw this issue 😢

update: and using EXPERIMENTAL_useSourceOfProjectReferenceRedirect: true, fixed the problem I was having in a mono repo with a lot of packages.

[Josh Goldberg ✨]

Eep, sad to hear that @shogunsea! Good idea on calling it out in FAQs. Filed #6363, +1.

[Shogun Sea]

Follow up on #2094 (comment)
Interestingly, the behavior of using using EXPERIMENTAL_useSourceOfProjectReferenceRedirect: true is same as commenting out all the references field from tsconfig, I did some search and found this article: https://turbo.build/blog/you-might-not-need-typescript-project-references

As it turns out, the TypeScript Language Server (in VSCode) and Type Checker can treat both a raw .ts or .tsx file as its own valid type declaration

So I'm guessing that's ^ what was happening under the hood, e.g. if we just set references to [], type checker just use whatever type information it found in its dependency modules through normal node module resolution.
So maybe one recommended workaround of this is just to have a different tsconfig with references field removed...? but it does seem like this will slow down the build time.
could someone confirm if this understanding is correct or not

[Alexander]

I have to disable this 5 rules on our monorepo setup:

    // Too much incorrect detections
    "@typescript-eslint/no-unsafe-assignment": ["off"],
    "@typescript-eslint/no-unsafe-return": ["off"],
    "@typescript-eslint/no-unsafe-member-access": ["off"],
    "@typescript-eslint/no-unsafe-call": ["off"],
    "@typescript-eslint/restrict-template-expressions": ["off"],

error, code, configs

Unsafe call of an `any` typed value.eslint@typescript-eslint/no-unsafe-call
(alias) typedKeys<T, keyof T>(collection: T): (keyof T)[]

server/lib/monitor/filtering.ts:

import { typedKeys } from "shared/utils/collections";
// ...
function withoutNullishAndEmptyStr<T extends GenericRecord>(obj: Nullable<T>): Either<typeof obj, T> {
  const res = {} as T;
  if (!checkNonNullable(obj)) return undefined as Either<typeof obj, T>;

  for (const key of typedKeys<T>(obj)) {
    if (obj[key] == null || obj[key] === "") continue;
    res[key] = obj[key] as T[keyof T];
  }

  return res;
}

shared/utils/collections.ts:

export function typedKeys<T extends GenericRecord, K = keyof T>(collection: T): K[] {
  return Object.keys(collection) as K[];
}

server/.eslintrc.cjs:

module.exports = {
  extends: "../.eslintrc.cjs",
  parserOptions: {
    project: "tsconfig.json",
    tsconfigRootDir: __dirname,
    sourceType: "module",
  },
};

I tried project: ["./tsconfig.json", "../shared/tsconfig.json"] with no effect

server/tsconfig.json:

{
  "compilerOptions": {
    "composite": true,

    // enable latest features
    "lib": ["esnext"],
    "module": "esnext",
    "target": "esnext",

    // if TS 5.x+
    // https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-0.html#--moduleresolution-bundler
    "moduleResolution": "bundler",
    "allowImportingTsExtensions": true,
    "noEmit": true,
    "moduleDetection": "force",

    // "jsx": "react-jsx", // support JSX
    "allowJs": true, // allow importing `.js` from `.ts`
    "esModuleInterop": true, // allow default imports for CommonJS modules

    // best practices
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "skipLibCheck": true,

    "paths": {
      "shared/*": ["../shared/*"],
      "@front/*": ["../src/*"]
    },
    "baseUrl": ".",
    "plugins": [
      { "name": "typescript-styled-plugin", "validate": false },
      // Fix import absolute paths
      { "transform": "typescript-transform-paths", "useRootDirs": true } // Transform paths in output .js files
      // { "transform": "typescript-transform-paths", "useRootDirs": true, "afterDeclarations": true } // Transform paths in output .d.ts files (Include this line if you output declarations files)
    ],
    "rootDirs": [".", "../src", "../shared"],
    "types": ["jest"],
    "resolveJsonModule": true
  },
  "ts-node": { /*...*/ },
  "include": [
    "./**/*",
    "./.eslintrc.cjs",
    "../jest.config.ts",
    // Don't include: "../src/utils/**/*", As it have browser specific, like utils/hooks.ts
    "../src/**/types.ts",
    "../src/**/config.ts",
    "../src/**/*.types.ts",
    "../src/**/*.d.ts",
    "../shared/**/*"
  ],
  "references": [{ "path": "../shared/tsconfig.json" }, { "path": "../src/tsconfig.json" }]
}

.eslintrc.cjs:

module.exports = {
  root: true,
  extends: [
    "react-app",
    "react-app/jest",
    "plugin:@typescript-eslint/recommended",
    "plugin:@typescript-eslint/recommended-requiring-type-checking",
  ],
  parser: "@typescript-eslint/parser",

  plugins: ["@typescript-eslint", "unused-imports"],

  rules: {
    "max-len": ["warn", { code: 120, ignoreComments: true, ignoreUrls: true }],
    indent: [
      "warn",
      2,
      {
        SwitchCase: 1,
        offsetTernaryExpressions: true,
      },
    ],
    "@typescript-eslint/ban-ts-ignore": "off", // It deprecated and renamed to @typescript-eslint/ban-ts-comment
    "react-hooks/exhaustive-deps": [
      "warn",
      {
        additionalHooks:
          "(useAsyncEffect|useCallbackDebounce|useCallbackThrottle|useDidUpdate|useMemoVal|useSyncRefState)",
      },
    ],

    //
    // Setup unused-imports
    "@typescript-eslint/no-unused-vars": ["off"],
    "no-unused-vars": "off", // or "@typescript-eslint/no-unused-vars": "off",
    "unused-imports/no-unused-imports": "error",
    "unused-imports/no-unused-vars": [
      "warn",
      {
        vars: "all",
        varsIgnorePattern: "^_",
        args: "after-used",
        argsIgnorePattern: "^_\\d?$",
        ignoreRestSiblings: true,
      },
    ],
  },
};

---

versions (latest)

node v16.16.0
"typescript": "^5.0.4",
"eslint": "^8.39.0",
"@typescript-eslint/eslint-plugin": "^5.59.1",
"@typescript-eslint/parser": "^5.59.1",

[Ernesto Stifano]

Hello guys, I understand that supporting TS Project References goes well beyond this, but is there a half-way solution to be able to lint code in a monorepo without needing to build packages? In other scenarios (like with Jest and Webpack) we use aliases to point to the source code directory instead of the built one (that does not exist), so all packages act as one and there is no need to build anything.

[swandir]

Hey @ernestostifano! I'm currently using it like that.

In my experience everything works as expected as long as:

• all tsconfigs have correct references (tsc itself is more forgiving here, it may work correctly even if you are missing some of the references);
• parserOptions.project option in the root .eslintrc lists all tsconfigs explicitly in the right order; that is, dependent-upon tsconfigs come before ones that depend on them;
• parserOptions.EXPERIMENTAL_useSourceOfProjectReferenceRedirect is enabled.

It might be that I'm not hitting certain edge cases though. But I was experiencing issues before and this configuration resolved them, so I thinks it's good enough.

P.S: Though performance is probably worse that it should be

[Ernesto Stifano]

Hi @swandir, thanks for your response!

I tried your suggestions and it seems to be working fine.

At first, to list all references explicitly in parserOptions.project what I did was to import my tsconfig.json into my ESLint config file and map the paths from there.

But then I tried without explicitly listing all references and it seems to work too. This is my config now:

parserOptions: {
    sourceType: 'module',
    ecmaVersion: 'latest',
    tsconfigRootDir: __dirname,
    project: './packages/*/tsconfig.json',
    EXPERIMENTAL_useSourceOfProjectReferenceRedirect: true
}

[swandir]

Yeah, it appears to work if compilerOptions do not differ between tsconfigs.

However, compilerOptions may affect type info gathered by the plugin. And when linting a file, the plugin seems to grab the first tsconfig from parserOptions.project what matches the file. Where "matcher the file" means that the file is a part of TS's compilation for this tsconfig. The issue is, the plugin does not use TS's build mode. So all the files from dependent-upon TS projects (referenced tsconfigs) are part of the same compilation.

As a result files may end up matched with wrong tsconfigs.

But as long as dependent-upon tsconfigs appear in the list before their dependencies, the plugin will see the correct tsconfig first.

I can be wrong on the exact details here, but this is how it appears to work.

[Jake Bailey]

Hopefully #6575 and #6754 will help address this, as this would effectively let ts-eslint piggy back on the project handling you get when you're using tsserver (so, VS Code opening your code). But, there are some challenges left.

[swandir]

So I've tried enabling the new EXPERIMENTAL_useProjectService option implemented in #6754 in my repro from #2094 (comment)

swandir/typescript-eslint-project-references@6249fc2

Now type-aware rules produce consistent results regardless of declaration files presence, yay!

[François Jacques]

Confirmed that the experimental flag fixes it as well!

[Karl Horky]

Thanks for this new EXPERIMENTAL_useProjectService flag 🙌 I can also confirm it's working.

Here's how I used it in the new ESLint Flat Config format (in the languageOptions.parserOptions object):

("type": "module" in package.json to use ESM config below)

eslint.config.js

import eslintTypescript from '@typescript-eslint/eslint-plugin';
import typescriptParser from '@typescript-eslint/parser';

/** @type {import('@typescript-eslint/utils/ts-eslint').FlatConfig.ConfigArray} */
const configArray = [
  {
    files: ['**/*.js', '**/*.jsx', '**/*.cjs', '**/*.mjs', '**/*.ts', '**/*.tsx', '**/*.cts', '**/*.mts'],
    languageOptions: {
      parser: typescriptParser,
      parserOptions: {
        project: './tsconfig.json',
        // typescript-eslint specific options
        warnOnUnsupportedTypeScriptVersion: true,
        EXPERIMENTAL_useProjectService: true,
      },
    },
    plugins: {
      '@typescript-eslint': {
        rules: eslintTypescript.rules,
      },
    },
    settings: {
      'import/resolver': {
        // Load <rootdir>/tsconfig.json
        typescript: {
          // Always try resolving any corresponding @types/* folders
          alwaysTryTypes: true,
        },
      },
    },
    rules: {
      // Warn on dangling promises without await
      '@typescript-eslint/no-floating-promises': ['warn', { ignoreVoid: false }],
    },
  },
];

export default configArray;

tsconfig.json

{
  "$schema": "https://json.schemastore.org/tsconfig",
  "files": [],
  "references": [
    { "path": "./tsconfig.root.json" },
    { "path": "./scripts/tsconfig.json" }
  ]
}

tsconfig.root.json

{
  "$schema": "https://json.schemastore.org/tsconfig",
  "compilerOptions": {
    "composite": true,
    "lib": ["DOM", "DOM.Iterable", "ESNext"],
    "target": "ES2015",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "resolveJsonModule": true,
    "esModuleInterop": true,
    "isolatedModules": true,
    "allowJs": true,
    "allowSyntheticDefaultImports": true,
    "downlevelIteration": true,
    "forceConsistentCasingInFileNames": true,
    "noEmit": true,
    "noFallthroughCasesInSwitch": true,
    "skipLibCheck": true,
    "strict": true,
    "incremental": true,
    "noUncheckedIndexedAccess": true
  },
  "include": ["**/*.ts", "**/*.tsx", "**/*.js", "**/*.jsx", "**/*.cjs", "**/*.mjs"],
  "exclude": ["node_modules", "scripts"]
}

scripts/tsconfig.json

{
  "$schema": "https://json.schemastore.org/tsconfig",
  "compilerOptions": {
    "composite": true,
    "lib": ["DOM", "DOM.Iterable", "ESNext"],
    "target": "ESNext",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "resolveJsonModule": true,
    "esModuleInterop": true,
    "isolatedModules": true,
    "allowJs": true,
    "allowSyntheticDefaultImports": true,
    "downlevelIteration": true,
    "forceConsistentCasingInFileNames": true,
    "noEmit": true,
    "noFallthroughCasesInSwitch": true,
    "skipLibCheck": true,
    "strict": true,
    "incremental": true,
    "noUncheckedIndexedAccess": true,
    "checkJs": true
  },
  "include": ["**/*.ts", "**/*.tsx", "**/*.js", "**/*.jsx", "**/*.cjs", "**/*.mjs"],
  "exclude": ["node_modules"]
}