cache/hashing is unclear and will invalidate caching on projects that do not have `node` dependencies

Question

cache/hashing is unclear and will invalidate caching on projects that do not have `node` dependencies

EelcoLos opened this issue a month ago · comments

Current Behavior

Running affected on projects should only have projects affected by the hashing concept stated by https://nx.dev/concepts/mental-model#computation-hashing-and-caching. However, when only changing a trivial item at top level lock files (yarn.lock/package-lock.json) seem to change the hash/cache of any project, regardless of input exclusion, or basically anything.

EDIT:
Although the example will show the trivial part, when actually updating packages, this actually affects the cache
Updated a slightly more real example in the comment below: #26915 (comment)

My tests are solely regarding dotnet projects in regards to integrated monorepos.

Expected Behavior

I would expect some control to state files not affecting

GitHub Repo

https://github.com/nx-dotnet/nx-dotnet

Steps to Reproduce

clone the repo
run yarn install
run npx nx show projects. This will show the a project called demo-csharp-models, which is solely a dotnet project.
update the yarn.lock file with something trivial like adding a '# test'comment on top of the file ie:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
# test

run npx nx show project --affected. the project demo-csharp-models is affected. This would be problematic already.
add the following inputs to the projects.json at demo\libs\csharp-models\project.json:

"inputs": ["default", "!{workspaceRoot}/yarn.lock"],

run npx nx show project --affected again. Still, the project is affected.

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 20.15.1
   OS     : win32-x64
   yarn   : 1.22.17

   nx (global)        : 19.2.0
   nx                 : 17.0.2
   @nx/js             : 17.0.2
   @nx/jest           : 17.0.2
   @nx/linter         : 17.0.2
   @nx/eslint         : 17.0.2
   @nx/workspace      : 17.0.2
   @nx/cypress        : 17.0.2
   @nx/devkit         : 17.0.2
   @nx/eslint-plugin  : 17.0.2
   @nx/node           : 17.0.2
   @nx/plugin         : 17.0.2
   @nx/react          : 17.0.2
   @nrwl/tao          : 17.0.2
   @nx/web            : 17.0.2
   @nx/webpack        : 17.0.2
   typescript         : 5.1.6
   ---------------------------------------
   Community plugins:
   @nx-plus/docusaurus          : 15.0.0-rc.0
   @trumbitta/nx-plugin-openapi : 1.12.1
   ---------------------------------------
   Local workspace plugins:
         @nx-dotnet/workspace-plugin
         @nx-dotnet/nx-ghpages
         @nx-dotnet/nxdoc
         @nx-dotnet/core

Failure Logs

No response

Package Manager Version

1.22.17

Operating System

macOS
Linux
Windows
Other (Please specify)

Additional Information

This is also the case for NX 18+ and project crystal inferred tasks.

{
	"plugin": "@nx-dotnet/core",
	"options": {
		"solutionFile": "Solution.sln",
		"inferProjects": true,
		"nugetPackages": {},
		"inferredTargets": {
			"build": {
				"inputs": [
					"production",
					"^production",
					"!{workspaceRoot}/packag*.*",
					"!{workspaceRoot}/**/packag*.*",
					"!{workspaceRoot}/package.json",
					"!{workspaceRoot}/package-lock.json"
				],
				"targetName": "build",
				"configurations": {
					"production": {
						"runtime": "linux-x64",
						"configuration": "Release",
						"noDependencies": false
					}
				}
			},
			"lint": "lint",
			"serve": "serve",
			"test": {
				"inputs": [
					"default",
					"^production",
					"!{workspaceRoot}/packag*.*",
					"!{workspaceRoot}/**/packag*.*",
					"!{workspaceRoot}/package.json",
					"!{workspaceRoot}/package-lock.json"
				],
				"targetName": "test",
				"cache": true,
				"configurations": {
					"ci": {
						"blame": false,
						"blameCrash": false,
						"logger": "trx"
					}
				}
			}
		},
		"ignorePaths": [],
		"tags": ["nx-dotnet"]
	}
}

I tried addressing this issue at multiple locations, but no actual solution is provided, not information shared on how to at least gain more insight:

I really like nx and try to advocate for using monorepos at my organization. However, there is a huge use of various platforms and languages. I feel though computation caching for monorepos is a must for my endavour to succeed in my organization

Eelco Los · Answer 1 · Fri Jul 12 2024 17:46:31 GMT+0800 (China Standard Time)

@AgentEnder @juristr is there something wrong at my understanding the Mental Model using external dependencies like dotnet and inputs? if so, how should the documentation reflect that?
We'd like to generalize the affected command in our workflows and not needing to use the exclude parameter to work around this issue. That'd defeat the purpose of this whole setup

Eelco Los · Answer 2 · Thu Jul 18 2024 14:27:55 GMT+0800 (China Standard Time)

I've been able to pinpoint it a little more closely for a more real-life scenario at https://github.com/EelcoLos/nx-tinkering .
So, the main branch has nx 19.5.0 . I installed all the packages and ran npx nx run-many -t build test on there.
Then i created a branch test-package-lock-dotnet, where I first tried the case above. This already showed the projects api-demo and api-demo.Test to be affected.
When running npx nx affected -t build test at that point (which is EelcoLos/nx-tinkering@3416d7d), the items are affected. However, the cache states there is a local cache hit.

So, upon pointing out a more real life scenario, I updated all npm packages today resulting in EelcoLos/nx-tinkering#2 .
This did trigger the cache to be void, even for api-demo and api-demo.Test.
My guess at this point (with some testing of my own) is that affecting dependencies (not devDependencies) trigger the cache to be void

Eelco Los · Answer 3 · Thu Jul 18 2024 14:37:38 GMT+0800 (China Standard Time)

I'm also hoping to get to know how to create your own hasher with a plugin. I'd like to create a plugin using a language not using node as well :)

Eelco Los · Answer 4 · Tue Jul 23 2024 00:00:52 GMT+0800 (China Standard Time)

I'm wondering if this is helping or not: What i'm noticing myself upon starting a new repo with adding @nx-dotnet/core, is that the package itself depends on @nx/js. That package is dependant on package.json?