Guidance on shipping ts within node_modules
johnnyreilly opened this issue · comments
Hello!
I'm raising an issue because I'm seeking guidance on whether TypeScript is intended to be used to ship .ts files inside node_modules. I'm asking as I'm one of the maintainers of ts-loader and this issue was raised on that very topic. It seems to have worked in the past (without any particular intention or effort on the part of ts-loader) but broke with TypeScript 2.0.
@basarat took a look into it and ended up reaching this conclusion:
I have the lost the will to implement it and would not recommend anyone to write a package that ships .ts + .js and instead people should use outDir + .d.ts + .js.
Again for those interested reasons, the following bad things happen if you ship .ts + .js files
- If your package is used in nodejs by someone using outDir, your package messes up other peoples outDir option.
- As new stricter TypeScript compiler options are implemented they cannot be used by people that use your package unless you update your package to compile with those options as well.
@basarat also suggested
We can provide an additional error message if path contains node_modules that says:
Since the file is in node_modules you should not need to recompile .ts files in node_modules and should contact the package author to request them to use --declaration --outDir.
I'm inclined to think this is a good idea and I was planning to implement this. Before I did so I wanted to see if this was in line with the way the TypeScript team intends the language to be used. I think this is probably encouraging good practice but I wondered if you could share a view on whether that's the case?
Can you elaborate on the issue, i could not get the details reading through the issue.
As for the recommendation, publish .js + .d.ts. this way your users do not have to compile your sources each time they need to refer to your library.
As Typescript user... Bobril-build currently depends on ts in node_modules, because it uses Typescript AST to recognize calls to translation module - modifying AST to produce message ids, or detecting used resources, or usages of images for creating of sprites. Yes in theory all this stuff could be parsed out of JS, but question would it have same robustness, error reporting, speed? I am also thinking about advanced minification (Closure style or better) which is impossible just from JS code. But upgrade to TS 2.0 was not all roses we had to fix some incompatibilities in our TS code, so I feel pain too.
@Bobris I am afraid i do not understand the underlying issue. can you explain the problem or share a repro i can look at?
Issue is that you fixing bugs in TypeScript compiler, making it stricter even without changing any TypeScript compiler options. This dir https://github.com/Bobris/Bobril/tree/7860bda9ac82520e2826ccbced38168eceb01cf6/package is compilable by TS 1.8.10 without errors, but not with TS 2.0.3.
So anybody using this in node_modules and upgrading TS will have hard time... That's why one of solutions is to not distribute TS, but just JS and D.TS which lowers probability of breakage.
I see. but if you publish sources, would not your have to compile with matching version of the tools you use, and with matching compiler options (e.g. --noImplicitAny, --noUnusedLocals, etc..) as well?
But I think with TS 2.0 even just D.TS would not save me, because I made it null strict, so D.TS now contains | undefined which TS 1.8 does not understand I think, forcing users to upgrade to 2.0 anyway.
It is not big deal for us because I control used build tool (mentioned bobril-build). Just want to say, I don't want to loose this option (distribute JS and TS) completely because I don't control IDE (VS Code).
And because of bobril-build - it unify used compiler options (at least on minimum needed), but you can set it more strict - and that's why all my modules needs to be compilable under most strict compiler options available too.
Distributing .ts files is a supported scenario. We have users who use this with npm link at design time, and we do not intend to break this.
The OP was about recommendation. my recommendation is not to ship your .ts files to external users.
Thanks for the response @mhegazy.
Can you elaborate on the issue, i could not get the details reading through the issue.
That's a fair point - I have a feeling that there might be more than one scenario being discussed in the thread. Speaking for myself this is not a scenario I look to use at all and I'm not totally certain I understand what each poster has asked for - I'm just trying to grapple with what is reasonable to support and what is not.
Perhaps rather than me assuming wrongly I'll ask the posters on the original issue to pitch in with their expectations. So @ahelmel , @Tiedye, @g3r4n, @sumeet70 , @skysteve, @stevejhiggs could you pitch in with what you are hoping for here and lets see if we can identify if you're all after similar behaviours or if there is more than one issue going on here.
@basarat - could you also pitch in with what you were planning to support and then abandoned? Just so we can gather everything in one place? Thanks! 🌷
sure, both me and @skysteve are actually working on the same project so I'll detail our exact scenario:
- We have a large internal core library that provides 90% of our functionality and is just a collection of ts files.
- This library has multiple entry points (we reference more that just the entry point as defined within the package.json)
- On top of this library are a number of custom implementations that build on the functionality provided by the core.
- We want the end result to be consumed by the browser and to be as small as possible. We bundle via webpack + ts-loader.
- Because life likes to hand out problems we need the resulting script to run on ie8 so typescript is set to compile down to es3.
Now, we've actually worked around the problem for now as oddly our issues only occurred when "allowJs" was set to true and now we are typescript all the way through this is no longer needed. But with allowJs set to true we received errors along the lines of Typescript emitted no output for.. and I believe @skysteve provided a simple case to reproduce this issue in TypeStrong/ts-loader#278
There are actually three issues. Personas : Publisher is the person publishing an npm module and Consumer is the person using it 🌹
noEmit for isSourceFileFromExternalLibrary
These files are external (default host.isSourceFileFromExternalLibrary == true see TypeStrong/ts-loader#365 (comment) for breakdown). That is the issue discussed by @stevejhiggs here #12358 (comment) and all the people in this thread TypeStrong/ts-loader#278. We could fix it in ts-loader (adding own host.isSourceFileFromExternalLibrary) but I gave up and suggesting that we should improve the error message in ts-loader telling people not to shop .ts files in node_modules. Hence prompting this question by @johnnyreilly about best practice ❤️
Old code, New compiler
Publisher's .ts files must conform to all the strict compiler options Consumer wants. This is fairly clear I believe. If you ship .ts files its just the way it has to be. This is the issue being discussed by
@Bobris here #12358 (comment)
outDir
I believe it messes the outDir for Consumer. This is purely psychic debugging but I think its a safe assumption (@tomitrescak is typestyle/typestyle#34 fixed after I stopped publishing .ts files right?)
Thanks everyone that has commented here; that's really helpful 🌹
@mhegazy, given you've said:
As for the recommendation, publish .js + .d.ts. this way your users do not have to compile your sources each time they need to refer to your library.
and also given the other scenarios outlined in this issue and @basarat's specific suggestion that, in the case where there is no emit, we execute the following code:
if (outputText === null || outputText === undefined) {
const additionalGuidance = filePath.indexOf('node_modules') !== -1
? "You should not need to recompile .ts files in node_modules and should contact the package author to request them to use --declaration --outDir. More https://github.com/TypeStrong/ts-loader/issues/278"
: "";
throw new Error(`Typescript emitted no output for ${filePath}.${additionalGuidance}`);
}Which will prompt the user that having node_modules packages which contain raw .ts files is a bad thing. Does that seem reasonable?
My gut feeling is that it is but I wanted to check with the TypeScript team before adding something that might propogate bad practice. (I don't think this will)
sounds reasonable. I should note that we have issue #11946 tracking emitting node_modules files if they are listed explicitly in the include/files list in the config file. this allows users who realy want to include the node_modules*.ts into their compilation.
Thanks @mhegazy - I'll plan to make the comment link back to this issue as this has more context.
Hi @mhegazy,
I'm just implementing the warning now.
I should note that we have issue #11946 tracking emitting node_modules files if they are listed explicitly in the include/files list in the config file. this allows users who realy want to include the node_modules*.ts into their compilation.
I'm mindful of the change and wanted to make our warning configurable for TypeScript 2.1 depending on whether a file is mentioned in include/files. Obviously with files we can just examine the tsconfig.json. However, with include we're reliant on glob resolution. Is there a simple way on the TypeScript API to find whether a file was resolved in include or similar? I'm aware of compiler.sys.getDirectories but that doesn't really provide what I'm looking for...
@stevejhiggs This is exactly what I am trying to do:
-
I want to have multiple pure typescript npm modules that I can easily import into my other projects.
-
I want to keep the source code as typescript, so I can use it in various projects that could potentially transpile to different module types or es versions.
-
I want to be able to mix ts files in the same project that target different platforms (for example a server folder for node and client for browser).
-
Note: I still want to be able to require and use js modules (especially when targeting node.js platform)
I thought I had it working, but now it is starting to breakdown.
What workflow do you use to achieve this?
@ricklove in the end we gave up. We got everything working for us but it became very obvious from reactions to this issue that it was not really a use case that was being considered and that we would always be fighting to tooling to achieve what we wanted.
Right now we are using declaration: true along with outdir to create a dist folder that contains just the es5 js + d.ts files, the package.json is then copied to the dist file and the result then published as the final package.
The resulting package is then consumable from anywhere. One improvement would be to set typescript to use es6 modules to allow tree shaking and we'll be doing this when we move our build pipeline to webpack 2.
@johnnyreilly , remember the issue I raise? #7398
You can't really distribute ts file before that is handled properly.
It is a systematic issue. If people start shipping ts files, the packages will be divided into two camps that are not compatible with each other.
@stevejhiggs Thanks for explaining that.
I'm doing a similar thing now. I am building to the "lib" folder and I have index.js at root pointed to the lib folder:
export * from './lib/'
I still have my original source in the package under src, but it isn't used for anything right now.
This seems to work pretty well.
Also, thanks for letting me know about webpack 2 and tree shaking with es6. I wasn't aware of that possibility and I'll research it more.
So TypeScript can't import its own source files (.ts) directly from node_modules? Do I understand it correctly?
This is a huge limitation, I believe. Let me explain why "just use builds" approach won't work at scale.
-
There is such thing as Cross-Platform where code may be built differently, depending on a platform you target. You want to apply minification, tree-shaking, uglification, source maps, polyfills and other transformations differently, depending on them. And now you tell me that's impossible to control in my app's build step?! That I have to ask library authors to ship everything I need, just in case?!
-
You cannot simply "provide all possible builds" for all possible platforms. First, it's a step back to pre-virtual machine age where you had to uploads "builds" for all imaginable Windows, Linux, FreeBSD, OS X versions. We know how that ended. Second there is a multiplication of cases for each plaftofm: source-maps, polyfills etc. (see a list above). So it's even worse.
-
And you cannot simply tell "then provide some universal ES6 semi-baked build" because it will require to ask EVERY person to follow this practice.
TypeScript should absolutely be able to use its own source files from its distribution system of choice.
It's just doesn't make sense otherwise. "Typings" files *.d.ts are a hack, – necessary to reuse JS ecosystem, but still a hack at a global scale.
Some people tell "it's like CoffeeScript which failed" – a weird argument IMO.
- A lot of people use NPM to distribute LESS, CSS, images and other non-JS files, especially after Webpack.
- There is a growing ecosystem of language-agnostic package managers (which ship different formats by definition).
- CoffeeScript failed because it had zero added value compared to ES6, not because it shipped its sources via NPM.
Docs at https://www.typescriptlang.org/docs/handbook/namespaces.html state:
Working with Other JavaScript Libraries
To describe the shape of libraries not written in TypeScript, we need to declare the API that the library exposes. Because most JavaScript libraries expose only a few top-level objects, namespaces are a good way to represent them.
We call declarations that don’t define an implementation “ambient”. Typically these are defined in .d.ts files. If you’re familiar with C/C++, you can think of these as .h files. Let’s look at a few examples.
If TypeScript is unable to consume non-JS libraries these paragraphs don't make any sense.
To describe the shape of libraries not written in TypeScript ...
– Imply you can write and distribute library in TypeScript. 😞
So TypeScript can't import its own source files (.ts) directly from node_modules? Do I understand it correctly?
you can, but it is not a recommended practice. .ts files are looked up first before .d.ts files. you need to think about where the build output will located (use --out or --outDir).
To elaborate more, the configuration you used when you are writing your library can be different than the configuration made by your consumer. That's why your consumer will have problems using your library.
The same partially applied to typings and is the root cause of #7398
So you can see you will face an even bigger problem if you distribute TS directly.
I mentioned about this here in that thread:
#7398 (comment)
To elaborate more, the configuration you used when you are writing your library can be different than the configuration made by your consumer. That's why your consumer will have problems using your library. The same partially applied to typings and is the root cause of #7398
Thanks for clarifications @unional but I still don't take the argument. Don't you see that the claim of
the configuration you used when you are writing your library can be different than the configuration made by your consumer. That's why your consumer will have problems using your library
applies to every other language (including compile-to-JS). Yet somehow people prefer to share sources, not builds. PureScript, Elm, ReasonML, etc. etc are source- not build-oriented. You don't distribute .js files with some magical .d.purs or .d.elm there.
So sorry for being annoying, but I think you should really explain what's so special about TypeScript here. Because your current stance of "it's obviously a bad idea" does not look convincing at all. Nothing "obvious" about it, it goes against the established practices, and while I believe you have some arguments – they should be exposed and highlighted in docs.
you can
@mhegazy somehow I'm not able to run the simplest example:
// node_modules/somelib/index.ts
export function add(x: number, y: number) {
return x + y
}// app.ts
import {add} from "somelib"
add(1, 2)$ ts-node app.ts
> SyntaxError: Unexpected token export
(function (exports, require, module, __filename, __dirname) { export function add (x: number, y: number) {
^^^^^^
SyntaxError: Unexpected token export
Tried different tsconfig settings, no luck so far. It looks like files in node_modules aren't compiled at all. If this is another issue – pls. point me to the thread.
@ivan-kleshnin I also would like to have source code that I can easily share through npm.
However, there is a huge problem with tsconfig differences. There are many settings that can drastically effect source code. (NoImplicitAny, StrictNullChecks, AllowJs, etc.).
So the compiler would have to allow each node module to have its own tsconfig settings, which I don't think is possible. *.d.ts acts as a buffer between those and allows the compiler to have a single setting context.
That being said, I would still like to have the ability to include ts source from some node_modules especially for my own projects. It would be nice if we could have a compiler flag to treat specific dependencies as if it were source code inside our context.
We use ts-node and have about 10 different pure ts modules that we import throughout our services and I just started upgrading from typescript@2.5.3 -> typescript@2.8.3 and ts-node@4.1.0 -> ts-node@6.0.3 and started running into this no emit issue (after much digging through ts-node). I probably would have run into it sooner but the default for ts-node for a while was transpile only which does not cause this issue (they just switched back to type checking by default again).
I have tried many different solutions, and the only one that works is using ts-node with --transpileOnly --skip-ignore flags which doesn't help with my type checking. It's not a complete killer because I can run tsc manually, but unfortunate.
I have tried the include in tsconfig.json but that creates other problems like missing node packages (because parent and child don't always share the same dependencies). So I would really appreciate a flag on getEmitOutput or something like that, that would allow emitting output for files in node_modules. I am fine if it remains an API only option so that others don't shoot themselves in the foot, but would appreciate the option.
One of the primary reason we like to have ts files in our node_modules internally is that it makes it much easier to use npm link for local development (without having to recompile the dependency every time we make a change in a dependency). It's just so much more seamless.
There is no such limitation in TypeScript compiler API and I am saying it from position of author of bobril-build which is mostly TypeScript compiler and bundler focused on Bobril framework. We have hundreds of TypeScript only modules. Whole trick is to provide custom CompilerHost and correctly implement resolveModuleNames.
This is now conditionally supported in ts-loader: https://github.com/TypeStrong/ts-loader#allowtsinnodemodules-boolean-defaultfalse as of 4.3.0: https://github.com/TypeStrong/ts-loader/releases/tag/v4.3.0
Down to the hard work of @aelawson
Is it possible for a library author to include TS source files (and sourcemaps pointing to them) in an NPM package, but to prevent those TS files from being re-compiled by default? If so, how to do it, and should this be the officially-recommended way to publish TS packages to NPM, as opposed to the "don't publish TS source at all" recommendation above?
Or should there be some other solution that doesn't include source in the NPM package but at least points dev tools towards the original source, similar to how Source Server does for Windows and Visual Studio? (Personally I don't like this solution because it seems more fragile than publishing source, but wanted to throw it out as an option in case preventing compilation of TS source is too hard.)
I'm asking because it seems like there are two separate questions discussed in this issue:
1 - should libraries publish .ts source files for compilation ?
2 - should libraries publish .ts source files for reference and debugging (not for compilation) ?
I understand the debate above around (1), but why is (2) problematic? Seems like every library should make it easy for dev tools to show developers the original, pre-transpiled source code of that library. Otherwise how will developers know where to set breakpoints in library functions whose only NPM-published "source" is confusing transpiled-for-ES5 code? How can developers step into original library code in the debugger if libraries use features like async/await whose ES5 transpiled equivalents make linear debug-stepping an exercise in frustration? How will users browse the actual source code (for inspiration, for curiosity, or when investigating possible library bugs) of the libraries that they're using without having to switch over to GitHub to browse the TS source there?
For example, here's the easy-to-understand, easy-to-debug TS source of the API.post() method from the aws-amplify package, which is the preferred client library for AWS Lambda functions:
async post(apiName, path, init) {
if (!this._api) {
try {
await this.createInstance();
} catch (error) {
return Promise.reject(error);
}
}
const endpoint = this._api.endpoint(apiName);
if (endpoint.length === 0) {
return Promise.reject('Api ' + apiName + ' does not exist');
}
return this._api.post(endpoint + path, init);
}
And here's the compiled version of the same code. Much more complicated and much less readable!
APIClass.prototype.post = function (apiName, path, init) {
return __awaiter(this, void 0, void 0, function () {
var error_2, endpoint;
return __generator(this, function (_a) {
switch (_a.label) {
case 0:
if (!!this._api) return [3 /*break*/, 4];
_a.label = 1;
case 1:
_a.trys.push([1, 3, , 4]);
return [4 /*yield*/, this.createInstance()];
case 2:
_a.sent();
return [3 /*break*/, 4];
case 3:
error_2 = _a.sent();
return [2 /*return*/, Promise.reject(error_2)];
case 4:
endpoint = this._api.endpoint(apiName);
if (endpoint.length === 0) {
return [2 /*return*/, Promise.reject('Api ' + apiName + ' does not exist')];
}
return [2 /*return*/, this._api.post(endpoint + path, init)];
}
});
});
};
If you spend a few minutes with two monitors looking back and forth between TS and the transpiled code, you can probably figure out where to set breakpoints. But having to do this definitely slows down debugging and intimidates newbies.
And if you thought setting breakpoints is hard, just try stepping into (in a debugger like VS Code) the library code, whose whose first action is to call an __awaiter helper function that was inserted by the transpiler. Holy crap the code below is confusing!
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : new P(function (resolve) { resolve(result.value); }).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
var __generator = (this && this.__generator) || function (thisArg, body) {
var _ = { label: 0, sent: function() { if (t[0] & 1) throw t[1]; return t[1]; }, trys: [], ops: [] }, f, y, t, g;
return g = { next: verb(0), "throw": verb(1), "return": verb(2) }, typeof Symbol === "function" && (g[Symbol.iterator] = function() { return this; }), g;
function verb(n) { return function (v) { return step([n, v]); }; }
function step(op) {
if (f) throw new TypeError("Generator is already executing.");
while (_) try {
if (f = 1, y && (t = op[0] & 2 ? y["return"] : op[0] ? y["throw"] || ((t = y["return"]) && t.call(y), 0) : y.next) && !(t = t.call(y, op[1])).done) return t;
if (y = 0, t) op = [op[0] & 2, t.value];
switch (op[0]) {
case 0: case 1: t = op; break;
case 4: _.label++; return { value: op[1], done: false };
case 5: _.label++; y = op[1]; op = [0]; continue;
case 7: op = _.ops.pop(); _.trys.pop(); continue;
default:
if (!(t = _.trys, t = t.length > 0 && t[t.length - 1]) && (op[0] === 6 || op[0] === 2)) { _ = 0; continue; }
if (op[0] === 3 && (!t || (op[1] > t[0] && op[1] < t[3]))) { _.label = op[1]; break; }
if (op[0] === 6 && _.label < t[1]) { _.label = t[1]; t = op; break; }
if (t && _.label < t[2]) { _.label = t[2]; _.ops.push(op); break; }
if (t[2]) _.ops.pop();
_.trys.pop(); continue;
}
op = body.call(thisArg, _);
} catch (e) { op = [6, e]; y = 0; } finally { f = t = 0; }
if (op[0] & 5) throw op[1]; return { value: op[0] ? op[1] : void 0, done: true };
}
};
I know there's value in having developers understand that transpilation is happening under the hood, and to learn advanced debugging techniques for transpiled code. And somewhere there's a CS professor who's overjoyed that thousands of developers have to really, really, really learn how a state machine works.
But it seems really unfair to novice developers to throw this much complexity in front of them if all they want to do is set a breakpoint in a library function or to step into a library function to understand why their code isn't working as expected.
Such a pity that we can't use pure ts packages.
In our company we have a monorepo setup that consists of around 27 packages written in TypeScript. Running tsc -w on all packages consumes around 6GB of RAM. Even if i run watch mode only on package that i am working on right now it consumes around 2GB of RAM beacuse of dependencies. I measured that tsc on a single file without dependencies consumes ~190MB of RAM. We cant use build mode because of legacy code that is full of type related erros. There is even more problems with build mode because none of webpack ts-loaders supports it. Why language server and compiler cant just work in build mode "in background"? I mean if we import some module from another package and there is tsconfig.json around this module, ts should apply this settings and give us right type information.
And how d.ts files are affected by compilator settings that was used to compile them? If for example we use keyofStringsOnly: true to compile declarations isn't it forces us to use this setting on consumer side? Sorry i can't check this right now. I mean what is so special about declaration files? How typescript can generate declarations that work for any compiler settings that consumer uses?
Anyway, wanted to say thanks for a great work of bringing static typing to js. Even when it creates problems it already solved alot more problems than created.
Is there any working answer? I have absolutely the same situation as @toddbluhm and really wanna work with my linked modules for local development. include sections does not work for me either.
@toddbluhm did you found a solution to this problem?
@vashchukmaksim The solution was already given for this issue in the comment right after mine.
Whole trick is to provide custom CompilerHost and correctly implement resolveModuleNames
Now implementing both of those was quite a bit more work than I had the time for, so I didn't do that. Instead, I just stuck with what was working for me in my original comment ts-node --transpileOnly --skip-ignore.
Apologies for resurrecting this old issue, but I feel the release of npm 7 necessitates some discussion on this.
Breaking up a repo into multiple smaller packages is a first class feature of node/npm via workspaces now. Being able to import pure .ts files from a different package and compile according to the requirements of the consumer should be a valid use case of typescript.
Say I have 2 bundled web apps, each with their own package.json and watch script. Lets call them App A and App B. App A will run on modern browsers and can safely target es6. App B however needs to run on a legacy browser so can only target es5.
(This is a very crude example, but there are plenty of other tsconfig options which could apply)
Assume I am following the monorepo/workspace pattern and break up what was previously a monolith up into smaller, more reusable packages. These packages aren't published publicly, they don't need to work in a non-typescript consumer and they don't need to emit JS files. They exist purely to be reused across by other pure typescript packages in this hypothetical workspace.
In order to make any common code packages compatible with both App A and App B, each needs to set up its own watch, Thus, if I have 1 common package, we will have 3 watches. 5 will result in 7, etc. I'd argue we shouldn't need anymore than 2 watches in this scenario.
As @professional-human-being has explained, this just doesn't scale in terms of performance.
Furthermore, compatibility will be a case of lowest common denominator, e.g. if App A can use latest JS features safely, but App B needs to target an older ES version, the common package will have to use App B's target for the compilation, bloating App A's build size unnecessarily.
I realise this is a bit of a moan without any solutions offered. I also realise there are a lot of excellent reasons for why this hasn't been resolved yet. However I genuinely love Typescript and just want it to be as future proof as possible. In my opinion, this has the slight potential to hamstring the language as workspace tooling and adoption increases.
As someone that ended up at this issue while figuring out how to deal with source maps in NPM packages, here is what I ended up with:
tsc --sourceRoot "https://raw.githubusercontent.com/<user>/<repo>/$(git rev-parse HEAD)/src/"points the sourceRoot property of the source maps to the latest commit inside of your GitHub repo. This allows tools to pick up the original source if needed.
By default, ts-loader will not compile .ts files in node_modules.
You should not need to recompile .ts files there, but if you really want to, use the allowTsInNodeModules option.
who can tell me what should i do