Support redirects
RyanCavanaugh opened this issue · comments
We should allow a folder structure which is
/some-lib
redirect.jsonThe contents of redirect.json would be something that tells types-publisher how to enlist and parse the linked repo. We can probably start with just git URLs and expand as needed
What do you think the specificity of this file should be? Per directory or per version or per commit? Directory seems to make sense for the most part. Does the redirect structure just need to follow the @types structure (E.g. versioned sub-folders, I think files must be the same name as the typed name?). Will typings be able to expand over multiple files and specify the main?
Has the structure of DefinitelyTyped been finalised now? I think the best approach to redirects, and one that keeps things consistent, would be to use git submodules. The advantage here is GitHub has a nifty feature that'll redirect when you click on another GitHub submodule. The only issue here is who has onus on updating the submodule references.
+1 for git submodules.
Currently we have a lot of high-quality typings in the registry that are well-maintained, have their own tests, issue tracker, PRs, docs, and CI. It is easy to contribute to them (no need to pull down all the other thousand typings), and the typings structure follow the library structure (instead of 10,000+ LOC in one file).
At the same time, the typings for these exact libraries are still maintained in DT, even though often outdated compared to types/. People are basically duplicating their efforts, because they might not even know that an equivalent in types/ exists. We really need some way to redirect them from DT, and I think submodules would be an excellent solution, because of the mentioned GitHub feature. They are generally frowned upon because the submodules don't get recursively cloned by default. But for DT, this would be exactly what we want - if you clone DT, you want to modify some typings in there. If you want to modify types/ typings, you can clone that repo directly.
We currently use subfolders for different versions of typings and could easily give them an individual package.json with a version field and proper @types/ name.
We would be happy to take PRs for this feature if anyone is interested in contributing. most of the infrastructure is already in place.
It would be nice if one of the TypeScript members could comment on the submodules proposal before putting work into a PR :)
The open question for submodules, as noted by @blakeembrey, is the update strategy. is the publisher responsible for always updating to latest? or is the package author responsible for this whenever they are ready to push a new version out? if it is the publisher how often should it look for these package changes? currently we have a webhook that is listening on definitelytyped, and ideally we would like keep it this way.
Currently, when we change on of the repos, we update the commit hash manually in the typings registry. That can be done really quickly with the GitHub online editor, but there were some ideas floating around for having a bot that does this or a typings publish command. For updating the git submodule, the typings maintainer should be responsible, just as he is for updating the registry entry. I guess you cannot do it online though, so it needs to happen with a PR. Updating happens afaik through git submodule update --remote when inside the submodule dir, but I dont know a lot about submodules. It may be worth to find out if this can be done programmatically through the GitHub API so typings publish can automatically open a PR at DT. @blakeembrey what do you think?
I'm not really sure, there's advantages on both sides and I believe I have presented each side (though offline, not particularly here). Hand editing is definitely made easier with text files, but it's possible we can make a bot to update submodules and submit PRs also. I imagine we'll want to do that flow so that tests can be run on the PR and an external change doesn't break DefinitelyTyped. Since both methods can be automated, sticking to the more valuable one (likely submodules, since the GitHub interface is friendly towards it) would be better.
@RyanCavanaugh So what is going on with this? Quoting yourself from microsoft/TypeScript#9184 (comment):
For large and popular libraries, I agree that having a separate repo is the way to go. We fully intend to support that scenario by allowing redirection from DefinitelyTyped to other repos so that well-maintained libraries can have their own repo.
We have 350 typings in the registry in total and of those are 200 typings in the @types org ranging from the types of node itself to the most-used libraries on NPM like express. These are well-maintained by a core team and can be improved with PRs aswell, and honestly from my own experience of much higher quality than the ones on DT. The issue is that with TS2 being stable now and the release notes actively telling people that "typings from the typings may not be compatible with TS2, so use npm", people will only use these huge >1000 lines per file typings from DT, that share an issue and PR tracker with 2000 other declarations. And as people use these, they will submit PRs and contribute to the typings on DT, that will eventually get replaced by a redirect. I feel you should really prioritize this feature much much higher, it feels like you are just putting this at the end of the backlog as a "nice to have" while people are duplicating their efforts everyday. You promised to have full support for this scenario and now TS2 is out and it doesn't seem like anyone has even started working on this... Some clear statement or ETA would be really nice 😐
I apologize that this hasn't happened sooner. We're the same engineers as on TypeScript and as you can see we just shipped 2.0, so things have been pretty busy. We're also spending a lot of time just reviewing DT PRs and this has been taking longer than expected during the transition period.
We'd totally accept a PR on this if you'd be willing to help implement it. Otherwise I think we're looking at 4-6 weeks out at the earliest.
It's been 6 weeks now, how is the progress?
We still have not looked at this. PRs are still welcomed.
@mhegazy I think the only thing stopping this would be the structure. Has that been formalized yet? If it has, I think the only change required for very basic support would be adding git submodule update --init --recursive to https://github.com/Microsoft/types-publisher/blob/ca97be3faedd30bb33e9c02245c55fb50a6feaff/src/get-definitely-typed.ts#L13.
For the structure, we just need to know what we need to make it work. Is types-publisher using package.json yet? How do we specify files to publish (we can follow the current tsconfig.json pattern)? How is multiple versions supported - we can make repos follow whichever pattern and the publisher will just work the same as long as submodules automatically update to the latest commit. At a later date, I can build some additional tooling around submodules to make contributing easier.
if we end up using submodules, then if the new repo has the same shape, nothing needs to be done here. it should just work.
the investment needed would be in getting a tool that runs tests and updates them. not sure what is the best experience here.
then if the new repo has the same shape
what shape exactly?
the investment needed would be in getting a tool that runs tests and updates them
types-publisher doesn't need to run tests, the repos themselves all have Travis.
I would imagine a typings publish command that uses the GitHub API to open a PR at DT. Which is the obvious disadvantage of submodules, someone needs to approve this PR. This hasn't been a problem in typings/registry so far though.
@mhegazy I can write a bot that does the updates part, or we can create a CLI later. Either way, having something seems better than nothing at this point because we're getting a number of issues asking for @types to be published which we can't control.
As for the testing tool, what would that need to look like? Most of our repos have individual tests already, plus the registry runs a final structural test. I imagine when the PR is opened that DefinitelyTyped would run its own test suite and as long as that also supports submodules we should be in business.
types-publisher doesn't need to run tests, the repos themselves all have Travis.
no. we need one place to run tests to maintain consistency. say you change node definition in a separate repro, we need to run all tests against all declarations that depend on node in Definitely typed before pushing a package.
The test tool is not in the types-publisher, but on DT.
As for the testing tool, what would that need to look like? Most of our repos have individual tests already,
I meant a tool that runs on regular basis, updates submodules, and runs tests, and if so pushes them.
what shape exactly?
the shape of a typical folder in DT.
If you add for support submodules wouldn't that work already - the tests should run on PR right?
I meant a tool that runs on regular basis, updates submodules, and runs tests, and if so pushes them.
See:
I can write a bot that does the updates part, or we can create a CLI later. Either way, having something seems better than nothing at this point because we're getting a number of issues asking for @types to be published which we can't control.
the shape of a typical folder in DT.
That's fine, but formalising this is important if things are going to become less centralised for contributors. Especially things like #3 which is important for us to maintain a matrix of versions for things like node.d.ts, we could either find ourselves maintaining multiple branches with multiple submodules or a single repo with folders contained, etc.
Either way, having something seems better than nothing at this point because we're getting a number of issues asking for @types to be published which we can't control.
Sorry if this seems obvious, but what is the urgency for this? and why not put it on DT?
no. we need one place to run tests to maintain consistency. say you change node definition in a separate repro, we need to run all tests against all declarations that depend on node in Definitely typed before pushing a package.
That doesn't make much sense to me. You essentially want to run the tests of every single of the 2000 typings on every PR? That is like running the tests of every of the millions of modules that depend on lodash for a PR to lodash.
When I change the node typings, it will result in a new version. As long as all other typings depend on the previous version, nothing breaks. Then humans can go in and update the definitions to adapt the latest API. Again, perfect analogy to a change in the lodash. You wouldn't require a PR to first change every dependent of lodash before merging the API change in. Or did I misunderstand you?
That's fine, but formalising this is important if things are going to become less centralised for contributors. Especially things like #3 which is important for us to maintain a matrix of versions for things like node.d.ts, we could either find ourselves maintaining multiple branches with multiple submodules or a single repo with folders contained, etc.
For #3, i would assume whatever works for a regular DT folder, will work for a mapped submodule. do you agree?
That doesn't make much sense to me. You essentially want to run the tests of every single of the 2000 typings on every PR
No. only on the subset of packages that depend on the one that changed. typically this is much smaller than 2000.
That is like running the tests of every of the millions of modules that depend on lodash for a PR to lodash.
I do not want to run it on all lodash dependencies. i want to run it on the ones we are publishing. that is much smaller.
As long as all other typings depend on the previous version, nothing breaks
they do not. they depend on "*"
Sorry if this seems obvious, but what is the urgency for this? and why not put it on DT?
We have 220 definitions that we maintain in @types, not to mention other peoples, and DefinitelyTyped gives us zero control over the ability to review issues or PRs to the definitions. It would immediately create fragmentation. We also can not iterate very quickly, which can be important like when heavy refactoring to larger definitions like node.d.ts or rethinkdb.d.ts.
Most importantly, once something is put directly in DefinitelyTyped the author loses ownership. This might not seem like a big deal, but as an author it is. The side effect of losing the author as an owner means that no one really owns the definition, which means that when it comes time to do updates to the definition there's no one really pushing for it unless it's vital. I've seen this a number of times without outdated definitions that no one has bothered updating because no one owns.
For #3, i would assume whatever works for a regular DT folder, will work for a mapped submodule. do you agree?
I think so, submodules should act exactly the same as part of the structure. The one thing we will likely need support for is package.json to specify our (type) dependencies. Has types-publisher been updated for that? (I can't recall what I saw around package.json).
Edit: Or do we need to follow double approaches - using references for DefinitelyTyped support and package.json for standalone tests?
No. only on the subset of packages that depend on the one that changed. typically this is much smaller than 2000.
That means when we do a breaking change in the node typings, we cannot merge it until every single node-dependent in DT is changed? That won't scale. Why not let the typings express a strict dependency, instead of *?
@felixfbecker I don't think that'll be a big problem for us, if the Express typings are wrong according to node.js we should definitely be updating them anyway.
@mhegazy Also, the reason we can't/won't copy into DefinitelyTyped is it just makes the process/workflow horrible. It means there's two possible places changes could be coming from. Every update we would need to 1. look in DefinitelyTyped, 2. copy back changes, 3. commit our change, 4. copy changes back to DefinitelyTyped. Going all in on DefinitelyTyped is not possible because of the previously mentioned reasons - it's unreasonable to expect an author to keep up with a mono-repo of thousands of definitions and unrelated issues just to maintain something. The project separation using GitHub affords us sanity, which allows us to do things like maintain separate issues on definitions, plan new features (e.g. types/rethinkdb#2) and use things like GreenKeeper to automatically know when something needs updating (e.g. typed-typings/npm-scmp#1).
The one thing we will likely need support for is package.json to specify our (type) dependencies. Has types-publisher been updated for that? (I can't recall what I saw around package.json).
The publisher will read package.json if one exists, and will load dependencies and peerDependencies from it. not sure if this is what you meant though.
Edit: Or do we need to follow double approaches - using references for DefinitelyTyped support and package.json for standalone tests?
I am not sure i understand the question
That means when we do a breaking change in the node typings, we cannot merge it until every single node-dependent in DT is changed? That won't scale. Why not let the typings express a strict dependency, instead of *?
we did this for a while. the issue here is that means that every time there is a change in a package, you need to republish all packages that depend on it so that they have the latest version. otherwise they do not get the updated version.
Also this created a lot of conflicts, since the user installs node@6.0, and one of their dependencies depends on node@4.0 and another one depends on node@4.2.
The observation here is most declaration files do add. and most dependencies are on latest. we do have the ability to "freeze" the dependency on a specific version if needed. but has not been needed so far.
Would expanding the pool of users with push rights to DT be an acceptable solution here?
also pinging @andy-ms
The publisher will read package.json if one exists, and will load dependencies and peerDependencies from it. not sure if this is what you meant though.
I think that's great.
Would expanding the pool of users with push rights to DT be an acceptable solution here?
For which? I don't think it solves all the reasons I don't push to DefinitelyTyped myself. It's just to hard to maintain. FWIW, I can already push to DefinitelyTyped, it's the management lifecycle which is impossible with DefinitelyTyped (can't manage issues well and can't keep up with any related PRs/issues as subscribing results in hundreds of unrelated issues a day).
Thanks for the explanation. my concern is fragmentation. Submodules should be fine here since we are really looking at one place when publishing, but makes it hard for users to search for issues, and get help.
we did this for a while. the issue here is that means that every time there is a change in a package, you need to republish all packages that depend on it so that they have the latest version. otherwise they do not get the updated version.
This pattern has been working fine for typings though. They will just use their own version of the typings until they are updated, and thanks to using NPM this could be done fully automatically using Greenkeeper. It would run the tests and after merge can directly open the PR at DT with whatever tool we come up with (like typings publish).
Also this created a lot of conflicts, since the user installs node@6.0, and one of their dependencies depends on node@4.0 and another one depends on node@4.2.
Node is a special case though. Iirc TS2 now will resolve typings dependencies so that every declaration can have there own declaration dependency version of module x, as long as they are in external module format. Node however is a global declaration, it is not a module, it is an environment. As such, in typings, it has always been a "global dependency" that users need to install manually so there is only one in every project. With NPM it should be a peerDependency (side note: People already are putting @types/node as a dependency everywhere and it already broke my builds in several projects in patch releases, see my comment here #173 (comment)). We cannot just assume that every declaration version of Node will work because the versions could have totally different API.
Thanks for the explanation. my concern is fragmentation. Submodules should be fine here since we are really looking at one place when publishing, but makes it hard for users to search for issues, and get help.
That's why I like the submodule idea over JSON files, because in GitHub you can click on a submodule and it will take you to the repository. Imo the current structure makes it hard to get help, because you share one issue tracker for 2000 declarations.
Would expanding the pool of users with push rights to DT be an acceptable solution here?
For merging our own updates to declarations in the types org, definitely.
they do not. they depend on "*"
Doesn't this break versioning? i.e. packageA@1.2.3 depends on packageB@2.0.0 and packageC@3.4.5 depends on packageB@1.0.0.
I understand that you will run tests on DT to make sure they don't break. However, major version change does break things.
That's why proper versioning is needed.
I have not used @types in TS2.0 enough so by understanding may be wrong.
Also, as @felixfbecker mentioned, we can use GreenKeeper for this purpose.
As for testing, simple compile test is not sufficient, IMO: https://github.com/typings/generator-typings#about-writing-tests-for-typings
If this is landed and microsoft/TypeScript#7661 is available, you can leverage tests in the source package, making testing a much easier effort.
Just a quick update on this issue.
Approach:
- Submodules seem to be what everyone likes so far, so likely go with submodules.
- No change to DT test/publish infrastructure seem to be needed;
- The expectation is whatever rules (folder structure, file headers, compiler version, style, etc..) apply to a DT folder, would apply to submodule linked package
- We may need to make some tooling updates to simplify the process, e.g. nightly update PRs, or automatic merges for submodule update PRs.
- Open question: what does licencing mean here?
- Would every package have its own licence?
- Can licences be conflicting?
- Or should we require that all packages be under the same licence?
Timing
- There a couple of changes we would like to do first before enabling this change first; namely publishing different versions of a package : #3 and adding linter to the test infrastructure.
- @andy-ms is working on the linter rules now, so we should have this wrapped up soon
- publishing different versions is the next one.
Open question: what does licencing mean here?
The TypeScript team could restrict accepted submodules to open licenses which I'm sure wouldn't be a problem. Something like MIT, ISC and Apache 2.0. Aside from that, taking the LICENCE file and license from package.json should suffice with these licenses. I would probably suggest a quick review from a legal team to make sure, maybe there needs to be a contribution agreement (but I'm sure that sort of thing could also apply to regular PRs) for republishing work onto various mediums (NPM).
@mhegazy Three notes on the approach that I think should be considered:
- Compiler version: coupling submodules to the compiler version will make it very difficult for DefinitelyTyped to update without changes to all submodules (when/if that happens). Probably best not to run tests on the submodules directly to avoid that
- We're using
package.jsonfor the header metadata information - could that be resolved before looking forindex.d.tsand trying to read the header? Most of our repos do not actually useindex.d.tsbecause we follow the source repo style for deep require compatibility. Also, the version is already inpackage.json. Not to mention, we usepackage.jsonanyway for people who need tonpm linkornpm installduring local updates that haven't synced back to DefinitelyTyped - Happy to follow the same style guidelines, but it needs to be published so modules can use it (we currently use https://github.com/typings/tslint-config-typings)
Just as an update, publishing different versions is the current task that @andy-ms is working on - as @mhegazy mentioned, this issue will be addressed following that.
I spent a couple of hours trying to implement this, but couldn't find a way to do a recursive clone with node-git. The docs are just horrible. The closest thing I found was nodegit/nodegit#925
Is there any reason we cannot simply exec a child process? It would be so much simpler and node-git really seems overkill for the simple use case of cloning a repository.
Can a maintainer please comment on whether it would be acceptable to just use a git child process?
Sorry for the delay. I originally had tried using the git command line but had problems running it on Azure. I'll have to investigate.
Bump. Ran into an issue today where I had to remove node from typings because a @types module I had pulled in @types/node. I would like to clean up typings all together but cannot because of this issue. @blakeembrey
@andy-ms did you find out anything? What Azure setup is this running in? Is it a Docker container?
Is there any way we could help to speed this up?
Apology
I really owe you guys an update of where we are on this. I'm sorry that I haven't been more communicative here. Discussions have been happening internally and we should have posted notes from these meetings here to keep everyone appraised of the situation.
Status
As it stands, we don't think we can accept redirects at this time. Let me outline the pros and cons to be explicit that we're thinking about all parties' interests here. I'll use "definition owners" to refer to people who might want to establish an external repo to develop a .d.ts file in isolation -- for example, node.d.ts is extremely high-impact and high-traffic, and in an ideal world really would have its own repo. "Us" / "we" refers to the TypeScript team / @types in general.
The primary problems faced by definition owners today, faced with contributing to DT instead, are:
- Turnaround for PRs. The PR backlog is still too long despite us contributing almost an entire full-time engineer's worth of time on this.
- Repo size. The repo is too big. It takes a long time to clone or do other git operations.
- Scoping. There isn't a clear correspondence between issues on the issue tracker and the underlying definitions.
- Control. Due to our accelerated PR review process, it's possible (though somewhat unlikely) that a badly-written commit might get merged into a file where "proper" definition owners would have rejected the PR
- Communication bandwidth. Subscribing to DT is impossibly noisy in terms of notifications, but many interested parties do want to know what's going on with key definition files.
Now let me go over the things we're thinking about from the types-publisher / DT management side
- Testing. It's critical that anyone installing from
@typesdoesn't get definition files which have errors in them. - Global updates. We sometimes need to change a large number of definition files as an atomic operation. This can happen because of breaking changes (rare), infrastructure changes (common though hopefully less so in the future), renaming (somewhat common), compiler improvements (e.g. the "excess property" checks we added found a bunch of "bugs" in the test suites), or other unforeseen reasons (unknown unknowns...). We also want to sometimes enable new TSLint rules for the entire DT repo and fix them "everywhere" all at once - see Andy's PR history for examples here.
- Invasive updates. There are also times when a non-TS/non-DT person needs to update many (but not "all") files in the repo. This scenario needs to work with
The main proposal on the table is having git submodules which point to specific commits of external repos, or something isomorphic to that. At face value, this would certainly solve the definition owners' problems, but there are some really bad trade-offs here. Let me review the main definition owners' problems and discuss what we think a good alternative is.
Solution: External repos, but no redirect
The model we'd like people to move toward is to have their own repo (either on GitHub or wherever else). A copy of the .d.ts file itself remains in DefinitelyTyped. This can be set up in whatever file layout the owners desire. When an update occurs, either a bot or a person sends a PR to DT to update the .d.ts file. A comment in the header of the .d.ts file can point to the external repo URL for human-reading purposes. Additionally we could have DT-bot warn when a PR changing one of these files doesn't match the source version in the originating repo, if that'd be useful.
This enables some great new scenarios as well -- maybe the external repo doesn't actually have a .d.ts file checked in, but is instead the output of some more complex build process, for example.
Why not just have external repos give commit rights to the DT org? This doesn't solve the "invasive updates" scenario for non-MS/non-DT people who need to make wide changes.
Addressing problems
Let's review the problems, for both sides, listed above, and how this proposal affects them.
Turnaround
When a hypothetical external repo updates, they'll have to submit a PR to DT anyway to move the submodule pointer forward. This PR will end up with the same latency problems as the PRs today, so that problem doesn't actually get solved. Now you might say: we should set up a bot to automatically merge these PRs, and that's a great idea. But this only fixes PR turnaround for external repos.
Instead: what we really want to do is have a well-defined set of "trust" rules that would allow for automatic merging of ✔CI PRs. This could take the form of a per-folder OWNERS.md which lists people whose PRs can be merged as soon as CI passing. We'd have to be defining the set of trusted people implicitly by picking the "true" owning repo for a .d.ts file anyway.
Repo Size
A 580 MB repo isn't great to work with. But for the sorts of definitions people are talking about moving out, this problem doesn't go away. Anyone making changes to a .d.ts file needs to make sure they don't break any dependents of the file, and the only (reasonable) way to know which files depend on you is to have the entire repo so that the test runner can find them.
Instead: external repos can optimistically accept PRs and run the tests only when pushing a PR to DT. This is basically the same situation you'd be in with submodule redirects anyway.
Scoping
This gets solved equally well whether the external repo is a submodule or not.
Control
I think we generally get the best of both worlds here. We absolutely have to be able to make atomic updates to the entire DT repo, but having an external repo with the authors' "official" copy makes clear where those updates are coming from.
This is an explicit trade-off: External repo owners will have to be capable of taking "reverse" merges from DT from time to time.
Communication
This gets solved equally well whether the external repo is a submodule or not.
Testing
As stated above, everyone needs to be testing the "entire world" when making definition file updates.
Global Updates
With submodule redirects, global updates would be nearly impossible. Giving commit rights to everyone who wants to make global updates is a decent mitigation, but we'd have to somehow enforce that this was set up correctly at all times. As stated above, we're introducing a different cost ("reverse" merges from DT to the external repo), but we consider this to be preferable given the relatively small number of external repos we expect to exist.
Invasive Updates
This is the tricky one that isn't solved by giving commit rights to DT people. A reasonably common occurrence is that someone needs to change the internal structure of an export = target so that it can be correctly augmented in other modules. If the thing being refactored is in a submodule and has externally-hosted dependencies, this becomes an absolute nightmare - the author would have to send a PR to the main module, then a separate failing PR to the dependent module, then once both of those merge, she sends a submodule update PR to DT. But between when the second and third PRs merge, neither the first nor second module can send a submodule update to DT. And somehow you'd need to communicate the fact that the projects are in this state while it's all occurring so that the changes don't get treated like a normal build break and reverted.
Summary
I hope this outlines that we've thought carefully about this. The externally-hosted repo solution should fix a great majority of the problems presented by the monorepo, while still allowing good testing, refactorability, and predictability for the DT maintainers.
I don't entirely agree. You're solution sounds basically like "copy and paste" when we could use git submodules for the identical process. The solution you propose, as a result, fixes neither the issues definition authors face nor does it solve issues that you face. I can see the compromises you're trying to make, but I don't feel they are the entirely right ones.
For instance, testing is something you can already achieve. This doesn't resolve any communication confusion and will continue to result in communication issues when users don't know where to log issues. The ability for these "global updates" to occur now means that external authors need to somehow be able to pull the reverse changes (which you did note). However, as you also noted, this may be difficult for authors to do as the DefinitelyTyped structure is very particular and might not match the repo (e.g. build tools).
There's a few little things, that if you're going down this road, would be good to see fixed. The NPM package would need to have the correct repo URLs, bugs URL, etc (pointing to the main repo and not DefinitelyTyped). An automated publishing tool to DefinitelyTyped and "bundler" (to make it possible for multi-file and non-index.d.ts input sources to work in DefinitelyTyped).
All in all, some assumptions you make such as "we consider this to be preferable given the relatively small number of external repos we expect to exist" seem incorrect. Even Typings, which effectively "lived" for less than a year and was never "official" compared to DefinitelyTyped, has almost 400 repos which is around 15% of DefinitelyTyped. Also, in the "invasive updates" section you seemed to explain how regular dependencies work "as a nightmare". I understand the want to not have to submit cascading PRs, but in reality that's rarely an issue (how often is a wrong definition depended on down the line) and is also easily solved by following correct semver.
I'd like to see some of the solutions that will automate this for maintainers, because at this point (10 months after the issue was brought up) and with no communication, I don't have enough faith that even with this proposal that there'll be enough movement without members from @typings building it - and we don't have enough information to know what it is you want built. At this point, I'd also like to give away ownership of @types because I'm not sure I'd be able to maintain it in this future.
Another issue with the proposed solution is it does not seem to take versioning into consideration at all.
Your proposal is combining all typings in DT and consider it as one single version.
That's not how npm and package works.
When you say that "We absolutely have to be able to make atomic updates to the entire DT repo", that should be a huge red flag.
UPDATE: I also notice that @types does not seem to solve version conflicts either.
This issue comes up in SO several times and the situation will just get worse if it is not addressed.
Another issue with the proposed solution is it does not seem to take versioning into consideration at all.
I don't see how versioning is solved any better by submodule redirects. The resulting cloned repo which we'd use to publish to @types remains unchanged. Can someone sketch out a spec of how this would somehow be different?
This doesn't resolve any communication confusion and will continue to result in communication issues when users don't know where to log issues.
How exactly do submodule pointers solve the "where do I log an issue" problem? Do you think people will go to DT and first examine a long list to see which files are from submodule redirects and which aren't?
easily solved by following correct semver
I'd like to see a specification of how this would work in a way that's useful. "semver" when a .d.ts file depends on a version of the TS compiler and a version of the underlying library and corrective revisions to the .d.ts file itself and has dependencies on other files (all of which are subject to the same matrix) is not at all clear to me. Literally any non-comment edit to a .d.ts file could break someone, so a strict semver interpretation for .d.ts files would be isomorphic to hard-version-locking every single definition (which, in practice, is really not what you want the vast majority of the time). I mean, we could have just shipped @types with every new publish being a new major version number, but I don't think this is a good solution at all.
Your proposal is combining all typings in DT and consider it as one single version.
Again, I don't see how submodule pointers changes that situation at all.
When you say that "We absolutely have to be able to make atomic updates to the entire DT repo", that should be a huge red flag.
Files in DefinitelyTyped depend on each other, and they depend on the surrounding infrastructure. This is like saying that "I have to be able to modify multiple .js files in my program as an atomic commit" is a red flag -- it isn't. No one would or could program in a world where every individual source file in their program had an independent floating semver and each file had to be submitted to a passing CI server before being checked in.
If we had taken the submodule approach being proposed here, @types could never have happened. We would never be able to successfully add new repo-wide lint rules to DefinitelyTyped. We would not be able to make the sorts of high-breadth improvements (e.g. removing thousands of implicit anys from the codebase) we did during the @types transition. It just wouldn't be tractable. I get that the people on this thread are highly invested in easy independent repos, but we're looking at this from the perspective of the ultimate quality that TypeScript developers get when they pull from @types, and for not painting ourselves into a corner for the future of DT. If we start doing submodule redirects now, nearly all innovation on the DefinitelyTyped infrastructure stops, and stops forever, because we won't be able to make DT-wide changes in a way that allows us to keep a successful CI build pushing to NPM without incurring enormous logistical costs.
How exactly do submodule pointers solve the "where do I log an issue" problem?
It's easier to discover the location. I do believe people go and open the repo to find the definition and fix it, how else would someone be submitting PRs? Yes, it's not ideal, but it is closer. As long as it's extremely clear and the repo URLs are fixes, as I mentioned, it's hopefully less of an issue.
I'd like to see a specification of how this would work in a way that's useful. "semver" when a .d.ts
I absolutely agree, and I believe I've mentioned this issue a dozen times now including when the initial proposal for NPM and DefinitelyTyped occurred. As far as NPM goes, it's not possible to achieve today, but perhaps we can see what we can do in #12? It might be a matter of petitioning how ^ and ~ works in relation to pre-release versions so you can achieve something like 1.2.0-1.0.1.
I don't see how versioning is solved any better by submodule redirects.
I agree, it isn't. I think the concern here is the same one I have - you're only looking at definitions as a "update everything as once" problem, hence disregarding underlying versioning problems that occur from skipping how dependencies interact. For instance, there's the exact concern you've mentioned that'll occur with regular NPM packages distributing definitions - are you going to start coping those into DefinitelyTyped and ask them to "reverse" port the changes you make?
No one would or could program in a world where every individual source file in their program had an independent floating semver
Agreed, but this isn't that world. It goes to my point above - you're treating DefinitelyTyped as a giant mono-repo instead of just a registry. It doesn't really fit your analogy, because in your analogy you aren't publishing every single file as individual packages that you intend for authors to install as dependencies. I understand the concern of wanting to update everything together, but forcibly relying on this principle disregards everyone else. It's not impossible to use dependencies, we've all been using NPM for years now - it's just a trade-off and one thousands of people prefer.
If we had taken the submodule approach being proposed here, @types could never have happened.
This just isn't true. For instance, repo-wide lint rules are not important to dependencies - I don't care how someone writes their dependency (mostly). The improvements are an understandable case, but that only occurred because you needed to migrate from a previously broken system. That hasn't happened in Typings and shouldn't happen again anyway. If it did, all you'd have to do is remove the modules that aren't playing ball or fork them yourself (or pull them inline) which is really the same thing you'd do anyway.
nearly all innovation on the DefinitelyTyped infrastructure stops, and stops forever
What innovation do you expect would be limited? I can't imagine any scenario where data is changing enough that this could occur. And if it is, there's a bigger underlying problem because you have another whole world of NPM authors publishing definitions independent of DefinitelyTyped already. I'd like my registry to be as dumb as possible - it only has one job, move a to b.
I think it's all doable over time, my primary concern now would be by who? It took 10 months to get here and some contributors have consistently failed (including myself) to get any face time with people making the plan for something they'll never use themselves - will it be another year before we have a workflow that is usable as a result?
Just to throw in a few thoughts here
For instance, repo-wide lint rules are not important to dependencies - I don't care how someone writes their dependency (mostly).
Lint rules and fixes go beyond stylistic preference. As an example, recall the excess properties checking pull request. From a glance, DefinitelyTyped/DefinitelyTyped#5151 and the related pull requests constituted changes to 145 declaration files. These sort of changes work much better when considered holistically.
I think the concern here is the same one I have - you're only looking at definitions as a "update everything as once" problem, hence disregarding underlying versioning problems that occur from skipping how dependencies interact.
I think the reason we do this is the simple reality of the problem: once a .d.ts file is modified, nobody is motivated to update the prior version so that dependencies can keep playing nicely. For most contexts, you wouldn't want to. If there's an objectively better .d.ts file, then the ideal state is that everyone upgrades. In practice, most people updating a declaration file are drive-by committers (as is often the case with most contributions in the open-source world). Here, the key is not to reject them, but to find a way to maximize their value.
Instead of finding out 2 weeks later from the one confident user who reported an issue about it, we find out immediately that something went wrong. This is the benefit of the approach.
I actually have to run now, but if I get the chance I'll respond further tonight.
If there's an objectively better .d.ts file, then the ideal state is that everyone upgrades.
This is not how package work.
If you want to use lodash@4.1.5 and I want to use lodash@2.3.4, they would have different typings and one is not better than the other. We should be able to get our respective typings, which do not depends on an abstracted "single commit" of everything else.
I somewhat understand your approach to avoid version conflicts, but that's not solving the problem to begin with.
We should solve the version problem instead.
The version problem is solved by #264. @DanielRosenwasser is talking about different typing versions within the same major-version library.
Lint rules and fixes go beyond stylistic preference
In this case, does that mean we also need to find a way to have the test suite copied into DefinitelyTyped as a result? And if so, we'd need to stick with whatever DefinitelyTyped is using anyway?
If there's an objectively better .d.ts file, then the ideal state is that everyone upgrades
Absolutely, but I'm thinking as a consumer here. I don't want to be upgraded automatically. I've moved maybe four repos to @types, and I don't ever remember to do --save-exact. As a result, even though there's an objectively better definition available, I've been forced to use it breaking CI, CD, etc. A new user might be blocked until I can update to the objectively better definition. I also have this issue also with TypeScript, having to remember to do ~ version ranges for the same reasons. The TypeScript ecosystem is the only place, for me, where this is a major issue and something I need to remember because it doesn't follow semver.
Anyway, I'll try to move that discussion to the other issue, I just wanted to make it clear that because you're operating in this closed ecosystem you are likely ignoring these sorts of issues that exist outside of that ecosystem (DefinitelyTyped). I also want to note that none of this testing is related to whether the files are physically within DefinitelyTyped or not, so the argument here is really mute and we should probably focus on improving the other aspects that are issues.
The version problem is solved by #264.
@andy-ms Absolutely, I saw that. I disagree on the approach, which is part of a separate overall problem. You're not listening to or involving the community in decisions that are primarily being driven by the community. I personally don't feel that any value I could have added to this endeavour has been extracted. There's going to need to be some tooling made available for people to automate any sort of publishing to DefinitelyTyped, and I hope that'll open more dialog.
@DanielRosenwasser is talking about different typing versions within the same major-version library.
I'm aware of it. It's the first concern I brought up when @types was being formulated. I never found a solution within the NPM ecosystem since you can't dual version in any way (even by hacking around with pre-releases), but we can continue that discussion in #12. It's definitely in everyone's interest to resolve it.
First of all, thank you Ryan for speaking up.
You perfectly identified the current issues with DT in your first list, and they are major ones that actively keep people from contributing to DT and hurt the ecosystem.
On first impression, both the submodules-solution and what I will call the copy+paste solution would solve these issues. Lets see how they compare in regard to the DT management side list.
-
Testing I don't know if you are aware, but all the typings in the types org have tests and linting that are run in CI. This ensures the validity of a particular definition and catches the majority of problems. But just like now or with the copy+paste solution, you still get global tests run in CI when the hash is updated in the DT repo. Whether other definitions should be able to depend on older versions of definitions like with
typings(which npm would support greatly and allows to iterate faster overall) is a different issue and I would rather not discuss this here.
If any other typings in DT break, you can make a single, atomic PR that both updates submodule hashes and typings that need to be adapted. Yes, you will first make the change in the external repo and might discover the required modification only upon opening the PR at DT, but this is true for both the submodules and copy+paste solutions. All you have to do is also do the required modifications to all typings, just like you do now in DT. Again, whether or not that's optimal or not is a different issue, but it's equal for the solutions proposed and the current state.
We can therefor definitely ensure that installing from@typesdoesn't get definition files which have errors in them. -
Global/Invasive updates This kind of seems like the biggest concern here and I actually think that submodules are better for this than copy+paste. If you do a global update to DT with the copy+paste solution, there is no automatic way to upstream these to the external repo. There isn't even a way to notify them.
One thing I would like to mention here is that most definitions in the types org are not a single file, but follow the source library. If I understand you correctly, you proposed that these would get squashed into one file automatically for the copy+paste solution. This of course has disadvantages, you cannot declare multiple modules (for deep imports) if you want to keep the definition in external module format. You also mentioned how typings could be the result of a build script. Your solution proposes a path to convert from external repo to a single DT definition, but certainly the other way around will not be possible. So any global changes you do would not get upstreamed and the source-product relationship between external repo and definition in DT would break.
Now lets see how this would work with submodules: When you clone DT, you would get all the external repos inside your clone as subfolders. Whatever you do when you do a global update right now, you can execute the exact same steps. You can then commit these changes for all the submodules directly with a single git command,git submodule foreach. All updates will be done with minimal work in both the external repos and DT at once.
The alternative is doomed to cause out-of-sync between external repos and DT. Not only for internal global updates: If people see that there is a .d.ts file in DT, they will make PRs to change it directly.
Since you keep mentioning it as an important example, every external repo can easily use a single shared TSLint config published to npm.
Since you keep mentioning it as an important example, every external repo can easily use a single shared TSLint config published to npm.
Which we already have https://github.com/typings/tslint-config-typings
If you want, feel free to take over or submit PR to your liking.
Let's take a fresh look at the issue.
Shareholders
We have these shareholders:
- author: who originally creates the typings
- contributor: who make changes to the typings
- consumer: who consume the typings
- publisher: who accept, update, and publish typings to npm
@types
Use Cases
Here are the use-case for each shareholder.
Feel free to add more.
Author
Much are already discussed and identified:
- Can focus and work on individual typings
- No extra noise from DT.
- Manage issues specific to the typings
- Small repo size
- Can create typings in small files to enable deep links
Contributor
- Can submit issue and PR for specific typings
- Need to discover the location of the typings repo (hence the suggestion of submodule)
- Can test (and add test to) the typings. Don't need to get the whole DT to do that.
Consumer
- Consume typings through npm
- Can discover the location of the typings repo to file issue (i.e. become contributor)
Publisher
- Ensure minimal standard is preserved
- Make wide-spread updates (* not necessary IMO, will address below)
My Solution (redirect with no submodule)
My understanding of the submodule suggestion is to make the external repo easily discoverable.
However, this can be achieved with a simple README.md
Furthermore, this README.md and redirect.json can be created and updated automatically in two ways: a cli tool or invoke a small publish service during npm publish script.
types-publisher
A types-publisher cli tool (cli) will be available to do several things:
- Ensure minimum quality by running lint and tests on the repo
- Mandates the typings repo in specific format (e.g. lint, test, CI)
- Scaffold a typings repo project
- Create and submit PR to DT to invoke the final validation and publishing process
- Create and update two files:
redirect.jsonandREADME.md(more detail below)
- Create and update two files:
At DT, a bot (bot) will run to do the following:
- Based on the information in
redirect.jsonand/orREADME.md, pull in the external repo and run tests against it. Add a comment on the test and linting result in the PR for review- Alternatively, the
clialready enforce CI exists and thebotwill get the correct CI result directly.
- Alternatively, the
- If the PR is merged, publish to
@typeswill start (same as before, but pulling the code from external repo in CI).
npm publish script
The external repo can be published by the author to npm if they wanted to.
This means the consumer need to update their tsconfig.json/typesRoot in order to read that typings, not ideal.
The author can also add a publish script that invokes a publish service which will create a PR to DT as in the cli above.
README.md
This README file contains a few information that makes external typings repo discovery very easy:
- Link to the external typings repo and issue page
- Link to the source npm page and repo
- Version of the typings
- Version of the source supported
- Version of TypeScript supported
- (optional) change log
- (optional) typings description (extracted from
<typings repo>/package.json/description) - (optional) Author and contributor
- (optional) Known issue
Testing
I don't think this is a problem to begin with. You can always git clone the external repo and run tests in it or against it.
Also, remember that external repo can use submodule the source repo so that when microsoft/TypeScript#7661 is available, the typings repo can be tested against the source directly.
Global/Invasive/wide-spread updates
I understand your desire to keep the typings clean and up to date.
However, making an update like this is a non-starter IMO.
While with every TypeScript update the usability of the typings improves and we have the desire to update and utilize the latest and greatest features, it simply is not the thing we should do or take on lightly.
Think about ES6 vs ES5 and CommonJS.
ES6 is great, but there are still many packages written in cjs and will not update to the new format.
Making global update like this simply forcing the consumer to update and use the latest TypeScript available.
From your standpoint, this may not be an issue, but you know what happen to big companies and large projects. Updating any toolchain is not something can be done quickly or easily.
Also, there is another issue of "ownership".
When you make a global update, should the authors of all typings affected be notified and review the PR? Will you be open to waiting for the response from all authors? Or do you think that since you are the author of TS so whatever you do with the typings is superior then what the author can produce and the author should just accept it?
"reverse" merge
The same argument for Global Update applies to "reverse" merge.
This should not be considered as there should be a single source of truth, which is the external repo.
By not including the typings in DT but relying on redirect.json and README.md, you don't have to deal with this issue at all.
Address a few more loose points:
Already mentioned the drawback of the copy-paste approach
we need one place to run tests to maintain consistency. say you change node definition in a separate repro, we need to run all tests against all declarations that depend on node in Definitely typed before pushing a package.
First of all, while maintain "consistency" is desirable, why do we want that?
npm packages existed for a long time without this "consistency".
Also, "consistency" of what? Updating to the latest TypeScript version? Or write the typings in a certain format?
The TypeScript version issue as addressed in the last comment and for the other one the ecosystem and the community will handle that automatically.
And about testing against all declarations that depend on that package, the same problem exists on any npm packages.
The JS community resolves that problem by following semver and using tools such as greenkeeper, without the need of any dictator.
On a related note, I remembered another reason why the copy-paste approach might be difficult overall and it does come back to knowing where to contribute. For instance, if the TypeScript team is allowed to make these sweeping "reverse" merges, does that mean everyone can? How would someone new to DefinitelyTyped otherwise discover that the definition is stored somewhere else? A header description in the file or README might work, but it'll definitely be overlooked at least once, but with these approaches you now have the issue that as a maintainer of DefinitelyTyped, you need to ensure every PR is to a non-linked definition. This might add just a minute to every PR you review, but it'll add up.
Thankfully most points seem to be solvable by building tooling around it, but someone needs to build that tooling. For instance, the above concerns could be resolved by having @dt-bot close PRs to non-primary sources automatically. Maybe it could also be updated to parse issues in the future too. Also, thankfully, even if we do go with copy-paste we can still change approaches in the future. It's backward-compatible, while submodules wouldn't be, and I definitely understand that.
If someone wants to take on the tooling job and make it usable, I'm sure everyone will get on board. In the end it hopefully shouldn't make too much difference. The concept of "reverse" merges would be the most devastating as a maintainer unless you can heavily automate this. I will not remember to, or do, check DefinitelyTyped every time before I do anything in my external repo. You'll definitely need to have the bot automate some of this because having duplicate feature PRs might turn into a separate issue - one where most maintainers won't be able to help from DefinitelyTyped because they have zero permissions to close those PRs/issues while they could manage the same issues on their own repo.
I'd like to thank the TS team for bringing this discussion in the open again; I feel this has brought a lot of new momentum.
A "customer feedback" about versioning:
http://stackoverflow.com/questions/43217220/using-typescript-definitions-files-from-definitely-typed-with-older-version-of-t
More thoughts on how to implement this:
types-publishershould remain a tool that just accesses the files in DefinitelyTyped and publishes them.- We may change it to reflect the redirect in the generated
README.mdandpackage.json.
- We may change it to reflect the redirect in the generated
- The default for adding a new package to DefinitelyTyped should be to not use a redirect.
- For redirects, have a
redirect.jsonfile, or use// Definitions:in the header for this instead.- I think the redirect URL is all we need, so probably don't need
redirect.json?
- I think the redirect URL is all we need, so probably don't need
- All redirect packages should give
dt-review-botpermission to make pull requests/issues and to listen on events.
Tool tasks
dt-review-bot (and dt-review-tool) should be updated to:
-
Automatically accept pull requests that have been approved by a package author and pass tests.
-
Automatically accept pull requests that come from a redirect and pass tests.
-
Watch for any pull requests to DefinitelyTyped that touches a package with a redirect.
- If a change affects only a redirect package, loudly warn.
- If it touches a redirect package along with many others, still notify, but this is the preferred way to make changes affecting multiple types.
- If it pull request is merged, make a pull request to the redirect package updating it there.
- Of course, we should update
README.mdandPULL_REQUEST_TEMPLATE.mdto warn people about this beforedt-review-botdoes.
-
Watch for changes to the
masterbranch of the redirect repository, and make pull requests to DefinitelyTyped.- But if the redirect repository has not updated all the latest changes from DefinitelyTyped, it should not do this.
This is the scenario when something is fixed on DT, and the external repository does not apply the fix and then makes some other change. - Per above, these pull requests should be automatically accepted once tests pass.
- But if the redirect repository has not updated all the latest changes from DefinitelyTyped, it should not do this.
Structure
A redirect package should look like:
index.d.tsandfoo-tests.ts: Identical to what's on DefinitelyTyped.- Meaning, there should be no "compile" step.
index.d.tsdoesn't have to be the only declaration file present, but it should exist and should have a header.
tslint.json: Should extend"dtslint/dtslint.json"instead of"../tslint.json".- We should update the
tslint.jsons on DefinitelyTyped to do the same, actually, so identical in both places.
- We should update the
tsconfig.json: Mostly the same.- But
baseUrlandtypeRootsshould not be set, because there is no parent directory. - When copying to DefinitelyTyped, these should be added.
- But
package.json:
One problem is that on DefinitelyTyped we currently use a file going by this name to specify dependencies.
We should change the name to something likeexternalDependencies.jsoninstead, since it's not a realpackage.jsonanyway.
package.jsonshould have"devDependencies": { "dtslint": "latest" }and"scripts": { "lint": "dtslint --dt" }.- Note:
"dtslint": "latest"opts you in to have your lint broken by any updates. But any time we updatedtslintwe will updateDefinitelyTyped, and you will get a pull request fromdt-review-botfixing it.
- Note:
LICENSE: Should presumably be the same as the one in DefinitelyTyped's root. Then we wouldn't need to copy it over.- Any other files: An external repository probably needs a README and whatnot, but these shouldn't be copied to DefinitelyTyped.
It's a bit tricky if a redirect has dependencies. I think these dependencies should be put in the package.json as in "@types/foo": "latest". package.json won't be copied over, and on DefinitelyTyped we will rely on baseUrl and typeRoots instead.
So, the steps to set up a redirect would be:
- Make a repo.
- Give
dt-review-botpermissions (to make issues and pull requests and watch for changes). - Copy files from DefinitelyTyped. Add additional files (see above).
- Make a pull request to
DefinitelyTypedadding the redirect.
Issues
People should prefer to make issues in the redirect repository, but in case they don't, dt-review-bot should notify them.
It should also warn on any issues that do not include the issue template.
Also, it would be nice if someone filing an issue just had to fill in the name of the package, and dt-review-bot could look up the authors.
Meaning, there should be no "compile" step.
To give you an example why compile step is important and beneficial, see typed-typings/npm-text-buffer#3 (comment)
In github.com/types/* repositories, we add the source package as dev dependency and actually run the tests (i.e. require compiling).
With greenkeeper, this allows the typings maintainer to get up to date immediately with the latest version.
I have written about why no compile step is not sufficient and mentioned that is a previous comment. See: https://github.com/typings/generator-typings#about-writing-tests-for-typings
Another approach is to just use github.com/types/*. Have a bot as an owner in the organization.
The bot will be notified by github that a new PR is available.
It can then get the PR, ensure the toolchain is there and run tests.
When the PR is merged, then the bot can publish a new version to npm.
Contributors will use a cli tool or web portal to create new repo under the organization.
This way we can utilize many features available to the npm ecosystem for free: dependency management, source linking, real test, green keeper, CI, PR, etc.
Another example of how important the compile step is:
https://travis-ci.org/types/npm-react-router/builds/254269560?utm_source=github_status&utm_medium=notification
The underlying library removed the method call. Without an actual running test against the actual library, you will have to wait until someone files an issue to fix it. With actual tests, you catch it right away with the help of greenkeeper.
This is a relatively old issue. I originally was in favor of it, but today I am no longer really sure it's needed.
Typescript reached a point where I see interest by Javascript projects to host their own types. I tend to present it as documentation: it reduces ambiguity compared to free-form text and lets some editors to better understand the lib (providing a better experience: code completion, quick doc, etc.). Gulp is gradually adding types to its dependencies, chai is in the process of bundling its types, chalk bundles them, sequelize too, etc.
Because of this, I think that it's simpler to convince maintainers to keep type definitions (and their tests) in the main repo and use DT as a fallback if the maintainer is inactive or not interested. Redirects may still help for these libs, but in practice I found that developers don't mind hosting their own types.
Adding typings directly to the libs is always the recommended way.
Even though that has its own challenge (one challenge is correctly maintaining sem-ver).
This has always been about redirect from DT so that it is easier to maintain the typings when they are not directly distributed with the libs.
Personally, I have not been contributing to DT for a long time precisely because of this.
Now when I need some typings that do not come with the libs nor DT, I just define them locally.
It just doesn't make sense to me to get the whole DT repo and run tests just for adding one typings.
Another reason is the use of global script files vs module files.
I understand TS team like to have a tighter control of the typings, but as npm packages are self-regulated, I believe typings packages should be the same.
And as I pointed out above, there are ways to improve both the quality of the typings and the experience of writing them without the need to contributing to a single mono repo.
Typings from @types are easily accessed globally in a project not using modules; however, I have not been able to find documentation around getting types from projects that include them for the aforementioned scenario.
I am using MSBuild, so changing typeRoots/paths is not supported w/o creating a tsconfig.json which I really don't want to do. Any idea how to resolve those types in this situation without using import?