Infrastructure Status Updates
andrewbranch opened this issue · comments
This issue is an announcement board for known issues with DefinitelyTyped infrastructure, including CI, @typescript-bot automation, and npm publishing. To keep the noise down for those who want to subscribe to this issue, I’m locking the comments, but if you have questions or would like to report potential problems, please do so in the DefinitelyTyped Gitter.
Status badges
- Most recent build type-checked/linted cleanly:
- All packages are type-checking/linting cleanly on typescript@next:
- All packages are being published to npm in under an hour:
- typescript-bot has been active on DefinitelyTyped
✅ Current status: Ok ✅
There are no known repository-wide infrastructure issues. If you’d like to report one, please do so in the DefinitelyTyped Gitter.
Ongoing typescript-bot incident
@typescript-bot had a number of issues today, primarily adding incorrect labels and posting irrelevant comments on closed PRs. The source of the issue has been fixed, but many PRs that saw activity today still have incorrect labels. I will write a script to reprocess all labels tomorrow, but in the meantime, certain types of activity on the PR, such as editing the description, will also cause the bot to refresh them.
✅ Resolved: typescript-bot incident
All PRs that had activity in the last 40 hours have been re-processed by the bot to ensure labels are accurate. If you think the state of your PR is being represented incorrectly, please ping me on the PR.
Expect continued tweaks to @typescript-bot’s behavior as we monitor the new system in production. If you have feedback or would like to report a problem, please ping me on the problematic PR or on the DefinitelyTyped Gitter.
Missing and duplicate reviewer pings
For some of the time the new bot has been running, it has failed to ping owners for reviews. We’ve since fixed the problem and have been reprocessing PRs updated in the last four days to ensure owners are made aware of them.
Unfortunately, in some cases, this reprocessing failed to deduplicate previous pings that did go through, so some reviewers may have been pinged multiple times unnecessarily. This is an artifact of trying to fix up old PRs on the fly, and doesn’t reflect the typical behavior of the bot as it processes future PRs in real time. Nonetheless, we’re sorry for the spam.
⚠️ Duplicate typescript-bot comments
A bug caused @typescript-bot to double-comment on many PRs.
We think most causes of duplicate comments have been fixed, but some will take longer to fix.
✅ Resolved: Duplicate typescript-bot comments.
A separate bug caused @typescript-bot to issue duplicate comments pinging reviewers to update their reviews if that reviewer left more than one comment on a given review or more than one review on a given commit. It has now been fixed.
We believe we have now accounted for all causes of duplicate comments. If you see evidence to the contrary, please ping me on the PR where it occurred.
🚨 Incorrect dtslint failures 🚨
We’re currently investigating an issue that’s causing dtslint (which is run in CI) to report superfluous errors for the strict-export-declare-modifiers and void-return rules.
⚠️ Incorrect dtslint failues: workaround deployed ⚠️
We’ve temporarily disabled the offending rules until a full fix is implemented.
✅ Resolved: Incorrect dtslint failures
The issue was caused by dtslint and tslint each using a different version of TypeScript. We’ve fixed the issue by using peerDependencies instead of dependencies in certain dependency paths, and are continuing to do so in additional dependency paths to prevent further issues.
✅ Resolved: CI and publishing down Wednesday 4 AM - 4 PM GMT.
(9 PM Tuesday - 9 AM Wednesday PDT).
A PR that failed CI was merged and blocked subsequent CI runs as well as publishing. The PR was correct but a broken suggested-change commit got merged right before the whole PR merged.
I added myself as the author in f64a42d. Broken PRs will need to merge from master to get this commit.
Incorrect parsing shouldn't block tests or publication, and normally doesn't. I filed a bug to track this failure.
✅ Resolved: Publishing down Thursday 3 AM - 8 PM GMT.
(8 PM Wednesday - 1 PM Thursday PDT).
Publishing is broken because of a change to react-bootstrap, which currently has a v1 subdirectory, but a toplevel directory with 0.32. Any change, such as #45183, will cause publishing to fail. The publisher gets confused when trying to get the newest version, returns undefined, and tries to read react-bootstrap/undefined/index.d.ts instead of react-bootstrap/v1/index.d.ts. I fixed the crash in microsoft/DefinitelyTyped-tools#30 but CI needs an additional fix to forbid PRs that create this inverted version structure.
I also have a PR that removes v1 — it's not really needed — but that fails CI for other reasons, which I have a PR to fix as well. microsoft/DefinitelyTyped-tools#24
But I haven't been able to finish that PR because Definitely Typed keeps failing.
✅ Resolved: CI down 7 AM 2021/03/17 - 6 PM 03/18 UTC.
(12 AM Wednesday - 11 AM Thursday PDT).
CI and other dtslint-runner-based tests ran out of memory because of a nightly version of Typescript that uses more memory to parse JSDoc comments. I mitigated the problem by pinning to Tuesday's nightly build of Typescript, and reverting the PR that caused the memory increase: microsoft/TypeScript#43302.
I'll revert DT's mitigation once Typescript's new nightly build has published.
✅ Resolved: Publishing down from around 1 PM 2021/03/23 UTC - 2021/03/25 2 AM UTC.
(6 AM 03/23 PST - 7 PM 03/24).
A change in notNeededPackage.json's format incorrectly updated a retry case for deprecating packages. The first time that case happened about 12 hours after the change, publishing was blocked.
I shipped a mitigation (microsoft/DefinitelyTyped-tools#219) and a fix (microsoft/DefinitelyTyped-tools#220), but the publisher took several hours to come back up because Azure Web App deployments can be very slow.
@andrewbranch and I decided this was a good time to switch to Azure Functions. That is done, but packages with a lot of files can take a long time to publish, which can block publishing. We're still working on performance and improvements which should fix that problem too.
✔️ Resolved: Publishing backlogged from around 10 PM 2021/07/07 UTC - 12 AM 2021/07/09.
(2 PM 07/07 PDT - 5 PM 07/08).
I merged a PR to upgrade thousands of packages to be backward compatible with 4.4's new flag, exactOptionalPropertyTypes. This requires adding | undefined to every optional property. The publisher cleared the backlog at the start of 07/09.