Extremely fast file change detector and task orchestrator for Node.js.
If you ever wanted something like Nodemon but more capable, then you are at the right place.
Basic usage:
npm install turbowatch
cat > turbowatch.ts <<'EOD'
import { watch } from 'turbowatch';
void watch({
project: __dirname,
triggers: [
{
expression: ['match', '*.ts', 'basename'],
name: 'build',
onChange: async ({ spawn }) => {
await spawn`tsc`;
},
},
],
});
EOD
npm exec turbowatchNote See logging instructions to print logs that explain what Turbowatch is doing.
Refer to recipes:
- Rebuilding assets when file changes are detected
- Restarting server when file changes are detected
- Retrying failing triggers
- Gracefully terminating Turbowatch
- Handling the
AbortSignal - Tearing down project
- Throttling
spawnoutput
| Turbowatch | Nodemon | |
|---|---|---|
| Node.js interface (scriptable) | ✅ | ❌1 |
| Graceful termination (teardown) | ✅ | ❌2 |
| Scriptable child processes (zx) | ✅ | ❌ |
| Retries | ✅ | ❌ |
| Debounce | ✅ | ❌ |
| Interruptible workflows | ✅ | ❌ |
| Concurrent workflows | ✅ | ❌ |
| Log grouping | ✅ | ❌ |
| Works with long-running processes | ✅ | ✅ |
| Works with build utilities and REPLs | ✅ | ✅ |
| Watch specific files or directories | ✅ | ✅ |
| Ignoring specific files or directories | ✅ | ✅ |
| Open source and available | ✅ | ✅ |
1 Undocumented
2 Nodemon only provides the ability to send a custom signal to the worker.
Turbowatch defaults are a good choice for most projects. However, Turbowatch has many options that you should be familiar with for advance use cases.
import {
watch,
type ChangeEvent,
} from 'turbowatch';
void watch({
// AbortController used to gracefully terminate the service.
// If none is provided, then Turbowatch will gracefully terminate
// the service when it receives SIGINT.
abortSignal: new AbortController().signal,
// Debounces triggers by 1 second.
// Most multi-file spanning changes are non-atomic. Therefore, it is typically desirable to
// batch together information about multiple file changes that happened in short succession.
// Provide { debounce: { wait: 0 } } to disable debounce.
debounce: {
wait: 1000,
},
// The base directory under which all files are matched.
// Note: This is different from the "root project" (https://github.com/gajus/turbowatch#project-root).
project: __dirname,
triggers: [
{
// Expression match files based on name.
// https://github.com/gajus/turbowatch#expressions
expression: [
'anyof',
['match', '*.ts', 'basename'],
['match', '*.tsx', 'basename'],
],
// Indicates whether the onChange routine should be triggered on script startup.
// Defaults to false. Set it to false if you would like onChange routine to not run until the first changes are detected.
initialRun: true,
// Determines what to do if a new file change is detected while the trigger is executing.
// If {interruptible: true}, then AbortSignal will abort the current onChange routine.
// If {interruptible: false}, then Turbowatch will wait until the onChange routine completes.
// Defaults to true.
interruptible: false,
// Name of the trigger. Used for debugging
// Must match /^[a-z0-9-_]+$/ pattern and must be unique.
name: 'build',
// Routine that is executed when file changes are detected.
onChange: async ({ spawn }: ChangeEvent) => {
await spawn`tsc`;
await spawn`tsc-alias`;
},
// Routine that is executed when shutdown signal is received.
onTeardown: async ({ spawn }) => {
await spawn`rm -fr ./dist`;
},
// Label a task as persistent if it is a long-running process, such as a dev server or --watch mode.
persistent: false,
// Retry a task if it fails. Otherwise, watch program will throw an error if trigger fails.
retry: {
retries: 5,
},
},
],
});To abstract the complexity of orchestrating file watch operations.
For context, we are using Turborepo. The reason this project came to be is because Turborepo does not have "watch" mode (issue #986).
At first, we attempted to use a combination of tsc --watch, concurrently and Nodemon, but started to run into things breaking left and right, e.g.
- services restarting prematurely (before all the assets are built)
- services failing to gracefully shutdown and then failing to start, e.g. because ports are in use
Furthermore, the setup for each workspace was repetitive and not straightforward, and debugging issues was not a great experience because you have many workspaces running in watch mode producing tons of logs. Many of the workspaces being dependencies of each other, this kept re-triggering watch operations causing the mentioned issues.
In short, it quickly became clear that we need the ability to have more control over the orchestration of what/when needs to happen when files change.
We started with a script. At first I added debounce. That improved things. Then I added graceful termination logic, which mostly made everything work. We still had occasional failures due to out-of-order events, but adding retry logic fixed that too... At the end, while we got everything to work, it took a lot of effort and it still was a collection of hacky scripts that are hard to maintain and debug.
In the end, Turbowatch is a toolbox for orchestrating and debugging file watch operations based on everything we learned along the way.
Note If you are working on a very simple project, i.e. just one build step or just one watch operation, then you don't need Turbowatch. Turbowatch is designed for monorepos or otherwise complex workspaces where you have dozens or hundreds of build steps that depend on each other (e.g. building and re-building dependencies, building/starting/stopping Docker containers, populating data, sending notifications, etc).
Turbowatch can be used to automate any sort of operations that need to happen in response to files changing, e.g.,
- You can run (and automatically restart) long-running processes (like your Node.js application)
- You can build assets (like Docker images)
The spawn function that is exposed by ChangeEvent is used to evaluate shell commands. Behind the scenes it uses zx. The reason Turbowatch abstracts zx is to enable auto-termination of child-processes when triggers are configured to be interruptible.
Expressions are used to match files. The most basic expression is match – it evaluates as true if a glob pattern matches the file, e.g.
Match all files with *.ts extension:
['match', '*.ts', 'basename']Expressions can be combined using allof and anyof, e.g.,
Match all files with *.ts or *.tsx extensions:
[
'anyof',
['match', '*.ts', 'basename'],
['match', '*.tsx', 'basename']
]Finally, not evaluates as true if the sub-expression evaluated as false, i.e. inverts the sub-expression.
Match all files with *.ts extension, but exclude index.ts:
[
'allof',
['match', '*.ts', 'basename'],
[
'not',
['match', 'index.ts', 'basename']
]
]This is the gist behind Turbowatch expressions. However, there are many more expressions. Inspect Expression type for further guidance.
type Expression =
// Evaluates as true if all of the grouped expressions also evaluated as true.
| ['allof', ...Expression[]]
// Evaluates as true if any of the grouped expressions also evaluated as true.
| ['anyof', ...Expression[]]
// Evaluates as true if a given file has a matching parent directory.
| ['dirname' | 'idirname', string]
// Evaluates as true if a glob matches against the basename of the file.
| ['match' | 'imatch', string, 'basename' | 'wholename']
// Evaluates as true if the sub-expression evaluated as false, i.e. inverts the sub-expression.
| ['not', Expression];Note Turbowatch expressions are a subset of Watchman expressions. Originally, Turbowatch was developed to leverage Watchman as a superior backend for watching a large number of files. However, along the way, we discovered that Watchman does not support symbolic links (issue #105). Unfortunately, that makes Watchman unsuitable for projects that utilize linked dependencies (which is the direction in which the ecosystem is moving for dependency management in monorepos). As such, Watchman was replaced with chokidar. We are hoping to provide Watchman as a backend in the future. Therefore, we made Turbowatch expressions syntax compatible with a subset of Watchman expressions.
import { watch } from 'turbowatch';
void watch({
project: __dirname,
triggers: [
{
expression: ['match', '*.ts', 'basename'],
name: 'build',
onChange: async ({ spawn }) => {
await spawn`tsc`;
await spawn`tsc-alias`;
},
},
],
});import { watch } from 'turbowatch';
void watch({
project: __dirname,
triggers: [
{
expression: [
'anyof',
['match', '*.ts', 'basename'],
['match', '*.graphql', 'basename'],
],
// Because of this setting, Turbowatch will kill the processes that spawn starts
// when it detects changes when it detects a change.
interruptible: true,
name: 'start-server',
onChange: async ({ spawn }) => {
await spawn`tsx ./src/bin/wait.ts`;
await spawn`tsx ./src/bin/server.ts`;
},
},
],
});Retries are configured by passing a retry property to the trigger configuration.
/**
* @property factor The exponential factor to use. Default is 2.
* @property maxTimeout The maximum number of milliseconds between two retries. Default is Infinity.
* @property minTimeout The number of milliseconds before starting the first retry. Default is 1000.
* @property retries The maximum amount of times to retry the operation. Default is 10. Seting this to 1 means do it once, then retry it once.
*/
type Retry = {
factor?: number,
maxTimeout?: number,
minTimeout?: number,
retries?: number,
}The default configuration will retry a failing trigger up to 10 times. Retries can be disabled entirely by setting { retries: 0 }, e.g.,
{
expression: ['match', '*.ts', 'basename'],
onChange: async ({ spawn }: ChangeEvent) => {
await spawn`tsc`;
},
retry: {
retries: 0,
},
},AbortController is used to gracefully terminate Turbowatch.
If none is provided, then Turbowatch will gracefully terminate the service when it receives SIGINT signal.
const abortController = new AbortController();
void watch({
abortSignal: abortController.signal,
project: __dirname,
triggers: [
{
name: 'test',
expression: ['match', '*', 'basename'],
onChange: async ({ spawn }) => {
// `sleep 60` will receive `SIGTERM` as soon as `abortController.abort()` is called.
await spawn`sleep 60`;
},
}
],
});
// SIGINT is the signal sent when we press Ctrl+C
process.once('SIGINT', () => {
abortController.abort();
});The abort signal will propagate to all onChange handlers. The processes that were initiated using spawn will receive SIGTERM signal.
Workflow might be interrupted in two scenarios:
- when Turbowatch is being gracefully shutdown
- when routine is marked as
interruptibleand a new file change is detected
Implementing interruptible workflows requires that you define AbortSignal handler. If you are using zx, such abstraction could look like so:
Note Turbowatch already comes with
zxbound to theAbortSignal. Just usespawn. Documentation demonstrates how to implement equivalent functionality.
import { type ProcessPromise } from 'zx';
const interrupt = async (
processPromise: ProcessPromise,
abortSignal: AbortSignal,
) => {
let aborted = false;
const kill = () => {
aborted = true;
processPromise.kill();
};
abortSignal.addEventListener('abort', kill, { once: true });
try {
await processPromise;
} catch (error) {
if (!aborted) {
console.log(error);
}
}
abortSignal.removeEventListener('abort', kill);
};which you can then use to kill your scripts, e.g.
void watch({
project: __dirname,
triggers: [
{
expression: ['match', '*.ts', 'basename'],
interruptible: false,
name: 'sleep',
onChange: async ({ abortSignal }) => {
await interrupt($`sleep 30`, abortSignal);
},
},
],
});onTeardown is going to be called when Turbowatch is gracefully terminated. Use it to "clean up" the project if necessary.
Warning There is no timeout for
onTeardown.
import { watch } from 'turbowatch';
const abortController = new AbortController();
void watch({
abortSignal: abortController.signal,
project: __dirname,
triggers: [
{
expression: ['match', '*.ts', 'basename'],
name: 'build',
onChange: async ({ spawn }) => {
await spawn`tsc`;
},
onTeardown: async () => {
await spawn`rm -fr ./dist`;
},
},
],
});
process.once('SIGINT', () => {
abortController.abort();
});When multiple processes are sending logs in parallel, the log stream might be hard to read, e.g.
redis:dev: 973191cf > #5 sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948 4.19MB / 9.60MB 22.3s
api:dev: a1e4c6a7 > [18:48:37.171] 765ms debug @utilities #waitFor: Waiting for database to be ready...
redis:dev: 973191cf > #5 sha256:d01ec855d06e16385fb33f299d9cc6eb303ea04378d0eea3a75d74e26c6e6bb9 0B / 1.39MB 22.7s
api:dev: a1e4c6a7 > [18:48:37.225] 54ms debug @utilities #waitFor: Waiting for Redis to be ready...
worker:dev: 2fb02d72 > [18:48:37.313] 88ms debug @utilities #waitFor: Waiting for database to be ready...
redis:dev: 973191cf > #5 sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948 5.24MB / 9.60MB 22.9s
worker:dev: 2fb02d72 > [18:48:37.408] 95ms debug @utilities #waitFor: Waiting for Redis to be ready...
redis:dev: 973191cf > #5 sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948 6.29MB / 9.60MB 23.7s
api:dev: a1e4c6a7 > [18:48:38.172] 764ms debug @utilities #waitFor: Waiting for database to be ready...
api:dev: a1e4c6a7 > [18:48:38.227] 55ms debug @utilities #waitFor: Waiting for Redis to be ready...In this example, redis, api and worker processes produce logs at almost the exact same time causing the log stream to switch between outputting from a different process every other line. This makes it hard to read the logs.
By default, Turbowatch throttles log output to at most once a second, producing a lot more easier to follow log output:
redis:dev: 973191cf > #5 sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948 4.19MB / 9.60MB 22.3s
redis:dev: 973191cf > #5 sha256:d01ec855d06e16385fb33f299d9cc6eb303ea04378d0eea3a75d74e26c6e6bb9 0B / 1.39MB 22.7s
redis:dev: 973191cf > #5 sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948 5.24MB / 9.60MB 22.9s
redis:dev: 973191cf > #5 sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948 6.29MB / 9.60MB 23.7s
api:dev: a1e4c6a7 > [18:48:37.171] 765ms debug @utilities #waitFor: Waiting for database to be ready...
api:dev: a1e4c6a7 > [18:48:37.225] 54ms debug @utilities #waitFor: Waiting for Redis to be ready...
api:dev: a1e4c6a7 > [18:48:38.172] 764ms debug @utilities #waitFor: Waiting for database to be ready...
api:dev: a1e4c6a7 > [18:48:38.227] 55ms debug @utilities #waitFor: Waiting for Redis to be ready...
worker:dev: 2fb02d72 > [18:48:37.313] 88ms debug @utilities #waitFor: Waiting for database to be ready...
worker:dev: 2fb02d72 > [18:48:37.408] 95ms debug @utilities #waitFor: Waiting for Redis to be ready...However, this means that some logs might come out of order. To disable this feature, set { throttleOutput: { delay: 0 } }.
Turbowatch uses Roarr logger.
Export ROARR_LOG=true environment variable to enable log printing to stdout.
Use @roarr/cli to pretty-print logs.
ROARR_LOG=true turbowatch | roarrThe biggest benefit of using Turbowatch is that it provides a single abstraction for all file watching operations. That is, you might get away with Nodemon, concurrently, --watch, etc. running in parallel, but using Turbowatch will introduce consistency to how you perform watch operations.
Many tools provide built-in watch functionality, e.g. tsc --watch. However, there are couple of problems with relying on them:
- Running many file watchers is inefficient and is probably draining your laptop's battery faster than you realize. Turbowatch uses a single server to watch all file changes.
- Native tools do not allow to combine operations, e.g. If your build depends on
tscandtsc-alias, then you cannot combine them. Turbowatch allows you to chain arbitrary operations.
Note There are some valid use cases for using native watch mode (e.g.
next dev). However, even in those cases you should consider wrapping those operations in Turbowatch for consistency, e.g.void watch({ project: __dirname, triggers: [ { expression: [ 'anyof', ['match', '*', 'basename'], ], // Marking this routine as non-interruptible will ensure that // next dev is not restarted when file changes are detected. interruptible: false, name: 'start-server', onChange: async ({ spawn }) => { await spawn`next dev`; }, }, ], });
I have seen concurrently used to "chain" watch operations such as:
concurrently "tsc -w" "tsc-alias -w"While this might work by brute-force, it will produce unexpected results as the order of execution is not guaranteed.
If you are using Turbowatch, simply execute one command after the other in the trigger workflow, e.g.
async ({ spawn }: ChangeEvent) => {
await spawn`tsc`;
await spawn`tsc-alias`;
},Turborepo currently does not have support for watch mode (issue #986). However, Turbowatch has been designed to work with Turborepo.
To use Turbowatch with Turborepo:
- define a persistent task
- run the persistent task using
--parallel
Example:
"dev": {
"cache": false,
"persistent": true
},turbo run dev --parallelNote We found that using
dependsOnwith Turbowatch produces undesirable effects. Instead, simply use Turbowatch rules to identify when dependencies update.
Note Turbowatch is not aware of the Turborepo dependency graph. Meaning, that your builds might fail at the first attempt. However, thanks to retries and debounce, it will start working after warming up. We are currently exploring how to reduce preventable failures. Please open an if you would like your ideas to be considered.