devcluster
A developer tool for running the Determined cluster.
Configuring and Running
You can run the code from the git root by running python -m devcluster
, or
install it into your python environment with pip install -e .
and then just
call devcluster
by itself.
The first time you run it, it will give you the option to automatically install
the default configuration file (see devcluster/example.yaml
).
By default, devcluster needs to run from the root of a determined
source
tree. You can do this in one of a few ways:
- always run
devcluster
from the root of yourdetermined
source tree - set
cwd: /path/to/determined
to rundevcluster
from anywhere - run
devcluster
with the-C /path/to/determined
option
You can either specify a config file via the --config
or -c
option, or
devcluster
will try to read the default config file at
~/.devcluster.yaml
.
devcluster
Works
How devcluster runs a cluster with a linear dependency graph of stages. With the
default config (devcluster/example.yaml
), those stages are:
DEAD
DB
MASTER
AGENT
Because of the linear dependency graph, the DB
stage can run by itself, the
MASTER
stage is only started after DB
is already up, and the AGENT
is only
started after the MASTER
is up. The DEAD
stage indicates that nothing
is running. The non-DEAD
stages correspond to the stages
setting in your
config file.
The status bar has two rows. The first row is the state row. It shows all of
the stages in your dependency graph. All active stages are highlighted orange.
There is also a 'target state', which is indicated by a less-than character
(<
). You set the target state by pressing the key corresponding to a
particular stage.
When the target state is modified, devcluster will walk up or down the stage list, starting or killing processes as needed to reach the target stage. In the default configuration, the MASTER and AGENT stages are rebuilt each time they are started.
A common pattern during development might be to:
- make a change to the master code
- press
1
to set the target state toDB
(killing theMASTER
and theAGENT
) - press
3
to set the target state back toAGENT
(rebuilding/restarting theMASTER
andAGENT
on the way).
The second row in the status bar is the logging row. Every stage collects a separate stream of logs, and each stream can be toggled by pressing the corresponding keys.
Keybindings
1
-3
set target states for the cluster (as shown in the status bar)!
-#
toggle logs for the corresponding stage (as shown in the status bar)q
orctrl
+c
once to quit, or twice to force-quit (which may leave dangling processes or docker containers laying around)u
/d
will scroll up/down, andU
/D
will scroll up/down faster Scrolling support is only partially line-aware; it will scroll by literal log chunks, which may each be more or less than a linex
will reset scrolling to continue following logs- Spacebar will emit a visual separator with a timestamp to the console stream.
- With the default configuration, the keys
p
,w
, andc
will trigger building the python harness, webui, and docs, respectively.
Server/Client Mode
You can run devcluster
as a headless server with various configurations for
listening for clients:
# listen on 0.0.0.0:1234
devcluster --quiet --listen 1234
# listen on localhost:1234
devcluster -q -l 127.0.0.1:1234
# listen on a unix socket
devcluster -q -l /path/to/socket
# listen on the default unix socket (/tmp/devcluster/sock)
devcluster -q
# configure multiple listeners
devcluster -q -l 1234 /path/to/socket
Sending a SIGINT
(via ctrl
+c
) or a SIGTERM
(via kill
) to the headless
server will close it.
You can then connect the UI to a headless server:
devcluster 1234
devcluster host:1234
devcluster /path/to/socket
In client mode, pressing q
or ctrl
+c
will only close the UI; it will not
affect the server.
Oneshot Mode
You can run devcluster
in a special oneshot mode (-1
/--oneshot
), suitable
for automated usage. In automated mode, the stdout
from devcluster
will be
suitable for redirecting to a file, and the stderr
will emit the line:
devcluster is up
to indicate when the cluster is ready for use.
Philosophy
devcluster
tries as much as possible to be a standalone tool for running
arbitrary versions of the Determined cluster. It is tested as far back as
v0.12.3.
Some defaults are kept up-to-date with tip-of-main to simplify the config file for the most-common use case, but in general preference was given to being configurable and explicit.
TODO
- config file reloading