richardelling / node-disk-manager

Kubernetes Storage Device Management

node-disk-manager

node-disk-manager aims to make it easy to manage the disks attached to the node. It treats disks as resources that need to be monitored and managed just like other resources like CPU, Memory and Network. It is a daemon which runs on each node, detects attached disks and loads them as Disk objects (custom resource) into Kubernetes.

While PVs are well suited for stateful workloads, the Disk objects are aimed towards helping hyper-converged Storage Operators by providing abilities like:

• Easy to access the inventory of Disks available across the Kubernetes Cluster.
• Predict failures on the Disks, to help with taking preventive actions.
• Allow for dynamically attaching/detaching Disks to a Storage Pod, without requiring a restart.

The design and implementation are currently in progress. The design is covered under this design proposal

Usage

A detailed usage documentation is maintained in the wiki.

Start Node Disk Manager DaemonSet

• Edit ndm-operator.yaml to fit your environment: Set the namespace, serviceAccount, configure filters in the node-disk-manager-config-map.
• Switch to Cluster Admin context and create the DaemonSet with kubectl create -f ndm-operator.yaml.

Using kubectl to fetch Disk Information

• kubectl get disks --show-labels displays the disks across the cluster, with kubernetes.io/hostname showing the node to which disk is attached.
• kubectl get disks -l "kubernetes.io/hostname=<hostname>" displays the disks attached to node with the provided hostname.
• kubectl get disk <disk-cr-name> -o yaml displays all the details of the disk captured by ndm for given disk resource.

Build Image

• go get or git clone node-disk-manager repo into $GOPATH/src/github.com/openebs/ with one of the below directions:

  • cd $GOPATH/src/github.com/openebs && git clone [e-mail]:openebs/node-disk-manager.git
  • cd $GOPATH/src/github.com/openebs && go get github.com/openebs/node-disk-manager

• Setup build tools:

  • By default node-disk-manager enables fetching disk attributes using udev. This requires udev develop files. For Ubuntu, libudev-dev package should be installed.
  • make bootstrap installs the required Go tools.
  • NDM uses SeaChest to probe for drive details. Use the following to setup the SeaChest libraries:

    pushd .
    cd ..
    git clone --recursive --branch Release-19.06.02 https://github.com/openebs/openSeaChest.git
    cd openSeaChest/Make/gcc 
    make release
    cd ../../
    sudo cp opensea-common/Make/gcc/lib/libopensea-common.a /usr/lib 
    sudo cp opensea-operations/Make/gcc/lib/libopensea-operations.a /usr/lib 
    sudo cp opensea-transport/Make/gcc/lib/libopensea-transport.a /usr/lib
    popd
    

• run make in the top directory. It will:

  • Build the binary.
  • Build the docker image with the binary.

• Test your changes

  • sudo -E env "PATH=$PATH" make test execute the unit tests.
  • make integration-test will launch minikube to run the tests. Make sure that minikube can be executed via sudo -E minikube start --vm-driver=none.

Push Image

By default travis pushes the docker image to openebs/node-disk-manager-amd64, with ci as well as commit tags. You can push to your custom registry and modify the ndm-operator.yaml file for your testing.

License

Inspiration

• Thanks to Daniel for setting up the go-based SMART library.
• Thanks to Humble, Jan and other from the Kubernetes Storage Community for reviewing the approach and evaluating the use-case.