Ceph Rados Block Device Docker VolumeDriver Plugin

Use Case: Persistent Storage for a Single Docker Container
- an RBD Image can only be used by 1 Docker Container at a time
Plugin is a separate process running alongside Docker Daemon
- plugin can only be configured for a single Ceph User (currently no good way to pass things from docker -> plugin, get only volume name)
- run multiple plugin instances for varying configs (ceph user, default pool, default size)
- can pass extra config via volume name to override default pool and creation size:
  - docker run --volume-driver rbd -v poolname/imagename@size:/mnt/disk1 ...
plugin supports all Docker VolumeDriver Plugin API commands:
- Create - can provision Ceph RBD Image in a pool of a certain size
  - controlled by --create boolean flag (default false)
  - default size from --size flag (default 20480 = 20GB)
- Mount - Locks, Maps and Mounts RBD Image to the Host system
- Unmount - Unmounts, Unmaps and Unlocks the RBD Image on request
- Remove - Removes (destroys) RBD Image on request
  - only called for docker run --rm -v ... or docker rm -v ...
  - action controlled by plugin's --remove flag, which can be one of three values:
    - ''ignore'' - the call to delete the ceph rbd volume is ignored (default)
    - ''rename'' - will cause image to be renamed with zz prefix for later culling
    - ''delete'' - will actually delete ceph rbd image (destructive)
- Get, List
for there is problems in "exclusive-lock" feature, it must be disabled
this plugin must run before docker daemon

Generate RPMs

$ sudo yum install -y rpm-build
$ make rpms

Plugin Setup

Plugin is a standalone process and places a Socket file in a known location. Generally need to start this before running Docker. It does not daemonize as it is expected to be controlled by systemd, so if you need it in the background, use normal shell process control (&).

The socket has a name (sans .sock) which is used to refer to the plugin via the --volume-driver=name docker CLI option, allowing multiple uniquely named plugin instances with different default configurations.

The default name for the socket is "rbd", so you would refer to --volume-driver rbd from docker.

General build/run requirements:

librados2-devel and librbd1-devel for go-ceph
/usr/bin/rbd for mapping and unmapping to kernel
/usr/sbin/mkfs.xfs for fs creation
/usr/bin/mount and /usr/bin/umount

Tested with Ceph version 10.2.2 on Centos 7.1 host with Docker 1.12.6 and docker api version 1.24.

Commandline Options

Usage of rbd-docker-plugin:
  -cluster string
        xtao ceph cluster (default "xtao")
  -config string
        Xtao ceph cluster config (default "/etc/ceph/xtao.conf")
  -create
        Can auto Create RBD Images (default true)
  -debug
        Debug output
  -fs string
        FS type for the created RBD Image (must be xfs now) (default "xfs")
  -go-ceph
        Use go-ceph library
  -logdir string
        Logfile directory (default "/var/log")
  -mount string
        Mount directory for volumes on host (default "/var/lib/docker-volumes")
  -name string
        Docker plugin name for use on --volume-driver option (default "rbd")
  -plugins string
        Docker plugin directory for socket (default "/run/docker/plugins")
  -pool string
        Default Ceph Pool for RBD operations (default "rbd")
  -remove value
        Action to take on Remove: ignore, delete or rename (default ignore)
  -size int
        RBD Image size to Create (in MB) (default: 20480=20GB) (default 20480)
  -use-nbd
        Use rbd-nbd to map RBD Image (default true)
  -user string
        Ceph user (default "admin")
  -version
        Print version

Running Plugin

Start with the default options:

socket name=rbd, pool=rbd, user=admin, logfile=/var/log/rbd-docker-plugin.log
no creation or removal of volumes

sudo rbd-docker-plugin

docker run --volume-driver rbd -v ...

For Debugging: send log to STDERR:

sudo RBD_DOCKER_PLUGIN_DEBUG=1 rbd-docker-plugin

Use a different socket name and Ceph pool

sudo rbd-docker-plugin --name rbd2 --pool liverpool
# docker run --volume-driver rbd2 -v ...

To allow creation of new RBD Images:

sudo rbd-docker-plugin --create

To allow creation and removal:

sudo rbd-docker-plugin --create --remove

Testing

Use with docker 1.8+ which has the --volume-driver support.

https://docker.com/

Alternatively, you can POST json to the socket to test the functionality manually. If your curl is new enough (v7.40+), you can use the --unix-socket option and syntax. You can also use this golang version instead:

go get github.com/Soulou/curl-unix-socket

Once you have that you can POST json to the plugin:

% sudo curl-unix-socket -v -X POST unix:///run/docker/plugins/rbd.sock:/Plugin.Activate
> POST /Plugin.Activate HTTP/1.1
> Socket: /run/docker/plugins/rbd.sock
> Content-Length: 0
>
< HTTP/1.1 200 OK
< Content-Type: appplication/vnd.docker.plugins.v1+json
< Date: Tue, 28 Jul 2015 18:52:11 GMT
< Content-Length: 33
{"Implements": ["VolumeDriver"]}


# Plugin started without --create:
% sudo curl-unix-socket -v -X POST -d '{"Name": "testimage"}' unix:///run/docker/plugins/rbd.sock:/VolumeDriver.Create
> POST /VolumeDriver.Create HTTP/1.1
> Socket: /run/docker/plugins/rbd.sock
> Content-Length: 21
>
< HTTP/1.1 500 Internal Server Error
< Content-Length: 62
< Content-Type: appplication/vnd.docker.plugins.v1+json
< Date: Tue, 28 Jul 2015 18:53:20 GMT
{"Mountpoint":"","Err":"Ceph RBD Image not found: testimage"}

# Plugin started --create turned on will create unknown image:
% sudo curl-unix-socket -v -X POST -d '{"Name": "testimage"}' unix:///run/docker/plugins/rbd.sock:/VolumeDriver.Create
> POST /VolumeDriver.Create HTTP/1.1
> Socket: /run/docker/plugins/rbd.sock
> Content-Length: 21
>
< HTTP/1.1 200 OK
< Content-Length: 27
< Content-Type: appplication/vnd.docker.plugins.v1+json
< Date: Fri, 14 Aug 2015 19:47:35 GMT
{"Mountpoint":"","Err":""}

Examples

If you need persistent storage for your application container, you can use a Ceph Block Device as a persistent disk.

You can provision the Block Device and Filesystem first, or allow a sufficiently configured Plugin instance create it for you. This plugin can create RBD images with XFS filesystem.

(Optional) Provision RBD Storage yourself

sudo rbd create --size 1024 foo
sudo rbd map foo => /dev/rbd1
sudo mkfs.xfs /dev/rbd1
sudo rbd unmap /dev/rbd1

Or Run the RBD Docker Plugin with --create option flag

sudo rbd-docker-plugin --create

Requesting and Using Storage

docker run --volume-driver=rbd --volume foo:/mnt/foo -it ubuntu /bin/bash
Volume will be locked, mapped and mounted to Host and bind-mounted to container at /mnt/foo
When container exits, the volume will be unmounted, unmapped and unlocked
You can control the RBD Pool and initial Size using this syntax:
- foo@1024 => pool=rbd (default), image=foo, size 1GB
- deep/foo => pool=deep, image=foo and default --size (20GB)
- deep/foo@1024 => pool=deep, image=foo, size 1GB
- pool must already exist

Misc

RBD Snapshots: sudo rbd snap create --image foo --snap foosnap
Resize RBD image:
- set max size: sudo rbd resize --size 2048 --image foo
- map/mount and then fix XFS: sudo xfs_growfs -d /mnt/foo

TODO

add cluster config options to support non-default clusters
figure out how to test
- do we need a working ceph cluster?
- docker containers with ceph?

Packaging

Using tpkg package to distribute and specify native package dependencies. Tested with Centos 7.1 and yum/rpm packages.

License

This project is using the MIT License (MIT), see LICENSE file.

yanghonggang / rbd-docker-plugin-nbd

Ceph Rados Block Device Docker VolumeDriver Plugin

Generate RPMs

Plugin Setup

Commandline Options

Running Plugin

docker run --volume-driver rbd -v ...

Testing

Examples

Misc

TODO

Links

Docker API Documentation

Code Examples and Libraries

Packaging

License

About

Languages