ceph rbd with docker volumes

Question

I just initialized a ceph instances within two differents servers

cluster 241b5d19-15f5-48be-b98c-285239d70038
 health HEALTH_WARN
        64 pgs degraded
        64 pgs stuck degraded
        64 pgs stuck unclean
        64 pgs stuck undersized
        64 pgs undersized
 monmap e3: 2 mons at {serv1=10.231.69.9:6789/0,serv2=10.231.69.34:6789/0}
        election epoch 6, quorum 0,1 serv1,serv2
    mgr no daemons active
 osdmap e10: 2 osds: 2 up, 2 in
        flags sortbitwise,require_jewel_osds,require_kraken_osds
  pgmap v22: 64 pgs, 1 pools, 0 bytes data, 0 objects
        68292 kB used, 1861 GB / 1861 GB avail
              64 active+undersized+degraded

with only mon and osd (I do not setup mds, rgw or CephFS).

I would use rbd to create a persisted shared storage for containers volumes, but I'm really confused on how to plug my osd inside docker.

I saw some rbd docker plugins exists:

1. https://github.com/yp-engineering/rbd-docker-plugin
2. https://github.com/AcalephStorage/docker-volume-ceph-rbd
3. https://github.com/contiv/volplugin

But none seems to be compatible with latest docker version or at least >= 1.13.

Thus I'm asking myself how can I achieve what I want, some solutions come on my mind but I'm really not sure which is best (or even if it's possible).

1. Use CephFS + standard docker filesystem mounting volume
2. Use rexray (flocker is no more maintained)
3. Install Ceph Object Gateway S3 and use existing docker S3 plugins
4. Other?

But 1. solution seems to be inelegant and will be harder to manage in larger environment (more than 2 servers).

Whereas 2. solution seems to be a great starting point but anyone have feedback?

Answer 1

4. You can extend Docker Engine with a v2 plugin for RBD volumes.

Docker volume plugin for RBD works with Docker Engine v2 plugin system. (we use it in with Swarm clusters)

Here an excerpt from the Readme:

install the plugin:

docker plugin install wetopi/rbd \
  --alias=wetopi/rbd \
  LOG_LEVEL=1 \
  RBD_CONF_POOL="ssd" \
  RBD_CONF_CLUSTER=ceph \
  RBD_CONF_KEYRING_USER=client.admin

create a volume:

docker volume create -d wetopi/rbd -o size=206 my_rbd_volume

docker volume ls
DRIVER              VOLUME NAME
local               069d59c79366294d07b9102dde97807aeaae49dc26bb9b79dd5b983f7041d069
local               11db1fa5ba70752101be90a80ee48f0282a22a3c8020c1042219ed1ed5cb0557
local               2d1f2a8fac147b7e7a6b95ca227eba2ff859325210c7280ccb73fd5beda6e67a
wetopi/rbd          my_rbd_volume

run a container with rbd volume:

docker run -it -v my_rbd_volume:/data --volume-driver=wetopi/rbd busybox sh