How to use a separate file system for the Docker daemon related files at start up on AWS EC2

Question

I would like to use a separate overlayfs file system volume for the Docker daemon on AWS EC2. I have been able to configure and set this up myself manually by logging in and running the commands. I would now like to create an AWS AMI that does this automatically on start up so I don't have to do this myself each time.

I've tried googling this but I can't seem to find an AMI that does this already. Could anyone offer a suggestion on the best way to go about baking this into my custom AMI?

Answer 1

I ended up working out how to do this myself. I'm posting my results in case this is of any use to anyone else.

Creating an AMI

Assuming a base AMI that already has docker installed and using systemctl to manage start up services. I ran the following commands to switch from the docker daemon from using devicemapper to using overlayfs. Furthermore the old /var/lib/docker/devicemapper directory is removed, and the flag --graph=/mnt/ specifies that directory /mnt will be used to store all the docker images, containers, and etc.

sudo service docker stop
sudo rm -rf /var/lib/docker/devicemapper
sudo sed -i -e "/ExecStart/s/$/ --storage-driver=overlay --graph=\/mnt\//" /lib/systemd/system/docker.service
sudo systemctl daemon-reload

Next create the directory where the file system will be mounted. Update /etc/fstab that it is mounted when the EC2 instance starts.

sudo mkdir -p /mnt
echo '/dev/xvdf       /mnt    ext4    defaults,nofail 0 2' | sudo tee --append /etc/fstab

Create an new amazon AMI with these changes. I used packer for this and found it quite straightforward.

EBS Snapshot

You'll need an already formatted EBS snaphot to attach to each launched EC2 instance. I did this by creating a EC2 instance with a new EBS volume, formatting the attached EBS volume. The AWS website has documentation, for example formatting as Ext4 is:

sudo mkfs -t ext4 device_name

Once you've done this you'll need to turn the EBS volume into an EBS snapshot. You can do this by going in the AWS, and clicking "EC2" -> "Volumes" (Left Menu). Then select the volume attached to the EC2 instance and select "Actions" -> "Create Snapshot". It'll then take a few minutes to create the snapshot.

Launching an instance

Once you've done all of this you should have the following:

• An AMI expect a mounted device at /dev/xvdf.
• A Docker daemon configured to save it's graph database to a volume under /mnt.
• An Ex4 formatted EBS snapshot to be used as this volume.

I then used the following BlockDeviceMappings configuration to mount this snapshot as a volume when launching the instance. Replace the all caps values with your values. You should use the AMI you created to launch the instance. I used the DeleteOnTermination : true because I don't care about the docker related files after I shutdown the instance.

"BlockDeviceMappings": [
  {
    "DeviceName": "/dev/sdf",
    "Ebs": {
      "SnapshotId": "SNAPSHOT_ID",
      "Encrypted": false,
      "DeleteOnTermination": true,
      "VolumeSize": VOL_SIZE
    }
  }
],

Trouble shooting

• Make sure to use an base AMI that can use EBS volumes
• Make sure your EBS volume is formatted with as a file system e.g. Ext4
• Check whether /dev/xvdf is the mounted device and adjust accordingly if not.
• See my packer github repository for the commands I used to build my AMI.