How do I avoid crushing Docker with too many containers?

Question

My Story

Once upon a time I was calling Docker from a PHP laravel queue to process thousands of media files. My code would create a new container for each task I wanted to complete (e.g. "process", "search", "slice" etc). Then I went to bed and disaster struck. I woke up to a 1TB log file, and a few hundred thousand containers in various states.

Docker doesn't work any more. As in, when I type docker version (or docker anything for that matter) it just sits there and stares. I would like to avoid reinstalling, but I can't remove the containers through the standard docker rm $(docker ps -a -q) because Docker has become very, very depressed.

My Questions

1. How can I remove docker containers if the docker daemon isn't responding?

2. My code is using Docker as a glorified executable using the docker API. What extra steps do I need my code take to clean after itself in future?

3. In general I have noticed that when I try to spin up too many (i.e. 20) containers at the same time, Docker was prone to timing out. I would love to be able to crank that dial up.

Some Information

Here's how I'm currently calling the daemon:

$> sudo docker daemon
INFO[0000] API listen on /var/run/docker.sock
INFO[0000] [graphdriver] using prior storage driver "aufs"

(Meanwhile in another terminal)
$> docker version
(insert infinite darkness here)

Answer 1

I did some digging.

How can I remove docker containers if the docker daemon isn't responding?

In Ubuntu, containers are stored in the /var/lib/docker/containers directory. Delete the contents of that directory and things are just that much better. This will live in other places depending on your system and installation configuration.

Also, for anybody curious, there were around 250k containers that had to be removed. Given the number of files I had to go with the INCREDIBLY DANGEROUS: ls /var/lib/docker/containers | xargs -n200 rm -rf <-- DO NOT TYPE THAT IN YOUR COMMAND LINE IF YOU DON'T KNOW WHAT YOU ARE DOING.

I'm using Docker as a glorified executable being called from code using their API. What extra steps do I need my code take to clean after itself in future?

If the container isn't going to be used again, you should officially "remove" it after you're done. On command line this would be done using docker rm $(containername) and in code it will completely depend on how docker is being accessed. I heard you can also run docker using docker run --rm which will clean up after closing.

I've found that even after attempting to remove programatically, there will be cases where containers don't close out. To prevent that I set up a cron that would regularly docker ps -a | awk '{print $1}' | xargs docker rm

I have noticed that when I try to spin up too many (i.e. 20) containers at the same time, Docker was prone to timing out.

This stopped once I cleared things out, however I still think that I should be using something more like Docker Swarm for this use case. In my case I decided to simply set up the production environment to remove the need for docker there (which is where it was using multiple threads).