Docker-ized Consul, Zookeeper and Kafka in Amazon-ECS

Question

I’m trying to deploy a dockerized Kafka (with Zookeeper) to AWS. I’ve combed through several resources, and got close. But when I ssh into my Kafka boxes in AWS, I can see that Kafka is exiting with -1. I can only assume that Kafka can’t reach Zookeeper, because I haven’t set up any name resolution. I’m looking at Consul as a solution for this (see here).

This looks like a decent resource. But I can’t even successfully run the official Consul docker image (in either server or agent mode). I can get this one to work (used here). But I’d like to use the official image if possible.

Has anyone gotten the official Consul image working with Kafka and Zookeeper?

A) To start, I think I’d like to get it working in docker-compose. This is my docker-compose.yml file, with me trying to have Kafka use Consul, to discover Zookeeper nodes.

Below is the relevant code chunk, which you can run with docker-compose up consul. This doesn't show any errors. But i) I can't reach http://localhost:8500 . And ii) How do I then point Kafka to Zookeeper and Kafka clients to Kafka?

version: '2'
services:

  consul:
    image: consul:0.8.3
    expose:
      - 8300
      - 8301
      - 8301/udp
      - 8302
      - 8302/udp
      - 8400
      - 8500
      - 8600
      - 8600/udp
    ports:
      - 8300:8300
      - 8301:8301
      - 8301/udp:8301/udp
      - 8302:8302
      - 8302/udp:8302/udp
      - 8400:8400
      - 8500:8500
      - 8600:8600
      - 8600/udp:8600/udp
    command: "agent -server -bind=0.0.0.0 -retry-join=0.0.0.0 -bootstrap-expect=1"

B) Then I can try to use it in AWS (via Terraform). Ideally, I’d like to have 10 Kafka nodes and 2 Zookeeper nodes. But that’s as a stretch goal.

EDIT

As @MattSchuchard pointed out, there is the option of Mesos, with Marathon and Chronos.

A) But I wanted to try out something that I thought would be more straightforward (i.e., Consul, Zookeeper, Kafka), before layering on another abstraction. Installing DCOS seems simplified with things like zutherb/terraform-dcos.

B) But going that route would mean changing the deployment workflow from i) just deploying ECS instances, to ii) deploying a DCOS cluster to AWS, then deploying docker images to that. If I could know that it were easy to deploy Consul, Zookeeper, Kafka, etc, containers to Marathon, then that would be good. Far as I can tell though, I’d have to use the docs cli, losing the declarative property of Terraform for my actual apps and services. Because as it stands, there’s no official Terraform provider for interacting with Marathon (nicgrayson/terraform-provider-marathon notwithstanding).

C) So I’ll take a look at Mesosphere DCOS. But hopefully there’s just a small config error with what I have up now.

Answer 1

Has anyone gotten the official Consul image working with Kafka and Zookeeper?

Yes, I have :) Please see https://github.com/dmstr/docker-roj/blob/master/data/example/discovery/consul/docker-compose.yml for an example config. Despite the mapped ports I am using network mode host for consul, which simply worked better.

You may take a look at the docs on how to setup a swarm with consul on AWS.

While this might not answer all your questions, it can still be a good resource to see how you can setup your swarm; since it's a pretty straightforward solution. Feel free to submit issues or PRs for roj.