How can I access my AWS MSK managed kafka queue from my local machine and EC2 instances in other regions

Question

I'm setting up a managed kafka queue on AWS MSK. I can't seem to get the security to work when connecting from a local machine and I can't work out if I can use security groups from one region to another.

I've gone through the information on setting up the security groups on the main documentation here. I still can't seem to connect to the broker though. I'm currently using kafka scripts from my local machine in the following way:

bin/kafka-console-producer.sh --broker-list "my-broker-ip:9092" --topic "some-topic"

but keep getting the result

[2019-01-28 12:06:13,278] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've associated my local IP with all the ports on the security group that I set up with my VPC and associated with the kafka queue but it doesn't seem to have helped. I also don't see how I can associate my boxes with the kafka queue as they are in different regions. Is this possible?

I'm currently using kafka scripts from my local machine in the following way:

bin/kafka-console-producer.sh --broker-list "my-broker-ip:9092" --topic "some-topic"

but keep getting the result

[2019-01-28 12:06:13,278] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've associated my local IP with all the ports on the security group that I set up with my VPC and associated with the kafka queue but it doesn't seem to have helped. I also don't see how I can associate my boxes with the kafka queue as they are in different regions. Is this possible?

I'm expecting to be able to connect my local producer code the kafka queue and observe the output rather than have it constantly reject the connection.

Answer 1

As far as I know, you have to access your MSK cluster from a client machine on EC2, and cannot do so from a local machine.