AWS MSK kafka Not authorized to access topics

Question

My Kafka cluster is IAM auth enabled. I am successfully able to produce and consume messages from topic test-topic2 by assuming the correct IAM role if the policy is as follows;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKafkaTopicWrite",
            "Effect": "Allow",
            "Action": [
                "kafka:*",
                "kafka-cluster:*"
            ],
            "Resource": "*"
        }
    ]
}

But now I want to narrow down the policy to a specific cluster, so I change it to following;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKafkaTopicWrite",
            "Effect": "Allow",
            "Action": [
                "kafka:*",
                "kafka-cluster:*"
            ],
            "Resource": [
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9",
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*",
                "arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/topic/test-topic2"
            ]
        }
    ]
}

I get following error on Producer side;

[2023-06-15 15:28:42,476] ERROR Error when sending message to topic test-topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-topic2]

And on consumer;

[2023-06-15 15:28:00,208] WARN [Consumer clientId=console-consumer, groupId=console-consumer-46486] Error while fetching metadata with correlation id 3 : {test-topic2=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2023-06-15 15:28:00,210] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-46486] Topic authorization failed for topics [test-topic2] (org.apache.kafka.clients.Metadata)
[2023-06-15 15:28:00,211] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-topic2]

What am I missing here ?

Answer 1

topic arn in your IAM policy seems incorrect, as per the documentation - https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#msk-iam-resources
Topic - arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/topic-name

so the topic arn in this case should be arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2