Kafka Connect running out of heap space

Question

After starting Kafka Connect (connect-standalone), my task fails immediately after starting with:

java.lang.OutOfMemoryError: Java heap space     at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)     at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)     at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)     at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)     at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)     at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)     at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:343)     at org.apache.kafka.common.network.Selector.poll(Selector.java:291)     at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)     at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)     at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)     at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:193)     at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)     at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)     at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)     at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:316)     at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:222)     at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170)     at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)     at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)     at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)     at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)     at java.lang.Thread.run(Thread.java:745) 

There's a mention of heap space in some Kafka documentation, telling you to try it with "the default" and only modifying it if there are problems, but there are no instructions to modify the heap space.

Answer 1

When you have Kafka problems with

java.lang.OutOfMemoryError: Java heap space 

it doesn't necessarily mean that it's a memory problem. Several Kafka admin tools like kafka-topics.sh will mask the true error with this when trying to connect to an SSL PORT. The true (masked) error is SSL handshake failed !

See this issue: https://issues.apache.org/jira/browse/KAFKA-4090

The solution is to include a properties file in your command (for kafka-topics.sh this would be --command-config) and to absolutely include this line:

security.protocol=SSL 

Answer 2

You can control the max and initial heap size by setting the KAFKA_HEAP_OPTS environment variable.

The following example sets a starting size of 512 MB and a maximum size of 1 GB:

KAFKA_HEAP_OPTS="-Xms512m -Xmx1g" connect-standalone connect-worker.properties connect-s3-sink.properties 

When running a Kafka command such as connect-standalone, the kafka-run-class script is invoked, which sets a default heap size of 256 MB in the KAFKA_HEAP_OPTS environment variable if it is not already set.