Kafka minimal (production) setup

Question

We plan to use Apache Kafka in our product as it seems to handle well horizontal scaling and shock absorbing - and our cloud environment requires these features.

But we also plan to deploy our product to envrironments, where RAM consumption is a priority. These environments will however not require big throughput. Let's say 1MB/s, with bigger deployments 5MB/s. Messages will be rather small, 1kB each.

We do not require HA in for these deployments, nor any perfect performance. We just need it to be stable, 24h retention (we'll have enough space on reasonable disks).

Would it be possible to run Kafka under these conditions with 1GB RAM? (or even less). If not, what will be the minimal setup?

Answer 1

Just in case anyone is also interrested in this question. We've tested single node Kafka with Xmx=1GB, ingesting twitter stream API (~50 messages a second, each 5-10kB).

We also tried to stop kafka consumer for some time, so that there is a lot of messages, when it gets started again. And we've achived 2500 messages a second on this side of Kafka.

Machines used: 2x t2.medium from AWS - one for ingest and broker, one for consumer.