Store images in Apache Kafka?

Question

Do any big users of Kafka (e.g. LinkedIn) store uploaded images in Kafka? I like the architectural simplicity of putting everything into the log, but I am concerned that it may not be workable in practice.

Answer 1

Theoretically there is no restriction as you could easily store images as binary in Kafka Queue. But there could be other issues .. I would try to put some thought on this

There is a message.max.bytes parameter in the consumer config and the default value is 1000000. This is done actually to prevent brokers from running out of memory because the consumer does not have option for streaming a message and has to allocate memory to be able to read the message. A workaround could be to compress messages in Kafka in order to save spaces.

The most suitable condition I could find is to write your own client implementation by passing pre-allocation of send&receive buffers, providing zero-copy file transfer over Kafka as seen in this discussion.

You need to check if it is possible to make such effort to achieve the same.