How do producers and consumers usually work when sending a video file?

Question

In my understanding, when I want to send a movie (4GB) to a Kafka broker, one producer will send that 4GB byte of a video file (serialized it) and send it to a kafka broker and many consumers who want to see that movie will consume that movie file.

I heard Netflix uses Kafka to send and watch movies. I am curious how they use producer, broker, and consumer. I'm using Netflix, and it's really fast. I want to know how they use Kafka.(especially how they use producers and consumers)

And as far as I know, when sending a video file, you need to encode it, and serialize it to send the data. (maybe encoding is serializing in this case?) Did I understand correctly? If I am missing something, could you give me some tips and guidance?

Answer 1

Netflix uses Kafka as part of its centralized data lineage solution. It is not using Kafka to encode, stream video contents. You can read more about how Kafka is being used here.

Now to answer your question on why its video streaming services are so fast. You'll need to understand how Netflix leverages aws resources like ec2, s3 and others to create a highly scalable, fault-tolerant microservice architecture.

On top of this Netflix works with ISPs to localize contents using a program called Netflix Open Connect. This allows them to cache the content locally which minimizes latency and saves on compute.

Answer 2

Kafka is a "Streaming Platform" but it's intended for streaming data and it's not designed to stream videos or audio.

While Netflix is using Kafka, it's not to stream videos to users but instead to process events in their backend, see their technology blog. Note that I'm not a Netflix employee nor I have any insider knowledge, it's just based on the information they disclosed publicly on their blog and at conferences.

That said, it's still possible to send a video file using a producer and receive it with a consumer but I don't think it's what you had in mind.