Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to create a Kafka Topic using Confluent.Kafka .Net Client

It seems like most popular .net client for Kafka (https://github.com/confluentinc/confluent-kafka-dotnet) is missing methods to setup and create Topics. When calling Producer.ProduceAsync() the topic is created automatically but I can't find a way to setup partitions, retention policy and other settings.

I tried to find any examples online but all I found just use defaults.

Maybe there is another .net client that I can use instead?

like image 632
Michael D. Avatar asked Apr 27 '18 13:04

Michael D.


People also ask

Does Kafka support C#?

Building the Kafka producer and Kafka consumer Net in both the producer and the consumer application. Incidentally, there are many providers available, but in this post we will be using kafka-net, a native C# client for Apache Kafka. You can install kafka-net via the NuGet package manager from within Visual Studio.

What is difference between Kafka and Confluent Kafka?

Apache Kafka is an open source message broker that provides high throughput, high availability, and low latency. Apache Kafka can be used either on its own or with the additional technology from Confluent. Confluent Kafka provides additional technologies that sit on top of Apache Kafka.

How do I create a Kafka consumer for a topic?

Construct a Kafka Consumer You also need to define a group.id that identifies which consumer group this consumer belongs. Then you need to designate a Kafka record key deserializer and a record value deserializer. Then you need to subscribe the consumer to the topic you created in the producer tutorial.


1 Answers

It is now available in latest release of Confluent.Kafka .Net client library.

See: https://github.com/confluentinc/confluent-kafka-dotnet/blob/b7b04fed82762c67c2841d7481eae59dee3e4e20/examples/AdminClient/Program.cs

        using (var adminClient = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = bootstrapServers }).Build())
        {
            try
            {
                await adminClient.CreateTopicsAsync(new TopicSpecification[] { 
                    new TopicSpecification { Name = topicName, ReplicationFactor = 1, NumPartitions = 1 } });
            }
            catch (CreateTopicsException e)
            {
                Console.WriteLine($"An error occured creating topic {e.Results[0].Topic}: {e.Results[0].Error.Reason}");
            }
        }
like image 115
Alexander Bortnik Avatar answered Oct 28 '22 06:10

Alexander Bortnik