It seems like most popular .net client for Kafka (https://github.com/confluentinc/confluent-kafka-dotnet) is missing methods to setup and create Topics.
When calling Producer.ProduceAsync()
the topic is created automatically but I can't find a way to setup partitions, retention policy and other settings.
I tried to find any examples online but all I found just use defaults.
Maybe there is another .net client that I can use instead?
Building the Kafka producer and Kafka consumer Net in both the producer and the consumer application. Incidentally, there are many providers available, but in this post we will be using kafka-net, a native C# client for Apache Kafka. You can install kafka-net via the NuGet package manager from within Visual Studio.
Apache Kafka is an open source message broker that provides high throughput, high availability, and low latency. Apache Kafka can be used either on its own or with the additional technology from Confluent. Confluent Kafka provides additional technologies that sit on top of Apache Kafka.
Construct a Kafka Consumer You also need to define a group.id that identifies which consumer group this consumer belongs. Then you need to designate a Kafka record key deserializer and a record value deserializer. Then you need to subscribe the consumer to the topic you created in the producer tutorial.
It is now available in latest release of Confluent.Kafka .Net client library.
See: https://github.com/confluentinc/confluent-kafka-dotnet/blob/b7b04fed82762c67c2841d7481eae59dee3e4e20/examples/AdminClient/Program.cs
using (var adminClient = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = bootstrapServers }).Build())
{
try
{
await adminClient.CreateTopicsAsync(new TopicSpecification[] {
new TopicSpecification { Name = topicName, ReplicationFactor = 1, NumPartitions = 1 } });
}
catch (CreateTopicsException e)
{
Console.WriteLine($"An error occured creating topic {e.Results[0].Topic}: {e.Results[0].Error.Reason}");
}
}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With