Installing librdkafka on Windows to support Python development

Question

A little background: I am working on some python modules that other developers on our team will use. A common theme of each module is that one or more messages will be published to Kafka. We intend at this time to use the Confluent Kafka client. We are pretty new to python development in our organization -- we have traditionally been a .NET shop.

The complication: while the code that we create will run on Linux (rhel 7), most of the developers will do their work on Windows.

So we need the librdkafka C library compiled on each developer machine (which has dependencies of its own, one of which is OpenSSL). Then a pip install of confluent-kafka should just work, which means a pip install of our package will work. Theoretically.

To start I did the install on my Linux laptop (Arch). I knew I already had OpenSSL and the other zip lib dependencies available, so this process was painless:

• git clone librdkafka repo
• configure, make and install per the README
• pip install confluent-kafka
• done

The install of librdkafka went into /usr/local:

/usr/local/lib/librdkafka.a
/usr/local/lib/librdkafka++.a
/usr/local/lib/librdkafka.so -> librdkafka.so.l
/usr/local/lib/librdkafka++.so -> librdkafka++.so.l
/usr/local/lib/librdkafka.so.l
/usr/local/lib/librdkafka++.so.l
/usr/local/lib/pkgconfig/rdkafka.pc
/usr/local/lib/pkgconfig/rdkafka++.pc
/usr/local/include/librdkafka/rdkafkacpp.h
/usr/local/include/librdkafka/rdkafka.h

Now the painful part, making it work on Windows:

• install precompiled OpenSSL
• git clone librdkafka repo
• open in VS2015
• install libz via NuGet
• build solution
• install to where???

This is where I'm stuck. What would a standard install on a Windows 7/8/10 machine look like?

I have the following from the build output, but no idea what should go where in order to make the pip install confluent-kafka "just work":

/librdkafka/win32/Release/librdkafka.dll
/librdkafka/win32/Release/librdkafka.exp
/librdkafka/win32/Release/librdkafka.lib
/librdkafka/win32/Release/librdkafkacpp.dll
/librdkafka/win32/Release/librdkafkacpp.exp
/librdkafka/win32/Release/librdkafkacpp.lib
/librdkafka/win32/Release/zlib.dll
<and the .h files back in the src>

Any recommendations on an install location?

Answer 1

I'm not sure where the ideal place to install on Windows would be, but I ran the following test with some success.

I copied my output and headers to C:\test\lib and C:\test\include, then ran a pip install with the following options:

 pip install --global-option=build_ext --global-option="-LC:\test\lib" --global-option="-IC:\test\include" confluent-kafka

Unfortunately, this doesn't quite work because the confluent-kafka setup does not support Windows at this time: https://github.com/confluentinc/confluent-kafka-python/issues/52#issuecomment-252098462