How to verify that Amazon Kinesis Python client is working

Question

I'm trying to build an Amazon Kinesis Python consumer using the KCL library for Python (https://github.com/awslabs/amazon-kinesis-client-python). I started by checking the sample code. I was able to run both producer and consumer scripts parts of the sample code, but I'm unable to verify if the data from my kinesis stream (with one shard) is being pushed to the sample Python consumer script, sample_kclpy_app.py.

I used the amazon_kclpy_helper.py to generate the Java command that would call the Python script through the sample.properties file. I ran the Java command, and I can see from the terminal output that the data from the Kinesis stream is being read. I added a print statement in the process_record function of the Python consumer script to check if the data was being pushed to it. But it doesn't show up in the terminal output.

I also tried using logging to generate STDOUT messages, as well as write to a file. I also added assert 0 line in the Python code to force fail the script and see an exception would appear in the log output. Then, I purposely added a syntax error in the Python code. However, it seems that all of these were not detected by the Java MultiLangDaemon, which just kept going and churning out INFO log messages.

What could be the problem? And is there a better way to check if the data is indeed being sent to the process_record function of the Python consumer script?

Answer 1

After numerous trial and error, I figured out that the Python script assigned to executableName should be visible from the Java KCL process that is calling it.

The quick fix is to set the PATH environmental variable to the directory of the script and make the script an executable file (e.g. chmod +x)

Answer 2

I was having the same issue where my code just seemed to never run. It wasn't until I looked at the Node version that I realized that executableName in sample.properties was the command that is run and not just the name of the python file. I changed executableName = sample_kclpy_app.py to executableName = python sample_kclpy_app.py and it started working as expected.