how to implement counters in hadoop streaming in python

Question

I am new to hadoop streaming. I have few filter conditions in my reduce code, I would like to know how many records pass this conditions. I come to know we can do this by writing custom counters. Can some body show point me how to write custom counters?

I am emitting three columns in mapper code, say a,b,c key is a, and value as list, which is like [b,c], To have an example from mapper code, it is like ['I'^['C','P']]

Here is my reduce code.

labels = ["a","b"]
for line in sys.stdin:
    l = line.strip().split("^")
    key = l[0]
    value = l[1]
    record = [key] + value
    records.append(record)
df = pd.DataFrame.from_records(records,columns=labels)
df = df((df['a'] == 'I') & (df['b'] == 'C'))

I would like to know how many records df contains, at reducer level.

Thank you.

Answer 1

You can simply print to stderr:

print >> sys.stderr, "reporter:counter: CUSTOM, NbRecords,1"

This will increment counter "NbRecords" in counters group "CUSTOM" by 1