How to remove empty rows from an Pyspark RDD

Question

I am having few empty rows in an RDD which I want to remove. How can I do it?

I tried the below but it is not working. I am still getting the empty rows

json_cp_rdd = xform_rdd.map(lambda (key, value): get_cp_json_with_planid(key, value)).filter(
            lambda x: x is not None).filter(
            lambda x: x is not '')

[u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'[{ "PLAN_ID": "d2031aed-175f-4346-af31-9d05bfd4ea3a", "CostTotalInvEOPAmount": 0.0, "StoreCount": 0, "WeekEndingData": "2017-07-08", "UnitTotalInvBOPQuantity": 0.0, "PriceStatus": 1, "UnitOnOrderQuantity": null, "CostTotalInvBOPAmount": 0.0, "RetailSalesAmount": 0.0, "UnitCostAmount": 0.0, "CostReceiptAmount": 0.0, "CostSalesAmount": 0.0, "UnitSalesQuantity": 0.0, "UnitReceiptQuantity": 0.0, "UnitTotalInvEOPQuantity": 0.0, "CostOnOrderAmount": null}]', u'', u'', u'', u'', u'', u'', u'', u'', u'']

Answer 1

is checks object identity not equality. In Python 2.x you could use !=

.filter(lambda x: x is not None).filter(lambda x: x != "")

but idiomatically you can use only a single filter with identity:

.filter(lambda x: x)

or directly with bool:

.filter(bool)

Answer 2

replaced filter(lambda x: x is not '') with filter(lambda x: x is not u'') and it worked out