Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

'list' object has no attribute 'map' in pyspark

I'm new in pyspark . I write this code in pyspark:

def filterOut2(line):
    return [x for x in line if x != 2]
filtered_lists = data.map(filterOut2)

but I get this error:

'list' object has no attribute 'map'

How do I perform a map operation specifically on my data in PySpark in a way that allows me to filter my data to only those values for which my condition evaluates to true?

like image 715
zahra Avatar asked Nov 03 '17 07:11

zahra


People also ask

What does list object has no attribute mean Python?

The Python "AttributeError: 'list' object has no attribute" occurs when we access an attribute that doesn't exist on a list. To solve the error, access the list element at a specific index or correct the assignment.

What does it mean when object has no attribute?

It's simply because there is no attribute with the name you called, for that Object. This means that you got the error when the "module" does not contain the method you are calling.


1 Answers

map(filterOut2, data) works:

>>> data = [[1,2,3,5],[1,2,5,2],[3,5,2,8],[6,3,1,2],[5,3,2,5],[4,1,2,5] ]
... def filterOut2(line):
...     return [x for x in line if x != 2]
... list(map(filterOut2, data))
...
[[1, 3, 5], [1, 5], [3, 5, 8], [6, 3, 1], [5, 3, 5], [4, 1, 5]]

map() takes exactly 1 argument (2 given)

Looks like you redefined map. Try __builtin__.map(filterOut2, data).

Or, use a list comprehension:

>>> [filterOut2(line) for line in data]
[[1, 3, 5], [1, 5], [3, 5, 8], [6, 3, 1], [5, 3, 5], [4, 1, 5]]
like image 137
Mike Müller Avatar answered Oct 26 '22 05:10

Mike Müller