How to I create split a line into pairs of words rather than singular words?

Question

I read a CSV file into an RDD in Jupyter and wanted to convert each line into a pair of words rather than singular words, and then to create tuples of the pairs of words, but have no idea how i should do it. The CSV file looks something like this:

Afghanistan, AFG
Albania, ALB
Algeria, ALG
American Somoa, ASA
Anguilla, AIA

I've tried this:

lines = sc.textFile(...) words = lines.flatMap(lambda line : line.split (" "))

but it doesn't return Albania, ALB as one tuple. Instead it counts Albania as one and ALB as another. Help please!

Answer 1

You need to use map instead of flatMap. You can create your list of tuples as follows:

result = sc.textFile("...").map(lambda line:tuple(line.split(",")))

result.collect() then returns:

[(u'Afghanistan', u' AFG'), 
 (u'Albania', u' ALB'), 
 (u'Algeria', u' ALG'), 
 (u'American Somoa', u' ASA'), 
 (u'Anguilla', u' AIA')]

Looking at this output, you may want to add unicode.strip to remove the leading spaces:

sc.textFile("....").
map(lambda line:tuple(map(unicode.strip,line.split(",")))).
collect()