Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Parallel map operations?

Does Scala provide a way to execute parallel map operations as part of the standard language?

For example, given:

scala> val a = List((1,2), (3,4), (3,6))
a: List[(Int, Int)] = List((1,2), (3,4), (3,6))

I can do:

scala> a.map(tup => tup._1 + tup._2)
res0: List[Int] = List(3, 7, 9)

However, to the best of my knowledge this maps the provided function over the list objects sequentially. Is there a built-in way to have the function applied to each element in a separate thread (or equivalent), and the results then gathered into a resulting list?

like image 706
csvan Avatar asked Nov 20 '13 10:11

csvan


People also ask

What is parallel mapping?

The Parallel mapping type projects content geometrically into the scene, as if 'virtually shooting content from an emitting rectangle. The mapped image does not increase in size the further away you go from the emitting rectangle. Instead, the image remains the same size, hence the term parallel.

What is parallel operation in computer?

Parallel processing is a method in computing of running two or more processors (CPUs) to handle separate parts of an overall task. Breaking up different parts of a task among multiple processors will help reduce the amount of time to run a program.

Does Python map run in parallel?

It supports asynchronous results with timeouts and callbacks and has a parallel map implementation. The built-in map() function allows you to apply a function to each item in an iterable. The Python process pool provides a parallel version of the map() function.


2 Answers

If you add par then you will get a parallel collection and operations on it will be processed in parallel. To convert back to a normal collection call a toList.

So your code would look like:

a.par.map(tup => tup._1 + tup._2).toList

Or a .seq to get a Sequential Collection (the opposite of a Parallel Collection).

a.par.map(tup => tup._1 + tup._2).seq

Also, check the documentation.

like image 124
Akos Krivachy Avatar answered Sep 19 '22 07:09

Akos Krivachy


par splits your list for processing over several threads. You can then regulate how the threading is done by modyfying the tasksupport member of the resultant ParSeq.

like image 26
mikołak Avatar answered Sep 22 '22 07:09

mikołak