Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is multithreading allowed on Spark/YARN?

Since YARN is responsible of managing the resources of Hadoop applications, like cores and processes, is a YARN application allowed to create new threads ?

I refer mainly to Spark applications running on YARN resource manager. I know that if you create a new thread no errors are thrown but, is it safe ?

like image 653
Nicola Ferraro Avatar asked Dec 26 '14 17:12

Nicola Ferraro


1 Answers

I reply to myself. YARN controls resources like memory and cores, not threads. Cores are controlled by changing OS specific parameters when creating containers (process affinity). Then, the JVM is running using a specified number of cores and you can create as many threads as you want.

like image 105
Nicola Ferraro Avatar answered Sep 23 '22 11:09

Nicola Ferraro