Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Apache Spark and node.js

I would like to call node.js program from spark-submit, using Spark API within JavaScript. Is there any implementation of this API in JavaScript?

I know one can call external mapper and reducer (written in JavaScript) from Hadoop, but I want to be able to run the complete program as one job from Spark.

like image 866
Diamond Dutch Avatar asked Mar 16 '23 13:03

Diamond Dutch


2 Answers

There is a new project called EclairJS that is providing a Spark API from Node and JavaScript, see https://github.com/EclairJS/eclairjs-node.

like image 138
DavidCF Avatar answered Mar 18 '23 03:03

DavidCF


I don't believe there is a current JavaScript spark wrapper, however I'd check out the information on the Spark pipe interface, this will allow you to use your JavaScript code from within Spark. Best of luck with your Spark & Javascript adventures (sadly there was an april fools joke about Spark & ScalaJS which makes searching for information on Spark + Javascript a little bit tricky).

like image 38
Holden Avatar answered Mar 18 '23 02:03

Holden