Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Ubuntu install apache spark via apt-get

I need to install spark and run it in standalone mode on one machine and looking for a straight forward way to install it via apt-get

I found how to do this with pyspark via pip here

I can not find any way to install spark with one terminal command.

Does installing pyspark install all the spark software?

I found instructions for how to install spark, and it's more complex

Is there a way to install spark with a similar one terminal command via apt-get?

EDIT

I found this explaining how to install spark using apt-get and i get the following error

E: Unable to locate package spark-core E: Unable to locate package spark-master E: Unable to locate package spark-worker E: Unable to locate package spark-history-server E: Unable to locate package spark-python

Thanks

like image 299
thebeancounter Avatar asked Jul 24 '17 09:07

thebeancounter


1 Answers

Before installing pyspark you must install Java8. This is described at

  • http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html
  • http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

For a fully automatic installation of Java8 on Ubuntu do

$ apt-get update
$ apt-get -y install software-properties-common
$ add-apt-repository -y ppa:webupd8team/java
$ echo debconf shared/accepted-oracle-license-v1-1 select true | debconf-set-selections
$ echo debconf shared/accepted-oracle-license-v1-1 seen true | debconf-set-selections
$ apt-get update
$ apt-get -y install oracle-java8-installer

(see https://newfivefour.com/docker-java8-auto-install.html)

Afterwards you can simply run pip install pyspark.

like image 170
asmaier Avatar answered Oct 07 '22 13:10

asmaier